Natural Language Query and Conversational Interface to Apache Spark
0 likes228 views
Apache Spark has been a great technology for processing and analyzing Big Data. However, it is not accessible to business users, who don’t have technical or programming skills. In this talk, I’ll talk about recent efforts in the space of “Conversational analytics”. This paradigm allows any user to ask text and voice questions, in natural language, of their data to a bot and receive back a natural language and visual result. A key technology is natural language to SQL translation, where we translate natural language queries from a user into Spark SQL queries that can go against a Databricks system, and that can be easily trained on different schemas and databases. This NLP technology needs to be further combined with dialog management, natural-language generation/narration, data understanding and modeling, augmented analytics and automated visualization generation in order to achieve the goal of “Conversational Analytics”. Using such a technology, a user can ask, in plain English, “How many cases of Covid were there in the last 2 months in states that had no social distancing mandates by type of transmission”, and then dig deeper into the results in a conversational manner to uncover hidden insights from Covid datasets in a Spark instance. We believe that having access to such data and insights at their fingertips can help users make appropriate decisions quickly, improve data literacy and even overcome the scourge of fake news for the general public.
More Related Content
What's hot (20)
Similar to Natural Language Query and Conversational Interface to Apache Spark (20)
More from Databricks (20)
Recently uploaded (20)
Natural Language Query and Conversational Interface to Apache Spark
- 1. Introduction to Auto Loader How to easily ingest PBs of data into your Delta lakehouse Pranav Anand Software Engineer at Databricks
- 2. s3:/logs Cloud Storage Input files containing unstructured and semi-structured data Final data Building a data pipeline
- 3. Building a data pipeline: Step one s3:/logs Cloud Storage Structured Streaming Input files containing unstructured and semi-structured data Structured tables Data Ingestion
- 4. s3:/logs Cloud Storage Structured Streaming Input files Structured tables “Here be dragons” Building a data pipeline: Step one
- 5. input path: s3:/logs Cloud Storage File Stream Source Final data a.json, b.json List on trigger at t = 0 Transformed data Challenge: Scalability
- 6. input path: s3:/logs Cloud Storage File Stream Source Final data c.json, a.json, b.json List on trigger at t = 5 Transformed data Challenge: Scalability Seen file paths a.json b.json
- 7. input path: s3:/logs Cloud Storage File Stream Source Final data c.json, a.json, b.json List on trigger at t = 5 Transformed data Challenge: Scalability Seen file paths a.json b.json Repeated listing is slow and expensive
- 8. input path: s3:/logs Cloud Storage File Stream Source Final data c.json, a.json, b.json List on trigger at t = 5 Transformed data Challenge: Scalability Seen file paths map a.json b.json In-memory map does not scale Repeated listing is slow and expensive
- 9. Challenge: Schema Cloud Storage { id: 5 name: “John” } { id: 7 name: “Amy” } a.json b.json Manually infer and set schema as id: Int name: String Ready to go! Input files
- 11. Challenge: Schema Cloud Storage Input files { id: 18, name: “Olivia” } { id: 23, name: “Alex”, age: 31 } e.json f.json Data loss! Manually handle this column. User intervention needed every time. How do I deal with all this?!
- 12. Auto Loader ▪ New Structured Streaming source ▪ Solves biggest ingestion challenges: ▪ Scalability ▪ Schema management
- 13. Challenge: Scalability File Notification Mode s3:/logs Cloud Storage Input files
- 14. Challenge: Scalability File Notification Mode s3:/logs Cloud Storage Input files a.json File notification generated for a.json a.json s3:/logs
- 15. Challenge: Scalability File Notification Mode Input files Cloud files source Pull a.json from queue Delete a.json from queue once ingested a.json s3:/logs Cloud Storage a.json File notification generated for a.json s3:/logs No listing! Files ingested as they arrive.
- 16. Challenge: Scalability File Notification Mode Input files Pull a.json from queue Delete a.json from queue once ingested a.json s3:/logs Cloud Storage a.json File notification generated for a.json s3:/logs Cloud files source Seen file paths map in RocksDB a.json RocksDB deduplication means no scalability limits
- 17. Challenge: Scalability Backfill Input files Pull a.json from cloud queue Delete a.json from queue once ingested a.json Cloud Storage a.json File notification generated for a.json Cloud files source Seen file paths map in RocksDB a.json A.json a.json, A.json Include existing files A.json Internal queue a.json A.json Existing files
- 18. Challenge: Scalability File Notification Mode spark .readStream .format(“cloudFiles”) .option(“cloudFiles.format”, “json”) .option(“cloudFiles.useNotifications”, “true”) .option(“cloudFiles.includeExistingFiles”, “true”) .load() ▪ No repeated listing ▪ Scalable to many millions of files ▪ Done simply
- 19. Challenge: Schema Cloud Storage Input files a.json Starting off more simply { id: 5 name: “John” } { id: 7 name: “Amy” } b.json Automatically infer and set schema as id: Int name: String Ready to go!
- 20. Challenge: Schema Cloud Storage Input files { id: 18, name: “Olivia” } { id: 23, name: “Alex”, age: 31 } e.json f.json Where we hit a dead end... How do I deal with all this?! Data loss! Manually handle this column. User intervention needed every time.
- 21. Challenge: Schema Cloud Storage Input files { id: 18, name: “Olivia” } { id: 23, name: “Alex”, age: 31 } e.json f.json What we can do about it How do I deal with all this?! + Auto Loader Data loss! Manually handle this column. User intervention needed every time.
- 22. Challenge: Schema Cloud Storage Input files { id: 18, name: “Olivia” } { id: 23, name: “Alex”, age: 31 } e.json f.json Self-sustaining stream - Evolve schema Schema was inferred as id: Int name: String New schema: id: Int name: String age: Int
- 23. Challenge: Schema Cloud Storage Input files { id: 18, name: “Olivia” } { id: 23, name: “Alex”, age: 31 } e.json f.json Self-sustaining stream - Evolve schema Schema was inferred as id: Int name: String New schema: id: Int name: String age: Int Set and forget! .option(“cloudFiles.schemaEvolutionMode”, “addNewColumns”)
- 24. Challenge: Schema Cloud Storage Input files { id: 18, name: “Olivia” } { id: 23, name: “Alex”, age: 31 } e.json f.json Self-sustaining stream: Rescue data Schema was inferred as id: Int name: String _rescued_data: String New schema: id: Int name: String _rescued_data: String Set and forget! .option(“cloudFiles.schemaEvolutionMode”, “rescue”)
- 25. Challenge: Schema Cloud Storage Input files { id: 18, name: “Olivia” } { id: 23, name: “Alex”, age: 31 } e.json f.json Self-sustaining stream: Rescue data Schema was inferred as id: Int name: String _rescued_data: String New schema: id: Int name: String _rescued_data: String Set and forget! .option(“cloudFiles.schemaEvolutionMode”, “rescue”) Other modes like “failOnNewColumns” and “none”
- 26. Let’s look at the code
- 27. In practice ▪ 50+ TB per day of logs ingested at Databricks using Auto Loader ▪ Customers have used Auto Loader to ingest 10s of PB of data
- 28. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.