SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Rohit Reddy, SingleStore
0 likes980 views
To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.
More Related Content
What's hot (20)
Similar to SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Rohit Reddy, SingleStore (20)
More from HostedbyConfluent (20)
Recently uploaded (20)
SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architecture | Rohit Reddy, SingleStore
- 1. — Kafka & SingleStore: Better Together to Power Modern Real-Time Data Architecture Rohit Reddy Principal Solutions Engineer SingleStore
- 2. 2 Increasing Focus on Cloud and Real-Time Analytics 90% 75% 30% By 2022, public cloud services will be essential for 90% of data and analytics innovation By 2022, 75% of all workloads will move to Hybrid-Cloud By 2025, nearly 30% of all data generated will be real-time Gartner Top 10 Trends in Data and Analytics for 2020 McKinsey - Unlocking business acceleration in a hybrid cloud world, Aug 2019 IDC - Data Age 2025
- 3. 3 — Traditional Data Architecture System of Record System of Engagement Data Warehouse ODS ETL CDC Reporting Visualization Batch Engine Real-Time Engine Data Lake
- 4. 4 — Modern Real-Time Data Architecture System of Record System of Engagement Smart Apps System of Insight System of Intelligence CDC
- 5. 5 The Unified Database for Fast Analytics Data Warehouse Operational Database Transactional Workloads Fast Queries | Large Data Size Aggregation Fast Lookup | High Concurrency Simplifies the support of diverse workloads by reducing operational complexity Analytical Workloads
- 6. 6 Ultra-Fast Ingest Parallel, high-scale streaming data Ingest Super-Low Latency High Concurrency Blazing Fast Queries Unparalleled Scalability Billions of events/sec for immediate availability Sub-second latencies with immediate consistency Millions of real-time queries across tens of thousands of users Fast Analytics Fast analytics on dynamic data for complex analytical queries SingleStore - Key Capabilities for Fast Analytics
- 7. 7 — Kafka & SingleStore Meet Demands of Operational Analytics Real-Time Millions of records per second Consistent Exactly-once semantic Distributed, Fault Tolerant Parallel ingest Developer-Friendly Pub-Sub & SQL
- 8. 8 — Anatomy of SingleStore Pipelines Sequence SingleStore pulls for changes from a data source system. SingleStore pulls the data into its memory space (no commit) where a transform can be applied. The data is committed in a transaction (and in parallel) Pipelines Kafka SingleStore Data can be directly inserted into tables or pre-processed by a stored procedure Write to Kafka
- 9. 9 SingleStore Pipelines Creation 10M UPSERTS PER SECOND WITH KAFKA + SINGLESTORE CREATE OR REPLACE PIPELINE load_trade_data AS LOAD DATA KAFKA 'hostname:9092/trades' WITH TRANSFORM ('score_data.py','','') -- optional INTO TABLE live_predictions -- directly into tables INTO PROCEDURE trade_proc -- via a stored procedure FIELDS TERMINATED BY ','; START PIPELINE load_trade_data; ; —
- 10. 10 — SingleStore Kafka Pipelines SingleStore Cluster
- 11. 11 — SingleStore Transforms ● Build transforms using any language! ● Transforms are an optional user-defined program that receives data from a pipeline’s extractor and outputs modified data (JSON, Avro, CSV) ○ Examples: Data modification, aggregation, feature engineering, model execution, and more! ● Linux distribution must have the required dependencies to execute the transform ● Data streamed to the transform is byte-length encoded Stream Transform Load
- 12. 12 — SingleStore Stored Procedures DELIMITER // CREATE OR REPLACE PROCEDURE tweets_proc(batch QUERY(tweet JSON)) AS BEGIN INSERT IGNORE INTO tweets(tweet_id, tweet_user_id, tweet_text) SELECT tweet::tweet_id, tweet::tweet_user_id, tweet::tweet_text FROM batch; INSERT INTO retweets_counter(user_id, num_retweets) SELECT tweet::retweet_user_id, 1 FROM batch WHERE tweet::retweet_user_id IS NOT NULL ON DUPLICATE KEY UPDATE num_retweets = num_retweets + 1; END // DELIMITER ; ; ● Preprocess incoming data: cleansing, aggregation, filtering… ● Dispatch to multiple tables ● Cross-reference with dimension tables ● Integrity check ● Push to Kafka
- 13. 13 — SingleStore Push to Kafka ● Allows users to leverage SingleStore as a true Operational Data Hub with downstream decisioning ● “SELECT … INTO KAFKA …” runs a SELECT query, constructs Kafka message for each row in the result set, and publishes the messages to a Kafka topic ● Includes every column value in the result set’s row and separates the column values by a delimiter ● Configure security credentials within the statement easily SELECT col1, col2, col3 FROM t ORDER BY col1 INTO KAFKA 'host.example.com:9092/test-topic' FIELDS TERMINATED BY ',' ENCLOSED BY '"' ESCAPED BY "t" LINES TERMINATED BY '}' STARTING BY '{';
- 14. 14 — SingleStore Confluent Kafka Connector ● SingleStore Kafka Connect Connector on the Confluent Hub ● Integration with Confluent Kafka Connect to stream data into SingleStore ● Management and deployment capabilities of Confluent make this incredibly easy to get started ● Cloud-first: Kafka Connector sits Kafka-side, eliminating many potential security constraints 5X THAN JDBC CONNECTOR FASTER
- 15. Real-time fraud analytics for Credit card swipes in less than 50ms. Real-time geospatial insights with massive concurrency to manage 24/7 operations 300K Events per second Streaming analytics to drive proactive care and real-time recommendations IoT Analytics ingesting and analyzing data from over 1.2 Million smart meters 13x data growth moving from batch to near-real time visibility and analytics 3500+ Users 1.2M Smart meters analyzed 10M Upserts per second Tier-1 US Bank 50ms Real-time Fraud Detection Top Energy Company
- 16. SingleStore is the Unified Database for Fast Analytics on Any Data, Anywhere
- 17. 17 Learn Your Way — Get Started with $500 in Free Credits Today Go to singlestore.com/managed-service-trial ● Learn by Reading ○ docs.singlestore.com ● Learn by Engaging Peers ○ singlestore.com/forum ● Learn by Watching ○ youtube.com/singlestore ● Learn through Training ○ training.singlestore.com
- 18. — 18 Thank You Sales Please fill out the form if you need to learn more. For immediate sales help, call us at 1-855-463-6775 or email us at team@singlestore.com. Enterprise Edition Support Are you encountering an issue and have an enterprise support? Submit a support request. U.S. Office Locations San Francisco (HQ) 534 Fourth Street San Francisco, CA 94107 Seattle 96 Union Street Seattle, WA 98101 Portland 700 SW Fifth Ave Portland, OR 97204