Exploring KSQL Patterns

Editor's Notes

• #5: Really, stream processing is still a pretty new discipline. We are only on the second generation of OSS tooling (depending on how you look at things), and most people who are building streaming systems are building their first. As a result, most stream processing requires a bunch of custom code, often deployed to specialized infrastructure, coded against specialized APIs. And hey, sometimes that’s what you gotta do, but having a declarative language and getting infrastructure problems out of your way is a good thing. KSQL aims to do both things.
• #6: Another way to put that, is that KSQL is a SQL engine for Kafka. It’s not a subset of ANSI SQL—it can’t be, since streaming systems deal with unbounded data sets and relational databases are fundamentally about bounded data sets, and that difference matters—but man, how great would it be to have a SQL-like language to describe stream processing computation you want done to the data you have stored in Kafka topics? (If you don’t know Kafka already, it’s a messaging system, and topics are just queues of messages. Basic stuff here, and don’t let it confuse you if you’re new to all of this.)
• #7: Where does it fit into my system? What is the language syntax like?
• #8: architecture diagram stuff goes into Kafka, KSQL processes it, it goes out KSQL takes the place of more complex options that have preceded it, like the Streams API or the Producer and Consumer API.
• #9: KSQL is familiar, but is also different in important ways. What is a stream? An unbounded sequence of facts. What is a table? A collection of evolving facts. We’ll see examples. Queries tend to run until you stop them. This is counterintuitive, but remember we’re dealing with streaming data here. There’s never a “last” record. Persistent queries are really stream processing programs that run in KSQL.
• #10: Ok, so we want to make ourselves a stream out of a topic we have in Kafka, how to start ? This is a lightweight abstraction on top of the topic. Note that the stream has metadata, but the metadata is extracted automatically from the topic.
• #11: It’s not an ad-hoc query language as such, but since you can define stream processing jobs with it, it’s certainly possible to use it to arbitrary filtering and projection on existing topics. You have to create streams first, fo course, but we’ve gone over that now.
• #12: Creating a table. Note that this is fundamentally tabular data: the key is the user_id, so each message in the topic is an update to that user’s record. We don’t need to specify the metadata, because it gets sucked in from the topic.
• #13: It’s not an ad-hoc query language as such, but since you can define stream processing jobs with it, it’s certainly possible to use it to arbitrary filtering and projection on existing topics. You have to create streams first, fo course, but we’ve gone over that now.
• #16: On the third bullet: often people build streaming pipelines with Kafka dumping data into C* or Elastic. Well, you’re probably going to need to do some work on the data along the way. No need to have a Spark Streaming job running now!
• #19: KSQL also turns out to be super-useful for housekeeping and administrative actions that would otherwise require a stream-transforming program of some sort to be written and tested, or changes in the underlying source data ssytems to produce to a topic in a different format in the first place. In this example we’re simply: taking all the records from the ‘clickstream’ topic and copying them into a new ‘views_by_userid’ topic, which we’ve asked to be written out in json format (notice that the inout stream could be in any other format KSQL can read), and explicitly asked for there to be 6 partitions of this output topic, for the record timestamps to be populated from the value of the ‘view_time’ field in the input topic And finally, the records should be distributed across the 6 partitions based on their ‘user_id’ All the options we’re specifying here have sensible defaults and can be omitted if you don’t want or need to override them
• #20: 4
• #22: When we’re looking to select tools for solving a particular tech problem in front of us we are always making trade-offs. In kafka-land, one interesting set of trade-offs to consider is this, fairly typical, spectrum: ranging from very flexible and low-level on the left side, using the original kafka client producer and consumer APIs – think of this as being at the level of ‘get-message’, ‘put message’ and you of course have to take care of many details of orchestrating these reads and writes yourself; up through something like the kafka streams api, shown in the center here, where we can hide a lot of lower-level implementation concerns and focus on using functions which operate on a stream of records as a whole – perhaps filtering or transforming every record that passes by in a more functional-programming style. The real shift when using this is in mindset, to a place where you think of passing functions to be run against everything in a stream rather than strictly iterating over the stream yourself. KSQL shifts it up another gear to a place where we can declaratively transform one or more streams into another stream, using syntax and ideas that may be more familiar. Notice how, as we go from left to right on this spectrum, each thing builds upon the preceding one – both conceptually and also literally in terms of implementation – each of these APIs is built around the preceding one
• #27: Leave resource mgmt. to dedicated systems such as k8s All running Engines share the processing load Technically, instances of the same Kafka Streams Applications Scale up/down without restart
• #29: This is open source, and you should get involved. You can check out the code on GitHub or play with the many examples there. Also, you are hereby solemnly adjured to join the Slack community and ask questions there!