Introducing Apache Kafka and why it is important to Oracle, Java and IT professionals (Tokyo, 13th November 2018, Oracle Groundbreakers APAC tour

What is Apache
Kafka & Why is
it important to
Oracle & Java &
IT
professionals?
The Event Fabric –
bringing IT together
What is Apache Kafka & Why is it important
µ
µ
Lucas Jellema, CTO of AMIS
Oracle Groundbreakers APAC Tour
Tokyo, November 13th 2018

Lucas Jellema
Architect / Developer
1994 started in IT at Oracle
2002 joined AMIS
Currently CTO & Solution Architect
こんばんは

It would be so nice if I could
publish my ideas and actions,
accessible near instantly for
everyone who is interested
Heck, I do not even know these people
and they may not know me [personally]
– just my pearls of wisdom. And if they
are late to the party, they can also
check out the historic archives of my
eloquence
Without fretting about the numbers of
readers involved and whether they are
in the same time zone as me and online
when I publish my messages – and
which device they use

• Decoupled communication
• 0, 1 or many followers
• Scalable number of messages (and parties)
• Reliable (mostly available, few messages lost)
• Full history
• Open: cross device, cross location
• Not Sub-second, near real-time fast
• Rate limited (#messages/minute)
• Size limited (140-280 characters)
• Format limited (text)
• Not for private interactions
• Not (really) for programmatic use

MongoDB
ORDERS
Oracle Database
DVX_ORDERS

7
MongoDB
ORDERS
Oracle Database
DVX_ORDERS

µLocally running
Node application
µ Oracle Application
Container Cloud
8
MongoDB
ORDERS
Oracle Database
DVX_ORDERS

What does the Twitter for System Driven Event Interaction
look like?
• Decoupled communication – organized per topic
• 0, 1 or many Consumers per Topic
• Reliable (distributed)
• Full history
• Open: libraries in many technologie & REST APIs

µ Oracle Application
Container Cloud
µLocally running
Node application
Oracle Event Hub
MongoDB
ORDERS
Oracle Database
DVX_ORDERS

What does the Twitter for System Driven Event Interaction
look like?
• Decoupled communication – organized per topic
• 0, 1 or many Consumers per Topic
• Reliable (distributed)
• Full history
• Open: libraries in many technologie & REST APIs
• Near real-time fast
• No Rate Limit
• No enforced size limit
• Anything goes (it’s all byte[])
• On premises or in cloud, private or trusted
• Very much for programmatic use

Products
Data Manipulation
Data
Retrieval

Special
Products
Product
Clusters
ProductsData Manipulation
Data Retrieval
Food
Stuff
Toys
Quick Product
Search Index
Product Store in
SaaS app
CQRS = Command Query Responsibility Segregation

Comand Query Responsbility Segregation = CQRS
Special
Products
Product Clusters
ProductsData Manipulation
Data Retrieval
Food Stuff
Toys
Quick Product Search
Index
Product Store in
SaaS app
Detect changes
Extract Data

Events
Producers
Consumers
Robust, Scalable, Fast,
History Retention
Containerized/Cloud-
enabled
Open

Messaging as we know it
• JMS, Oracle Advanced Queuing, IBM MQ, MS MQ, RabbitMQ, MQTT,
XMPP, WebSockets, Apache Active MQ, AMQP , …
• Challenges
• Costs
• Scalability (size and speed)
• (lack of) Distribution (and therefore availability)
• Complexity of infrastructure
• Message delivery guarantees
• Lack of technology openness
• Deal with temporarily offline consumers
• Retain history

Introducing Apache Kafka
• ..- 2010 – creation at Linkedin
• Message Bus | Event Broker
• High volume, low latency, highly reliable, cross technology
• Scalable, distributed, strict message ordering, ….
• 2011/2012 – open source under the Apache Incubator/ Top Project
• Kafka is used by many large corporations:
• Walmart, Cisco, Netflix, PayPal, LinkedIn, eBay, Spotify, Uber, Sift
Science, Zalando, The New York Times, Airbnb, Coursera, ING Bank,…
• And embraced by many software vendors & cloud providers
• Client libraries available for Node, Java, C/C++, Python, Ruby, PHP, Go,
Rust, .NET, Perl, Scala DSL, Clojure, Swift and more
• Commercial backing by and Enterprise support from Confluent

Producers
Consumers
tcp
tcp

Producers
Consumers
Topic

Kafka terminology
• Topic
• Message
• == ByteArray
• Broker
• Producer
• Consumer
Producer Consumer
Topic
Broker
Key
Value
Time
Message

Producers
Consumers
Topic
Broker
Key
Value
Time

Consuming
• Messages are available to consumers only when they have been committed
• Kafka does not push
• Unlike JMS
• Read does not destroy
• Unlike JMS Topic
• (some) History available
• Offline consumers can catch up
• Consumers can re-consume from the past
• Delivery Guarantees
• Ordering maintained
• At-least-once (per consumer) by default; at-most-once and exactly-once
can be implemented

Producers
Consumers
Topic
Broker
tcp
tcp
Consumer Group
Partition

What’s so special?
• Durable
• Scalable
• High volume
• High speed
• Available
• Distributed
• Open
• Quick start
• Free (no license costs)
• “Self Fulfilling Prophecy”
(positive feedback loop feeding from buzz around Kafka)
• Eco system, tools/libraries/resources, cloud services

Ecosystem – Kafka and Friends
• Clients
• CLI
• Java, Node/JavaScript, C/C++, Python, Go, Rust, .NET, Ruby, Clojure, Perl, …
• REST Proxy
• Kafka Schema Registry
• Kafka Connect – read data and change events from many sources and/or write
to many targets
• Also see Debezium
• Kafka Streams
• Kafka KSQL

Tweet!
#codeone

Select from <stream of tweet events>
select text
, author
, timestamp
from tweets
Where tag = 'codeone'
<--- streaming data

Select Running Count
from <stream of tweet events>
select tag
, count(*) tweet_count
from tweets
group
by tag

Tweets on
#CodeOne #java
#oraclecode
Tweets
Topic
Oracle Cloud
Event HubApplication
Container
TWEET_COUNT
Topic
Running
Tweets
Aggregation
Client
Client
Client
Client
IoT metrics from
hundreds of devices
User actions & click
events from webshop
Live Traffic EventsMicroservices chatter
Social Media events
(Facebook,
Whatsapp, …)
IT Operations –
monitoring metrics
µ
µ
µ
µ

Tweets on #JEEConf
#java #oraclecode
Tweets
Topic
Oracle Cloud
Event HubApplication
Container
TWEET_COUNT
Topic
Running
Tweets
Aggregation
Client
Client
Client
Client
IoT metrics from
hundreds of devices
User actions & click
events from webshop
Live Traffic EventsMicroservices chatter
Social Media events
(Facebook,
Whatsapp, …)
IT Operations –
monitoring metrics
µ
µ
µ
µ

Final Demo
• Microservice

The Microservice
API
HTTP REST/JSON

Extended API of microservice
• Deployment API
• Injectable dependencies – reference to cache, logging, storage URL, …
• Configurable meta-data – run time parameters, log level, credential (key)
• Interaction API
• REST Resources & Operations – query and URL parameters, message formats
• Events Consumed – alternative way to call | activate a microservice
• Reference to entry in Event Catalog
• May include reference to shared Cache Resource
• Events Produced – alternative output from microservice
• Event can be an asynchronous response to a stateless consumer
API

Microservices State
Cache
RDBMS
Document
Store
NoSQL
Generic Platform for running microservices
Event Hub
Big Data
Block
Storage
LDAP

Bounded context of microservices
• A micoservice needs to be able to run independently
• It needs to contain & own all data required to run
• It cannot depend on other microservices
API
Customer
APIUI
OrderCustomerModified event

Order Microservice
Demo – Maintaining Derived Data in Bounded Context using
Data Change Events communicated cross cloud over Kafka
Application
Container
Customer Microservice
Customers
Topic
Event Hub
Application
Container
DBaaS

Use cases for events
• Data Synchronization, (near real-time) replication, CQRS
• Event Sourcing
• Microservices
• Decoupled interaction
• Workflow & collaboration
• Bounded Context synchronization
• Serverless Functions
• Fast and Streaming Data – for example from IoT
• (near) Real Time User Interfaces
• with SSE WebSocket based push to browser or mobile push notifications
• …

Getting Started with Apache Kafka
• Get Access to an Apache Kafka Cluster environment
• Download, Install, Configure and Run
• Pull and Run a Docker Container (optionally on a Kubernetes cluster)
• Sign up for a Managed Apache Kafka Cloud Service (e.g. Oracle Event Hub)
• Configure Topic(s) & Partition(s) (optional)
• Get an Apache Kafka Client Library
• For Java, Spring Boot, Node, Python, …
• Or expose Apache Kafka REST Proxy service
• Develop Client Application to
• Produce Events to Kafka Topic
• Consume Events from Kafka Topic
• Optionally
• Use Schema Registry to design, publish and enforce event payload structure
• Use Kafka Connect[ors] to integrate with specific source or sink technology
• Use tooling for monitoring, tuning, reporting, administration, …
Topic

Develop Client Applications
• Leveraging a precreated Event Hub Managed Kafka Topic in the cloud
• Simple Node client
• To publish message to Apache Kafka
• using kafka-node npm module
• Simple Java client
• To consume messages from Apache Kafka
• using org.apache.kafka.kafka_2.12 Maven dependency
Topic

Kafka Node Client Producer
• package.json
• npm install

Kafka Java Client - Consumer
• mvn init
• pom.xml

Kafka Java Client - Consumer

Kafka Java Client – Consumer (3)
• Run

Oracle embracing Apache Kafka
• Event Hub Cloud Service = Managed Apache Kafka platform
• Managed Topics have been announced too
• Oracle Cloud Infrastructure: Telemetry & Streaming
• Read from Kafka Topic in Oracle Database SQL queries
• And publish to Kafka Topic from Oracle Database PL/SQL
• Kafka as source for Data Integration Platform (Golden Gate and ODI)
• Data Pipeline with Data Hub (Apache Cassandra) & Event Hub
• Oracle Service Bus Kafka Adapter
• Integration Cloud
• Stream Analytics (aka Stream Explorer fka Oracle Event Processor)
• Oracle Native Container and Microservices Platform
• Fn Serverless Platform – triggered by and publishing to
• Visual Builder & JET - real time push based on Apache Kafka
• In general – the bridge between on premises  [public] Cloud

Summary
• => == =>
• Apache Kafka is emerging as platform of choice for message exchange in a world of
• Microservices
• CQRS and Data Source Synchronization
• Clouds
• Fast Data (IoT) and Streaming Analysis
• Real time data integration & distribution
• Oracle is rapidly embracing Apache Kafka on various levels
• Getting started with Apache Kafka is not very hard at all
• The platform is open source – and has broad client support (Java, Node, …)
• Many resources are available – tutorials, blog article, demonstrations, presentation
slides and recordings of conference sessions, samples on GitHub

Thank you!
ありがとうございました
• Blog: technology.amis.nl
• Email: lucas.jellema@amis.nl
• : @lucasjellema
• : lucas-jellema
• : www.amis.nl, info@amis.nl

Introducing Apache Kafka and why it is important to Oracle, Java and IT professionals (Tokyo, 13th November 2018, Oracle Groundbreakers APAC tour

Recommended

More Related Content

What's hot (20)

Similar to Introducing Apache Kafka and why it is important to Oracle, Java and IT professionals (Tokyo, 13th November 2018, Oracle Groundbreakers APAC tour (20)

More from Lucas Jellema (20)

Recently uploaded (20)

Introducing Apache Kafka and why it is important to Oracle, Java and IT professionals (Tokyo, 13th November 2018, Oracle Groundbreakers APAC tour

Editor's Notes