Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWS

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Customer Journey with Streaming Data on AWS
Amazon Web Services (AWS) offers over 165 fully featured cloud services from data
centers globally. AWS launched its first data streaming service, Amazon Kinesis Data
Streams, over five years ago. Now, customers are using streaming data across most
AWS services including two that support running Apache Flink, Amazon EMR and
Amazon Kinesis Data Analytics. In this keynote, we will describe how customers and
their use of streaming data has evolved on AWS. We will look at how streaming data
and Apache Flink are used externally and internally on AWS, and where we see usage
of Apache Flink growing.

Rahul Pathak
General Manager of Databases, Analytics, and Blockchain
Amazon Web Services
Customer Journey with Streaming
Data on AWS

Building with our customers
50+ fully managed streaming
capabilities deployed globally in 22
AWS Regions
2019
2013
First fully managed streaming
service,
Amazon Kinesis Data Streams
2018
Support for Apache Flink
based apps in
Amazon Kinesis Data Analytics
Support for Apache Flink in
Amazon EMR
2016

Streaming data on AWS in 2013
Internal and external customer struggling with high volume data
• Low latency, continuous data capture
• Durable storage to quickly get data from
unreliable sources
• Scale to cost effectively handle lots of data
AWS Metering
and Billing

What did customers look like in the past?
The “Average” customer… was attracted by ease of use
• Java developer
• Had one application processing 10s of millions of events per day
• Application performed extract, buffer, and load to Amazon S3
The “Large” customer… was attracted by performance and scale
• Had distributed systems experience and was familiar with Hadoop
• Had two applications processing billions of events per day
• Application performed advanced ETL like joins

Early streaming data customers on AWS
Streaming extract
and load to Amazon
S3
50 billion daily
ad impressions,
sub-50 ms
responses
100 GB/day
clickstreams from
250+ sites

Supercell Delivers World-Class Mobile Games
Supercell is a Finnish game company known for the hit
games Clash of Clans, Hay Day, and Boom Beach.
The world of gaming never sleeps
... We owe every player a great
experience, and AWS is our
platform to make that happen.
Sami Yliharju
Services Lead, Supercell
”
“ • Started with a non-
streaming architecture
• Use streaming data for
faster ETL and analytics
• Started with archival apps
and kept adding use cases

Supercell’s data pipeline circa 2012

Supercell’s streaming pipeline circa 2018

Simplify common use cases
Every customer built a streaming delivery app
• Load streaming data into streams, data lakes and
warehouses
• Zero administration and seamless elasticity
• Direct-to-data store integration
• Serverless continuous data transformations

Make advanced use cases accessible
Few customers were able to move to real-time analytics
• Analyze data streams in real time
• Interact with streaming data in real-time using Apache
Flink-based apps
• Build fully managed and serverless stream processing
applications

Apache Flink at AWS
Customers run Apache Flink on AWS on different services with varying degree
of flexibility and management
Amazon Kinesis
Data Analytics
Amazon Elastic
Kubernetes Service
Amazon EMR

Streaming data on AWS today

Streaming data on AWS in 2019
<40%
delivery
apps
3
apps
3
data
stores

What do customers do today?
The “Average” customer… is attracted by the ease of use and rich features
• New to streaming data (~50% of customers are still net new to streaming)
• Has several applications processing 100s of millions of events per day
The “Large” customer… is attracted to the above but needs flexibility and elasticity
• Uses many languages including SQL, Java, Python
• Has 10s of applications processing 10s of billions of events per day
The “Platform” customer is…attracted to all of the above plus performance
• Builds abstracted platforms on top of streaming services
• Has 100s of applications processing trillions of events per day

Streaming services are foundational at Amazon
Amazon Go
video analytics
Amazon.com
online catalog
Amazon
CloudWatch
logs
Amazon
S3 events
AWS
metering

Customers built a wide variety of use cases
Near-real-time
home valuation
(Zestimates)
Live clickstream
dashboards refreshed
under 10s
1 billion events per
week from
connected devices
Real-time
game events
analytics
Built event driven,
micro services arch
StreamHub
Online stylist
processing
10 million
events/day
Facilitate
communications
between 100+
microservices
IoT predictive
analytics
Log analytics for
real-time “single
pane of glass”
Serverless event bus
and ingestion
pipeline

Autodesk makes software for people who make things
Autodesk, a leading provider of 3D design and
engineering software, wants to do more than create
and deliver software.
“Ultimately, we are improving our
software products and offering
better service to our customers
because of the real-time visibility
we’re getting into log data.”
Tommy Li
Senior Software Architect, Autodesk
”
“ • Provides cloud services for
its design software
• Uses streaming analytics to
monitor and improve their
customer experience

Autodesk’s Streaming Architecture
Amazon Kinesis
Data Streams
Amazon EC2
Amazon Elastic Container
Service
AWS Lambda
Amazon Kinesis
Data Firehose
Amazon Kinesis
Data Analytics
Amazon Elasticsearch
Service
Amazon Athena
Amazon CloudWatch

AWS Data Center Services (DCS) uses Apache Flink
DCS maintains and monitors electrical and physical
topologies for all AWS Data Centers
“We chose Apache Flink because it
provided simple, extensible interfaces and
scalability. We chose Kinesis Data Analytics
because of its guaranteed uptime, simple
deployment mode, and lower ops cost.
AWS Data Center Team
”
“ • Write software to interface
with equipment in data
centers
• Includes electrical power
draw, water usage, weather,
fan speeds, and host
temperatures
• Use insights to drive down
cost of data center
operations

AWS Data Center Services
• Data is captured from Kinesis Data
Streams and CDC from a noSQL store
(Amazon DynamoDB streams)
• Analytics are calculated using Apache
Flink on Kinesis Data Analytics
• Analytics include drift in circuit breaker
settings to power utilization and much
more

What will customers look like in the future?
The “Average” customer… doesn’t even know they are using Flink or a
streaming data service
The “Large” customer… has teams across the company with varying levels of
technical sophistication
The “Platform” customer is… any company, no longer requiring teams of
engineers and years of investment
You may already be here but streaming is still new for most

Apache Flink at AWS in the future
• Flink is the fastest growing framework reading Kinesis Data
Streams
• Usage is still relatively small compared to simplest of
solutions (e.g. KafkaConsumer, Kinesis Clients) running on EC2
• Excited to work with community of further simplifying
running Flink both on AWS and anywhere else

It is still Day 1

Thank you!

Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWS

Recommended

More Related Content

What's hot (20)

Similar to Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWS (10)

More from Flink Forward (20)

Recently uploaded (20)

Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWS