Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022

The Journey of
InﬂuxDB
Paul Dix
Founder & CTO
@pauldix

1.0 2.0 IOx
2013-2019 2019-2022 2022-

1.0
2.0
IOx
2013-2019
2019-2022
2022-

Talk themes • What are time series?
• What is a time series
database? (TSDB)

Time series are a method for
interpreting and understanding
data

[[23.2, t1], [64.6, t2], [54.1, t3], [77.8, t4], [62.1, t5], [99.6, t6],
…]

Two ways to think
about TSDBs:
• Store values & metadata
directly
• Compute on the ﬂy from
raw data

Round Robin Database Structure
t0 marker
t
1
t2
t20
Length: 20
Delta: 60s
Start Epoch: ...

mysql.bytes_received 1287333217 327810227706 schema=foo host=db1
mysql.bytes_sent 1287333217 6604859181710 schema=foo host=db1
Metric name Epoch Value Tags

TSDB that store
time series
• Time series data
• Metadata
• Hierarchical (Graphite)
• Tags (OpenTSDB)

InﬂuxDB 0.8 data model
C1 C2 C3 C4 C5 C6 time
measurement
1
measurement2
C1 C2 C3 C4 time

select value from response_times where time > now() - 1h limit 1000;

select value from response_times
where time > now() - 1h and server = ‘foo’
limit 1000;
Range scan

API changes: tags and line protocol

API changes: tags and line protocol
foo,tag1=bar,tag2=asdf field1=23i,field2=2.4,field3=“hello”,field4=true
12344321
measurement name Fields Timestamp
Tags

TSM Data
Block 1
Series 1 Series 2
Block 2 Block 3
Series 3
Series 4
Block 1 Block 2 Block 3 Block 4

Inverted Index
weather,city=Denver,state=CO,zip=80222 temp=62.3,humidity=32
weather,city=Bellevue,state=WA,zip=98007 temp=50.7,humidity=76
weather,city=Brooklyn,state=NY,zip=11249 temp=58.2,humidity=55
weather,city=Denver,state=CO,zip=80222#temp – 1
weather,city=Denver,state=CO,zip=80222#humidity - 2
weather,city=Bellevue,state=WA,zip=98007#temp - 3
weather,city=Bellevue,state=WA,zip=98007#humidity – 4
weather,city=Brooklyn,state=NY,zip=11249#temp – 5
weather,city=Brooklyn,state=NY,zip=11249#humidity - 6
Line Protocol
Series

Inverted Index – Posting Lists
Measurements Fields Tags
weather – [1,2,3,4,5,6] temp – [1,3,5]
humidity – [2,4,6]
city=Denver – [1,2]
city=Bellevue – [3,4]
city=Brooklyn – [5,6]
state=CO – [1,2]
state=NY – [5,6]
state=WA – [3,4]
zip=11249 – [5,6]
zip=80222 – [1,2]
zip=98007 – [3,4]
select temp from weather where time > now() - 1h and state = “NY”;

weather – [1,2,3,4,5,6] temp – [1,3,5] state=NY – [5,6]
select avg(temp) from weather where time > now() - 1h and state = “NY”;
Intersect these postings lists = 5

weather – [1,2,3,4,5,6] temp – [1,3,5]
select avg(temp) from weather where time > now();
[1, 3,
5]

Time Series Blocks
Lookup - [4, 73, 128, 222, 245, 278, 333, 561]

Design Tradeoﬀs
• Fast Low Cardinality Lookups & Execution
• Inverted Index maintenance on Ingest
• Sorted time series key space
• Expensive and Slow Higher Cardinality

InﬂuxDB had evolved to become a
TSDB that stored time series

2.0 - Flux, all-in-one, cloud ﬁrst
delivery

The problem with time series: you
summarize everything before you
get to the database

http_request_duration_seconds_bucket{le=“0.1”} 23
http_request_duration_seconds_bucket{le=“0.5”} 30
http_request_duration_seconds_bucket{le=“2”} 34
http_request_duration_seconds_bucket{le=“+Inf”} 130
http_request_duration_seconds_sum
http_request_duration_seconds_count

http_request_duration_seconds_bucket{host=“foo”, le=“0.1”} 23
http_request_duration_seconds_bucket{host=“foo”, le=“0.5”} 30
http_request_duration_seconds_bucket{host=“foo”, le=“2”} 34
http_request_duration_seconds_bucket{host=“foo”, le=“+Inf”} 130
http_request_duration_seconds_sum{host=“foo”}
http_request_duration_seconds_count{host=“foo”}

http_request_duration_seconds_bucket{host=“foo”, le=“0.1”, service=“asdf”} 23
http_request_duration_seconds_bucket{host=“foo”, le=“0.5” , service=“asdf”} 30
http_request_duration_seconds_bucket{host=“foo”, le=“2” , service=“asdf”} 34
http_request_duration_seconds_bucket{host=“foo”, le=“+Inf” , service=“asdf”} 130
http_request_duration_seconds_sum{host=“foo”, service=“asdf”}
http_request_duration_seconds{count host=“foo”, service=“asdf”}

http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.1”, service=“asdf”} 23
http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“0.5” , service=“asdf”} 30
http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“2” , service=“asdf”} 34
http_request_duration_seconds_bucket{customer=“1423”, host=“foo”, le=“+Inf” , service=“asdf”} 130
http_request_duration_seconds_sum{customer=“1423”, host=“foo”, service=“asdf”}
http_request_duration_seconds_count{customer=“1423”, host=“foo”, service=“asdf”}

Pull represents full state every
scrape

Push an event
http_request customer=“1423”, host=“foo”, le=“0.1”, service=“asdf”, response_time=23

What about
tracing? • Level of detail so high you
must sample

Match precision to
use case
• Query log
• Auditing
• Usage metering
• Analytics of many kinds

Columnar databases solve
performance problems for storing
events

Back to Tables
City State Zip Temp Humidity Time
Denver CO 80222 62.3 32 1
Denver CO 80222 61.2 32 2
Bellevue WA 98007 50.7 76 1
Bellevue WA 98007 50.5 76 2
Brooklyn NY 11249 58.2 55 1
Brooklyn NY 11249 57.0 55 2
Tags Timestamp
Fields

Back to Tables
Denver CO 80222 62.3 32 1
Denver CO 80222 61.2 32 2
Bellevue WA 98007 50.7 76 1
Bellevue WA 98007 50.5 76 2
Brooklyn NY 11249 58.2 55 1
Brooklyn NY 11249 57.0 55 2
select avg(temp) from weather where time < 100 and state = “NY”;

Solving the performance problem
• Per column compression (dictionary, RLE)
• Vectorized execution
• Parallelism
• Partitioning
• Pruning

Per-column Compression
Denver CO 80222 62.3 32 1
Denver CO 80222 61.2 32 2
Bellevue WA 98007 50.7 76 1
Bellevue WA 98007 50.5 76 2
Brooklyn NY 11249 58.2 55 1
Brooklyn NY 11249 57.0 55 2
Repeated values: Dictionary Encoding, RLE

Partitioned Layout
measurement
1
C1 C2 C3 C4 time
2022-11--01 2022-11--02
C1 C2 C3 C4 time

Query Properties
• Execution against all rows from included partitions
• Accessing a single series through brute force
• 1B rows/sec/core

Partitioned Layout
measurement2
C1 C2 C3 time
2022-11--01 2022-11--02
region=west region=east
C1 C2 C3 time

Problems to Solve
• Wide Tables (many columns)
• Sparse Tables
• Repartitioning (automatic)
• Secondary Indexes

Cloud Databases Solve for Scale

InﬂuxDB IOx
• Columnar database
• SQL natively supported
• Cloud Database
• Federated at the Edge

Available in InﬂuxDB Cloud Today!

InﬂuxDB IOx is a TSDB for storing
AND calculating time series

Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022

Recommended

More Related Content

What's hot (20)

Similar to Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022 (20)

More from InfluxData (20)

Recently uploaded (20)

Paul Dix [InfluxData] The Journey of InfluxDB | InfluxDays 2022