Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
8 likes2,451 views
From the makers of the popular Sematext Logsene log management service - how to build reliable, high-speed log processing pipelines
More Related Content
What's hot (20)
Viewers also liked (20)
Similar to Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka (20)
More from Sematext Group, Inc. (13)
Recently uploaded (20)
Building Resilient Log Aggregation Pipeline with Elasticsearch & Kafka
- 1. Building Resilient Log Aggregation Pipeline Using Elasticsearch and Kafka Rafał Kuć @ Sematext Group, Inc.
- 3. Next 30 minutes… Log shipping - buffers - protocols - parsing Central buffering - Kafka - Redis Storage & Analysis - Elasticsearch - Kibana - Grafana
- 4. Log shipping architecture File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 5. Focus: Elasticsearch File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 13. Daily indices are a good start 2016.11.18 2016.11.19 2016.11.22 2016.11.23. . . Indexing is faster for smaller indices Deletes are cheap Search can be performed on indices that are needed Static indices are cache friendly indexing most searches
- 14. Daily indices are a good start 2016.11.18 2016.11.19 2016.11.22 2016.11.23. . . Indexing is faster for smaller indices Deletes are cheap Search can be performed on indices that are needed Static indices are cache friendly indexing most searches We delete whole indices
- 21. Slice using size Predictable searching and indexing performance Better indices balancing Fewer shards Easier handling of spiky loads Less costs because of better hardware utilization
- 22. Proper Elasticsearch configuration Keep index.refresh_interval at maximum possible value 1 sec -> 100%, 5 sec -> 125%, 30 sec -> 175% You can loosen up merges - possible because of heavy aggregation use - segments_per_tier -> higher - max_merge_at_once-> higher - max_merged_segment -> lower All prefixed with index.merge.policy } higher indexing throughput
- 25. Hot – cold architecture ES hot ES cold ES cold -Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold
- 26. Hot – cold architecture logs_2016.11.22 ES hot ES cold ES cold -Dnode.attr.tag=hot -Dnode.attr.tag=cold -Dnode.attr.tag=cold curl -XPUT localhost:9200/logs_2016.11.22 -d '{ "settings" : { "index.routing.allocation.exclude.tag" : "cold", "index.routing.allocation.include.tag" : "hot" } }'
- 27. Hot – cold architecture logs_2016.11.22 ES hot ES cold ES cold indexing
- 28. Hot – cold architecture logs_2016.11.22 logs_2016.11.23 ES hot ES cold ES cold indexing
- 29. Hot – cold architecture logs_2016.11.22 logs_2016.11.23 ES hot ES cold ES cold indexing move index after day ends curl -XPUT localhost:9200/logs_2016.11.22/_settings -d '{ "index.routing.allocation.exclude.tag" : "hot", "index.routing.allocation.include.tag” : "cold" }'
- 30. Hot – cold architecture logs_2016.11.23 logs_2016.11.22 ES hot ES cold ES cold indexing
- 31. Hot – cold architecture logs_2016.11.23 logs_2016.11.24 logs_2016.11.22 ES hot ES cold ES cold indexing
- 32. Hot – cold architecture logs_2016.11.23 logs_2016.11.24 logs_2016.11.22 ES hot ES cold ES cold indexing move index after day ends
- 33. Hot – cold architecture logs_2016.11.24 logs_2016.11.22 logs_2016.11.23 ES hot ES cold ES cold indexing
- 35. Hot – cold architecture summary ES cold Optimize costs – different hardware for different tier Performance – use case optimized hardware Isolation – long running searches don’t affect indexing
- 42. Focus: Centralized Buffer File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 45. Kafka & topics security_logs access_logs app1_logs app2_logs Kafka stores data in topics written on disk
- 50. Things to remember when using Kafka Scales by adding more partitions not threads The more IOPS the better Keep the # of consumers equal to # of partitions Replicas used for HA and FT only Offsets stored per consumer – multiple destinations easily possible
- 51. Focus: Shipper File Shipper File Shipper File Shipper Centralized Buffer ES ES ES ES ES ES ES ES ES data
- 53. Buffers performance & availability batches & threads when central buffer is gone
- 54. Buffer types Disk || memory || combined hybrid approach On source || centralized App Buffer App Buffer file or local log shipper easy scaling – fewer moving parts often with the use of lightweight shipper App App Kafka / Redis / Logstash / etc… one place for all changes extra features made easy (like TTL) ES ES
- 56. Protocols UDP – fast, cool for the application, not reliable TCP – reliable (almost) application gets ACK when written to buffer Application level ACKs may be needed HTTP RELP Beats Kafka Logstash, rsyslog, Fluentd Logstash, rsyslog Logstash, Filebeat Logstash, rsyslog, Filebeat, Fluentd
- 57. Choosing the shipper application rsyslog Elasticsearch http socket memory & disk assisted queues
- 59. What about OS? Say NO to swap Set the right disk scheduler CFQ for spinning disks deadline for SSD Use proper mount options for ext4 noatime nodirtime data=writeback, nobarier For bare metal check CPU governor disable transparent huge pages /proc/sys/vm/nr_hugepages=0