Tuning Elasticsearch Indexing Pipeline for Logs

Editor's Notes

• #3: Rafał starts and passes mic to Radu
• #4: Rafal slide – describe the talk brefily !!! Ask people how many of the audience used the tools
• #5: Radu slide we did some tests, we’ll share configs and benchmarks – here are the versions Logstash 1.5 – the final version will be up soon Rsyslog 8.9 – the current stable (note: most distros come with 5.x or 7.x) ES is a search engine based on Apache Lucene Current version is 1.5, next major is 2.0 with lots of changes. Many related to Lucene 5.0 Not the only tools for logging, there are many other tools, both open source and commercial, that can receive logs, parse them, buffer them and index them
• #6: Rafal slide
• #7: Rafal slide * Ask how many people know about Logstash
• #8: Rafal slide
• #9: Rafal slide
• #10: Radu Assume we want to centralize syslog Forward syslog via TCP/UDP on a port to Logstash On the Logstash side, you can use the TCP input to listen to that port and parse syslog messages You’d use the ES output to forward to ES you can use a Java binary, but HTTP is better Logstash comes with a template for ES index, but for perf tests we’ll use our own Specify where (index,type – like a DB and a table)
• #11: Radu - 1.3 CPUs
• #12: Radu – segue to tuning, pass the mic
• #13: Rafal Flush size – 1000 lowered from default 5000
• #14: Rafal
• #15: Rafal
• #16: Rafal Syslog is just TCP + Grok We changed that and we are not parsing the syslog format exactly – we wanted to parse additional things and wanted to show how to parse unstructured data
• #21: The bound was: - hardware (high CPU usage) - JSON lines codec is not parallelized, while GROK is - But if you want to do your homework you can do another run with JSON filter instead of codec and that will give the possibility of parallelization
• #22: Radu Many people hate it, maybe because of docs I like it because it’s light and fast and has surprisingly rich functionality
• #23: Like Logstash, it’s modular, you can use inputs to get data in, message modifiers to parse data and outputs to pass it on The flow of data is a bit different Inputs may have multiple threads, and they write to a main queue On the main queue, worker threads can do filtering, format messages using templates (will talk later) and run actions (parsing/output) You can have action queues as well, with their own threads => async You can have rulesets, which let you separate flows of input – parse – output (e.g. One ruleset for local logs, one for remote logs)
• #24: Typical setup is to have it on each server, push to ES directly, buffer if necessary
• #25: Load modules Impstats is for monitoring, then tcp and ES Start the tcp listener Template – how the JSON that we send to ES will look like Action – send to ES, using the template, specify index/type, use bulks, retry on failure
• #28: Bigger memory buffer Increase bulk size Moar worker threads
• #29: Not using more because ES is using the rest – Rafal will talk about that in a bit RAM has increased because of the queue size
• #30: Clear win
• #31: But not really apples for apples, because rsyslog has dedicated syslog parsers Still, not only for syslog, can parse unstructured data via mmnormalize Refer to a rulebase, which looks much like grok patterns, with two differences: Normally, patterns like number or date aren’t regexes but specific parsers. Faster but less flexible. The one above is equivalent to the Logstash grok seen earlier Builds a parse tree on startup, helps with speed if you have many rules
• #32: Radu
• #35: More throughput with less CPU usage
• #37: Before moving on, one more thing: in production you probably want to use disk assisted queues instead of in-memory queues like the ones we had here. DA queue is in-memory queue that can spill to disk. Specify that via file name and give it a threshold Spilling is smart: Normally in memory When it reaches high watermark it starts writing to disk, but it does so in batches, so resumes to memory when lowwatermark Side-benefit: can save and reload memory queue contents when restarting rsyslog
• #38: Rafal
• #39: Rafał Index a document It goes to ES first to transaction log next to inverted index It is replicated on transaction log level
• #40: Rafał
• #41: Rafał
• #42: Rafał
• #43: Rafał
• #44: Rafał
• #45: Rafał
• #46: Rafał Throttling – the default is 20, we are using 200, so we are actually going for 10 times more (we are usind SSD drives here)
• #47: Rafał
• #48: Rafał Cheaper filters and aggregations are on top The more expensive are at the bottom
• #49: Radu Index as fast as we can How much data we can put in a single index at a decent indexing rate before searches took too long a good practice is to have time-based indices (e.g. Keep logs for a week, have one per day). We want to benchmark that + separating indexing load from search load by putting today’s index on different nodes than the „old” ones
• #50: Rafal Rate slowly goes down, because merges happen and because the index is slowly getting bigger
• #51: Rafał 40-50 m @ 20 seconds Most expensive query takes 20 sec on average Filters (quick ones) takes subseconds Some aggs takes up to 5 seconds on average
• #52: Rafał Spikes because of merges, the big spike is because the merge happen and after the merge the queries are actually faster Most expensive queries take 15 seconds
• #53: Radu Want to benchmark TB indices. Because: Indexing is better because of merging Searching recent data is better because idx is smaller Deleting entired indices is better But what granularity? Use-cases for small (high indexing, small retention, CPU contraint) vs big (low idx, high retention, mem constraint) granularity doesn’t affect cold search perf
• #54: Rafal Tell about hot and cold setup The drop is because cold nodes were full
• #55: Rafal
• #57: Radu