Fluentd and Docker - running fluentd within a docker container
8 likes5,621 views
Walks through the basics of Fluentd, as run on Docker. Introduces a demo where fluentd is run in a docker container.
More Related Content
What's hot (20)
Viewers also liked (10)
Similar to Fluentd and Docker - running fluentd within a docker container (20)
More from Treasure Data, Inc. (16)
Recently uploaded (20)
Fluentd and Docker - running fluentd within a docker container
- 1. FLUENTD: UNIFIED LOGGING LAYER John Hammink August 11, 2015 Docker San Francisco Meetup
- 2. About Me • A recovering software & QA engineer turned digital artist once interested in fractals; • now into data visualization based on large datasets rendered directly to GPU (RGL, various Python GL libraries, etc.) • github: jammink2; twitter: rijksband
- 3. Tweet NOW! “At #svde learning how to collect more event data using #Fluentd”
- 4. WHAT’S FLUENTD? An extensible & reliable data collection tool simple core + plugins buffering, HA (failover), load balancing, etc. like syslogd
- 5. What’s Fluentd? > Data collector for unified logging layer > Streaming data transfer based on JSON > Written in Ruby > Gem based various plugins > http://www.fluentd.org/plugins > Working in production > http://www.fluentd.org/testimonials
- 7. ✓ duplicated code for error handling... ✓ messy code for retrying mechnism... Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system bash scripts ruby scripts rsync log file bash python scripts custom loggger cron other custom scripts...
- 9. Blueflood MongoDB Hadoop Metrics Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Your system filter / buffer / route
- 10. extensible
- 11. CORE PLUGINS • Divide & Conquer • Buffering & Retries • Error Handling • Message Routing • Parallelism • Read Data • Parse Data • Buffer Data • Write Data • Format Data Common Concerns Use Case Specific
- 12. architecture
- 13. INTERNAL ARCHITECTURE “input-ish” “output-ish” Input Parser Buffer Output FormatterFilter
- 14. Internal Architecture (Simplified) Input Buffer Output Plugin Plugin Plugin 2012-02-04 01:33:51 myapp.buylog { “user”: ”me”, “path”: “/buyItem”, “price”: 150, “referer”: “/landing” } time tag record
- 15. Architecture :: Input plugins Input HTTP+JSON (in_http) File tail (in_tail) Syslog (in_syslog) ... Plugin ✓ Receive logs ✓ Or pull logs from data sources ✓ in non-blocking manner
- 16. Architecture :: Output plugins Plugin ✓ Write or send event logs Output File (out_file) Amazon S3 (out_s3) MongoDB (out_mongo) ...
- 17. Architecture :: Buffer plugins Plugin ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safety Buffer Memory (buf_memory) File (buf_file)
- 18. Architecture :: Buffer plugins Plugin ✓ Improve performance ✓ Provide reliability ✓ Provide thread-safety chunk chunk chunk output Input
- 21. DIVIDE & CONQUER & RETRY error retry error retry retry retry
- 22. reliable process
- 23. THIS?
- 24. OR THIS?
- 25. M X N → M + N Nagios MongoDB Hadoop Alerting Amazon S3 Analysis Archiving MySQL Apache Frontend Access logs syslogd App logs System logs Backend Databases buffer/filter/route
- 26. use cases
- 28. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag backend.apache </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to ES and HDFS <match backend.*> type mongo database fluent collection test </match>
- 31. # logs from a file <source> type tail path /var/log/httpd.log format apache2 tag web.access </source> # logs from client libraries <source> type forward port 24224 </source> # store logs to ES and HDFS <match *.*> type copy <store> type elasticsearch logstash_format true </store> <store> type webhdfs host namenode port 50070 path /path/on/hdfs/ </store> </match>
- 32. FLUENTD ON KUBERNETES (NOV 2015)
- 33. FLUENTD LOGGING DRIVER (APR 2015)
- 34. Tweet Again! “Happy v1 #k8s and congrats #Fluentd for becoming a #docker logging driver”
- 35. DEMO: FLUENTD + DOCKER
- 36. THANK YOU! AND TREASURE DATA IS HIRING! WWW.TREASUREDATA.COMC/CAREERS