SlideShare a Scribd company logo
logstash
Infrastructure Coders
Melbourne
April 2013
David Lutz
@dlutzy
What does logstash do?
It does "stuff" with log files.
Typical day (or night) in the life of a
sysadmin...
Something's wrong.
Check the log files.
How?
grep
cat
grep
sed
awk
tail
sort
and pipes
lots of pipes
Infrastructure coders logstash
Fine if you have one server. But
what if you have 10 or 100 or
1000
for i in `seq 1 10` ; do ssh server$i blah blah; done
cluster ssh
Splunk perhaps?
Problems with Splunk...
1.eats log files
2. digests data
3. spits it out
into other apps
inputs
● amqp
● drupal_dblog
● eventlog
● exec
● file
● ganglia
● gelf
● gemfire
● generator
● heroku
● irc
● log4j
● lumberjack
● pipe
● redis
● relp
● sqs
● stdin
● stomp
● syslog
● tcp
● twitter
● udp
● xmpp
● zenoss
● zeromq
filters
● alter
● anonymize
● checksum
● csv
● date
● dns
● environment
● gelfify
● geoip
● grep
● grok
● grokdiscovery
● grokdiscovery
● json
● kv
● metrics
● multiline
● mutate
● noop
● split
● syslog_pri
● urldecode
● xml
● zeromq
outputs
● amqp
● boundary
● circonus
● cloudwatch
● datadog
● elasticsearch
● elasticsearch_http
● elasticsearch_river
● email
● exec
● file
● ganglia
● gelf
● gemfire
● graphite
● graphtastic
● http
● internal
● irc
● juggernaut
● librato
● loggly
● lumberjack
● metriccatcher
● mongodb
● nagios
● nagios_nsca
● null
● opentsdb
● pagerduty
● pipe
● redis
● riak
● riemann
● sns
● sqs
● statsd
● stdout
● stomp
● syslog
● tcp
● websocket
● xmpp
● zabbix
● zeromq
How to: install logstash
wget http://logstash.objects.
dreamhost.com/release/logstash-
1.1.9-monolithic.jar
easy!
How to: run logstash
java -jar logstash-1.1.9-monolithic.
jar agent -f logstash.conf -- web
easy!
How to: get some apache logs in
input {
tcp {
type => "apache"
port => 3333
}
}
How to: get some apache logs in
tail -f /var/log/apache2/access.log |
nc localhost 3333
How to: digest the logs
filter {
grok {
type => "apache"
pattern => "%{COMBINEDAPACHELOG}"
}
date {
type => "apache"
timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
}
}
How to: output to elasticsearch
output {
elasticsearch {
embedded => false
}
}
How to: output to elasticsearch
and graphite via statsd
output {
elasticsearch {
embedded => false
}
statsd {
increment => "apache.response.%{response}"
}
}
Ad

More Related Content

What's hot (20)

Project update - heat (up to pike-1)
Project update - heat (up to pike-1)Project update - heat (up to pike-1)
Project update - heat (up to pike-1)
Rico Lin
 
100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...
100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...
100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...
Lucidworks
 
Trace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdminsTrace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdmins
Sysdig
 
My talk from PgConf.Russia 2016
My talk from PgConf.Russia 2016My talk from PgConf.Russia 2016
My talk from PgConf.Russia 2016
Alex Chistyakov
 
Heat project onboarding
Heat  project onboardingHeat  project onboarding
Heat project onboarding
Rico Lin
 
How Openstack is Built
How Openstack is BuiltHow Openstack is Built
How Openstack is Built
Anton Weiss
 
Configuring Syslog by Octavio
Configuring Syslog by OctavioConfiguring Syslog by Octavio
Configuring Syslog by Octavio
Rowell Dionicio
 
Writing External Rsyslog Plugins
Writing External Rsyslog PluginsWriting External Rsyslog Plugins
Writing External Rsyslog Plugins
Rainer Gerhards
 
The Popper Experimentation Protocol and CLI tool
The Popper Experimentation Protocol and CLI toolThe Popper Experimentation Protocol and CLI tool
The Popper Experimentation Protocol and CLI tool
Ivo Jimenez
 
Redis深入浅出
Redis深入浅出Redis深入浅出
Redis深入浅出
ruoyi ruan
 
OpenStack - heat on boarding
OpenStack - heat on boardingOpenStack - heat on boarding
OpenStack - heat on boarding
Rico Lin
 
Week1
Week1Week1
Week1
reneedv
 
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)
Ontico
 
Performance is a feature! - Developer South Coast - part 2
Performance is a feature!  - Developer South Coast - part 2Performance is a feature!  - Developer South Coast - part 2
Performance is a feature! - Developer South Coast - part 2
Matt Warren
 
Harry Potter and the Daemons of Berkeley
Harry Potter and the Daemons of BerkeleyHarry Potter and the Daemons of Berkeley
Harry Potter and the Daemons of Berkeley
Alex Chistyakov
 
Infrastructure as code might be literally impossible
Infrastructure as code might be literally impossibleInfrastructure as code might be literally impossible
Infrastructure as code might be literally impossible
ice799
 
Debugging Complex Systems - Erlang Factory SF 2015
Debugging Complex Systems - Erlang Factory SF 2015Debugging Complex Systems - Erlang Factory SF 2015
Debugging Complex Systems - Erlang Factory SF 2015
lpgauth
 
Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicum
Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicumBsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicum
Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicum
Scott Tsai
 
Node collaboration - Exported Resources and PuppetDB
Node collaboration - Exported Resources and PuppetDBNode collaboration - Exported Resources and PuppetDB
Node collaboration - Exported Resources and PuppetDB
m_richardson
 
Building GUI App with Electron and Lisp
Building GUI App with Electron and LispBuilding GUI App with Electron and Lisp
Building GUI App with Electron and Lisp
fukamachi
 
Project update - heat (up to pike-1)
Project update - heat (up to pike-1)Project update - heat (up to pike-1)
Project update - heat (up to pike-1)
Rico Lin
 
100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...
100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...
100 Billion Documents And Counting: Rebuilding Message Search at Slack - Josh...
Lucidworks
 
Trace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdminsTrace everything, when APM meets SysAdmins
Trace everything, when APM meets SysAdmins
Sysdig
 
My talk from PgConf.Russia 2016
My talk from PgConf.Russia 2016My talk from PgConf.Russia 2016
My talk from PgConf.Russia 2016
Alex Chistyakov
 
Heat project onboarding
Heat  project onboardingHeat  project onboarding
Heat project onboarding
Rico Lin
 
How Openstack is Built
How Openstack is BuiltHow Openstack is Built
How Openstack is Built
Anton Weiss
 
Configuring Syslog by Octavio
Configuring Syslog by OctavioConfiguring Syslog by Octavio
Configuring Syslog by Octavio
Rowell Dionicio
 
Writing External Rsyslog Plugins
Writing External Rsyslog PluginsWriting External Rsyslog Plugins
Writing External Rsyslog Plugins
Rainer Gerhards
 
The Popper Experimentation Protocol and CLI tool
The Popper Experimentation Protocol and CLI toolThe Popper Experimentation Protocol and CLI tool
The Popper Experimentation Protocol and CLI tool
Ivo Jimenez
 
Redis深入浅出
Redis深入浅出Redis深入浅出
Redis深入浅出
ruoyi ruan
 
OpenStack - heat on boarding
OpenStack - heat on boardingOpenStack - heat on boarding
OpenStack - heat on boarding
Rico Lin
 
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)
Ontico
 
Performance is a feature! - Developer South Coast - part 2
Performance is a feature!  - Developer South Coast - part 2Performance is a feature!  - Developer South Coast - part 2
Performance is a feature! - Developer South Coast - part 2
Matt Warren
 
Harry Potter and the Daemons of Berkeley
Harry Potter and the Daemons of BerkeleyHarry Potter and the Daemons of Berkeley
Harry Potter and the Daemons of Berkeley
Alex Chistyakov
 
Infrastructure as code might be literally impossible
Infrastructure as code might be literally impossibleInfrastructure as code might be literally impossible
Infrastructure as code might be literally impossible
ice799
 
Debugging Complex Systems - Erlang Factory SF 2015
Debugging Complex Systems - Erlang Factory SF 2015Debugging Complex Systems - Erlang Factory SF 2015
Debugging Complex Systems - Erlang Factory SF 2015
lpgauth
 
Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicum
Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicumBsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicum
Bsdtw17: mariusz zaborski: case studies of sandboxing base system with capsicum
Scott Tsai
 
Node collaboration - Exported Resources and PuppetDB
Node collaboration - Exported Resources and PuppetDBNode collaboration - Exported Resources and PuppetDB
Node collaboration - Exported Resources and PuppetDB
m_richardson
 
Building GUI App with Electron and Lisp
Building GUI App with Electron and LispBuilding GUI App with Electron and Lisp
Building GUI App with Electron and Lisp
fukamachi
 

Similar to Infrastructure coders logstash (20)

Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
Tim Bunce
 
Logstash: Get to know your logs
Logstash: Get to know your logsLogstash: Get to know your logs
Logstash: Get to know your logs
SmartLogic
 
Elastic search
Elastic searchElastic search
Elastic search
Torstein Hansen
 
Luigi presentation NYC Data Science
Luigi presentation NYC Data ScienceLuigi presentation NYC Data Science
Luigi presentation NYC Data Science
Erik Bernhardsson
 
Syslog.ppt
Syslog.pptSyslog.ppt
Syslog.ppt
ifsharahmad
 
Scaling an ELK stack at bol.com
Scaling an ELK stack at bol.comScaling an ELK stack at bol.com
Scaling an ELK stack at bol.com
Renzo Tomà
 
High performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodbHigh performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodb
Wei Shan Ang
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logs
Jeremy Cook
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in style
DefconRussia
 
Zero mq logs
Zero mq logsZero mq logs
Zero mq logs
Tomas Doran
 
Security Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budgetSecurity Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budget
Juan Berner
 
Handout: 'Open Source Tools & Resources'
Handout: 'Open Source Tools & Resources'Handout: 'Open Source Tools & Resources'
Handout: 'Open Source Tools & Resources'
BDPA Education and Technology Foundation
 
Introduction to Docker (and a bit more) at LSPE meetup Sunnyvale
Introduction to Docker (and a bit more) at LSPE meetup SunnyvaleIntroduction to Docker (and a bit more) at LSPE meetup Sunnyvale
Introduction to Docker (and a bit more) at LSPE meetup Sunnyvale
Jérôme Petazzoni
 
17 Linux Basics #burningkeyboards
17 Linux Basics #burningkeyboards17 Linux Basics #burningkeyboards
17 Linux Basics #burningkeyboards
Denis Ristic
 
Network Automation: Ansible 101
Network Automation: Ansible 101Network Automation: Ansible 101
Network Automation: Ansible 101
APNIC
 
The Accidental DBA
The Accidental DBAThe Accidental DBA
The Accidental DBA
PostgreSQL Experts, Inc.
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
Jérôme Petazzoni
 
Stripe CTF3 wrap-up
Stripe CTF3 wrap-upStripe CTF3 wrap-up
Stripe CTF3 wrap-up
Stripe
 
Elk stack @inbot
Elk stack @inbotElk stack @inbot
Elk stack @inbot
Jilles van Gurp
 
Docker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12XDocker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12X
Jérôme Petazzoni
 
Application Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.keyApplication Logging in the 21st century - 2014.key
Application Logging in the 21st century - 2014.key
Tim Bunce
 
Logstash: Get to know your logs
Logstash: Get to know your logsLogstash: Get to know your logs
Logstash: Get to know your logs
SmartLogic
 
Luigi presentation NYC Data Science
Luigi presentation NYC Data ScienceLuigi presentation NYC Data Science
Luigi presentation NYC Data Science
Erik Bernhardsson
 
Scaling an ELK stack at bol.com
Scaling an ELK stack at bol.comScaling an ELK stack at bol.com
Scaling an ELK stack at bol.com
Renzo Tomà
 
High performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodbHigh performance json- postgre sql vs. mongodb
High performance json- postgre sql vs. mongodb
Wei Shan Ang
 
Turbo charge your logs
Turbo charge your logsTurbo charge your logs
Turbo charge your logs
Jeremy Cook
 
Kettunen, miaubiz fuzzing at scale and in style
Kettunen, miaubiz   fuzzing at scale and in styleKettunen, miaubiz   fuzzing at scale and in style
Kettunen, miaubiz fuzzing at scale and in style
DefconRussia
 
Security Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budgetSecurity Monitoring for big Infrastructures without a Million Dollar budget
Security Monitoring for big Infrastructures without a Million Dollar budget
Juan Berner
 
Introduction to Docker (and a bit more) at LSPE meetup Sunnyvale
Introduction to Docker (and a bit more) at LSPE meetup SunnyvaleIntroduction to Docker (and a bit more) at LSPE meetup Sunnyvale
Introduction to Docker (and a bit more) at LSPE meetup Sunnyvale
Jérôme Petazzoni
 
17 Linux Basics #burningkeyboards
17 Linux Basics #burningkeyboards17 Linux Basics #burningkeyboards
17 Linux Basics #burningkeyboards
Denis Ristic
 
Network Automation: Ansible 101
Network Automation: Ansible 101Network Automation: Ansible 101
Network Automation: Ansible 101
APNIC
 
Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)Introduction to Docker (as presented at December 2013 Global Hackathon)
Introduction to Docker (as presented at December 2013 Global Hackathon)
Jérôme Petazzoni
 
Stripe CTF3 wrap-up
Stripe CTF3 wrap-upStripe CTF3 wrap-up
Stripe CTF3 wrap-up
Stripe
 
Docker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12XDocker and Containers for Development and Deployment — SCALE12X
Docker and Containers for Development and Deployment — SCALE12X
Jérôme Petazzoni
 
Ad

More from David Lutz (7)

2014 devops conferences
2014 devops conferences2014 devops conferences
2014 devops conferences
David Lutz
 
Which watcher watches CloudWatch
Which watcher watches CloudWatch Which watcher watches CloudWatch
Which watcher watches CloudWatch
David Lutz
 
antifragility - DevOps Melbourne November 2013
antifragility - DevOps Melbourne November 2013 antifragility - DevOps Melbourne November 2013
antifragility - DevOps Melbourne November 2013
David Lutz
 
Devops, Dungeons & Dragons
Devops, Dungeons & Dragons Devops, Dungeons & Dragons
Devops, Dungeons & Dragons
David Lutz
 
Monitoring and metrics in the cloud
Monitoring and metrics in the cloudMonitoring and metrics in the cloud
Monitoring and metrics in the cloud
David Lutz
 
Graphite
GraphiteGraphite
Graphite
David Lutz
 
Kanban thank you maam
Kanban thank you maamKanban thank you maam
Kanban thank you maam
David Lutz
 
2014 devops conferences
2014 devops conferences2014 devops conferences
2014 devops conferences
David Lutz
 
Which watcher watches CloudWatch
Which watcher watches CloudWatch Which watcher watches CloudWatch
Which watcher watches CloudWatch
David Lutz
 
antifragility - DevOps Melbourne November 2013
antifragility - DevOps Melbourne November 2013 antifragility - DevOps Melbourne November 2013
antifragility - DevOps Melbourne November 2013
David Lutz
 
Devops, Dungeons & Dragons
Devops, Dungeons & Dragons Devops, Dungeons & Dragons
Devops, Dungeons & Dragons
David Lutz
 
Monitoring and metrics in the cloud
Monitoring and metrics in the cloudMonitoring and metrics in the cloud
Monitoring and metrics in the cloud
David Lutz
 
Kanban thank you maam
Kanban thank you maamKanban thank you maam
Kanban thank you maam
David Lutz
 
Ad

Recently uploaded (20)

The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
SOFTTECHHUB
 
machines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdfmachines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdf
AmirStern2
 
Build With AI - In Person Session Slides.pdf
Build With AI - In Person Session Slides.pdfBuild With AI - In Person Session Slides.pdf
Build With AI - In Person Session Slides.pdf
Google Developer Group - Harare
 
Cybersecurity Threat Vectors and Mitigation
Cybersecurity Threat Vectors and MitigationCybersecurity Threat Vectors and Mitigation
Cybersecurity Threat Vectors and Mitigation
VICTOR MAESTRE RAMIREZ
 
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
João Esperancinha
 
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Mike Mingos
 
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Markus Eisele
 
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
Ivano Malavolta
 
Reimagine How You and Your Team Work with Microsoft 365 Copilot.pptx
Reimagine How You and Your Team Work with Microsoft 365 Copilot.pptxReimagine How You and Your Team Work with Microsoft 365 Copilot.pptx
Reimagine How You and Your Team Work with Microsoft 365 Copilot.pptx
John Moore
 
Artificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptxArtificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptx
03ANMOLCHAURASIYA
 
AI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamsonAI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamson
UXPA Boston
 
fennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solutionfennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solution
shallal2
 
Top 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptx
Top 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptxTop 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptx
Top 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptx
mkubeusa
 
Unlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web AppsUnlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web Apps
Maximiliano Firtman
 
Viam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdfViam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdf
camilalamoratta
 
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Wonjun Hwang
 
Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?
Eric Torreborre
 
Com fer un pla de gestió de dades amb l'eiNa DMP (en anglès)
Com fer un pla de gestió de dades amb l'eiNa DMP (en anglès)Com fer un pla de gestió de dades amb l'eiNa DMP (en anglès)
Com fer un pla de gestió de dades amb l'eiNa DMP (en anglès)
CSUC - Consorci de Serveis Universitaris de Catalunya
 
Zilliz Cloud Monthly Technical Review: May 2025
Zilliz Cloud Monthly Technical Review: May 2025Zilliz Cloud Monthly Technical Review: May 2025
Zilliz Cloud Monthly Technical Review: May 2025
Zilliz
 
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Cyntexa
 
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
The No-Code Way to Build a Marketing Team with One AI Agent (Download the n8n...
SOFTTECHHUB
 
machines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdfmachines-for-woodworking-shops-en-compressed.pdf
machines-for-woodworking-shops-en-compressed.pdf
AmirStern2
 
Cybersecurity Threat Vectors and Mitigation
Cybersecurity Threat Vectors and MitigationCybersecurity Threat Vectors and Mitigation
Cybersecurity Threat Vectors and Mitigation
VICTOR MAESTRE RAMIREZ
 
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
Could Virtual Threads cast away the usage of Kotlin Coroutines - DevoxxUK2025
João Esperancinha
 
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Optima Cyber - Maritime Cyber Security - MSSP Services - Manolis Sfakianakis ...
Mike Mingos
 
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Enterprise Integration Is Dead! Long Live AI-Driven Integration with Apache C...
Markus Eisele
 
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
On-Device or Remote? On the Energy Efficiency of Fetching LLM-Generated Conte...
Ivano Malavolta
 
Reimagine How You and Your Team Work with Microsoft 365 Copilot.pptx
Reimagine How You and Your Team Work with Microsoft 365 Copilot.pptxReimagine How You and Your Team Work with Microsoft 365 Copilot.pptx
Reimagine How You and Your Team Work with Microsoft 365 Copilot.pptx
John Moore
 
Artificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptxArtificial_Intelligence_in_Everyday_Life.pptx
Artificial_Intelligence_in_Everyday_Life.pptx
03ANMOLCHAURASIYA
 
AI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamsonAI-proof your career by Olivier Vroom and David WIlliamson
AI-proof your career by Olivier Vroom and David WIlliamson
UXPA Boston
 
fennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solutionfennec fox optimization algorithm for optimal solution
fennec fox optimization algorithm for optimal solution
shallal2
 
Top 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptx
Top 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptxTop 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptx
Top 5 Benefits of Using Molybdenum Rods in Industrial Applications.pptx
mkubeusa
 
Unlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web AppsUnlocking Generative AI in your Web Apps
Unlocking Generative AI in your Web Apps
Maximiliano Firtman
 
Viam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdfViam product demo_ Deploying and scaling AI with hardware.pdf
Viam product demo_ Deploying and scaling AI with hardware.pdf
camilalamoratta
 
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Kit-Works Team Study_아직도 Dockefile.pdf_김성호
Wonjun Hwang
 
Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?Shoehorning dependency injection into a FP language, what does it take?
Shoehorning dependency injection into a FP language, what does it take?
Eric Torreborre
 
Zilliz Cloud Monthly Technical Review: May 2025
Zilliz Cloud Monthly Technical Review: May 2025Zilliz Cloud Monthly Technical Review: May 2025
Zilliz Cloud Monthly Technical Review: May 2025
Zilliz
 
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Everything You Need to Know About Agentforce? (Put AI Agents to Work)
Cyntexa
 

Infrastructure coders logstash

  • 2. What does logstash do? It does "stuff" with log files.
  • 3. Typical day (or night) in the life of a sysadmin... Something's wrong. Check the log files. How?
  • 8. Fine if you have one server. But what if you have 10 or 100 or 1000 for i in `seq 1 10` ; do ssh server$i blah blah; done cluster ssh Splunk perhaps? Problems with Splunk...
  • 9. 1.eats log files 2. digests data 3. spits it out into other apps
  • 10. inputs ● amqp ● drupal_dblog ● eventlog ● exec ● file ● ganglia ● gelf ● gemfire ● generator ● heroku ● irc ● log4j ● lumberjack ● pipe ● redis ● relp ● sqs ● stdin ● stomp ● syslog ● tcp ● twitter ● udp ● xmpp ● zenoss ● zeromq
  • 11. filters ● alter ● anonymize ● checksum ● csv ● date ● dns ● environment ● gelfify ● geoip ● grep ● grok ● grokdiscovery ● grokdiscovery ● json ● kv ● metrics ● multiline ● mutate ● noop ● split ● syslog_pri ● urldecode ● xml ● zeromq
  • 12. outputs ● amqp ● boundary ● circonus ● cloudwatch ● datadog ● elasticsearch ● elasticsearch_http ● elasticsearch_river ● email ● exec ● file ● ganglia ● gelf ● gemfire ● graphite ● graphtastic ● http ● internal ● irc ● juggernaut ● librato ● loggly ● lumberjack ● metriccatcher ● mongodb ● nagios ● nagios_nsca ● null ● opentsdb ● pagerduty ● pipe ● redis ● riak ● riemann ● sns ● sqs ● statsd ● stdout ● stomp ● syslog ● tcp ● websocket ● xmpp ● zabbix ● zeromq
  • 13. How to: install logstash wget http://logstash.objects. dreamhost.com/release/logstash- 1.1.9-monolithic.jar easy!
  • 14. How to: run logstash java -jar logstash-1.1.9-monolithic. jar agent -f logstash.conf -- web easy!
  • 15. How to: get some apache logs in input { tcp { type => "apache" port => 3333 } }
  • 16. How to: get some apache logs in tail -f /var/log/apache2/access.log | nc localhost 3333
  • 17. How to: digest the logs filter { grok { type => "apache" pattern => "%{COMBINEDAPACHELOG}" } date { type => "apache" timestamp => "dd/MMM/yyyy:HH:mm:ss Z" } }
  • 18. How to: output to elasticsearch output { elasticsearch { embedded => false } }
  • 19. How to: output to elasticsearch and graphite via statsd output { elasticsearch { embedded => false } statsd { increment => "apache.response.%{response}" } }
  翻译: