Monitoring Kubernetes with Elasticsearch Services - Ted Jung, Consulting Architect, Elastic :: AWS Summit Seoul 2019

1
Ted Jung
Consulting Architect
Observable Kubernetes
with Elastic Stack

2
Ted, Jung
Consulting Architect @Elastic

3
• Intro to Elastic Stack
• Kubernetes (challenges, visibility)
• Logging and Monitoring Kubernetes
• APM tracing
• Demo
Agenda

44
Intro to Elastic Stack
● Elastic stack
● Elastic Elasticsearch Service

5
Elastic Stack
Store, Search, & AnalyzeElasticsearch
Visualize & ManageKibana
Beats IngestLogstash
SaaS
Elasticsearch
Service
Self Managed
Elastic Cloud
Enterprise
Standalone
Deployment
The Elastic Stack

6
US West
(Oregon)
US West
(N. Cal)
US East (N.
Virginia)
EU
(Ireland)
EU
(Germany)
S. America
(East)
Asia Pacific
(Sydney)
Asia Pacific
(Tokyo)
Asia Pacific
(Singapore)
Elasticsearch
The Official Hosted Elasticsearch & Kibana Solution
AWS 9Regions
VM or On-Prem VM or On-Prem
The Self Managed Elasticsearch & Kibana Solution

77
Kubernetes
● Challenges
● Visibility

9
Container
Runtime
Kubernetes is…
§ Kubernetes is complicated
§ Kubernetes is extremely dynamic, which makes monitoring & logging very challenging
§ All components provide metrics
§ On top of all this there are a ton of different providers

10
Kubernetes Visibility Challenges…

11
Three Pillars of Observability
Elastic brings a holistic solution
for Kubernetes observability in one platform.

1212
Logging
● Cluster level logging
● Service logging (eg, nginx, mysql)
● Custom application logging

13
• Need for a logging solution
– Kubernetes does not have a native solution
– kubectl logs is too hard for large clusters
• Cluster-level logging
– Logs have separate storage and lifecycle independent of nodes, pods and containers
– Kubernetes provides no native storage solution for log data
• Application-level logging
– Complicated
– Packaged applications (eg. nginx)
– Custom applications
Kubernetes Logging

14
kubectl create -f filebeat-kubernetes.yaml
Cluster
Service
Application
Enriched Logging & Better Log Collection
with Filebeat
• Cluster level logging
• Service level logs (eg. Nginx logs)
– Auto-discovery
• Application logs
– Cluster level logging (stdout, stderr)
– Custom log file

15
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- condition:
contains:
kubernetes.container.image: "nginx"
config:
- module: nginx
access: # For nginx access log
prospector:
type: docker
containers.ids:
- "${data.kubernetes.container.id}"
Filebeat Auto-Discovery

16
• Kubernetes module
• Monitors pods and services
– Cluster, pod & container metrics
– Application metrics through auto-discovery
(eg. Nginx)
• Metrics sources - Cover them ALL
– Kubelet (heapster, cAdvisor)
– kube-state-metric
– Kubernetes events
– Prometheus module (beta)
• Curated infra UI
– Dedicated Kibana app
Comprehensive Metrics Collection with
Metricbeat

17
Metricbeat Modules
Simplify collection and visualization of common metrics
● Aerospike module
● Apache module
● Ceph module
● Couchbase module
● Docker module
● Dropwizard module
● Elasticsearch module
● Etcd module
● Golang module
● Graphite module
● HAProxy module
● HTTP module
● Jolokia module
● Kafka module
● Kibana module
● Kubernetes module
● kvm module
● Logstash module
● Memcached module
● MongoDB module
● Munin module
● MySQL module
● Nginx module
● PHP_FPM module
● PostgreSQL module
● Prometheus module
● RabbitMQ module
● Redis module
● System module
● uwsgi module
● vSphere module
● Windows module
● ZooKeeper module

18
Filebeat Modules
Simplify collection and visualization of common metrics

21
Application Monitoring with Elastic APM
Adding end-user experience and application-level monitoring to the stack
RUM

2222
Observable Kubernetes
Elastic Stack:
Three Pillars of Observability in One Platform
● Logging
● Metrics
● APM Tracing

24
filebeat
metricbeat
packetbeat
heartbeat
Visualization
indexing
Demo Env
Store

25
Master&WorkerNode 1
Metricbeat
Filebeat
Worker Node 2
Metricbeat
Filebeat
Worker Node 3
Metricbeat
Filebeat
Heartbeat, Packetbeat
Kubernetes Daemonset for Beat

26
Simple deployment templates to gain insights faster
One-click upgrades of the Elastic Stack
Centrally manage all your deployments
Enhanced security and performance for all deployments
Elastic Elasticsearch Service

27
Howto deploy
Create deployment
Cloud(AWS)
Region
Demo Env

29
Access URL: http://**.**.**.**:?????/
Demo Application

Monitoring Kubernetes with Elasticsearch Services - Ted Jung, Consulting Architect, Elastic :: AWS Summit Seoul 2019

Recommended

More Related Content

What's hot (20)

Similar to Monitoring Kubernetes with Elasticsearch Services - Ted Jung, Consulting Architect, Elastic :: AWS Summit Seoul 2019 (20)

More from Amazon Web Services Korea (20)

Recently uploaded (20)

Monitoring Kubernetes with Elasticsearch Services - Ted Jung, Consulting Architect, Elastic :: AWS Summit Seoul 2019