Migrating a legacy logging system: Etsy’s journey to Elastic Cloud

Jul 14, 20200 likes1,956 views

See how the Observability team at Etsy uses Elastic Cloud on Google to provide their developers with a top-class logging solution.

Logs: Etsy’s Journey to Elastic Cloud
Migrating a legacy logging system to Elasticsearch service on Elastic Cloud
Stefano Vita <svita@etsy.com>

Agenda
1
Overview
2
Why?
3
How?
4
What did go well?
5
What we wish we knew?

Etsy is the global
marketplace for
unique and
creative goods. It’s
home to a universe
of special,
extraordinary
items, from unique
handcrafted pieces
to vintage
treasures.

Use cases of
Elasticsearch
• Convenient way to analyze log events
• Alerting
• Reporting

Logging numbers
8 Tb 70k 1.6k
Data Indexed
daily
Events
ingested every
second
Hosts sending
logs to the
pipeline

Etsy’s Decision to Migrate and Key
Considerations

Why migrate Etsy’s
logging system? ● Etsy was migrating entirely to
Google Cloud
● Elasticsearch is a complex system
that requires specialized knowledge
(especially in a logging use case)
● Elasticsearch 2.4 old and
unmaintained (EOL date was
02/2018)
A few reasons...

Why migrate Etsy’s
logging system? ● Alert fatigue for the whole team
● Maintaining Elasticsearch infra is
NOT observability
● Data center shutdown
A few reasons...

Key considerations
Migration must not impact
developers’ day-to-day
work
Business as usual
Migration must be time
efficient (data center
shutdown)
Time
Migration must reduce
infrastructure
management from the
team
Reduce TOIL

Process Options
1. Move all logs to Elasticsearch
service on Elastic Cloud
2. Move only critical logs to
Elasticsearch service on Elastic
Cloud
3. Move to our Google Cloud
infrastructure using ECE (Elastic
Cloud Enterprise)
4. Move to our Google Cloud
infrastructure manually

Alternatives
● Splunk
● Stackdriver
● <name logging solution>
Considerations:
● Too many intrusive solutions for
developers
● We didn’t want to throw away the
Elasticsearch knowledge we built
over the year
● Not enough time to prototype and
roll out a change that big

Challenges
● Move stack from 2.4 to 7.x
○ Logstash 2.x can’t talk to
Elasticsearch > 6.x
○ Identify and replace
deprecated settings in
Elasticsearch
○ Learn new features
○ Deploy changes safely
● Keep two systems running in
parallel for some time

Migration Timeline
03/2018
Gathered cluster size
and wrote ﬁrst options
draft
Prod data
migrated;
Beta testing
started!
10/2019
Users fully migrated to the
new setup
01/202012/2018
Finalized
options
Contract
signed!
03/201902/2019
Prepared
migration
plan
06/2019
Dev data
migrated

Migration Successes
● Met our deadline
● Elastic support and consultants are helpful
● Happy developers
● Returning teams

Migration Successes
● Better observability into the stack
● Easier and safer management of indices and logstash pipelines
● Create, grow and shrink clusters is way easier
● Better isolation of the stream of data

What we wished we knew?
● Sizing an ES cluster is an art
○ One needs to consider volume AND throughput
● Noisy neighbors
● Support SLAs are not ideal when developing
○ Initial response on SEV3 is 1 business day
● Elastic Cloud is not just an endpoint
○ We are still responsible for indices management

What’s next?
● Improvement on the logging pipeline
● Analyze use cases and recommend best practices in Etsy

Elastic Observability provides a full-stack monitoring solution with features including: - Support for ingesting metrics, logs and traces from applications, services, databases and infrastructure across hosts, VMs and containers. - Easy addition of new data sources through built-in integrations and support for multiple ingest methods and protocols. - Capabilities for interacting with and visualizing metrics and log data through dashboards, visualizations and flexible alerting. - Long term, reliable storage of observability data through Elasticsearch and capabilities like index lifecycle management and data rollups.

T-Mobile and ElasticElasticsearch

This document summarizes presentations given by three T-Mobile employees on how they use the Elastic stack to support customer experiences. Calum Lawler discusses using Elasticsearch to analyze social messaging conversations and calculate metrics to optimize customer care response times. Michael Mitchell explains how they use Raspberry Pis and the Elastic stack for remote device testing. Jon Soini talks about moving beyond dashboards to dynamic Canvas visualizations for sharing Elastic insights.

Elastic on a Hyper-Converged Infrastructure for Operational Log AnalyticsElasticsearch

Machine Learning for Anomaly Detection, Time Series Modeling, and MoreElasticsearch

What’s Evolving in the Elastic StackElasticsearch

Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...Elasticsearch

Elastic Cloud Enterprise in Azure with DevonElasticsearch

Devon Energy is a leading independent oil and natural gas exploration and production company. Hear about their journey to augment and eventually replace their legacy SIEM solution with a homegrown analytics and automation platform. See the details of moving from using on-prem open source Elasticsearch to being the first ever user to run Elastic Cloud Enterprise in Azure. Plus, learn how the team uses Elasticsearch optimizations in their security telemetry pipeline, hear about cloud native deployment models, and see how the Logstash transform functions and using Kibana as frontend for security and operational logs have helped deliver big wins.

Log Monitoring and Anomaly Detection at Scale at ORNLElasticsearch

Larry Nichols presented on Oak Ridge National Laboratory's transition from Splunk to Elastic Stack for log monitoring and anomaly detection at scale. Some key points: - ORNL manages over 20,000 endpoints and ingests over 1.5TB of log data daily into its Elastic Stack deployment. - Elastic Stack provides increased search speed, security, and integration capabilities compared to Splunk at a lower overall cost. - ORNL leverages Elastic Stack, Kafka, NiFi and other tools for real-time data streaming and ingestion across multiple clusters for production, development and research. - The Situ anomaly detection platform, deployed within Elastic Stack, helps analysts detect unknown attacks and suspicious behavior within

Improving search at Wellcome CollectionElasticsearch

KeynoteElasticsearch

ElasticON is a search company that provides the power of a single stack, cloud and hybrid solutions, and innovations to enable search, observability, and security. It offers the Elastic Agent, a unified data shipper, and Fleet for centralized ingestion and management. Kibana Lens provides an intuitive way to explore data. Searchable snapshots allow searching across cold and frozen indexes for cost-effective archiving and compliance. Schema on read provides flexibility for new data sources and handling changes.

Logging, Metrics, and APM: The Operations TrifectaElasticsearch

Hunting for Evil with the Elastic StackElasticsearch

Building a reliable and cost effect logging system at Box Elasticsearch

Zero Latency: Building a Telemetry Platform on the Elastic StackElasticsearch

Artik cloud deview 2016NAVER D2

The document discusses Samsung ARTIK Cloud, an open data exchange platform for the Internet of Things (IoT). It allows users to easily create and connect IoT devices, collect and massage device data, and build new services and applications. The platform represents things in the cloud, facilitates interoperability across devices and clouds, and enables data to be transformed and accessed through APIs and SDKs. It also ensures user privacy by allowing users to control and grant access to their own data.

Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsElasticsearch

Industrial production process visualization with the Elastic Stack in real-ti...Elasticsearch

Protecting Your Cluster from Your HumansElasticsearch

Elasticsearch on AzureElasticsearch

This document discusses the partnership between Elastic and Microsoft Azure and highlights several products and services: 1. Elastic provides solutions for logs, metrics, application performance monitoring, uptime monitoring, security information and event management, and endpoints on the Elastic Stack that can be deployed on Azure in various ways. 2. The Elasticsearch Service on Azure is highlighted as the best way to deploy Elasticsearch, Kibana, and Elastic solutions on Azure with benefits like being hosted, secure, compliant, and always up-to-date. 3. Elastic Observability for Azure provides out-of-the-box support for Azure logs and metrics with integration for Azure Monitor and pre-built Kibana dashboards.

Capgemini: Observability within the Dutch governmentElasticsearch

The Dutch government relies on a complex mix of technologies to deliver digital services. This makes it difficult to monitor performance and identify issues when they arise. Capgemini implemented Elastic solutions to provide observability across the heterogeneous infrastructure. This allowed problems to be traced and resolved in minutes rather than days. The improved visibility has enhanced operational stability and reduced breakdowns. Capgemini sees continued growth in demand for Elastic technologies from both government and commercial customers.

Grab: Building a Healthy Elasticsearch EcosystemElasticsearch

Turning Evidence into Insights: How NCIS Leverages Elastic Elasticsearch

Empower Your Security Practitioners with Elastic SIEMElasticsearch

Logging, Metrics, and APM: The Operations Trifecta (P)Elasticsearch

Fineo Technical Overview - NextSQL for IoTJesse Yates

Fineo is a turn-key data management platform for enterprise IoT that provides a NoSQL time-series database integrated with an analytics warehouse. It offers insights with 10x lower cost and the ability to scale to 100x more data. Fineo provides a "simple" big data deployment through its web scale architecture, security/compliance features, and one-click ETL tools to enable faster adoption and lower complexity.

Sharing our best secrets: Design a distributed system from scratchAdelina Simion

The document summarizes a system design workshop for designing a note-taking application called TechyNotes. The workshop covers defining system requirements and interfaces, discussing database and storage options, designing initial and revised system architectures, and addressing scalability bottlenecks. Attendees learn a repeatable process for system design and discuss technologies like databases, load balancing, caching, and queues.

Data Day Texas 2017: Scaling Data Science at Stitch FixStefan Krawczyk

At Stitch Fix we have a lot of Data Scientists. Around eighty at last count. One reason why I think we have so many, is that we do things differently. To get their work done, Data Scientists have access to whatever resources they need (within reason), because they’re end to end responsible for their work; they collaborate with their business partners on objectives and then prototype, iterate, productionize, monitor and debug everything and anything required to get the output desired. They’re full data-stack data scientists! The teams in the organization do a variety of different tasks: - Clothing recommendations for clients. - Clothes reordering recommendations. - Time series analysis & forecasting of inventory, client segments, etc. - Warehouse worker path routing. - NLP. … and more! They’re also quite prolific at what they do -- we are approaching 4500 job definitions at last count. So one might be wondering now, how have we enabled them to get their jobs done without getting in the way of each other? This is where the Data Platform teams comes into play. With the goal of lowering the cognitive overhead and engineering effort required on part of the Data Scientist, the Data Platform team tries to provide abstractions and infrastructure to help the Data Scientists. The relationship is a collaborative partnership, where the Data Scientist is free to make their own decisions and thus choose they way they do their work, and the onus then falls on the Data Platform team to convince Data Scientists to use their tools; the easiest way to do that is by designing the tools well. In regard to scaling Data Science, the Data Platform team has helped establish some patterns and infrastructure that help alleviate contention. Contention on: Access to Data Access to Compute Resources: Ad-hoc compute (think prototype, iterate, workspace) Production compute (think where things are executed once they’re needed regularly) For the talk (and this post) I only focused on how we reduced contention on Access to Data, & Access to Ad-hoc Compute to enable Data Science to scale at Stitch Fix. With that I invite you to take a look through the slides.

More Related Content

What's hot (20)

Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...Elasticsearch

Elastic Cloud Enterprise in Azure with DevonElasticsearch

Log Monitoring and Anomaly Detection at Scale at ORNLElasticsearch

Improving search at Wellcome CollectionElasticsearch

KeynoteElasticsearch

Logging, Metrics, and APM: The Operations TrifectaElasticsearch

Hunting for Evil with the Elastic StackElasticsearch

Building a reliable and cost effect logging system at Box Elasticsearch

Zero Latency: Building a Telemetry Platform on the Elastic StackElasticsearch

Artik cloud deview 2016NAVER D2

Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsElasticsearch

Industrial production process visualization with the Elastic Stack in real-ti...Elasticsearch

Protecting Your Cluster from Your HumansElasticsearch

Elasticsearch on AzureElasticsearch

Capgemini: Observability within the Dutch governmentElasticsearch

Grab: Building a Healthy Elasticsearch EcosystemElasticsearch

Turning Evidence into Insights: How NCIS Leverages Elastic Elasticsearch

Empower Your Security Practitioners with Elastic SIEMElasticsearch

Logging, Metrics, and APM: The Operations Trifecta (P)Elasticsearch

Fineo Technical Overview - NextSQL for IoTJesse Yates

Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elasti...Elasticsearch

Elastic Cloud Enterprise in Azure with DevonElasticsearch

Log Monitoring and Anomaly Detection at Scale at ORNLElasticsearch

Improving search at Wellcome CollectionElasticsearch

KeynoteElasticsearch

Logging, Metrics, and APM: The Operations TrifectaElasticsearch

Hunting for Evil with the Elastic StackElasticsearch

Building a reliable and cost effect logging system at Box Elasticsearch

Zero Latency: Building a Telemetry Platform on the Elastic StackElasticsearch

Artik cloud deview 2016NAVER D2

Better Search and Business Analytics at Southern Glazer’s Wine & SpiritsElasticsearch

Industrial production process visualization with the Elastic Stack in real-ti...Elasticsearch

Protecting Your Cluster from Your HumansElasticsearch

Elasticsearch on AzureElasticsearch

Capgemini: Observability within the Dutch governmentElasticsearch

Grab: Building a Healthy Elasticsearch EcosystemElasticsearch

Turning Evidence into Insights: How NCIS Leverages Elastic Elasticsearch

Empower Your Security Practitioners with Elastic SIEMElasticsearch

Logging, Metrics, and APM: The Operations Trifecta (P)Elasticsearch

Fineo Technical Overview - NextSQL for IoTJesse Yates

Similar to Migrating a legacy logging system: Etsy’s journey to Elastic Cloud (20)

Sharing our best secrets: Design a distributed system from scratchAdelina Simion

Data Day Texas 2017: Scaling Data Science at Stitch FixStefan Krawczyk

Big data @ Hootsuite analtyicsClaudiu Coman

Enterprise Data World 2018 - Building Cloud Self-Service Analytical SolutionDmitry Anoshin

This session will cover building the modern Data Warehouse by migration from the traditional DW platform into the cloud, using Amazon Redshift and Cloud ETL Matillion in order to provide Self-Service BI for the business audience. This topic will cover the technical migration path of DW with PL/SQL ETL to the Amazon Redshift via Matillion ETL, with a detailed comparison of modern ETL tools. Moreover, this talk will be focusing on working backward through the process, i.e. starting from the business audience and their needs that drive changes in the old DW. Finally, this talk will cover the idea of self-service BI, and the author will share a step-by-step plan for building an efficient self-service environment using modern BI platform Tableau.

Tips and Tricks for Migrating to Exchange OnlineSteve Goodman

Integrating ArchivesSpace and Archivematica at the Bentley Historical LibraryMax Eckard

Max Eckard, Lead Archivist for Digital Initiatives at the Bentley Historical Library, will cover the Bentley's integration of ArchivesSpace and Archivematica to streamline digital archiving workflows. He will highlight the decision-making process behind integrating both systems, things he wishes he’d known then that he knows now, goals for the future, and other tips and tricks. In his role at the Bentley Historical Library, Max oversees the digitization program, digital curation activities, web archives, and associated infrastructure.

[Virtual Meetup] Using Elasticsearch as a Time-Series Database in the Endpoin...Anna Ossowski

Elasticsearch is used as a time series database to store historical data from ThousandEyes' Endpoint Agent. It was chosen over other options like MongoDB and InfluxDB for its ability to scale horizontally, create complex reports, and answer unexpected questions. The architecture involves ingesting data from agents into Kafka and then using Elasticsearch connectors to load it into Elasticsearch. Various applications then query Elasticsearch to power dashboards and analytics. Lessons learned include having separate clusters per product and using filters before aggregations to improve query performance. Future plans include scaling the cluster and evaluating routing to co-locate related data.

slidesChristopher Bishop

ArchiveShuttle is a software solution that provides automated archive migrations between different archive systems. It uses a modular architecture that is scalable, flexible, and allows for faster migrations compared to traditional methods. The software can migrate archive data on-premises, to the cloud, or using appliances. It has automated workflows to migrate archive data for active users as well as departed "leaver" users in a way that handles licensing costs for cloud archives like Office 365.

Data Day Seattle 2017: Scaling Data Science at Stitch FixStefan Krawczyk

What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...Edureka!

( ELK Stack Training - https://www.edureka.co/elk-stack-trai... ) This Edureka tutorial on What Is ELK Stack will help you in understanding the fundamentals of Elasticsearch, Logstash, and Kibana together and help you in building a strong foundation in ELK Stack. Below are the topics covered in this ELK tutorial for beginners: 1. Need for Log Analysis 2. Problems with Log Analysis 3. What is ELK Stack? 4. Features of ELK Stack 5. Companies Using ELK Stack

Log aggregation: using Elasticsearch, Fluentd/Fluentbit and Kibana (EFK)Lee Myring

Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudStreamsets Inc.

Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudRick Bilodeau

How to Develop and Operate Cloud First Data PlatformsAlluxio, Inc.

Alluxio Online Meetup Feb 11, 2020 Speakers: Du Li, Electronic Arts Bin Fan, Alluxio In cloud-based software stacks, there are varying degrees of automation across different layers: infrastructure, platform, and application. The mismatch in automation often breaks balance in devops, causing ops nightmares in platforms and applications. This talk will overview two projects at Electronic Arts (EA) that address the mismatch by data orchestration: One project automatically generates configurations for all components in a large monitoring system, which reduces the daily average number of alerts from ~1000 to ~20. The other project introduces Alluxio for caching and unifying address space across ETL and analytics workloads, which substantially simplifies architecture, improves performance, and reduces ops overheads.

Tips and tricks for complex migrations to SharePoint OnlineAndries den Haan

This document provides tips and strategies for large-scale migrations to SharePoint Online. It discusses typical challenges such as dealing with large volumes of dark data from multiple sources and designing a futureproof target architecture. The document recommends rationalizing data by classifying it and identifying migration scenarios. It also demonstrates tools for inventory and analysis, and recommends maximizing automation through a migration pipeline and factory approach. Bulk migrations can be performed using tools like ShareGate that support mapping and automation.

Disenchantment: Netflix Titus, Its Feisty Team, and DaemonsC4Media

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Gmuwlg. Andrew Spyker talks about Netflix's feisty team’s work across container runtimes, scheduling & control plane, and cloud infrastructure integration. He also talks about the demons they’ve found on this journey covering operability, security, reliability and performance. Filmed at qconsf.com. Andrew Spyker worked to mature the technology base of Netflix Container Cloud (Project Titus) within the development team. Recently, he moved into a product management role collaborating with supporting Netflix infrastructure dependencies as well as supporting new container cloud usage scenarios including user on-boarding, feature prioritization/delivery and relationship management.

Data Science Machine Lerning Bigdat.pptxPriyadarshini648418

Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate.[2] Though used sometimes loosely partly due to a lack of formal definition, the best interpretation is that it is a large body of information that cannot be comprehended when used in small amounts only.[Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate.[2] Though used sometimes loosely partly due to a lack of formal definition, the best interpretation is that it is a large body of information that cannot be comprehended when used in small amounts only.[Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate.[2] Though used sometimes loosely partly due to a lack of formal definition, the best interpretation is that it is a large body of information that cannot be comprehended when used in small amounts only.[Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate.[2] Though used sometimes loosely partly due to a lack of formal definition, the best interpretation is that it is a large body of information that cannot be comprehended when used in small amounts only.[Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate.[2] Though used sometimes loosely partly due to a lack of formal definition, the best interpretation is that it is a large body of information that cannot be comprehended when used in small amounts only.[Big data primarily refers to data sets that are too large or complex to be dealt with by traditional data-processing application software. Data with many entries (rows) offer greater statistical power, while data with higher complexity (more attributes or columns) may lead to a higher false discovery rate.[2] Though used sometimes loosely partly due to a lack of formal definition, the best interpretation is that it is a large body of informa

O365Con19 - Tips and Tricks for Complex Migrations to SharePoint Online - And...NCCOMMS

This document provides tips and guidance for large-scale migrations to SharePoint Online. It discusses typical challenges like dealing with terabytes of unstructured "dark data" and the need to rationalize and classify data. The document recommends engaging stakeholders and designing a future-proof target architecture. It also demonstrates migration tools and techniques like using pipelines to automate and optimize the migration process. The key takeaway is that large-scale migrations require optimizing processes through automation since common tasks become time-consuming at large volumes of data.

Tips and tricks for complex migrations to SharePoint OnlineAndries den Haan

Behind the Scenes at Coolblue - Feb 2017Pat Hermens