MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive

Aug 19, 20191 like266 views

MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented.

MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive

Atlas Data Lake
Technical Deep-Dive
Subhead for the presentation goes here
Craig Wilson, Senior Staff Engineer, MongoDB

State of Affairs
Businesses have a humongous amount of data
• IDC predicts that by 2025 global data will reach 175 Zettabytes and 49% of it will reside in the public
cloud.
Cloud storage is cost-effective
Cloud storage is hard to operationalize

A New Service Offered by MongoDB Atlas
Access long-term data
Query long-term data
Analyze long-term data

Requirements
Look and act like MongoDB
Access customer’s data securely
Handle queries over vast amounts of data
Handle long-running queries
Efficient use of resources

Language
Must be able to communicate with our drivers
Written in Go
Implemented a TCP server
Used mongo-go-driver’s wireprotocol package
Used mongo-go-driver's bson package

Security
Must have the same security as MongoDB
Users configured in Atlas
Implemented MongoDB’s security model
Require the use of TLS + SNI(Server Name Indicator)

Behavior
Must behave like MongoDB
Implemented commands for a read-only server
Used the server’s aggregation engine

Security: Customers
Customers have complete control
Provide us with an IAM Role
Configure your buckets
Configure your users in Atlas

Security: Atlas
Atlas controls access to your data
Storage of IAM Role
Temporary Credentials

Configuration
Customers control their data layout
Stores
Databases, Collections
DataSources
CollectionCollection
Store Store
DataSource DataSource
DataSource

Configuration: File Formats
• BSON (gzipped)
• JSON (gzipped)
• Avro (gzipped)
• CSV/TSV (gzipped)
• Parquet
• XLSX

Configuration (S3 Bucket): ent-archive
/archive/customers
- a-m.json
- n-z.json
/archive/invoices
- 2019
- 1.parquet
- 2.parquet
- 2018
- 1.parquet
- 2017.json.gz
- 2016.json.gz

Configuration: Store
s3 : {
name: "ent-archive",
bucket: "ent-archive",
region: "us-east-1",
prefix: "/archive/"
}

Configuration: Data
history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}

Configuration: Data (Future)
history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}.json.gz
}, {
store: "atlas",
db: "customers",
collection: "invoices"
}]
}

Processing
MQL à Distributed MQL
Parse
Parallelize
Distribute

Architecture
Atlas
Control
Control
Plane
Compute
Plane
Data
Plane
DataLake
Frontend
DataLake
Agent
Load Balancer
Load Balancer
DataLake
Frontend
DataLake
Agent
Load Balancer
Load Balancer
DataLake
Frontend
DataLake
Agent
Load Balancer
Load Balancer

Architecture
Atlas
Control
Control
Plane
Compute
Plane
Data
Plane
DataLake
Frontend
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent

Query Example: $limit
Map:
{ $match: { year: { $gt: 2000 } } }
{ $limit: 10 }
Reduce:
{ $limit: 10 }
{ $match: { year: { $gt: 2000 } } }
{ $limit: 10 }

$Query Example: $group Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "$amount" }, totalAvg_count: { $sum: 1 } } } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" } } } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } } { $group: { _id: "$year", totalAvg: { $avg: "amount" } } }$

Future
More supported MongoDB operators.
$out
$merge
Geo operators
Full Text Search

Future
Integrations
Atlas
Microsoft Azure
Google Cloud

Craig Wilson
Senior Staff Engineer, MongoDB

The document discusses centralized logging with the Elastic Stack. It describes the components of the Elastic Stack including Beats, Logstash, Elasticsearch, and Kibana. It explains how Beats collects log data and sends it to Logstash for ingestion into Elasticsearch for storage, search, and analysis. Kibana is used for visualizing and managing the logged data. The document provides an example of using the Elastic Stack with Filebeat, Logstash, and Elasticsearch for log ingestion and analysis of access logs from a web server. It discusses benefits like reduced costs using a hot-warm architecture with the Elastic Stack.

Presto Summit 2018 - 03 - Starburst CBOkbajda

Social Data and Log Analysis Using MongoDBTakahiro Inoue

The document describes the speaker's role as a data scientist at a social game company. It outlines their analytic architecture which uses MongoDB to store and analyze social data and access logs from their mobile game. Hadoop is used for pre-processing raw log data before loading it into MongoDB. MapReduce operations in MongoDB are used to aggregate and analyze the data to calculate metrics like daily/hourly pageviews and unique users. The results are stored in normalized collections to enable further analysis and visualization of billions of data records.

Kafka as an Eventing System to Replatform a Monolith into Microservices confluent

(Madhulika Tripathi, Intuit) Kafka Summit SF 2018 Breaking down monolithic applications into smaller manageable microservices can be a tough challenge. But the benefits are many. Faster changes, developer productivity, maintainability, scalability and high performance are a few of the motivators that make companies undertake this difficult journey. At Intuit, we have our fair share of monolithic applications. One such application is Quickbooks Online, our accounting product for small businesses. In order to decompose the application, we needed to create new services, and reduce footprint of data in the monolith by moving it to new services in a phased manner. As more and more data and services keep moving out of the monolith, this data now distributed across multiple microservices needs to be synchronized in near real time to provide a seamless and fast experience to the customers of our product. To achieve this, we are using Kafka as our eventing backbone that can aid us in keeping distributed data in sync, without compromising performance and user experience. Guaranteed publishing of financial events with no loss, high accuracy and performance is of utmost importance as majority of Intuit products deal with highly sensitive, financial data. Strong ordering guarantees is another important criteria that Kafka can provide with low latency and high throughput. Use cases for data and streaming analytics, insights, personalization, machine-learning-based predictions, can all be unlocked by adopting Kafka as our distributed streaming platform. This talk will take you through Intuit’s journey of building a distributed, asynchronous system using Kafka. Specifically about the choices made, challenges faced, the adaptations clients had to make and how we see Kafka powering our future!

Redis OverviewHoang Long

This document provides an overview of Redis, including its creation, features, and persistence methods. Redis is an open-source, in-memory key-value store that was created by Salvatore Sanfilippo to solve the problem of storing real-time page view data from multiple websites. It features different data types that can be stored as keys, master-slave replication for availability, and two persistence methods: RDB, which takes periodic snapshots of the dataset, and AOF, which logs all write operations to reconstruct the dataset from disk on startup.

.NET Fest 2017. Константин Проскурдин. Marten как хранилище документов для .N...NETFest

This document discusses Marten, an ORM for .NET that allows for document-oriented data access to PostgreSQL. It begins with an overview of microservices and document databases, then discusses PostgreSQL's support for JSON and JSONB data types. It introduces Marten and demonstrates its basic features for querying and updating JSONB documents in PostgreSQL from .NET code. The presentation also covers more advanced Marten capabilities like indexing, profiling, and tenancy support, with examples.

MongoDB + SpringNorberto Leite

A Cheapskates Guide to AWS v2.0Michael Soh

Presented at the AWS Washington DC User Group and updated for September 25 While AWS is very affordable, it can get really costly for the average at-home user. The free tier is a great way to get started but how do you use AWS long-term while keeping costs low? Michael Soh, unashamed cheap skate and the unofficial chief innovation engineer at the Inova Translational Medicine Institute, shares his journey into AWS and how the average user can get the most out of their dollars. This work is released via the CC-BY-SA license.

Cloud talkUTD Computer Security Group

This document provides an introduction to cloud computing, including what cloud is, its benefits and drawbacks, common cloud service models (SaaS, PaaS, IaaS), major cloud providers, and common cloud computing services. Key cloud computing services discussed include compute services (like AWS EC2 and Google Compute Engine), databases, storage, and additional AI/ML and serverless services. The document also highlights some free cloud credits and resources available for students.

.Net Distributed CachingPaul Fryer

Distributed caching improves performance by storing data in multiple locations for faster retrieval. It works by storing copies of data on multiple machines to improve scalability and availability. There are different scopes for caching like application, tenant, user, and session data. The cache key identifies the data and includes the scope and uniqueness. The time to live determines how long cached data remains valid before being refreshed, with lower times improving consistency at the cost of performance. Code examples demonstrate caching patterns for services and tracking statistics across distributed caches.

Iceberg: a fast table format for S3DataWorks Summit

Netflix’s Big Data Platform team manages data warehouse in Amazon S3 with over 60 petabytes of data and writes hundreds of terabytes of data every day. With a data warehouse at this scale, it is a constant challenge to keep improving performance. This talk will focus on Iceberg, a new table metadata format that is designed for managing huge tables backed by S3 storage. Iceberg decreases job planning time from minutes to under a second, while also isolating reads from writes to guarantee jobs always use consistent table snapshots. In this session, you'll learn: • Some background about big data at Netflix • Why Iceberg is needed and the drawbacks of the current tables used by Spark and Hive • How Iceberg maintains table metadata to make queries fast and reliable • The benefits of Iceberg's design and how it is changing the way Netflix manages its data warehouse • How you can get started using Iceberg Speaker Ryan Blue, Software Engineer, Netflix

A Cheapskates Guide to AWSMichael Soh

Originally presented at the Arlington AWS Meetup on May 23, 2018. While AWS is very affordable, it can get really costly for the average at-home user. The free tier is a great way to get started but how do you use AWS long-term while keeping costs low? Michael Soh, unashamed cheap skate and the unofficial chief innovation engineer at the Inova Translational Medicine Institute, shares his journey into AWS and how the average user can get the most out of their dollars. This work is released via the CC-BY-SA license

Using MongoDB For BigData in 20 MinutesAndrás Fehér

This document provides an overview of using MongoDB for big data applications. It describes what MongoDB is, including that it is a open-source, document-oriented NoSQL database. It then covers some key MongoDB concepts and features like replica sets for high availability, sharding for horizontal scaling, and using MapReduce and Apache Spark with MongoDB. The document concludes that MongoDB is a fast, flexible, and scalable database solution for big data, though deploying sharded clusters requires planning.

Building Pinterest Real-Time Ads Platform Using Kafka Streams confluent

Building Pinterest Real-Time Ads Platform Using Kafka Streams (Liquan Pei + Boyang Chen, Pinterest) Kafka Summit SF 2018 In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business. The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.

Python and MongoDB as a Market Data Platform by James BlackburnPyData

This document discusses using Python and MongoDB as a scalable platform for storing time series market data. It outlines some of the challenges of storing different types and sizes of financial data from various sources. The goals are to access 10 years of 1-minute data in under 1 second, store all data types in a single location, and have a system that is fast, complete, scalable, and supports agile development. MongoDB is chosen as it matches well with Python and allows for fast, low latency access. The system implemented uses MongoDB to store data in a way that supports versioning, arbitrary data types, and efficient storage and retrieval of pandas DataFrames. Performance tests show significant speed improvements over SQL and other tick databases for accessing intra

AmazonRedshiftAhasan Habib

Amazon Redshift is a data warehousing tool developed by Amazon that allows for storing and analyzing large amounts of data. It uses a columnar data storage model and massively parallel processing to enable extremely fast queries. Redshift can be easily accessed through a web service API and supports standard SQL syntax. It is a cloud-based solution that provides high security and scalability for analytical workloads.

Big Data at Tube: Events to Insights to ActionMurtaza Doctor

GraphTalk München - Einführung in GraphdatenbankenNeo4j

This document discusses using Neo4j for logistics and parcel routing applications. It describes a large logistics carrier that needed a new parcel routing system to handle up to 5 million parcels per day with real-time routing capabilities. Neo4j was able to provide the required extreme availability, performance, and scalability through its graph database approach since a logistics network can be modeled as a graph. Queries were also simplified compared to relational databases for the complex routing needs. The flexible graph data model could also better reflect real-world data variations.

Introduction to Azure DocumentDBRadenko Zec

This document provides an introduction to Microsoft Azure DocumentDB. It discusses how DocumentDB is a non-relational or NoSQL database that stores data in JSON documents. It also overview how DocumentDB provides scalability, high availability, and fast performance for large document workloads. Key features of DocumentDB discussed include its resource and interaction models, indexing, consistency options, querying capabilities, and support for JavaScript transactions.

Introduction to new high performance storage engines in mongodb 3.0Henrik Ingo

This document provides an introduction to new high performance storage engines being introduced in MongoDB 2.8, including WiredTiger. It discusses how WiredTiger provides improved performance over MMAPv1 for both read and write heavy workloads through features like document level locking, write optimized data structures, and compression. The document also outlines different configuration options and tunables for WiredTiger to optimize performance based on factors like fitting the working data set in cache or disk.

Redis Day TLV 2018 - RediSearch AggregationsRedis Labs

This document discusses RediSearch aggregations, which allow processing search results to produce statistical insights. Aggregations take a search query, group and reduce the results, apply transformations, and sort. Key steps include filtering results, grouping and reducing with functions like count and average, applying expressions, and sorting. Examples show finding top GitHub committers and visits by hour. Scaling aggregations to multiple nodes requires pushing processing stages to nodes and merging results, such as summing counts or taking list intersections.

Exploring the replication and sharding in MongoDBIgor Donchovski

Redundancy and high availability are the basis for all production deployments. Database systems with large data sets or high throughput applications can challenge the capacity of a single server like CPU for high query rates or RAM for large working sets. Adding more CPU and RAM for vertical scaling is limited. Systems need horizontal scaling by distributing data across multiple servers. MongoDB supports horizontal scaling through sharding. Each shard consist of replica set that provides Redundancy and high availability.

The Next Generation Software Stack: MeteorMongoDB

Meteor is the next take on agile development on the full JavaScript stack. Based on established JavaScript tools like Node, JQuery and Underscore, it still brings a fresh and integrated approach. And MongoDB is very much its heart: Minimongo implements a client side MongoDB API for manipulating your data model; Transparent replication of data between client and server; Using WebSockets, MongoDB oplog events replicate immediately to all clients, making it simple to do distributed applications "Google Docs style."

MongoDB .local Chicago 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

More Related Content

What's hot (20)

Kafka as an Eventing System to Replatform a Monolith into Microservices confluent

Redis OverviewHoang Long

.NET Fest 2017. Константин Проскурдин. Marten как хранилище документов для .N...NETFest

MongoDB + SpringNorberto Leite

A Cheapskates Guide to AWS v2.0Michael Soh

Cloud talkUTD Computer Security Group

.Net Distributed CachingPaul Fryer

Iceberg: a fast table format for S3DataWorks Summit

A Cheapskates Guide to AWSMichael Soh

Using MongoDB For BigData in 20 MinutesAndrás Fehér

Building Pinterest Real-Time Ads Platform Using Kafka Streams confluent

Python and MongoDB as a Market Data Platform by James BlackburnPyData

AmazonRedshiftAhasan Habib

Big Data at Tube: Events to Insights to ActionMurtaza Doctor

GraphTalk München - Einführung in GraphdatenbankenNeo4j

Introduction to Azure DocumentDBRadenko Zec

Introduction to new high performance storage engines in mongodb 3.0Henrik Ingo

Redis Day TLV 2018 - RediSearch AggregationsRedis Labs

Exploring the replication and sharding in MongoDBIgor Donchovski

The Next Generation Software Stack: MeteorMongoDB

Kafka as an Eventing System to Replatform a Monolith into Microservices confluent

Redis OverviewHoang Long

.NET Fest 2017. Константин Проскурдин. Marten как хранилище документов для .N...NETFest

MongoDB + SpringNorberto Leite

A Cheapskates Guide to AWS v2.0Michael Soh

Cloud talkUTD Computer Security Group

.Net Distributed CachingPaul Fryer

Iceberg: a fast table format for S3DataWorks Summit

A Cheapskates Guide to AWSMichael Soh

Using MongoDB For BigData in 20 MinutesAndrás Fehér

Building Pinterest Real-Time Ads Platform Using Kafka Streams confluent

Python and MongoDB as a Market Data Platform by James BlackburnPyData

AmazonRedshiftAhasan Habib

Big Data at Tube: Events to Insights to ActionMurtaza Doctor

GraphTalk München - Einführung in GraphdatenbankenNeo4j

Introduction to Azure DocumentDBRadenko Zec

Introduction to new high performance storage engines in mongodb 3.0Henrik Ingo

Redis Day TLV 2018 - RediSearch AggregationsRedis Labs

Exploring the replication and sharding in MongoDBIgor Donchovski

The Next Generation Software Stack: MeteorMongoDB

Similar to MongoDB .local Houston 2019: MongoDB Atlas Data Lake Technical Deep Dive (20)

MongoDB .local Chicago 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB World 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local Bengaluru 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local London 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

This document summarizes MongoDB Atlas Data Lake, a new service that allows customers to access, query, and analyze long-term data stored in AWS S3 buckets. It implements MongoDB's query language and security model to provide a familiar interface for working with structured data in object storage. The service is read-only, distributed, and optimized to handle queries over vast amounts of data efficiently using MongoDB's aggregation engine. Customers maintain full control over their data and how it is configured and accessed.

MongoDB .local Munich 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

IBM THINK 2018 - IBM Cloud SQL Query IntroductionTorsten Steinbach

Torsten Steinbach and Chris Glew present IBM Cloud Query, a serverless analytics service that allows users to run ANSI SQL queries against data stored in cloud object storage. Some key points: - IBM Cloud Query allows users to query data in various open formats like CSV, Parquet, and JSON stored in cloud object storage using SQL, with results also stored in object storage. - It has a pay-per-query pricing model with no infrastructure to manage. Queries can be run via a web console, REST API, or Python client. - The presentation outlines the architecture and provides examples of using Cloud Query for log analytics, data exploration, and building serverless data pipelines with Cloud Functions.

MongodbThiago Veiga

MongoDB is an open-source document database that provides high performance, high availability, and automatic scaling. It stores data in flexible, JSON-like documents, enabling storage of data with complex relationships easily and supporting polyglot persistence. MongoDB can be used for applications such as content management systems, user profiles, logs, and more. It provides indexing, replication, load balancing and aggregation capabilities.

Developing hybrid applications with informixIBM_Info_Management

The document summarizes a presentation on developing hybrid applications with Informix. It discusses how the Informix wire listener allows unified access to JSON, relational, and time series data through MongoDB and REST APIs. It enables applications to execute SQL statements and perform joins across different data sources. Sample applications demonstrate basic CRUD operations using MongoDB and REST interfaces with Informix.

Webinar: The Anatomy of the Cloudant Data LayerIBM Cloud Data Services

Instrumenting and Scaling Databases with EnvoyDaniel Hochman

Every request to a database at Lyft is proxied by Envoy, providing complete visibility into the L3/L4 aspects of database interactions. This allows engineers to easily visualize changes to a database's load profile and pinpoint the root cause if necessary. Lyft has also open-sourced codecs for MongoDB, DynamoDB, and Redis. Protocol codecs in combination with custom filters yield benefits ranging from operation-level observability to horizontal scalability via sharding. Using Envoy for this purpose means that enhancements are implemented once and usable across a polyglot stack. The talk demonstrates Envoy's utility beyond traditional RPC service interactions in the network.

Lessons learned mongodb to redhsift - meetup July 1st Tel AvivRoie Shavit

AWS Well Architected-Info Session WeCloudDataWeCloudData

This document provides an overview of Big Data on AWS and discusses key concepts related to architecting Big Data solutions on AWS. It covers topics such as data security, scalability, performance efficiency, cost optimization, operational excellence, reliability, and disaster recovery. It includes examples of AWS services for Big Data like Amazon S3, DynamoDB, Redshift, EMR, and provides sample questions related to choosing the right AWS services for scenarios and designing Big Data architectures.

Virtual training intro to InfluxDB - June 2021InfluxData

In this training webinar, we will walk you through the basics of InfluxDB – the purpose-built time series database. InfluxDB has everything you need from a time series platform in a single binary – a multi-tenanted time series database, UI and dashboarding tools, background processing and monitoring agent. This one-hour session will include the training and time for live Q&A. What you will learn Core concepts of time series databases An overview of the InfluxDB platform How to ingesting and query data in InfluxDB

Solving enterprise challenges through scale out storage & big compute finalAvere Systems

Google Cloud Platform, Avere Systems, and Cycle Computing experts will share best practices for advancing solutions to big challenges faced by enterprises with growing compute and storage needs. In this “best practices” webinar, you’ll hear how these companies are working to improve results that drive businesses forward through scalability, performance, and ease of management. The slides were from a webinar presented January 24, 2017. The audience learned: - How enterprises are using Google Cloud Platform to gain compute and storage capacity on-demand - Best practices for efficient use of cloud compute and storage resources - Overcoming the need for file systems within a hybrid cloud environment - Understand how to eliminate latency between cloud and data center architectures - Learn how to best manage simulation, analytics, and big data workloads in dynamic environments - Look at market dynamics drawing companies to new storage models over the next several years Presenters communicated a foundation to build infrastructure to support ongoing demand growth.

MongoDB World 2018: Bumps and Breezes: Our Journey from RDBMS to MongoDBMongoDB

The document summarizes the journey of migrating from an RDBMS to MongoDB. It describes the pre-MongoDB RDBMS environment, reasons for choosing MongoDB, and the evolution of the MongoDB environment over time. The evolution involved some bumps in configuring databases and applications, but also many breezes like improved performance, flexibility and scalability. Benchmarking showed MongoDB could handle more concurrent users. Future plans include using MongoDB 4.0 features and further optimizing sharding performance.

Cloud-based Data Lake for Analytics and AITorsten Steinbach

IBM's Cloud-based Data Lake for Analytics and AI presentation covered: 1) IBM's cloud data lake provides serverless architecture, low barriers to entry, and pay-as-you-go pricing for analytics on data stored in cloud object storage. 2) The data lake offers SQL-based data exploration, transformation, and analytics capabilities as well as industry-leading optimizations for time series and geospatial data. 3) Security features include customer-controlled encryption keys and options to hide SQL queries and keys from IBM.

MongoDB Evenings DC: Get MEAN and Lean with Docker and KubernetesMongoDB

This document discusses running MongoDB and Kubernetes together to enable lean and agile development. It proposes using Docker containers to package applications and leverage tools like Kubernetes for deployment, management and scaling. Specifically, it recommends: 1) Using Docker to containerize applications and define deployment configurations. 2) Deploying to Kubernetes where services and replication controllers ensure high availability and scalability. 3) Treating databases specially by running them as "naked pods" assigned to labeled nodes with appropriate resources. 4) Demonstrating deployment of a sample MEAN stack application on Kubernetes with MongoDB and discussing future work around experimentation and blue/green deployments.

Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI

This document summarizes Satoshi Tagomori's presentation on Treasure Data, a data analytics service company. It discusses Treasure Data's use of Ruby for various components of its platform including its logging (Fluentd), ETL (Embulk), scheduling (PerfectSched), and storage (PlazmaDB) technologies. The document also provides an overview of Treasure Data's architecture including how it collects, stores, processes, and visualizes customer data using open source tools integrated with services like Hadoop and Presto.

IBM THINK 2019 - Self-Service Cloud Data Management with SQL Torsten Steinbach

SQL is a powerful language to express data transformations. But did you know that you can also use IBM Cloud SQL to convert data between various data formats and layouts on disks? In this session, you will see the full power of using SQL Query to move and transform your cloud data in an entirely self-service fashion. You can specify any data format, layout or partitioning with a simple SQL statement. See how you can move and transform terabytes of data in the cloud in a very scalable fashion and still being charged only for the individual SQL movement and transformation jobs without having standing costs.

MongoDB .local Chicago 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB World 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local Bengaluru 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local London 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB .local Munich 2019: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

IBM THINK 2018 - IBM Cloud SQL Query IntroductionTorsten Steinbach

MongodbThiago Veiga

Developing hybrid applications with informixIBM_Info_Management

Webinar: The Anatomy of the Cloudant Data LayerIBM Cloud Data Services

Instrumenting and Scaling Databases with EnvoyDaniel Hochman

Lessons learned mongodb to redhsift - meetup July 1st Tel AvivRoie Shavit

AWS Well Architected-Info Session WeCloudDataWeCloudData

Virtual training intro to InfluxDB - June 2021InfluxData

Solving enterprise challenges through scale out storage & big compute finalAvere Systems

MongoDB World 2018: Bumps and Breezes: Our Journey from RDBMS to MongoDBMongoDB

Cloud-based Data Lake for Analytics and AITorsten Steinbach

MongoDB Evenings DC: Get MEAN and Lean with Docker and KubernetesMongoDB

Data Analytics Service Company and Its Ruby UsageSATOSHI TAGOMORI

IBM THINK 2019 - Self-Service Cloud Data Management with SQL Torsten Steinbach

More from MongoDB (20)

MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB

This presentation discusses migrating data from other data stores to MongoDB Atlas. It begins by explaining why MongoDB and Atlas are good choices for data management. Several preparation steps are covered, including sizing the target Atlas cluster, increasing the source oplog, and testing connectivity. Live migration, mongomirror, and dump/restore options are presented for migrating between replicasets or sharded clusters. Post-migration steps like monitoring and backups are also discussed. Finally, migrating from other data stores like AWS DocumentDB, Azure CosmosDB, DynamoDB, and relational databases are briefly covered.

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB

MongoDB Kubernetes operator and MongoDB Open Service Broker are ready for production operations. Learn about how MongoDB can be used with the most popular container orchestration platform, Kubernetes, and bring self-service, persistent storage to your containerized applications. A demo will show you how easy it is to enable MongoDB clusters as an External Service using the Open Service Broker API for MongoDB

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB

Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: Common components of an IoT solution The challenges involved with managing time-series data in IoT applications Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

MongoDB SoCal 2020: MongoDB Atlas Jump StartMongoDB

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB

Our clients have unique use cases and data patterns that mandate the choice of a particular strategy. To implement these strategies, it is mandatory that we unlearn a lot of relational concepts while designing and rapidly developing efficient applications on NoSQL. In this session, we will talk about some of our client use cases, the strategies we have adopted, and the features of MongoDB that assisted in implementing these strategies.

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB

Encryption is not a new concept to MongoDB. Encryption may occur in-transit (with TLS) and at-rest (with the encrypted storage engine). But MongoDB 4.2 introduces support for Client Side Encryption, ensuring the most sensitive data is encrypted before ever leaving the client application. Even full access to your MongoDB servers is not enough to decrypt this data. And better yet, Client Side Encryption can be enabled at the "flick of a switch". This session covers using Client Side Encryption in your applications. This includes the necessary setup, how to encrypt data without sacrificing queryability, and what trade-offs to expect.

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB

When you need to model data, is your first instinct to start breaking it down into rows and columns? Mine used to be too. When you want to develop apps in a modern, agile way, NoSQL databases can be the best option. Come to this talk to learn how to take advantage of all that NoSQL databases have to offer and discover the benefits of changing your mindset from the legacy, tabular way of modeling data. We’ll compare and contrast the terms and concepts in SQL databases and MongoDB, explain the benefits of using MongoDB compared to SQL databases, and walk through data modeling basics so you feel confident as you begin using MongoDB.

MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB

The document discusses guidelines for ordering fields in compound indexes to optimize query performance. It recommends the E-S-R approach: placing equality fields first, followed by sort fields, and range fields last. This allows indexes to leverage equality matches, provide non-blocking sorts, and minimize scanning. Examples show how indexes ordered by these guidelines can support queries more efficiently by narrowing the search bounds.

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB

The document describes a methodology for data modeling with MongoDB. It begins by recognizing the differences between document and tabular databases, then outlines a three step methodology: 1) describe the workload by listing queries, 2) identify and model relationships between entities, and 3) apply relevant patterns when modeling for MongoDB. The document uses examples around modeling a coffee shop franchise to illustrate modeling approaches and techniques.

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB

Virtual assistants are becoming the new norm when it comes to daily life, with Amazon’s Alexa being the leader in the space. As a developer, not only do you need to make web and mobile compliant applications, but you need to be able to support virtual assistants like Alexa. However, the process isn’t quite the same between the platforms. How do you handle requests? Where do you store your data and work with it to create meaningful responses with little delay? How much of your code needs to change between platforms? In this session we’ll see how to design and develop applications known as Skills for Amazon Alexa powered devices using the Go programming language and MongoDB.

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB

Il n’a jamais été aussi facile de commander en ligne et de se faire livrer en moins de 48h très souvent gratuitement. Cette simplicité d’usage cache un marché complexe de plus de 8000 milliards de $. La data est bien connu du monde de la Supply Chain (itinéraires, informations sur les marchandises, douanes,…), mais la valeur de ces données opérationnelles reste peu exploitée. En alliant expertise métier et Data Science, Upply redéfinit les fondamentaux de la Supply Chain en proposant à chacun des acteurs de surmonter la volatilité et l’inefficacité du marché.

MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDBMongoDB

Chaque entreprise devient une entreprise de logiciels, fournissant des solutions client pour accéder à une variété de services et d'informations. Les entreprises commencent maintenant à valoriser leurs données et à obtenir de meilleures informations pour l'entreprise. Un défi crucial consiste à s'assurer que ces données sont toujours disponibles et sécurisées pour être conformes aux objectifs commerciaux de l'entreprise et aux contraintes réglementaires des pays. MongoDB fournit la couche de sécurité dont vous avez besoin, venez découvrir comment sécuriser vos données avec MongoDB.