MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic)

Jul 7, 2014Download as pptx, pdf3 likes8,508 views

This document discusses MongoDB implementation at Medtronic Energy and Component Center (MECC) to address data management challenges. MECC manufactures components for medical devices and generates a large volume of operational data from various sources. Previously, this data was stored in spreadsheets and relational databases, making it difficult to analyze. MongoDB was implemented to provide a flexible schema for storing component manufacturing and test data as documents. This has allowed for faster querying of complete historical data and improved reporting and analytics. While some gaps remain around enterprise acceptance and tool integration, MongoDB has provided benefits over the previous data management approaches.

MongoDB at
Medtronic
Jeff Lemmerman
Matt Chimento
1
Medtronic Energy and Component Center
*Disclaimer: We are not currently managing any Device Patient Data in MongoDB

Agenda
• Intro
• Challenge
• Approach
• Opportunities
• Takeaways
| MDT Confidential2

3
Medtronic Energy and Component Center
• MECC est. 1976
• Manufacture components for devices
• Census – 1200 Employees
• Plant Size – 190,000 Square Feet
40,000 Manufacturing
15,000 R&D Labs
38,000 Office
97,000 Common, Support,
Warehouse

Where stored?
6
Big Data: Volume, Variety, Velocity

Data Management Vision
| MDT Confidential8

| MDT Confidential9
Reporting and Analytics Variety
Operations Dashboards SPC Charts
Component Summary Time Series

| MDT Confidential10
Component Dataset In Excel

Ideal Future State
Medtronic Confidential14

Ideal Future State
Medtronic Confidential16

| MDT Confidential18
Simple, Fast Queries – Complete History

Medtronic Confidential
19
Component Time Series Data

Collect and Store Raw Time Series Data
Medtronic Confidential
20

Our experience with MongoDB
• Consulting/Training has been excellent
• Support agreement has been under-utilized
– Emails for security updates etc. are prompt
– Release cycle is frequent
• MongoDB Monitoring Service
– Potential concerns storing db stats externally
– MMS can now be hosted locally
• MongoDB Certification now available
– Udacity course, “Data Wrangling with MongoDB”

Gaps
• Enterprise acceptance of “new” approach
• Integration with off-the-shelf reporting and
analytics tools
• User interface for managing MongoDB cluster
• Developer familiarity with JSON and MongoDB
• 21 CFR Part 11 Compliance – Audit Tracing

Key Features
• Data stored as documents (JSON-like BSON)
– Flexible-schema
– In schema design, think about optimizing for read vs. storage
• Full CRUD support (Create, Read, Update, Delete)
– Atomic in-place updates
– Ad-hoc queries: Equality, RegEx, Ranges, Geospatial
• Secondary indexes
• Replication – redundancy, failover
• Sharding – partitioning for read/write scalability
• Terminology
– Collection = Table
– Index = Index
– Document = Row
– Column = Field
– Joining = Embedding & Linking

Creating Components
• .insert() will always try to create new document
• .save() if _id already exists will update
• If document doesn’t have _id field it is added
26

Updating Components
• $set keyword used for partial updates
• Without $set keyword entire document is replaced
• {multi : true} to update multiple documents
29

Deleting Components
30
Works like .find()
Drops collection
Drops database

Updating Components – Save()
33
Save sends entire document back to server

Updating Components – Update()
34
Update only sends changes

Deleting Components
35
Needed to add reference to Repo class

EBSCO Information Services (EBSCO) is the leading provider of electronic journals, magazines, eBooks, audioBooks, and online research content for libraries, including hundreds of research databases, historical archives, point-of-care medical reference, and corporate learning tools serving millions of end users at tens of thousands of institutions worldwide. The EBSCO platform is a widely used platform serving the needs of researchers at all levels in academic institutions, schools, public libraries, hospitals, medical institutions, corporations and government institutions. Data is our business, and delivering new products quickly is our competitive advantage. We build hundreds of data products and accelerating the analysis, transformation of new datasets translates to revenue and competitiveness. And since our data is so varied, using MognoDB to store data flexibly and JSON Studio to analyze this data allows us to deliver products to market faster. In this session we will describe this process that helped us expedite delivery of new datasets, and give real examples of how data is used, analyzed and processed.

Top 5 Things to Know About Integrating MongoDB into Your Data WarehouseMongoDB

1) The document discusses integrating MongoDB, a NoSQL database, with Teradata, a data warehouse platform. 2) It provides 5 key things to know about the integration, including how Teradata can pull directly from sharded MongoDB clusters and push data back. 3) Use cases are presented where the operational data in MongoDB can provide context and analytics capabilities for applications, and the data warehouse can enrich the operational data.

MongoDB and the Internet of ThingsMongoDB

Hype, buzzword, threat; however you want to characterize it, the Internet of Things (IoT) is here. IoT scenarios that were hypothetical only a few years ago are real today. Still thinking along the line of fleet management and temperature measurements? You’re out. Endless possibilities of IoT applications are surfacing every day, from the connected cow (huh?) to things that monitor and analyze your daily life (really?). In this webinar, we will discuss architecture of IoT data management solutions and the challenges that arise. We will explore how MongoDB features provide solutions to those problems. Time permitting, we will demonstrate an IoT Cloud service built on top of MongoDB.

Big Data Day LA 2016/ NoSQL track - MongoDB 3.2 Goodness!!!, Mark Helmstetter...Data Con LA

AmazonRedshiftAhasan Habib

Amazon Redshift is a data warehousing tool developed by Amazon that allows for storing and analyzing large amounts of data. It uses a columnar data storage model and massively parallel processing to enable extremely fast queries. Redshift can be easily accessed through a web service API and supports standard SQL syntax. It is a cloud-based solution that provides high security and scalability for analytical workloads.

An Intro to Elasticsearch and KibanaObjectRocket

The State of the Data Warehouse in 2017 and BeyondSingleStore

The document provides an overview of the changing analytic environment and the evolution of the data warehouse. It discusses how new requirements like performance, usability, optimization, and ecosystem integration are driving the adoption of a real-time data warehouse approach. A real-time data warehouse is described as having low latency ingestion, in-memory and disk-optimized storage, and the ability to power both operational and machine learning applications. Examples are given of companies using a real-time data warehouse to enable real-time analytics and improve business processes.

Converging Database Transactions and Analytics SingleStore

Improve your SQL workload with observabilityOVHcloud

La majeure partie du SI d'OVH repose sur des bases de données relationnelles (PostgreSQL, MySQL, MariaDB). En termes de volumétrie cela représente 400 bases pesants plus de 20To de données réparties sur 60 clusters dans deux zones géographiques le tout propulsant 3000 applications. Comment tout voir dans notre parc ? Mieux encore, comment faire pour que tout le monde puisse suivre l'activité de sa base de données ? C'est le challenge que nous nous sommes fixés, un an après nous pouvons partager notre expérience. Et si l'observability n'était pas juste un buzzword, mais avait un réel impact sur la production ?

Data platform architecture principles - ieee infrastructure 2020Julien Le Dem

This document discusses principles for building a healthy data platform, including: 1. Establishing explicit contracts between teams to define dependencies and service level agreements. 2. Abstracting the data platform into services for ingesting, storing, and processing data in motion and at rest. 3. Enabling observability of data pipelines through metadata collection and integration with tools like Marquez to provide lineage, availability, and change management visibility.

When to Use MongoDBMongoDB

- MongoDB is well-suited for systems of engagement that have demanding real-time requirements, diverse and mixed data sets, massive concurrency, global deployment, and no downtime tolerance. - It performs well for workloads with mixed reads, writes, and updates and scales horizontally on demand. However, it is less suited for analytical workloads, data warehousing, business intelligence, or transaction processing workloads. - MongoDB shines for use cases involving single views of data, mobile and geospatial applications, real-time analytics, catalogs, personalization, content management, and log aggregation. It is less optimal for workloads requiring joins, full collection scans, high-latency writes, or five nines u

Benefits of Using MongoDB Over RDBMSsMongoDB

Rapid Development and Performance By Transitioning from RDBMSs to MongoDB Modern day application requirements demand rich & dynamic data structures, fast response times, easy scaling, and low TCO to match the rapidly changing customer & business requirements plus the powerful programming languages used in today's software landscape. Traditional approaches to solutions development with RDBMSs increasingly expose the gap between the modern development languages and the relational data model, and between scaling up vs. scaling horizontally on commodity hardware. Development time is wasted as the bulk of the work has shifted from adding business features to struggling with the RDBMSs. MongoDB, the premier NoSQL database, offers a flexible and scalable solution to focus on quickly adding business value again. In this session, we will provide: - Overview of MongoDB's capabilities - Code-level exploration of the MongoDB programming model and APIs and how they transform the way developers interact with a database - Update of the exciting features in MongoDB 3.0

Webinar: An Enterprise Architect’s View of MongoDBMongoDB

The document provides an overview of MongoDB and how it addresses the requirements of modern applications and enterprises. It discusses how traditional databases struggle with new demands around dynamic schemas, large volumes of data, and agile development. MongoDB supports these requirements through features like document data structures, horizontal scaling, and high performance. Case studies demonstrate how MongoDB has helped organizations build real-time views of customer data, virtualize legacy systems, and improve data distribution. The document concludes by discussing best practices for enterprise adoption of MongoDB.

Architecting Data in the AWS EcosystemSingleStore

Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"Fwdays

Dmitry Lavrinenko is a solutions architect who specializes in blockchain for identity management, big data, and related technologies. He proposes a vendor-agnostic SaaS platform that utilizes fast data technologies like continuous loading, parallel processing, and data consolidation. The solution would include a data warehouse, processing, analytics, visualization, machine learning, and identity management capabilities. Blockchain provides benefits like cryptographic security, privacy, consensus, audibility and smart contracts for managing identities. The proposed architecture features data lakes, batch and speed layers, serving layers, storage, functions, streaming, and machine learning components.

Building Pinterest Real-Time Ads Platform Using Kafka Streams confluent

Building Pinterest Real-Time Ads Platform Using Kafka Streams (Liquan Pei + Boyang Chen, Pinterest) Kafka Summit SF 2018 In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business. The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.

MongoDB Operations for DevelopersMongoDB

MongoDB on AzureNorberto Leite

Introducing MongoDB Stitch, Backend-as-a-Service from MongoDBMongoDB

Watch this webinar to learn about our new Backend as a Service (BaaS) – MongoDB Stitch. MongoDB Stitch lets developers focus on building applications rather than on managing data manipulation code, service integration, or backend infrastructure. Whether you’re just starting up and want a fully managed backend as a service, or you’re part of an enterprise and want to expose existing MongoDB data to new applications, Stitch lets you focus on building the app users want, not on writing boilerplate backend logic. This webinar will cover the what, why, and how of MongoDB Stitch. We’ll cover everything from the features it provides to the architecture that makes it possible. By the end of the session, you should understand how Stitch can kickstart your new project or take your existing application to the next level. Attendees will learn: - The basics of MongoDB Stitch and how to use it for new projects or to expose existing data to new applications - How to control what data and services individual users can access - How to integrate your favorite services with your MongoDB application without writing extra code

MongoDB for Time Series Data Part 1: Setting the Stage for Sensor ManagementMongoDB

The document discusses schema design considerations for time series traffic monitoring data stored in MongoDB. It evaluates alternatives like storing a document per event, per minute, or per hour. Storing aggregated data improves write performance by changing inserts to updates, improves analytic queries by reducing document reads, and reduces memory requirements by decreasing index size. The best schema depends on the read and write workload of the specific application.

Cignex mongodb-sharding-mongodbdaysMongoDB APAC

This document discusses MongoDB sharding as a case study for scaling MongoDB. It provides background on CIGNEX Datamatics and their big data analytics practice. It then describes a use case of 7 million users accessing digital assets across 8 devices each. It recommends MongoDB due to its flexibility and performance. The solution involves sharding across multiple MongoDB nodes to distribute the data and handle the high volume of concurrent requests. Benchmarking shows that sharding significantly improves performance of inserts and updates over non-sharded architecture. The key takeaway is that sharding is very effective but requires careful planning, benchmarking, and choice of shard key.

Data Pipline Observability meetup Omid Vahdaty

This document discusses the need for observability in data pipelines. It notes that real data pipelines often fail or take a long time to rerun without providing any insight into what went wrong. This is because of frequent code, data, dependency, and infrastructure changes. The document recommends taking a production engineering approach to observability using metrics, logging, and alerting tools. It also suggests experiment management and encapsulating reporting in notebooks. Most importantly, it stresses measuring everything through metrics at all stages of data ingestion and processing to better understand where issues occur.

MongoDB & Hadoop - Understanding Your Big DataMongoDB

Big Data is the evolution of supercomputing for commercial enterprise and governments. Originally the domain of companies operating at Internet scale, today Big Data connects organizations of all sizes with discovery about their patterns, and insights into their business. But understanding the differences between the plethora of new technologies can be daunting. Graph / columnar / key value store / document are all called NoSQL, but which is best? How does Hadoop play in this ecosystem - its low cost and high efficiency have made it very popular, but how does it fit? In this webinar, we will explore: The full spectrum of Big Data Hadoop and MongoDB: friends or frenemies? Differences between Systems of Record and Systems of Engagement MongoDB customer examples of Systems of Engagement

An afternoon with mongo db new delhiRajnish Verma

The document summarizes MongoDB as a modern database designed to solve problems of volume, velocity, and variety of data that traditional relational databases are not well-suited for. It highlights key MongoDB features like scalability, flexible schemas, and high availability. The document also discusses how MongoDB compares favorably to other databases in security capabilities and is a good fit for applications involving user data management, content delivery, and mobile apps.

Sizing Your MongoDB ClusterMongoDB

Jay Runkel presented a methodology for sizing MongoDB clusters to meet the requirements of an application. The key steps are: 1) Analyze data size and index size, 2) Estimate the working set based on frequently accessed data, 3) Use a simplified model to estimate IOPS and adjust for real-world factors, 4) Calculate the number of shards needed based on storage, memory and IOPS requirements. He demonstrated this process for an application that collects mobile events, requiring a cluster that can store over 200 billion documents with 50,000 IOPS.

"Introduction to Kx Technology", James Corcoran, Head of Engineering EMEA at ...Dataconomy Media

"Introduction to Kx Technology", James Corcoran, Head of Engineering EMEA at First Derivatives About the Author: James is Senior Vice President, Fast Data Solutions at Kx where he has worked as a developer since 2009. In his career to date, he has worked in the algorithmic trading space at many of the world’s top financial institutions using Kx - a low latency technology for analysing time series data. He is a certified Professional Risk Manager and holds a masters in Quantitative Finance from University College Dublin. In recent years he has built systems for clients ranging from start-ups to blue chip companies in data intensive industries such as pharma, utilities and telco.

MongoDB and the Internet of ThingsMongoDB

- MongoDB is well-suited for IoT applications due to its ability to handle large volumes of variable data from sensors, perform analytics on both real-time and historical data, and scale horizontally to support growing workloads. - Its flexible document model accommodates changing sensor schemas and nested/complex data structures from devices, while secondary indexes enable expressive queries. - Time series data from sensors can be optimized in MongoDB using bucketing which improves write performance, storage usage, and analytics capabilities.

MongoDB San Francisco 2013: Storing eBay's Media Metadata on MongoDB present...MongoDB

This session will be a case study of eBay’s experience running MongoDB for project Zoom, in which eBay stores all media metadata for the site. This includes references to pictures of every item for sale on eBay. This cluster is eBay's first MongoDB installation on the platform and is a mission critical application. Yuri Finkelstein, an Enterprise Architect on the team, will provide a technical overview of the project and its underlying architecture.

MongoDB at MedtronicMongoDB

This document summarizes a presentation about data management in a regulated environment. It introduces two speakers from Medtronic who discuss the challenges of making changes to applications that require FDA approval. These include needing to revalidate applications every time a change is made. The presentation then discusses why MongoDB may help, including its flexibility and performance, but that noSQL does not mean no schema. It also notes gaps that need to be addressed like enterprise acceptance of new approaches and integrating MongoDB with reporting tools and making it 21 CFR Part 11 compliant.

RDBMS RamSon Queen

The document discusses components and concepts related to relational database management systems (RDBMS). It defines RDBMS and DBMS, and explains that an RDBMS is based on the relational model. The key components of an RDBMS include file manager, database manager, query processor, data dictionary, DML pre-compiler, and DDL compiler. It also discusses different types of relations, relational data integrity, database languages, and advantages and disadvantages of RDBMS.

More Related Content

What's hot (20)

Improve your SQL workload with observabilityOVHcloud

Data platform architecture principles - ieee infrastructure 2020Julien Le Dem

When to Use MongoDBMongoDB

Benefits of Using MongoDB Over RDBMSsMongoDB

Webinar: An Enterprise Architect’s View of MongoDBMongoDB

Architecting Data in the AWS EcosystemSingleStore

Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"Fwdays

Building Pinterest Real-Time Ads Platform Using Kafka Streams confluent

MongoDB Operations for DevelopersMongoDB

MongoDB on AzureNorberto Leite

Introducing MongoDB Stitch, Backend-as-a-Service from MongoDBMongoDB

MongoDB for Time Series Data Part 1: Setting the Stage for Sensor ManagementMongoDB

Cignex mongodb-sharding-mongodbdaysMongoDB APAC

Data Pipline Observability meetup Omid Vahdaty

MongoDB & Hadoop - Understanding Your Big DataMongoDB

An afternoon with mongo db new delhiRajnish Verma

Sizing Your MongoDB ClusterMongoDB

"Introduction to Kx Technology", James Corcoran, Head of Engineering EMEA at ...Dataconomy Media

MongoDB and the Internet of ThingsMongoDB

MongoDB San Francisco 2013: Storing eBay's Media Metadata on MongoDB present...MongoDB

Improve your SQL workload with observabilityOVHcloud

Data platform architecture principles - ieee infrastructure 2020Julien Le Dem

When to Use MongoDBMongoDB

Benefits of Using MongoDB Over RDBMSsMongoDB

Webinar: An Enterprise Architect’s View of MongoDBMongoDB

Architecting Data in the AWS EcosystemSingleStore

Дмитрий Лавриненко "Blockchain for Identity Management, based on Fast Big Data"Fwdays

Building Pinterest Real-Time Ads Platform Using Kafka Streams confluent

MongoDB Operations for DevelopersMongoDB

MongoDB on AzureNorberto Leite

Introducing MongoDB Stitch, Backend-as-a-Service from MongoDBMongoDB

MongoDB for Time Series Data Part 1: Setting the Stage for Sensor ManagementMongoDB

Cignex mongodb-sharding-mongodbdaysMongoDB APAC

Data Pipline Observability meetup Omid Vahdaty

MongoDB & Hadoop - Understanding Your Big DataMongoDB

An afternoon with mongo db new delhiRajnish Verma

Sizing Your MongoDB ClusterMongoDB

"Introduction to Kx Technology", James Corcoran, Head of Engineering EMEA at ...Dataconomy Media

MongoDB and the Internet of ThingsMongoDB

MongoDB San Francisco 2013: Storing eBay's Media Metadata on MongoDB present...MongoDB

Viewers also liked (20)

MongoDB at MedtronicMongoDB

RDBMS RamSon Queen

The Best of Both Worlds: Speeding Up Drug Research with MongoDB & Oracle (Gen...MongoDB

1) Genentech uses MongoDB to speed up drug research by allowing flexible data collection and analysis of genetic testing results. 2) MongoDB integrates easily with their existing RDBMS and allows loading of CSV files directly into flexible document structures for analysis. 3) This helps Genentech introduce new genetic tests and instruments more quickly, in weeks rather than months, potentially saving lives.

Practice Fusion & MongoDB: Transitioning a 4 TB Audit Log from SQL Server to ...MongoDB

Presented by Michael Poremba, Director of Data Architecture, Practice Fusion. Michael will share technical insights and key lessons learned by the Practice Fusion audit team, which can be leveraged for your own MongoDB projects. Key topics include: security, schema design, scalability, data migration, disaster recovery, and organizing the team of technical experts. Practice Fusion decided to move the primary audit system for their Electronic Health Record (EHR) from SQL Server to MongoDB. The goal was to provide a scale-out data store for a system that was under IO pressure. The project required moving 4 TB of production data from a traditional database schema into a document data store while the system was in operation with a peak transaction throughput of up to 1,000 writes per second. The audit system is a tier-1 component mandated by law, and any interruption in the availability of the audit system would result in a system outage. BIO: Michael Poremba is Director of Data Architecture at Practice Fusion, a cloud-based electronic health records (EHR) service used by over 100,000 health care providers. He has over 20 years of experience in software engineering with expertise in architecture, performance, and scalability of transactional database applications.

MongoDB and the Connectivity Map: Making Connections Between Genetics and Dis...MongoDB

This document discusses the Connectivity Map (CMap), a resource for exploring connections between genetics and disease. CMap contains over 1.4 million gene expression profiles from experiments with over 15 cell types and 12,488 compounds. It has been cited in over 1,200 scientific reports and has over 19,000 registered users. The document outlines CMap's data model for storing experimental signatures and metadata in a way that is flexible, scalable, and optimized for querying. It also discusses APIs for accessing and analyzing CMap data, as well as tools for tasks like predicting drug function and finding novel drug targets.

Webinar: Electronic Health Records (EHRs) and MongoDB - Advancing the Data Pl...MongoDB

NewWave is an IT services provider that uses MongoDB for various healthcare projects. It stores electronic health records and other structured/unstructured data in MongoDB to take advantage of its scalability. NewWave integrates MongoDB with other technologies like Hadoop, ElasticSearch and OpenGeo to enable analytics, spatial search and data exchange between systems. Rules engines and BPM are used to automate workflows using MongoDB as the rules and data repository. NewWave aims to expand MongoDB use to social media integration and multi-cloud environments.

Michael Poremba, Director, Data Architecture at Practice FusionMongoDB

Practice Fusion, the largest cloud-based electronic health records (EHR) system in the US, used by more than 100,000 health care providers managing over 100 million patient medical records, faced the need to move their four terabyte HIPAA audit reporting system off of a relational database. Practice Fusion selected MongoDB for their new HIPAA audit reporting system. Learn how the team designed and implemented a highly scalable system for storing protected health information in the cloud. This case study covers the move from a relational database to a document database; data modeling in JSON; sharding strategies; indexing; sharded cluster design supporting high availability and disaster recovery; performance testing; and data migration of billions of historical audit records.

A Translational Medicine Platform at SanofiMongoDB

Sanofi is using MongoDB as a central data repository to support its translational medicine platform by enabling researchers to store, organize, collaborate on, link, visualize, and analyze omics data, clinical records, and other research information in order to more efficiently conduct clinical trials and develop personalized medicines. The MongoDB implementation provides benefits like reduced development time, improved data quality and understanding for scientists, and increased flexibility and performance for IT.

MongoDB Europe 2016 - Warehousing MongoDB Data using Apache Beam and BigQueryMongoDB

What happens when you need to combine data from MongoDB along with other systems into a cohesive view for business intelligence? How do you extract, transform, and load MongoDB data into a centralized data warehouse? In this session we’ll talk about Google BigQuery, a managed, petabyte-scale data warehouse, and the various ways to get MongoDB data into it. We’ll cover managed options like Apache Beam and Cloud Dataflow as well as other tools that can help make moving and using MongoDB data easy for business intelligence workloads.

IMPLEMENTASI DATA WAREHOUSE GUNA MEMBANTU PETERNAK SAPI DAN KUD DALAM MENGELO...Udayana University

Accelerate Pharmaceutical R&D with Big Data and MongoDBMongoDB

This document provides a summary of a presentation on using MongoDB and big data technologies to accelerate pharmaceutical research and development at AstraZeneca. The presentation discusses: - AstraZeneca's focus on using next generation sequencing and big data to predict drug effectiveness and find associations between gene sequences and drug responses. - Pilot projects using MongoDB to store and query unstructured genomic and clinical trial data at scale in a flexible document format. - How these pilots helped prove the value of NoSQL databases for enabling faster exploration and analysis of large, complex datasets by researchers. - Future visions for using experimental management systems and big data analytics to integrate multiple data types and power predictive analytics across AstraZeneca's drug development pipelines

Attacking MongoDBМихаил Фирстов

MongoDB at eBayMongoDB

eBay has been using MongoDB to power several large scale applications due to its ability to scale horizontally and handle high volumes of data and queries. Some key applications summarized include search suggestions which handles high volume lookups, a cloud manager which tracks resource states, and a media metadata store which will store tens of terabytes of image data partitioned across multiple shards. MongoDB provides benefits like flexible schemas, automatic shard rebalancing, and proximity-aware replication which help eBay meet performance and scalability needs for these systems.

MongoDB basics in RussianOleg Kachan

Storing eBay's Media Metadata on MongoDB, by Yuri Finkelstein, Architect, eBayMongoDB

Migrating from RDBMS to MongoDBMongoDB

The document discusses migrating from an RDBMS to MongoDB. It covers determining if a migration is worthwhile based on evaluating current pain points and target value. It also discusses the roles and responsibilities that will change during a migration, including data architects, developers, DBAs and more. Bulk migration techniques are reviewed including using mongoimport to import JSON data. System cutover is also mentioned as an important part of the migration process.

Creating an E-Commerce ScorecardSAP Ariba

The document discusses creating an e-commerce scorecard. It includes presentations from three speakers: Jamie Werve from Insight, Emile Naguib from Office Depot, and Gerald Clarke from IBM. They discuss their companies' e-commerce journeys and metrics for measuring e-commerce performance. Key points include Insight's evolution in e-commerce since 1996 and their current scorecard measures. Office Depot outlines site and implementation metrics. IBM discusses their complex business model and balanced scorecard approach measuring financial, customer, operational, and people dimensions. The presentations provide lessons on developing an effective e-commerce scorecard.

Webinar: MongoDB and Analytics: Building Solutions with the MongoDB BI ConnectorMongoDB

MongoDB is known for being a developers database of choice, but what about data analysts? MongoDB 3.2 has introduced the MongoDB BI Connector – to allow users to connect to an instance using their analytics tool of choice. Now users of Tableau, QlikView, Excel, Cognos, and countless others can connect to MongoDB and immediately begin building reporting solutions. In this webinar, we will cover the architecture needed to use the BI Connector with MongoDB. We will also demonstrate how to build reports with your data.

Hadoop Summit 2013 : Continuous Integration on top of hadoopWisely chen

Wisely Chen and Neal Lee presented on continuous integration on Hadoop. They discussed automating the software development process, including automatically running unit tests on code commits, performing tests on staging environments that mimic production, and deploying to production. They demonstrated tools for unit testing Hadoop jobs, including PigUnit and Vaidya for performance diagnosis, and how these tools can integrate with a continuous integration system to automatically run tests with each code change.

Building a Scalable and Modern Infrastructure at CARFAXMongoDB

The document discusses CARFAX's transition from a proprietary key-value store to MongoDB. It describes CARFAX's production MongoDB environment including 12 sharded servers with 128GB RAM each. It details how CARFAX loads millions to billions of records per day via distributed processing. It also discusses implementing high availability reads through tagging data centers and replica sets to scale to millions of reports per day. The presentation emphasizes automating processes and sharding early and often for scaling their MongoDB deployment.

MongoDB at MedtronicMongoDB

RDBMS RamSon Queen

The Best of Both Worlds: Speeding Up Drug Research with MongoDB & Oracle (Gen...MongoDB

Practice Fusion & MongoDB: Transitioning a 4 TB Audit Log from SQL Server to ...MongoDB

MongoDB and the Connectivity Map: Making Connections Between Genetics and Dis...MongoDB

Webinar: Electronic Health Records (EHRs) and MongoDB - Advancing the Data Pl...MongoDB

Michael Poremba, Director, Data Architecture at Practice FusionMongoDB

A Translational Medicine Platform at SanofiMongoDB

MongoDB Europe 2016 - Warehousing MongoDB Data using Apache Beam and BigQueryMongoDB

IMPLEMENTASI DATA WAREHOUSE GUNA MEMBANTU PETERNAK SAPI DAN KUD DALAM MENGELO...Udayana University

Accelerate Pharmaceutical R&D with Big Data and MongoDBMongoDB

Attacking MongoDBМихаил Фирстов

MongoDB at eBayMongoDB

MongoDB basics in RussianOleg Kachan

Storing eBay's Media Metadata on MongoDB, by Yuri Finkelstein, Architect, eBayMongoDB

Migrating from RDBMS to MongoDBMongoDB

Creating an E-Commerce ScorecardSAP Ariba

Webinar: MongoDB and Analytics: Building Solutions with the MongoDB BI ConnectorMongoDB

Hadoop Summit 2013 : Continuous Integration on top of hadoopWisely chen

Building a Scalable and Modern Infrastructure at CARFAXMongoDB

Similar to MongoDB as a Data Warehouse: Time Series and Device History Data (Medtronic) (20)

Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauMongoDB

Pairing your real-time operational data stored in a modern database like MongoDB with first-class business intelligence platforms like Tableau enables new insights to be discovered faster than ever before. Many leading organizations already use MongoDB in conjunction with Tableau including a top American investment bank and the world’s largest airline. With the Connector for BI 2.0, it’s never been easier to streamline the connection process between these two systems. In this webinar, we will create a live connection from Tableau Desktop to a MongoDB cluster using the Connector for BI. Once we have Tableau Desktop and MongoDB connected, we will demonstrate the visual power of Tableau to explore the agile data storage of MongoDB. You’ll walk away knowing: - How to configure MongoDB with Tableau using the updated connector - Best practices for working with documents in a BI environment - How leading companies are using big data visualization strategies to transform their businesses

Getting Started with MongoDB Using the Microsoft Stack MongoDB

Speaker: John Randolph, Sr. Software Developer, Gexa Energy Level: 100 (Beginner) Track: Developer Gexa has implemented several applications using MongoDB as a document repository storing multiple types of files (PDF, XLS, CSV, etc.). This entry level session is intended to share what we’ve learned in developing and deploying our first applications in an on premise, Microsoft environment. We’ll provide architectural and development information about what we’ve done. The focus is to help get your projects up-to-speed more quickly. This will be useful to teams moving from pilot to production and for developers getting started with the .Net MongoDB drivers. Plenty of code samples will be shown. We’ll discuss our successful engagement with MongoDB Consulting to help us design and deploy a high-quality production environment. What You Will Learn: - Ideas how to store and retrieve documents of different sizes, types, and volumes. We’ll describe the storage, partitioning and indexing techniques used that provide sub-second retrieval from collections with over 100 million records. - The issues addressed moving to production, including: backup, disaster recovery, SSL, using replica sets, implementing authorization and authentication, changing default setting, and creating a full path-to-production set of environments. - A successful pattern for building applications with .Net, providing teams some ideas to jump-start their development along with tips and tricks for using the .Net drivers.

Mongo db 3.4 OverviewNorberto Leite

MongoDB Administration 101MongoDB

The document provides an overview of MongoDB administration including its data model, replication for high availability, sharding for scalability, deployment architectures, operations, security features, and resources for operations teams. The key topics covered are the flexible document data model, replication using replica sets for high availability, scaling out through sharding of data across multiple servers, and different deployment architectures including single/multi data center configurations.

Enterprise architectsview 2015-aprMongoDB

RedHat MRG and Infinispan for Large Scale Integrationprajods

Introduction to MongoDB Basics from SQL to NoSQLMayur Patil

This document provides an introduction to databases and MongoDB. It discusses the purpose of databases, types of databases including relational and non-relational, and the relational model. It then focuses on MongoDB, describing its basics like JSON and BSON storage formats, CRUD terms, features around performance, availability and scaling, and some limitations. Popular applications of databases are also listed.

Webinar: What's New in MongoDB 3.2MongoDB

MongoDB World 2018: Breaking the Mold - Redesigning Dell's E-Commerce PlatformMongoDB

This document discusses Dell's redesign of its e-commerce platform by moving from a relational database to MongoDB. It outlines the advantages of a resource-oriented architecture and document stores. It then provides details of Dell's DCQO application that was migrated from an XML-based relational database to MongoDB, including the architecture, schema design, data migration strategy, and operations management tools.

MongoDB Evening Austin, TX 2017MongoDB

MongoDB 3.4 is a multi-model database that supports documents, relational data, key-value, and graph structures. It features new capabilities like faceted navigation for advanced analytics, views for mission critical applications, and intra-cluster compression. MongoDB also provides enterprise tools like Ops Manager for high resolution monitoring, Compass for visual data exploration, and connectors for BI and SQL tools.

Webinar: “ditch Oracle NOW”: Best Practices for Migrating to MongoDBMongoDB

Government GraphSummit: And Then There Were 15 StandardsNeo4j

Todd Pihl PhD., Technical Project Mgr. & Mark Jensen, Director of Data Managements and Interoperability, National Institute of Health, Frederick National Labs for Cancer Research Data repositories such as NCI’s Cancer Research Data Commons receive data that use a variety of data models and vocabularies. This presents a significant obstacle to finding and using the data outside of their original purpose. In this talk we’ll show how using Neo4j allows different data models to be represented and mapped to each other, giving data managers a new way to provide harmonized data to their users.

Neo4j: What's Under the Hood & How Knowing This Can Help You Neo4j

Neo4j provides a concise summary of how graph databases have evolved and their advantages over traditional databases. Specifically, graph databases can handle billions of connections between data points and enable queries that can traverse thousands of relationships between nodes, providing answers in milliseconds rather than minutes. This level of connected data insight allows for real-time fraud detection, recommendations, knowledge graphs, and other applications that require understanding relationships in large, dynamic datasets.

ICIECA 2014 Paper 05Association of Scientists, Developers and Faculties

This document discusses a study on using a separation-based approach for analyzing big data from various sources. It proposes a system with four main components: a main switch to separate relevant data types, a key generator to encrypt data, a subnet switch master to assign work to map workers, and map workers that process assigned data. This bottom-up approach aims to more easily analyze large amounts of data by starting with smaller subsets. The document also covers challenges of big data, existing grid computing systems, Hadoop tools, and concludes the proposed framework would improve upon previous methods for big data analysis.

EM12c: Capacity Planning with OEM MetricsMaaz Anjum

Ops Jumpstart: MongoDB Administration 101MongoDB

Chromatography Data System: Chromeleon Goes Mass SpectrometryChromatography & Mass Spectrometry Solutions

Chromeleon CDS software now supports mass spectrometry (MS) instrument control and data processing, allowing laboratories to integrate MS into their chromatography data system (CDS) workflow. Key features include native MS instrument drivers for remote control and monitoring, MS-specific data organization and visualization tools, a suite of MS data processing tools including extracted ion chromatogram creation and library searching, and reporting objects tailored for MS data. The integrated CDS approach provides advantages like single software validation, enhanced data security, and use of Chromeleon's compliance and data processing features for MS data.

Data Discovery and Metadatamarkgrover

MongoDB Evenings Chicago - Find Your Way in MongoDB 3.2: Compass and BeyondMongoDB

Budapest Spring MUG 2016 - MongoDB User GroupMarc Schwering

Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauMongoDB

Getting Started with MongoDB Using the Microsoft Stack MongoDB

Mongo db 3.4 OverviewNorberto Leite

MongoDB Administration 101MongoDB

Enterprise architectsview 2015-aprMongoDB

RedHat MRG and Infinispan for Large Scale Integrationprajods

Introduction to MongoDB Basics from SQL to NoSQLMayur Patil

Webinar: What's New in MongoDB 3.2MongoDB

MongoDB World 2018: Breaking the Mold - Redesigning Dell's E-Commerce PlatformMongoDB

MongoDB Evening Austin, TX 2017MongoDB

Webinar: “ditch Oracle NOW”: Best Practices for Migrating to MongoDBMongoDB

Government GraphSummit: And Then There Were 15 StandardsNeo4j

Neo4j: What's Under the Hood & How Knowing This Can Help You Neo4j

ICIECA 2014 Paper 05Association of Scientists, Developers and Faculties

EM12c: Capacity Planning with OEM MetricsMaaz Anjum

Ops Jumpstart: MongoDB Administration 101MongoDB

Chromatography Data System: Chromeleon Goes Mass SpectrometryChromatography & Mass Spectrometry Solutions

Data Discovery and Metadatamarkgrover

MongoDB Evenings Chicago - Find Your Way in MongoDB 3.2: Compass and BeyondMongoDB

Budapest Spring MUG 2016 - MongoDB User GroupMarc Schwering

More from MongoDB (20)

MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB

This presentation discusses migrating data from other data stores to MongoDB Atlas. It begins by explaining why MongoDB and Atlas are good choices for data management. Several preparation steps are covered, including sizing the target Atlas cluster, increasing the source oplog, and testing connectivity. Live migration, mongomirror, and dump/restore options are presented for migrating between replicasets or sharded clusters. Post-migration steps like monitoring and backups are also discussed. Finally, migrating from other data stores like AWS DocumentDB, Azure CosmosDB, DynamoDB, and relational databases are briefly covered.

MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB

MongoDB Kubernetes operator and MongoDB Open Service Broker are ready for production operations. Learn about how MongoDB can be used with the most popular container orchestration platform, Kubernetes, and bring self-service, persistent storage to your containerized applications. A demo will show you how easy it is to enable MongoDB clusters as an External Service using the Open Service Broker API for MongoDB

MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB

MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB

MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB

Time series data is increasingly at the heart of modern applications - think IoT, stock trading, clickstreams, social media, and more. With the move from batch to real time systems, the efficient capture and analysis of time series data can enable organizations to better detect and respond to events ahead of their competitors or to improve operational efficiency to reduce cost and risk. Working with time series data is often different from regular application data, and there are best practices you should observe. This talk covers: Common components of an IoT solution The challenges involved with managing time-series data in IoT applications Different schema designs, and how these affect memory and disk utilization – two critical factors in application performance. How to query, analyze and present IoT time-series data using MongoDB Compass and MongoDB Charts At the end of the session, you will have a better understanding of key best practices in managing IoT time-series data with MongoDB.

MongoDB SoCal 2020: MongoDB Atlas Jump StartMongoDB

MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB

Our clients have unique use cases and data patterns that mandate the choice of a particular strategy. To implement these strategies, it is mandatory that we unlearn a lot of relational concepts while designing and rapidly developing efficient applications on NoSQL. In this session, we will talk about some of our client use cases, the strategies we have adopted, and the features of MongoDB that assisted in implementing these strategies.

MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB

Encryption is not a new concept to MongoDB. Encryption may occur in-transit (with TLS) and at-rest (with the encrypted storage engine). But MongoDB 4.2 introduces support for Client Side Encryption, ensuring the most sensitive data is encrypted before ever leaving the client application. Even full access to your MongoDB servers is not enough to decrypt this data. And better yet, Client Side Encryption can be enabled at the "flick of a switch". This session covers using Client Side Encryption in your applications. This includes the necessary setup, how to encrypt data without sacrificing queryability, and what trade-offs to expect.

MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB

MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB

MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB

When you need to model data, is your first instinct to start breaking it down into rows and columns? Mine used to be too. When you want to develop apps in a modern, agile way, NoSQL databases can be the best option. Come to this talk to learn how to take advantage of all that NoSQL databases have to offer and discover the benefits of changing your mindset from the legacy, tabular way of modeling data. We’ll compare and contrast the terms and concepts in SQL databases and MongoDB, explain the benefits of using MongoDB compared to SQL databases, and walk through data modeling basics so you feel confident as you begin using MongoDB.

MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB

MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB

The document discusses guidelines for ordering fields in compound indexes to optimize query performance. It recommends the E-S-R approach: placing equality fields first, followed by sort fields, and range fields last. This allows indexes to leverage equality matches, provide non-blocking sorts, and minimize scanning. Examples show how indexes ordered by these guidelines can support queries more efficiently by narrowing the search bounds.

MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB

MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB

The document describes a methodology for data modeling with MongoDB. It begins by recognizing the differences between document and tabular databases, then outlines a three step methodology: 1) describe the workload by listing queries, 2) identify and model relationships between entities, and 3) apply relevant patterns when modeling for MongoDB. The document uses examples around modeling a coffee shop franchise to illustrate modeling approaches and techniques.

MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB

MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.

MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB

Virtual assistants are becoming the new norm when it comes to daily life, with Amazon’s Alexa being the leader in the space. As a developer, not only do you need to make web and mobile compliant applications, but you need to be able to support virtual assistants like Alexa. However, the process isn’t quite the same between the platforms. How do you handle requests? Where do you store your data and work with it to create meaningful responses with little delay? How much of your code needs to change between platforms? In this session we’ll see how to design and develop applications known as Skills for Amazon Alexa powered devices using the Go programming language and MongoDB.

MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB

MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB

Il n’a jamais été aussi facile de commander en ligne et de se faire livrer en moins de 48h très souvent gratuitement. Cette simplicité d’usage cache un marché complexe de plus de 8000 milliards de $. La data est bien connu du monde de la Supply Chain (itinéraires, informations sur les marchandises, douanes,…), mais la valeur de ces données opérationnelles reste peu exploitée. En alliant expertise métier et Data Science, Upply redéfinit les fondamentaux de la Supply Chain en proposant à chacun des acteurs de surmonter la volatilité et l’inefficacité du marché.