Data Lake

Storing business content has always been a point of contention, and often frustration, within businesses of all types. Should content be stored in folders? Should prefixes and suffixes be used to identify file versions? Should content be divided by department or specialty? The list goes on and on.

The issue stems from the fact that many companies start to implement document or file management systems with the best of intentions but don't have the foresight or infrastructure in place to maintain the initial data organization.

Out of the dire need for organizing the ever increasing volume of data, data lakes were born.

A data lake is a centralized repository that allows you to store structured, semistructured, and unstructured data at any scale.

Data lakes promise the ability to store all data for a business in a single repository. You can leverage data lakes to store large volumes of data instead of persisting that data in data warehouses. Data lakes, such as those built in Amazon S3, are generally less expensive than specialized big data storage solutions such as on premise Hadoop systems. That way, you only pay for the specialized solutions when using them for processing and analytics and not for long-term storage. Your extract, transform, and load (ETL) and analytic process can still access this data for analytics. 

Benefits of a data lake on AWS

• Are a cost-effective data storage solution. You can durably store a nearly unlimited amount of data using Amazon S3.
• Implement industry-leading security and compliance. AWS uses stringent data security, compliance, privacy, and protection mechanisms.
• Allow you to take advantage of many different data collection and ingestion tools to ingest data into your data lake. These services include Amazon Kinesis for streaming data and AWS Snowball appliances for large volumes of on-premises data.
• Help you to categorize and manage your data simply and efficiently. Use AWS Glue to understand the data within your data lake, prepare it, and load it reliably into data stores. Once AWS Glue catalogs your data, it is immediately searchable, can be queried, and is available for ETL processing.
• Help you turn data into meaningful insights. Harness the power of purpose-built analytic services for a wide range of use cases, such as interactive analysis, data processing using Apache Spark and Apache Hadoop, data warehousing, real-time analytics, operational analytics, dashboards, and visualizations.

Using Amazon EMR with data lakes

Businesses have begun realizing the power of data lakes. Businesses can place data within a data lake and use their choice of open source distributed processing frameworks, such as those supported by Amazon EMR. Apache Hadoop and Spark are both supported by Amazon EMR, which has the ability to help businesses easily, quickly, and cost-effectively implement data processing solutions based on Amazon S3 data lakes.

BUSINESS CHALLENGE

SOLUTION

Data lake on AWS                                                                         

Traditional data storage and analytic tools can no longer provide the agility and flexibility required to deliver relevant business insights. That’s why many organizations are shifting to a data lake architecture.

A data lake on AWS can help you do the following:

- Collect and store any type of data, at any scale, and at low cost

- Secure the data and prevent unauthorized access

- Catalog, search, and find the relevant data in the central repository

- Quickly and easily perform new types of data analysis

- Use a broad set of analytic engines for one-time analytics, real-time streaming, predictive analytics, AI, and machine learning

Case Study :

Moderna Therapeutic Case Study (amazon.com)

Credits : Amazon Web Services