Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data

Mar 22, 20181 like995 views

You like to use R, and you need to use big data. dplyr, one of the most popular packages for R, makes it easy to query large data sets in scalable processing engines like Apache Spark and Apache Impala. But there can be pitfalls: dplyr works differently with different data sources—and those differences can bite you if you don’t know what you’re doing. Ian Cook is a data scientist, an R contributor, and a curriculum developer at Cloudera University. In this webinar, Ian will show you exactly what you need to know about sparklyr (from RStudio) and the package implyr (from Cloudera). He will show you how to write dplyr code that works across these different interfaces. And, he will solve mysteries: Do I need to know SQL to use dplyr? When is a “tbl” not a “tibble”? Why is 1 not always equal to 1? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems? 3 things to learn: Do I need to know SQL to use dplyr? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems?

1© Cloudera, Inc. All rights reserved.
dplyr Interfaces to Large-Scale Data
Ian Cook
@ianmcook
ian@cloudera.com

2© Cloudera, Inc. All rights reserved.
Mission for Cloudera: Provide a platform for data analysts, data scientists to
efficiently query, analyze, model large-scale data in clusters, cloud storage
• By distributing Apache Spark, Apache Impala, other tools
• By enabling productive use of these tools
Python and R users often have difficulty moving from smaller data to large-scale
distributed data
• Familiar packages, methods don’t work the same way on distributed data
Context

3© Cloudera, Inc. All rights reserved.
Poll question

4© Cloudera, Inc. All rights reserved.
]
SQLPySpark
SparkR
SQL
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API

5© Cloudera, Inc. All rights reserved.
Poll question

6© Cloudera, Inc. All rights reserved.
]
SQLPySpark
SparkR
SQL
dplyr

7© Cloudera, Inc. All rights reserved.
dplyr provides a set of verbs that perform common data manipulation steps
• select() to select columns
• filter() to filter rows
• arrange() to order rows
• mutate() to create new columns
• summarise() to aggregate
• group_by() to perform operations by group
dplyr works on local data and with remote data sources
• For remote sources, dplyr commands are translated into SQL
dplyr

8© Cloudera, Inc. All rights reserved.
Poll question

9© Cloudera, Inc. All rights reserved.
Demonstration
Example code at
github.com/ianmcook/dplyr-examples

10© Cloudera, Inc. All rights reserved.
dplyr SQL backends
dplyr
↕
dbplyr
↕
dplyr SQL backend package*
↕
DBI
↕
DBI-compatible interface package
↕
database driver or connector
↕
database/engine
* optional

11© Cloudera, Inc. All rights reserved.
• Provides a SQL backend to dplyr for Spark
• Also exposes the MLlib API and a subset of the Spark DataFrames API
• Developed by RStudio
spark.rstudio.com
sparklyr

12© Cloudera, Inc. All rights reserved.
• Provides a SQL backend to dplyr for Impala
• Uses ODBC or JDBC to connect to Impala
• Developed at Cloudera
tiny.cloudera.com/implyr
implyr
implyr

13© Cloudera, Inc. All rights reserved.
Five tips for using dplyr
with SQL data sources

14© Cloudera, Inc. All rights reserved.
Use show_query()
1

15© Cloudera, Inc. All rights reserved.
filter() early
arrange() late
2

16© Cloudera, Inc. All rights reserved.
Check your data types
3

17© Cloudera, Inc. All rights reserved.
Know your SQL engine
4

18© Cloudera, Inc. All rights reserved.
Know when to collect()
5

19© Cloudera, Inc. All rights reserved.
Questions?
Ian Cook
@ianmcook
ian@cloudera.com

20© Cloudera, Inc. All rights reserved.
Cloudera Data Science Workbench
More information
tiny.cloudera.com/cdsw
OnDemand training
tiny.cloudera.com/cdsw-training

The document discusses Cloudera's certification program and training options. It provides information on: 1) The different certification levels like CCA and CCP and their focus areas like administration, development, and data analytics. 2) The benefits of Cloudera training like its alignment with best practices, experienced instructors, and global coverage. 3) The various training modalities like public/private classes, virtual training, and on-demand options. 4) Suggested curricula for different roles and tips for studying for and taking the certification exams.

Unlock Hadoop Success with Cloudera Navigator OptimizerCloudera, Inc.

Supercharge Splunk with Cloudera Cloudera, Inc.

Cloudera can help optimize Splunk deployments by providing more cost-effective scalability, increased data flexibility, and enhanced analytics capabilities. Cloudera can ingest data from Splunk indexes and apply enrichment using open-source machine learning before storing the data in its data hub. This provides a single platform for advanced analytics like SQL and Python/R scripts across both historical and new data. Initial use cases include offloading event data from Splunk to reduce costs and loading additional context sources to gain better insights.

Solr consistency and recovery internalsCloudera, Inc.

Big Data FundamentalsCloudera, Inc.

Discover the origins of big data, discuss existing and new projects, share common use cases for those projects, and explain how you can modernize your architecture using data analytics, data operations, data engineering and data science. Big Data Fundamentals is your prerequisite to building a modern platform for machine learning and analytics optimized for the cloud. We’ll close out with a live Q&A with some of our technical experts as well. Stretch your brain with a packed agenda: Open source software Data storage Data ingestion Data analytics Data engineering IoT and life after Lambda architectures Data science Cybersecurity Cluster management Big data in the cloud Success stories

A deep dive into running data analytic workloads in the cloudCloudera, Inc.

This document discusses running data analytic workloads in the cloud using Cloudera Altus. It introduces Altus, which provides a platform-as-a-service for analyzing and processing data at scale in public clouds. The document outlines Altus features like low cost per-hour pricing, end-user focus, and cloud-native deployment. It then describes hands-on examples using Altus Data Engineering for ETL and the Altus Analytic Database for exploration and analytics. Workload analytics capabilities are also introduced for troubleshooting and optimizing jobs.

Data Science and CDSWJason Hubbard

Part 2: Cloudera’s Operational Database: Unlocking New Benefits in the CloudCloudera, Inc.

Simplifying Hadoop with RecordService, A Secure and Unified Data Access Path ...Cloudera, Inc.

SFHUG presentation from February 2, 2016. One of the key values of the Hadoop ecosystem is its flexibility. There is a myriad of components that make up this ecosystem, allowing Hadoop to tackle otherwise intractable problems. However, having so many components provides a significant integration, implementation, and usability burden. Features that ought to work in all the components often require sizable per-component effort to ensure correctness across the stack. Lenni Kuff explores RecordService, a new solution to this problem that provides an API to read data from Hadoop storage managers and return them as canonical records. This eliminates the need for components to support individual file formats, handle security, perform auditing, and implement sophisticated IO scheduling and other common processing that is at the bottom of any computation. Lenni discusses the architecture of the service and the integration work done for MapReduce and Spark. Many existing applications on those frameworks can take advantage of the service with little to no modification. Lenni demonstrates how this provides fine grain (column level and row level) security, through Sentry integration, and improves performance for existing MapReduce and Spark applications by up to 5×. Lenni concludes by discussing how this architecture can enable significant future improvements to the Hadoop ecosystem. About the speaker: Lenni Kuff is an engineering manager at Cloudera. Before joining Cloudera, he worked at Microsoft on a number of projects including SQL Server storage engine, SQL Azure, and Hadoop on Azure. Lenni graduated from the University of Wisconsin-Madison with degrees in computer science and computer engineering.

Faster Batch Processing with Cloudera 5.7: Hive-on-Spark is ready for productionCloudera, Inc.

It’s no secret that Apache Spark is becoming the successor to MapReduce for data processing in Hadoop. With it’s easy development, flexible API, and performance benefits, Spark is a powerful data processing engine that has quickly gained popularity within the community. On the other hand Hive continues to be the most widely used data warehouse/ETL engine with large scale adoption across enterprises. Therefore, it’s imperative to enable Spark as the underlying execution engine for Hive to seamlessly allow existing and future Hive workloads to leverage the advantages of Spark. With the recent release of Cloudera 5.7, we have delivered on this goal by adding support for Hive-on-Spark. Data engineers and ETL developers can now transition from MR to Spark for their Hive workloads seamlessly thereby benefitting from the advantages of Spark without any disruption on their end. Join Santosh Kumar, Senior Product Manager at Cloudera, and Rui Li, Apache Hive committer and engineer at Intel, as we discuss: An Introduction to Spark and its advantages over MR An introduction of Hive-on-Spark: Goals and Design Principles Migrating to HoS and a live demo Configuring and tuning for batch workloads What’s next for both tools

A Community Approach to Fighting Cyber ThreatsCloudera, Inc.

Part 3: Models in Production: A Look From Beginning to EndCloudera, Inc.

The document discusses the different roles involved in developing machine learning models from beginning to end. It describes the typical workflow as including data engineering to prepare data, exploratory data science to develop models, and operational model deployment to production applications. It provides examples of tasks for each role such as data engineers ingesting and transforming sensor data, data scientists building and evaluating predictive models, and model deployment engineers validating models and creating APIs.

Extreme Sports & Beyond: Exploring a new frontier in data with GoProCloudera, Inc.

GoPro is a powerful global brand, thanks in large part to its innovative cameras and accessories that capture moments other cameras just miss: surfing in Maui, skiing in Tahoe, recording your child’s first steps. And today, the company is nearly as well known for its user-generated social and content networks. Join us for this special webinar hosted by Tableau, Trifacta, and Cloudera—featuring GoPro. We’ll dive into GoPro’s data strategy and architecture, from ingest and processing to data prep and reporting, all on AWS.

Part 1: Lambda Architectures: Simplified by Apache KuduCloudera, Inc.

Spark One Platform WebinarCloudera, Inc.

This document discusses Cloudera's initiative to make Spark the standard execution engine for Hadoop. It outlines how Spark improves on MapReduce by leveraging distributed memory and having a simpler developer experience. It also describes Cloudera's investments in areas like management, security, scale, and streaming to further Spark's capabilities and make it production-ready. The goal is for Spark to replace MapReduce as the execution engine and for specialized engines like Impala to handle specific workloads, with all sharing the same data, metadata, resource management, and other platform services.

Cloudera Altus: Big Data in the Cloud Made EasyCloudera, Inc.

Cloudera Altus makes it easier for data engineers, ETL developers, and anyone who regularly works with raw data to process that data in the cloud efficiently and cost effectively. In this webinar we introduce our new platform-as-a-service offering and explore challenges associated with data processing in the cloud today, how Altus abstracts cluster overhead to deliver easy, efficient data processing, and unique features and benefits of Cloudera Altus.

How Data Drives Business at Choice HotelsCloudera, Inc.

3 Things to Learn: -How data is driving digital transformation to help businesses innovate rapidly -How Choice Hotels (one of largest hoteliers) is using Cloudera Enterprise to gain meaningful insights that drive their business -How Choice Hotels has transformed business through innovative use of Apache Hadoop, Cloudera Enterprise, and deployment in the cloud — from developing customer experiences to meeting IT compliance requirements

Gartner Data and Analytics Summit: Bringing Self-Service BI & SQL Analytics ...Cloudera, Inc.

For self-service BI and exploratory analytic workloads, the cloud can provide a number of key benefits, but the move to the cloud isn’t all-or-nothing. Gartner predicts nearly 80 percent of businesses will adopt a hybrid strategy. Learn how a modern analytic database can power your business-critical workloads across multi-cloud and hybrid environments, while maintaining data portability. We'll also discuss how to best leverage the increased agility cloud provides, while maintaining peak performance.

Hadoop on Cloud: Why and How?Cloudera, Inc.

The document discusses running Hadoop on the cloud using Cloudera Director. It begins with an introduction of the speaker and Cloudera Director. Several common architectural patterns for running Hadoop in the cloud are presented, including using object storage and running short-term ETL/modeling clusters versus long-term analytics clusters. The presentation envisions a future with a more portable, self-service, self-healing, and granularly secure experience for managing Hadoop in the cloud.

Data Science at Scale Using Apache Spark and Apache HadoopCloudera, Inc.

This document provides information about a data science course taught using Apache Spark and Apache Hadoop. It introduces the instructors Sean Owen and Tom White and describes what data science is and the roles of data scientists. Data scientists have skills in engineering, statistics, and business domains. The document discusses why companies need data scientists due to the growth of data and its value. It presents the tools used in data science, including Apache Spark, and how Spark can be used for both investigative and operational analytics. The course teaches a complete data science problem process through hands-on examples using tools like Hadoop, Python, R, Hive, and Spark MLlib.

Analyzing Hadoop Data Using Sparklyr Cloudera, Inc.

The document discusses how Sparklyr allows data scientists to access and work with data stored in Cloudera Enterprise using the popular RStudio IDE. It describes the challenges data scientists face in accessing secured Hadoop clusters and limitations of notebook environments. Sparklyr integration with RStudio provides a familiar environment for data scientists to access Hadoop data and compute using Spark, enabling distributed data science workflows directly in R. The presentation demonstrates how to analyze over a billion records using Spark and R through Sparklyr.

Multi-Tenant Operations with Cloudera 5.7 & BTCloudera, Inc.

One Hadoop, Multiple Clouds - NYC Big Data MeetupAndrei Savu

Part 2: Apache Kudu: Extending the Capabilities of Operational and Analytic D...Cloudera, Inc.

Hadoop Distributed File System (HDFS) Encryption with Cloudera Navigator Key ...Cloudera, Inc.

This document provides an overview of Cloudera's Navigator Key Trustee, which is a key management server that acts as a proxy between CDH components and an external key store. It discusses how Key Trustee uses encryption zone keys stored in an external hardware security module to encrypt data encryption keys, which are then used to encrypt data at rest in HDFS. The document also covers Key Trustee's architecture, deployment considerations, access control lists, and troubleshooting steps.

Hadoop Hadoop & Spark meetup - AltiscaleMark Kerzner

Altiscale provides a big data-as-a-service platform based on Apache Hadoop and related technologies like Spark, Hive, and Tez. Interest in big data is growing rapidly but many independent implementations fail. Altiscale aims to help with its experienced team and fully managed platform that offers fast time to value, scalability, security, and lower total cost of ownership. The platform core is Apache open source components like Hadoop, Spark, Hive and Tez. Altiscale handles administration of the Hadoop cluster including hardware, upgrades, tuning, and addressing failures so customers can focus on their data and jobs.

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)Cloudera, Inc.

In this workshop, we will look outside the box and help expand the problem space to include issues you may not have thought were possible before Big Data. From Near Real Time (NRT) recommendation engines, loan applications to churn detection, Big Data is answering new questions and providing organisations with a competitive edge through revenue increase, cost savings and risk mitigation. We will take a special look at the role the Cloud can play in elevating your analytics environment. We will discuss real world examples of how Big Data answers these questions and does it at a lower cost outlay.

dplyr Interfaces to Large-Scale DataCloudera, Inc.

Part 2: A Visual Dive into Machine Learning and Deep Learning  Cloudera, Inc.

More Related Content

What's hot (20)