NextGen Apache Hadoop MapReduce
49 likes5,416 views
Arun C Murthy, Founder and Architect at Hortonworks Inc., talks about the upcoming Next Generation Apache Hadoop MapReduce framework at the Hadoop Summit, 2011.
More Related Content
What's hot (20)
Viewers also liked (20)
Similar to NextGen Apache Hadoop MapReduce (20)
More from Hortonworks (20)
Recently uploaded (20)
NextGen Apache Hadoop MapReduce
- 1. Next Generation of Apache Hadoop MapReduceArun C. Murthy - Hortonworks Founder and Architect@acmurthy (@hortonworks)Formerly Architect, MapReduce @ Yahoo!8 years @ Yahoo!© Hortonworks Inc. 2011June 29, 2011
- 2. Hello! I’m Arun…Architect & Lead, Apache Hadoop MapReduce Development Team at Hortonworks (formerly at Yahoo!)Apache Hadoop Committer and Member of PMCFull-time contributor to Apache Hadoop since early 2006
- 3. Hadoop MapReduce TodayJobTrackerManages cluster resources and job schedulingTaskTrackerPer-node agentManage tasks
- 4. Current LimitationsScalabilityMaximum Cluster size – 4,000 nodesMaximum concurrent tasks – 40,000Coarse synchronization in JobTrackerSingle point of failure Failure kills all queued and running jobsJobs need to be re-submitted by usersRestart is very tricky due to complex stateHard partition of resources into map and reduce slots© Hortonworks Inc. 20115
- 5. Current LimitationsLacks support for alternate paradigmsIterative applications implemented using MapReduce are 10x slower. Example: K-Means, PageRankLack of wire-compatible protocols Client and cluster must be of same versionApplications and workflows cannot migrate to different clusters© Hortonworks Inc. 20116
- 6. RequirementsReliabilityAvailabilityScalability - Clusters of 6,000-10,000 machinesEach machine with 16 cores, 48G/96G RAM, 24TB/36TB disks100,000+ concurrent tasks10,000 concurrent jobsWire CompatibilityAgility & Evolution – Ability for customers to control upgrades to the grid software stack.© Hortonworks Inc. 20117
- 7. Design CentreSplit up the two major functions of JobTrackerCluster resource managementApplication life-cycle managementMapReduce becomes user-land library© Hortonworks Inc. 20118
- 8. Architecture
- 9. ArchitectureResource ManagerGlobal resource schedulerHierarchical queuesNode ManagerPer-machine agentManages the life-cycle of containerContainer resource monitoringApplication MasterPer-applicationManages application scheduling and task executionE.g. MapReduce Application Master© Hortonworks Inc. 201110
- 10. Improvements vis-à-vis current MapReduceScalability Application life-cycle management is very expensivePartition resource management and application life-cycle managementApplication management is distributedHardware trends - Currently run clusters of 4,000 machines6,000 2012 machines > 12,000 2009 machines<16+ cores, 48/96G, 24TB> v/s <8 cores, 16G, 4TB>© Hortonworks Inc. 201111
- 11. Improvments vis-à-vis current MapReduceFault Tolerance and Availability Resource ManagerNo single point of failure – state saved in ZooKeeperApplication Masters are restarted automatically on RM restartApplications continue to progress with existing resources during restart, new resources aren’t allocatedApplication MasterOptional failover via application-specific checkpointMapReduce applications pick up where they left off via state saved in HDFS© Hortonworks Inc. 201112
- 12. Improvements vis-à-vis current MapReduceWire Compatibility Protocols are wire-compatibleOld clients can talk to new serversRolling upgrades© Hortonworks Inc. 201113
- 13. Improvements vis-à-vis current MapReduceInnovation and AgilityMapReduce now becomes a user-land libraryMultiple versions of MapReduce can run in the same cluster (a la Apache Pig)Faster deployment cycles for improvementsCustomers upgrade MapReduce versions on their scheduleUsers can customize MapReduce e.g. HOP without affecting everyone!© Hortonworks Inc. 201114
- 14. Improvements vis-à-vis current MapReduceUtilizationGeneric resource model MemoryCPUDisk b/wNetwork b/wRemove fixed partition of map and reduce slots© Hortonworks Inc. 201115
- 15. Improvements vis-à-vis current MapReduceSupport for programming paradigms other than MapReduceMPIMaster-WorkerMachine LearningIterative processingEnabled by allowing use of paradigm-specific Application MasterRun all on the same Hadoop cluster© Hortonworks Inc. 201116
- 16. SummaryMapReduce .Next takes Hadoop to the next levelScale-out even furtherHigh availabilityCluster Utilization Support for paradigms other than MapReduce© Hortonworks Inc. 201117
- 17. Status – June, 2011Feature completeRigorous testing cycle underwayScale testing at ~500 nodesSort/Scan/Shuffle benchmarksGridMixV3!Integration testingPig integration complete!Coming in the next release of Apache Hadoop!Beta deployments of next release of Apache Hadoop at Yahoo! in Q4, 2011© Hortonworks Inc. 201118
- 19. Thank You.© Hortonworks Inc. 2011