Introduction to Apache Spark

Editor's Notes

• #3: Typical map-reduce paradigm, these steps are also common to spark. All the results of map is stored on the disk for safety. Bonus point always avoid shuffling. But, the problem is when you have multi stage map-reduce jobs.
• #4: For chain of map-reduce each subsequent step’s output is stored on the disk.
• #5: HDFS distributed file system
• #6: Spark is like an engineering student. Siddarth took a similar talk A group of engineering students whenever asked for assignments, practical records and what not. They just nod along. Planning for each weekend doing nothing. But, at the moment the professor asks at the end of the semester and threatens them that they will lose marks. At, that moment engineering students actually starts working. Lazy evaluation: To trigger the execution, an action is a must. Immutable and Read-only: RDDs are immutable, which means unchangeable over time. That property helps to maintain consistency when we perform further computations. As we can not make any change in RDD once created, it can only get transformed into new RDDs. This is possible through its transformations processes. Cacheable: We can store all the data in persistent storage, memory, and disk. Memory (most preferred) and disk (less Preferred because of its slow access speed). Partitioned: Each dataset is logically partitioned and distributed across nodes over the cluster. They are just partitioned to enhance the processing, Not divided internally. This arrangement of partitions provides parallelism. Fault tolerance: While working on any node, if we lost any RDD itself recovers itself. When we apply different transformations on RDDs, it creates a logical execution plan. The logical execution plan is generally known as lineage graph. As a consequence, we may lose RDD as if any fault arises in the machine. So by applying the same computation on that node of the lineage graph, we can recover our same dataset again. As a matter of fact, this process enhances its property of Fault Tolerance. Location Stickiness: That DAG(Directed Acyclic Graph) scheduler use to place computing partitions on. DAG helps to manage the tasks as much close to the data to operate efficiently. This placing of data also enhances the speed of computations. 
• #7: Small slide with image to drive the point.Partitioning is the key to parallelism.
• #8: Small slide with image to drive the point.Holds lineage to parent RDD can only be transformed.
• #9: Small slide with image to drive the point.Just plans doesn’t compute unless the answer is asked for.
• #11: SOME RANDOM IDEAS. HAVE TO BE TIED TOGETHER IN A HANDS ON Some online spark workbook for them to try outCreating a bare RDD from list, from csv file Parallelizing in RDD sc.parallelize(0 to 9) sc.parallelize(0 to 90 by 10) # Word count example and when Debug string is printed one can see lineage val wordCount = sc.textFile("README.md").flatMap(_.split("\\s+")).map((_, 1)).reduceByKey(_ + _) wordCount: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[21] at reduceByKey at <console>:24 scala> wordCount.toDebugString # Number of partitions wordCount.getNumPartitions
• #12: SOME RANDOM IDEAS. HAVE TO BE TIED TOGETHER IN A HANDS ON Some online spark workbook for them to try outCreating a bare RDD from list, from csv file Parallelizing in RDD sc.parallelize(0 to 9) sc.parallelize(0 to 90 by 10) # Word count example and when Debug string is printed one can see lineage val wordCount = sc.textFile("README.md").flatMap(_.split("\\s+")).map((_, 1)).reduceByKey(_ + _) wordCount: org.apache.spark.rdd.RDD[(String, Int)] = ShuffledRDD[21] at reduceByKey at <console>:24 scala> wordCount.toDebugString # Number of partitions wordCount.getNumPartitions