Improving Python and Spark Performance and Interoperability with Apache Arrow

Editor's Notes

• #3: We are a quantitative hedge fund based in New York City
• #6: Spark build-in functions overs basic math opeartors and functions, such as ‘mean’, ‘stddev’, ‘sum’
• #7: This is in comparison with build-in spark functions such as mean and sum where the python code gets translated into java code and executed in JVM Grouped based udf doesn’t exist now. Not sure about calling: lambda values: np.mean(np.array(values)) “grouped based udf”
• #8: 60x is computed by: T1 = df.agg(F.sum(df.v1)) T2 = df.withColumn(‘v2’, when(df.v1 > 0, 1.0).otherwise(-1.0)).agg(F.sum(‘v2’) T3 = df.withColumn(‘v3’, F.udf(lambda x: 1.0 if x > 0 else -1.0, DoubleType())(df.v1)).agg(F.sum(“v3”)) (T3 – T1) / (T2 – T1)
• #9: Grouping here can be by some id, category, or time based grouping, daily average
• #10: Groups are materialized is one of the performance issue. However, this is not the one we focus on here. For instance, groupBy().agg(collect_list()) fully materialized the group. It takes about 1.5 seconds to do that, while the normalization examples takes more than 25 seconds
• #11: Groups are materialized is one of the performance issue. However, this is not the one we focus on here. For instance, groupBy().agg(collect_list()) fully materialized the group. It takes about 1.5 seconds to do that, while the normalization examples takes more than 25 seconds
• #12: (complicated code. Audience should not read the code. The presenter shows the boiler plate code vs actually code. The point here is show audience how difficult it is to do such things)
• #13: (complicated code. Audience should not read the code. The presenter shows the boiler plate code vs actually code. The point here is show audience how difficult it is to do such things)
• #14: (complicated code. Audience should not read the code. The presenter shows the boiler plate code vs actually code. The point here is show audience how difficult it is to do such things)
• #23: Scatter/gather I/O also unknown as vectored I/O is the a method where single procedure call reads data from a stream and write to multiple buffers
• #29: To illustrate further, here is the profiling result of python worker during one UDF evaluation. Without profiling, It took about 2 seconds to process 2M doubles. And most of the time is spent in serialization and udf evalution
• #30: So we made some changes to the PySpark
• #32: Note: Without profile, 4 seconds -> 2 seconds Let’s now compare Scalar vs Vectorized UDF. For 2 M doubles, the runtime of the python worker goes from 2 seconds to about 0.14 seconds, that’s a 15x speed up.
• #33: Note: Without profile, 4 seconds -> 2 seconds Let’s now compare Scalar vs Vectorized UDF. For 2 M doubles, the runtime of the python worker goes from 2 seconds to about 0.14 seconds, that’s a 15x speed up.
• #34: Note: Without profile, 4 seconds -> 2 seconds Let’s now compare Scalar vs Vectorized UDF. For 2 M doubles, the runtime of the python worker goes from 2 seconds to about 0.14 seconds, that’s a 15x speed up.
• #41: Groups are materialized is one of the performance issue. However, this is not the one we focus on here. For instance, groupBy().agg(collect_list()) fully materialized the group. It takes about 1.5 seconds to do that, while the normalization examples takes more than 25 seconds
• #42: Make it clear before and after
• #45: Less time on this slide RPC: Generic way to exchange data. IPC: Share memory Integration: