Data Source V2 API in Spark 3.0 - Part 3: In-Memory Data Source
Spark 3.0 is a major release of Apache Spark framework. It’s been in preview from last December and going to have a stable release very soon. As part of a major release, Spark has a habit of shaking up API’s to bring it to latest standards. There will be breaking changes also in these API’s. One of such API is Data source V2 API.
Data Source V2 API, a new data source API for spark, was introduced in spark 2.3. Then it’s been updated in spark 2.4. I have written detailed posts on the same here.
This API is going to be completely changed in Spark 3.0. Spark rarely change an API this frequently in between releases. But as data source is heart of the framework, they are improved constantly. Also in spark 2.4, these API’s were marked evolving. This means they are meant to be changed in the future.
The usage of the data sources has not changed in 3.0. So if you are a user of the third party data sources you don’t need to worry. These changes are geared mainly towards the developer of these sources. Also, all the sources are written V1 API going to work even in 3.0. So if your source is not updated, no need to panic. It’s going to work without the latest optimizations.
These new changes in V2 API brings more control to data source developer and better integration with spark optimiser. Moving to this API makes third party sources more performant. So in these series of posts I will be discussing the new Data source V2 API in 3.0.
This is third post in the series where we discuss about building simple in-memory data source. You can read all the posts in the series here.
In-Memory Data Source
In this post we are going to build a data source which reads the data from an array. It will have single partition. This simple example helps us to understand how to implement all interfaces we discussed in last blog.
The below are the steps to implement a simple in memory data source.