Why is Iceberg choosing to deperecate, positional delete in MoR?

If you followed the latest journey of Apache Iceberg, then you might have heard about introducing Deletion Vector(DV) like Delta Lake and deprecating positional deletes from the MoR table.

Apache Iceberg’s format spec 3 introduces a major change to how row-level deletes are handled in Merge-on-Read (MoR) tables. In Iceberg, format spec 2, MoR supports two types of deletes.

1. Positional Deletes
2. Equality Deletes

In Iceberg v2, in most of the query engines, positional delete files were used to mark specific row positions as deleted. In v3, these positional delete files are being deprecated and replaced by deletion vectors.

This shift is driven by practical scalability issues observed with positional deletes, and the new deletion vector approach is designed to solve those problems.

While using Iceberg v2, I found that positional deletes are an effective way to manage deletes in open table formats like Iceberg. Most query engines, such as Apache Spark, Trino, and our own e6data, strongly support positional deletes during writing. In contrast, Apache Flink supports equality deletes.

Upon hearing the announcement about the introduction of a Deletion Vector (DV) and the gradual deprecation of positional deletes in the MoR table, I became curious to explore deeper as a Data Engineer. For me, positional deletes or equality deletes used to work really well.

Let's take a deep dive into this announcement and gain a thorough understanding of it together.

We'll start from the very beginning. I've planned a series of blogs to explore this topic in depth. But, by the end of this blog post, you will grasp the following key aspects of Apache Iceberg.

• What are COW and MOR? How to configure these?
• Type of deletes in MoR: Positional Deletes and Equality Deletes
• How does Positional Deletes work in Iceberg V2?
• Trade-Off Between Partition-Level and File-Level Deletes in Positional Deletes?
• Issues with Positional Deletes & why it is being deprecated in V3
• Introduction of Deletion Vector and how it can solve deletion better?

To enhance Apache Iceberg's read and write performance, a crucial aspect is how row-level updates are managed. CoW (Copy-on-Write) and MOR (Merge-on-Read) are the two strategies that can be configured in an Iceberg table to manage row-level updates.

Understanding how these strategies function internally provides you with the capability to define them in the initial phases of your Iceberg table designs, ensuring your tables remain performant over time.

Copy-on-Write:

• With this approach, whenever there are changes to the Apache Iceberg table, either updates or deletes, the data file associated with the updated records will be duplicated and updated.
• A new snapshot of the Iceberg table will be created, pointing to the newer version of the data files. This makes the overall writing slower.
• There might be situations where concurrent writes are needed with conflicts, so a retry has to happen, which increases the write time even more.
• On the other hand, when reading the data, there is no extra process needed. The query will retrieve data from the latest version of the data.
• CoW is mostly used when there is a need for faster reads, but the speed of the writer can be a little slower.

Merge-on-Read:

• In this approach, when updates or deletions are made to the Iceberg table, the existing data files are not rewritten. Instead of rewriting the entire data files, delete operations generate delete files.
• During reading, these delete files are applied on the fly, which filters out outdated records. While this method reduces write amplification, it may degrade read performance due to the extra processing involved.
• The query engine, which acts as the consumer of Apache Iceberg, is responsible for merging the delete parquet files with the actual parquet files to provide the updated or latest data.
• By default, in Iceberg format specification version 2, tables are organized in Copy-on-Write (CoW) mode. To implement Merge-on-Read, you need to set specific properties for the table.

spark.sql("""
      CREATE TABLE demo.learn_iceberg.ankur_ice_3 (
          id   INT,
          data STRING
     ) USING iceberg
     TBLPROPERTIES (
         'write.format.default' = 'parquet',
         'write.delete.mode' = 'merge-on-read',
         'write.update.mode' = 'merge-on-read',
         'write.merge.mode' = 'merge-on-read',
         'format-version' = '2'
     )
""")        

Properties for configuring

• write.delete.mode : Approach to use for delete transactions
• write.update.mode: Approach to use for update transactions
• write.merge.mode: Approach to use for merge transactions

Please remember that these properties represent the specifications, and whether they function as intended depends on whether the query engine being used respects them. If it does not, you may encounter unexpected results. In MoR, query engine or cosumer application of is responsible for all the operations.

In the screenshot of the data folder for my Apache Iceberg table, you will notice that there are two Parquet files. The original Parquet file was created at 9:38 AM. After applying the DELETE command at 9:39 AM, a delete parquet file with the extension ...deletes.parquet was generated.

Keep in mind that Delete Files are only supported by Iceberg v2 tables not in Iceberg v1.

I believe you now have a good understanding of Copy-on-Write (CoW) and Merge-on-Read (MoR). Let's move on to the next important topic: positional and equality deletes in Apache Iceberg. We will also discuss why positional deletes are being replaced by Deletion Vectors in the Iceberg v3 specification.

Delete types in MoR: Postional Delete & Eqaulity Delete

Delete files are used to track which records in a dataset have been logically deleted and should be ignored by the query engine when accessing data from an Iceberg Table.

These delete files are created within each partition, based on the specific data file from which records have been logically deleted or updated.

There are two types of delete files, categorized by how they store information about the deleted records:

Positional Delete Files

These files record the exact positions of the deleted records within the dataset. They keep track of both the file path of the data file and the positions of the deleted records within that file.

A position delete file lists row positions in specific data files that should be considered deleted.

At read time, the query engine merges these delete files with the data files to mask out deleted rows on the fly using information present in metadata and column information like file_path and position(pos).

While this approach avoids expensive rewrites on every delete, it introduces several design and performance drawbacks that have been observed at scale. We will discuss this in detail.

Let's try to take a quick look at Equlity Deletes before deepening a dive into understanding the constraints with Postional Deletes.

Equality Delete Files

Equality Delete Files stores the value of one or more columns of the deleted records. These column values are stored based on the condition used while deleting these records.

When I began to understand delete files, the most intriguing question that arose was: "How can I choose or configure my table to use either positional or equality delete files when handling row-level updates in PySpark?"

Actually, there is no support in Apache Spark for equality deletes at the time of writing to Iceberg. I have researched that Apache Flink supports writing delete files in equality deletes but I have not tried it yet.

Alrighty, finally as now we know about both the delete files, let's take a look at the next discussion on why positional deletes are being deprecated.

How Positional Deletes Worked (and Why They Were Complex)

Positional delete files in Iceberg mark individual rows as deleted based on their position within a data file. Each entry in a positional delete file contains a data file path and a row position (row ordinal) within that file, and optionally the row data itself.

For example, suppose we have a data file data-file-1.parquet with 100 rows. If we delete two specific rows from it in a MoR table, Iceberg might produce a small delete file with contents like:

Each line means “row at position X in data-file-1.parquet is deleted”.

At query time, the engine will merge this delete information with the actual Parquet file: as it reads data-file-1.parquet, it will skip or filter out the rows at positions 0 and 102, effectively hiding them from query results.

This merging happens using building an in-memory bitmap & there is a huge cost of serializing the parquet files to BitMap at both read & write levels or time. At read time, the query engine merges these delete files with the data files to mask out deleted rows on the fly. While this approach avoids expensive rewrites on every delete, it introduces several design and performance drawbacks that have been observed at scale.

Firstly we have to understand in their initial development release Apache Iceberg format spec 2, they started with Partition-Scoped Delete Files and then switched to File-Scoped Delete Files.

Let's try to understand the pros and cons of these strategies and understand them with some examples.

1. Trade-Off Between Partition-Level and File-Level Deletes

Partition-Scoped Delete Files:

• All deletes within a partition can be grouped into one file. This minimizes the number of delete files on disk, but it means each file covers multiple data.
• For example, if Partition P1 has data files A and B, a single partition-level delete file might record deletions for both. Reading one data file then requires scanning delete entries for all files in that partition, not just the target.
• In our example, a query reading file A in P1 would still have to read the combined delete file (which contains entries for A and B) and then discard B’s entries as irrelevant. This results in wasted I/O and processing – the query fetches delete data for files it doesn’t actually need, just because those deletes were consolidated for storage efficiency.

File-Scoped Delete Files:

• Each data file gets its own dedicated position delete file when rows in it are deleted.
• This target deletes very precisely: a reader of file A will load only A’s delete file (and ignore B’s).
• The trade-off is many more small files. In the example, deleting one row in A and one in B would produce two separate tiny delete files, instead of one combined file.
• At scale, file-scoped deletes can lead to a proliferation of files and metadata entries. This approach “may require a more aggressive approach for delete file compaction” to keep the file count under control.

Iceberg’s current format (spec v2) allows both strategies, but neither is ideal on its own.

Partition-level deletes reduce file count at the cost of extra read overhead; file-level deletes minimize read overhead but explode the number of files to manage.

In practice, users often face a no-win decision: either suffer the read-time inefficiency of coarse-grained delete files or incur the metadata bloat and operational hassle of millions of fine-grained delete files.

The design even permits multiple delete files per data file (e.g. if you perform many separate delete operations on the same file), which further multiplies file counts unless periodically compacted

2. Read I/O Overheads During Query Execution

Reading data in a MoR table with positional deletes involves extra I/O and computation that grows with the number of delete files:

A). Extra file reads per data file:

• For each data file that a query task reads, the engine must also open and read all corresponding position delete files to know which rows to skip.
• This is done by loading the delete file contents (typically stored as Parquet/Avro/ORC lists of positions) and building an in-memory bitmap of deleted rows.
• Only then can the data file be scanned with those rows filtered out.
• The Iceberg documentation notes that readers are required to “open all matching delete files and merge them” into a deletion bitset before scanning the data.
• This means if a data split has, say, 5 small delete files applicable (not uncommon if multiple deletions happened over time and no compaction combined them), the reader performs 5 additional file reads and merges five lists/bitmaps in memory.
• These overheads add up linearly with the number of delete files per task.

B). Irrelevant data reads:

• As discussed earlier, if using partition-scoped deletes, a single delete file can cover many files. In such cases, a reader will pull in that delete file (which might be relatively large) even if only a small fraction of its entries apply to the specific data file being read.
• The irrelevant entries are ultimately discarded, but the work of fetching and parsing them is already done.
• This is pure overhead – effectively reading data that you immediately throw away. For instance, if a partition delete file is 50 MB and only 5 MB of it pertains to the one data file your task is scanning, 45 MB of I/O and parsing work is wasted.
• This inefficiency is the price of the partition-level optimization (fewer files on disk at the cost of heavier reads).

C). Many small file opens:

• Conversely, with file-scoped deletes, you avoid reading unrelated deletes but incur the cost of opening many tiny files.
• Each file open on a distributed storage (HDFS, S3, etc.) has non-negligible latency and overhead.
• In fact, internal benchmarks noted that the difference in size between compact versus verbose delete data is often “less than the open file cost” for small.
• In plainer terms, reading N small delete files can be slower than reading one combined file of size equal to their sum, because the setup/seek time dominates. Thus, a flurry of small delete files (which is common if many fine-grained updates were done) can dramatically increase query overhead simply due to connection and file-read setup costs for each file.

D). Runtime filtering cost:

• Once loaded, the delete information is applied via an in-memory bitmap (Roaring Bitmap) to filter rows.
• This step is actually quite efficient – Iceberg’s vectorized reader can check the bitmap with minimal per row.
• The main hit to performance comes before that point: opening, reading, and materializing all those bitmaps.
• If deletes are numerous, the time spent preparing deletion bitmaps can be significant. For example, if a table has a large fraction of its rows deleted (but not yet compacted), the delete files might collectively be huge – approaching the size of the data itself.
• In an extreme case, reading data with 50% of rows deleted could double the scan workload (you read 100% of the data file and nearly 50% worth of delete entries).
• Even though each piece is processed efficiently, the total I/O is much higher than scanning a file with no deletes.

Summarized impact: Every positional delete introduces overhead on read: an extra file to open and a list of positions to process. With small numbers of deletes this overhead is negligible, but at scale (hundreds of thousands or millions of deletes spread across many files) it becomes a major factor. The approach essentially offloads the delete merge work to query time, affecting I/O throughput and latency for queries on heavily updated tables.

3. Accumulation of Delete Files and Maintenance Burden

Because position deletes remain separate from the data, they accumulate with each update/delete operation. The Iceberg spec does not require automatically merging new deletes with existing ones during writes.

This means that if you execute many delete or update commands, your table will collect a growing pile of delete files over time.

For example, if you delete a few rows every day in a given partition without cleanup, after 30 days you could have 30 separate delete files just for that partition (and many more across the table). Iceberg relies on the user or an external process to regularly compact these deletes, i.e. perform maintenance:

• Minor compaction: consolidate multiple small delete files into one larger delete file.
• Major compaction: merge deletes into the data files themselves by rewriting data (converting MoR to Copy-on-Write for those parts).

If such maintenance is not done, performance degrades quickly.

Every additional delete file adds overhead to future reads and writes. Anecdotally, tables that underwent frequent updates without compaction saw query slowdowns as the engine had to juggle ever-growing lists of delete files.

"users have to manually invoke actions to rewrite position deletes" If users fail to provide adequate maintenance... write and read performance can degrade quicker than desired.

In other words, consistent performance depends on constant vigilance – running periodic cleanup jobs, which adds operational complexity. This is a shortcoming because an efficient storage format should ideally manage metadata growth internally.

Relying on external housekeeping means a risk of human error or lag: a lapse in running compaction can leave the table with thousands of tiny delete files and significantly slower reads.

Example:

In one production scenario, a petabyte-scale Iceberg table partitioned by date experienced deletions across many partitions daily. Because each day’s partition had its own delete file(s), after weeks the table ended up with tens of millions of tiny delete files on disk.

Even though partition-scoped deletes were used to minimize file count per partition, the sheer number of partitions (each with at least one delete file) led to an explosion of files.

This large collection of delete files not only burdened the metadata and storage system (many small objects causing overhead on the filesystem), but also made query planning and execution increasingly sluggish. Such a situation demands aggressive compaction to merge or remove delete files – essentially fighting the format’s tendencies to keep performance in check.

4. Manifest and Metadata Growth:

• Each positional delete file is listed in Iceberg’s manifest files (metadata). As the count of delete files grows, manifests inflate (“manifest bloat”), and tracking these in each snapshot’s metadata becomes heavier.
• For example, 1000 small delete files might add 1000 entries to manifests. This not only makes the metadata files larger on disk but also means the Iceberg planner (catalog) has to process more entries when you refresh a table or plan a query.
• Tools like Trino and Spark have to load snapshot metadata, so a bloated manifest can increase planning time and even memory usage on the driver/coordinator.
• We have to observe that too many metadata files (including delete files) can increase memory pressure since the file lists are often cached & indexed in memory for performance.

5. Dangling Deletes and Rewrites:

• Positional deletes also complicate data file rewrites. If a data file is rewritten (for example, due to compaction or optimizing file sizes), ideally any delete entries for the original file should be retired as well. Iceberg ensures snapshot isolation, so it won’t drop delete files until the rewrite is committed and older snapshots are expired.
• However, in practice, there have been cases where “dangling” position delete entries remain in metadata referencing files that are no longer.
• This requires an extra cleanup step (Iceberg provided the rewrite_position_delete_files procedure to handle this by removing deletes for non-existent data files and merging the rest). All of this adds operational complexity.

In summary, while positional deletes do achieve per-row deletion without rewriting whole files, they introduce non-trivial overhead in terms of extra files, extra metadata, and runtime merge work. These costs accumulate with each delete operation, making large-scale incremental deletes increasingly expensive to manage.

I think with all the above examples, and discussion you might have understood the complications and shortcomings of using Positional Deleted in the MoR table of Apache Iceberg.

Now, let's try understanding the Deletion Vector(DV) which is being introduced in Apache Iceberg format-spec 3 which will slowly replace these positional deletes.

What Are Deletion Vectors (Iceberg v3’s Solution)?

Deletion vectors (DVs) are the new mechanism in Iceberg format v3 that replaces positional delete files. A deletion vector is essentially a bitmap of deleted row positions for a given data file. Instead of storing a list of positions in a separate file, a DV records the same information as a binary bitset: if a bit is “1” (or is present in the bitmap), the corresponding row position is deleted.

How deletion vectors work:

• Each data file that has deletions will have at most one deletion vector associated with it in a given. The deletion vector maps the positions within that specific data file that should be considered deleted. For example, if data-file-1.parquet has rows 0 and 102 deleted (as in our earlier example), the deletion vector for data-file-1.parquet would have bits 0 and 102 set to indicate those positions are deleted. All other bits (positions) are unset.
• Deletion vectors are stored as binary blobs (using a format based on Roaring Bitmap, a compressed bitmap representation). These blobs are not stored as standalone arbitrary files in the table’s data path; instead, they are stored in a special structured file (using the Puffin format, which is a container for auxiliary binary data in Iceberg). Multiple deletion vector bitmaps can be packed into one Puffin file for efficiency.
• The table metadata (manifests) will then reference the Puffin file along with an offset and length for each bitmap. In other words, instead of listing a bunch of small delete files, the manifest can list something like: “data-file-1.parquet has a deletion vector at offset X in delete_vectors.puffin (length Y bytes)”.
• Because only one vector per data file is allowed, any time new deletions are added for that data file, the vector needs to be updated (or replaced) rather than appending another separate vector.
• In practice, adding a deletion in a new commit will merge the bitmap (union the new deleted position(s) with the existing bitmap) and write out a new combined deletion vector. This ensures there is still just one contiguous bitmap representing all deletions for that file. The older vector may be discarded or superseded in the new snapshot. This design means we don’t stack up multiple layers of delete info for one file – it’s always consolidated.
• The Iceberg spec defines these deletion vector blobs and their metadata. The manifest entries for delete files now include a file_path (which points to the Puffin file), plus the content_offset and content_size that locate the bitmap within that file file.
• It also stores which data file the vector belongs to. Essentially, the manifest directly links a data file to a specific bitmap of its deleted rows.

The introduction of deletion vectors is a deliberate effort to optimize and simplify row-level deletes.

Ryan Blue (Iceberg co-creator) & Anton Okolnychyi explained that the Iceberg community, in designing v3, worked with the Delta Lake team (who had implemented a similar idea) to get this right. I have just loved the way Anton, has explained all these concepts in this year's Apache Iceberg Summit.

The goal is to remove specific rows from data files without rewriting files but with far less overhead. In fact, deletion vectors achieve the same logical outcome as positional deletes – hiding deleted rows at read time – but store the information more compactly and efficiently.

I think this blog has got really really long😅. Let me stop here and in the next blog let's try discussing more about deletion vectors, more in depth.

Till then, thank you very much for your precious time as a reader.

Keep learning, do deep dive and enjoy the engineering :)