Efficient Column Updates in Apache Iceberg

Efficient Column Updates in Apache Iceberg

Author: Anurag Mantripragada (anuragmantripragada@apache.org)

Authors from Column Families proposal: Péter Váry (pvary@apache.org)  Gábor Kaszab (gaborkaszab@apache.org)

(with input from Anton Okolnychyi, Daniel Weeks and Parth Chandra and many community members)

Background & Motivation

Historically designed for append-heavy workloads and immutable objects, modern data lakes are increasingly serving as the backbone for AI/ML workflows like vector stores and feature stores. Unlike traditional business intelligence data, these workloads are characterized by high dimensionality (tables with thousands of columns), heterogeneous content (mixing structured data with unstructured text, images, and embeddings), and asynchronous data lifecycles.

In this new paradigm, data is rarely static. A single "row" is often composed of base attributes that rarely change, combined with hundreds of derived attributes (e.g., embeddings, classification scores, tokens) that evolve rapidly and independently. This mismatch between the physical storage layout (row-oriented grouping) and the logical update patterns (column-oriented evolution) creates inefficiencies in current table formats such as Apache Iceberg and file formats such as Apache Parquet.

Current State

The core limitation in current Apache Iceberg implementation (with file formats like Parquet) is the coupling of columns within a file. While columnar formats allow efficient reads by skipping unnecessary columns, they enforce rigid writes. Current Iceberg implementation provides two ways to update data:

1. Copy-on-Write (COW): Updating a single column such as refreshing a model score requires identifying all data files containing the target rows and rewriting them entirely. This leads to write amplification.

2. Merge-on-Read (MOR): While MOR mitigates immediate rewrites by logging changes to delete files, it currently operates at row granularity. An update is essentially a DELETE of the old row and an INSERT of the new row. Even if only one field changes, the delete file must store the complete new row state to facilitate merging. This prevents the "surgical" updates required for efficient ML pipelines.

Impact on ML/AI Use Cases

The row-granularity limitation becomes particularly problematic for ML/AI workloads:

• Feature Backfilling & Column Updates: A common workflow involves adding a new feature column (e.g., a model embedding) to a petabyte-scale table.
• Model Score Updates:  Refreshing prediction scores after model retraining involves updating a subset of score columns in wide tables.
• Embedding Refresh: Updating vector embeddings in wide feature tables causes the entire row to be rewritten.
• Incremental Feature Computation: Daily batch jobs that compute and update 5-10 features out of 200 total features making daily updates cost-prohibitive at petabyte scale.

In all the above cases, the cost of adding or updating one column is proportional to the size of the entire table, not just the column being changed. This proposal attempts to address the write amplification problem in Iceberg by introducing column-level updates, enabling engines to write the updated columns to separate column files ^{[a][b][c][d][e][f]}and efficiently stitch the column files during read time to materialize all the rows of the table.

Goals

• Reduce write amplification on column updates i.e write only the updated column values.
• Preserve read efficiency (column stats and pruning capabilities) for queries projecting the updated columns.
• Leverage V4 architecture (build on Iceberg Single File Commits and Column Stats Improvementsproposals)

Non-goals

Proposal

The proposal is to achieve column-level updates through Iceberg metadata enhancements alone, without requiring modifications to Parquet libraries. The proposal is based on the the in-flight Iceberg Single File Commitsproposal.

Note: A Parquet-native approach was evaluated but discarded to keep that design at table format level (see Discarded Options).

Core Concept: Column Files

• A Column File is a standard data file that contains only the data for the updated columns (e.g., score, weight).
• A column file always applies to a single base file but a base file can have multiple column files as columns evolve.

Approach 1: Iceberg Metadata Updates

The key principle of this approach is that column updates are handled purely through metadata mutations, leaving underlying base data files untouched to prevent costly write amplification. This is achieved by logically associating new column files with existing base files at the manifest level.

Based on the scope of the operation, the engine could choose either of the following:

• Metadata copy-on-write: This method involves rewriting the data file entries to add updated column file references. While this approach deviates from the V4 principle of single-file commits and immutable entries in a root manifest, it is a worthwhile trade-off for the particular updates this design addresses: updating all rows of one or more columns. The key advantage is that this is only a metadata mutation, underlying data files themselves are not rewritten. 

• (Discarded) Metadata merge-on-read: This approach leverages V4 Manifest Deletion Vectors (MDVs) to invalidate the entries in the original leaf manifest and write a new leaf manifest with the new column files.

1.1 Spec extensions for in-line column manifest updates

We add a new struct field `column_files` to the ContentEntry structure.

V4 column_files_struct 

Note: Tracking fields are carried on the parent TrackedFile entry via the V4 Tracking struct (see TrackedFile schema()). When a column file is added, the parent entry's latest_column_file_snapshot_id records when the column file became part of the table state.

1.2 Metadata representation

There is at most one column_files_struct entry per field_id for a given base file. When the same field is updated again, the writer produces a new column file that carries over all previous values and updates the old column_file_struct.

The following examples illustrate typical column UPDATE operations and their corresponding representation.

1.2.1 ADD COLUMN

1.2.2 UPDATE multiple columns (possible with this new spec)

1.2.3: Multiple non-overlapping column update files,

1.2.4: (Not Allowed) overlapping field_ids across multiple column updates

This is not allowed. Each field_id must appear in at most one column_files_struct entry. To update field 5 again, the writer must produce a new column file that carries over the values from the previous column file for field 5, replacing the old entry.

1.3 Writing Column Files

When an engine processes a column update statement, the write operation proceeds in the following phases:

1.3.1 Planning Phase:

The engine identifies all base data files that will be affected by examining the root manifest. The engine produces logical updates by reading the base file and maintaining its position in a tuple (new_value, _pos, file_path), which identifies the new value, its ordinal row position (_pos),and the base file it belongs to (file_path). The engine then co-locates the updates by distributing them by file_path. Ordering the rows before the write phase by file_path and row position is also required.

1.3.2 Write Phase:

For each base file, the engine writes a new "column file" containing only the updated column data. The critical decision here is the row alignment strategy (described in detail below).

1.3.3 Row Alignment Strategy

A column update file MUST adopt a fully-aligned representation, ensuring strict positional alignment with its associated base file. This requirement implies that the column file contains precisely the same record_count as the base file, where every row i in the update file maps directly to row i in the base data. Readers can then execute efficient positional substitution: when a projected field is mapped to a column file, the engine simply retrieves row i from the update file rather than the base.

The rationale for this approach is that by specifying this representation, we avoid forcing all implementations to handle the more complex position based stitching with the base file.

Deleted positions are filled with NULL. To permit this, every leaf column in a column update file is nullable, even when the corresponding table or base column is required (non-null). This strategy keeps min/max bounds clean.

1.3.5 Column File Full Representation

For each field_id represented, a column update file must be be fully-aligned, matching the base_file.record_count precisely by providing a physical value for every row position:

• Updated live rows: The entry stores the newly computed state.
• Non-updated live rows: The original value is carried over via a copy-on-write mechanism. This allows the reader to resolve the entire column from the update file.
• Deleted rows: A NULL filler is utilized. While writers SHOULD apply nulls for known-deleted rows during the write phase, this remains a recommendation rather than a requirement to account for concurrent DV applications; the DV eventually ensures row exclusion while preserving file integrity.

Every write operation is required to include the _pos field within the corresponding update files because it facilitates debugging. Benchmarks show that it compresses well in Parquet so it is not a significant overhead.

1.4 Commit Phase

1.4.1 Replaced and modified manifest update

The commit process for a column update involves traversing the manifest tree of the preceding snapshot to locate manifest entries where the base file is affected. For every target entry, the commit operation generates a corresponding pair of entries within the new snapshot:

• The original entry is changed to a REPLACED state (Tracking.status = REPLACED). While it no longer represents the active state for that base file, it is preserved to support time-travel and change-detection queries. Its Tracking.snapshot_id remains associated with the snapshot where it was last modified.
• A successor entry sharing the same data_file_path is created with MODIFIED status. This entry includes the updated column_files list, and its Tracking.snapshot_id is updated to the current snapshot.

This REPLACED/MODIFIED pair can be co-located within the same manifest file. Alternatively, the writer may retain the REPLACED entry in its current leaf manifest marking the status change via a Manifest Delete Vector (MDV) while writing the MODIFIED entry to the root manifest, a new leaf manifest, or a different existing leaf. Engines determine the optimal layout based on commit-cost trade-offs, and readers reconcile the pair across manifests by joining on the data_file_path. Both implementations ensure consistent commit semantics.

1.4.1.1 Figure: Column update commit (leaf-rewrite layout)

1.4.1.2  Figure: Column update across mixed root and leaf manifest entries

1.4.1.3  Figure: Column update with MDV-based non-colocated manifest

1.4.2 (Discarded) Merge-on-read manifest update

As an alternative to rewriting manifests, engines can use a merge-on-read approach that leverages V4 Manifest Delete Vectors (MDVs). This strategy avoids modifying existing leaf manifests especially for partial updates. An update operation will invalidate the data file entries in existing leaf manifests using MDVs and write new manifests with the updated entries with the associated column update files.

1.4.2.1 Diagram

1.5 Planning

Query planning with inline column metadata follows Iceberg's standard two-phase planning model. No additional phase is required because column file mappings are pre-resolved at write time.

1. Manifest Entry Resolution

Planning starts by reading the root manifest and resolving manifest entries. Leaf data manifests are identified for processing. Since column update references are embedded directly in data file entries in the manifest, no separate metadata is needed in this phase.

2. File Pruning with Inline Stats

For each leaf manifest, the planner evaluates entries using their content_stats directly because they were  updated at write time to reflect merged statistics from both the base file and all column updates. Entries that cannot match the query predicate are pruned. For entries that survive pruning, the column_files list is already present and fully resolved, no cross-manifest join or additional scanning is required.

For each surviving entry, a FileScanTask is created containing a DataFile object. This object encapsulates the base file path and the column_files list directly from the entry. When a column_files entry exists with the field id of a projected field, it reads from the column file, otherwise from the base file. Since each field_id appears in at most one column file, no conflict resolution is needed.

1.5.1 Equality Delete handling

Writers must reconcile equality deletes by materializing them as DVs at the column update's sequence number. This makes the original equality delete obsolete and avoids the read-time interleaving..

1.6 Parquet reader changes (in Iceberg)

The FileScanTask will be enhanced with a columnMapping field to map field IDs to files. A new ColumnStitchingReader will efficiently combine columns in a batch.^[k]

1.6.1 Java Pseudo-code

1.7 Compaction 

RewriteDataFiles operation merges all associated column files into the newly rewritten base file. This "collapses" the updates into a single physical file.

1.8 Change Detection for Column Updates

For change detection, see Appendix

1.9 Advantages

• This method maintains the complexity of the column files and retains all necessary planning metadata within Iceberg.
• The planning overhead remains minimal because the copy-on-write manifests allow for the new column files to be updated and the inline column statistics for those updated columns to be written directly.

1.10 Disadvantages

• The approach requires a full metadata rewrite via a copy-on-write (COW) strategy. However, because this targets a scenario where every base data file has sister files, the cost of rewriting manifests is low relative to the operation's total expense.

2. Interfaces

2.1 SQL

• Triggering a column update can be as simple as running the following SQL:
  `UPDATE tbl SET col_a = func_a(), col_b = func_b();`
• Basically, an UPDATE statement without any WHERE^[l] filter can trigger writing column update files.
• However, we might want to respect the user's choice of `write.update.mode`. E.g. For `copy-on-write` the users expect that writes are more expensive, however, there is no overhead for reads. With the current proposal, the reads do have some performance penalty as the result rows have to be stitched together from various files.
• To give more flexibility we can introduce a 3rd value for `write.update.mode` to enable column updates (regardless of full or partial writes).

2.2 Java API

Open Questions

1. Should we do all the work in Iceberg metadata layer or parquet layer (Approach 1 vs Approach 2)

1. We have decided to do it on the Iceberg side. See Efficient Column Updates in Iceberg discarded options tab for details.

2. Equality delete handling (Equality deletes will not be supported when column updates exist)

1. Equality deletes make the reader logic complicated as the column updates change the state of a row while equality deletes depend on the previous state. Should we disallow column updates on columns that have an equality delete?
2. With the MoR approach, if the new manifests written by the column update operation get a new sequence number then technically we undelete rows if there are equality deletes.

3. Stacked updates (Newer update will carry over the older update into a single column file per field id)

1. For ADD COLUMN, should we allow updating the newly added column^[m]
2. For UPDATE column, should we allow subsequent updates?

1. We will always carry over the updates to the new update file and rewrite the column_files fields of older entries to remove the filed_id that was updated.

4. Future work on column families. (The current design does not prevent us from using column families in the future)

1. Supporting column families is an extension to this proposal where we could have rigid requirements for what columns should constitute a column file. The current design does not prevent us from using column families in the future.

5. Should this design support partial column updates. i.e column updates that impact a subset of rows?

1. Row level MoR style partial updates
2. Can we have file-level partial updates (i.e all rows in the file are updated)

6. What should be the row alignment strategy for column update files?

1. Dense Column Update Files

This strategy involves creating column update files with a row count and row group structure identical to the base file. The primary benefit is highly efficient reads, as a scan task for a base file's row group directly corresponds to the same row group in the column file, both sharing the same number of rows. A potential drawback is that the differing data profiles between the base file and the column files may lead to a suboptimal physical layout for the column update files. Additionally, the column stats need some adjustment with this approach.

1. POC done on fully aligned column files.

1. Empty spaces filled with nulls
2. One challenge is the trailing null values cannot be detected by just reading the column files. We need a way to pass the (file_path, num_rows) to the readers.
3. Parquet allows null values in fields

                  Dense representation has two options

1. Option A: Positional matching with dummy values                                                                                                        Write filler values at deleted positions so the column file has exactly base_file.record_count rows. Possible fill strategies:                          

• NULLs: Set each field to NULL at deleted positions. Parquet supports per-field nulls. Simple for writers but skews null_count statistics.
• Same value as base file: Carry over whatever value existed in the base file for that row. Preserves stats but adds read-before-write cost.            
• Last valid value (backfill): Similar to Lance's DeletionRestorer — fill with the nearest valid value to keep column stats tight. Adds write-path buffering logic.                                                                                                                                                                                                                                                

Pros:                                                                                                                                                  

• Stitching with base file is straightforward (pure positional)                                                                                        
• Position column (_pos) may not be needed in the update file at all  

                          Cons:                                                                                                                                                  

• Writing values that will be discarded at read time
• Filler values may skew stats (num_nulls, avg_length, value_count)  
• Some filler values could make expressions invalid (e.g., division by zero)
• Extra write-path complexity to determine fill values

   2.   Option B: Don't write deleted rows

Column files contain only non-deleted rows (rows that are live after applying DVs). The file has fewer rows than the base file.                                                                                                                                                               

          Pros:                                                                                                                                                  

•   Everything in the update file is valid data      
•   No skewed stats                                                                                                                                      
•   No extra write-path complexity for computing filler values                                                                                                                               

          Cons:                                                                                                                                                  

• Position column (_pos) is required in the update file to map rows back to base file positions
• Stitching is more complex, especially for vectorized readers
• Stats

Alignment in non-java engines

• The two representations have asymmetric read-side cost across language ecosystems. Spark's ColumnVector is an abstract class with virtual dispatch, so a custom stitching subclass can redirect per-row reads (getInt(rowId) → choose base vs column file) with no memory copy. Concrete Arrow array types in arrow-rs, PyArrow, and arrow-go do not support this yet.
• See https://lists.apache.org/thread/jbh1gbrso5h6l4by9rh9poy2cjjtb8j0   

                 b.   Sparse Column Update (Dense Representation chosen)

In this strategy, column files only contain rows from the updates column values, they do not contain updates from deleted rows or rows that did not match the update condition. For this to work, the writer needs to materialize the row position (_pos) in the column update file to map the value back to the original row. The drawback is that reader implementation is significantly more complex. It must perform a join on the _pos column to stitch rows and must also handle scan splits that do not align with the column file's row group boundaries.

1. POC in progress for spare representation
2. Change detection is potentially easier with sparse representation.

7. Efficient CDC - See Appendix for the design

1. Iceberg Single File Commitsdescribes the algorithm for change detection at manifest level. A new REPLACED is introduced which enables the ability to easily distinguish between pure DELETEs of entries and cases where there’s a change and the original entry should still be considered live. This REPLACED state helps with filtering manifests and data files that are non-needed for the CDC reads.
2. For row level change detection, the sparse representation already has _pos filed (which is similar to a DV). For CDC, the reader first finds the replaced manifest using replaced bitmap and the for entries that are REPLACED in the manifest finds the column_files.

8. Compaction and small files problem

1. Instead of one column update file per base file which could cause several tiny column files in small columns, should we pack multiple batches of updates into column files, essentially decoupling the base file and column file mapping and instead have larger column files with row ranges to read from the same column file?
   Example:

1. Initially, we have base_file_1.parquet (rows 1-1000) and base_file_2.parquet (rows 1001-2000). If we update the "score" column across both files and pack those updates into a single larger file, packed_col_A.parquet

 {

    "data_file_path": "base_file_1.parquet",

    "column_updates": [

      {

        "field_id": 12,

        "update_file_path": "packed_col_A.parquet",

        "row_range": "0-1000"

      }

    ]

  },

  {

    "data_file_path": "base_file_2.parquet",

    "column_updates": [

     {

        "field_id": 12,

        "update_file_path": "packed_col_A.parquet",

        "row_range": "1001-2000"

    }

    ]

  }

Appendix:

Column Update Mechanisms in Other Data Lake Systems

Lance

Lance implements column-level data evolution through a fragment-based architecture where each Fragment contains multiple DataFiles, with each DataFile storing a distinct subset of columns identified by field IDs. When new columns are added via add_columns() or merge(), Lance writes only the new column data to a separate DataFile within the same Fragment. The DataFile.fields array in the manifest tracks which columns reside in which file.

To maintain alignment when columns are added to Fragments with deletions, the DeletionRestorer inserts placeholder values at deleted row positions, ensuring new DataFiles have the same physical row count as existing ones. This design means write time involves computing new column values and writing a new DataFile with updated Fragment metadata, while read time involves opening multiple DataFile readers and positionally merging their batches.

Apache Paimon

Apache Paimon implements column-level updates for append tables through its Data Evolution feature, which uses row ID tracking to enable positional stitching of columns stored in separate files. Each DataFileMeta contains a firstRowId field and a writeCols list specifying which columns the file contains—the Snapshot.nextRowId field tracks the global next available row ID across commits.

Statistics are stored per-file in SimpleStats and when multiple files cover the same row range, DataEvolutionFileStoreScan.evolutionStats() builds composite statistics using DataEvolutionRow and DataEvolutionArray structures that map each output field to its source file and position via rowOffsets[] and fieldOffsets[] arrays—this enables pruning predicates to be evaluated against merged stats from multiple files.

At read time, DataEvolutionSplitRead uses RangeHelper.mergeOverlappingRanges() to group files by overlapping firstRowId ranges, then creates a DataEvolutionFileReader that wraps multiple inner readers; DataEvolutionIterator.next() synchronously advances all readers and populates a DataEvolutionRow that redirects field accesses to the appropriate source row. Alignment is guaranteed by validating that all files in a merge group have identical rowCount and firstRowId values. This means write time involves writing only the updated columns to new files with the target firstRowId range (files are small, containing only delta columns), while read time performs lightweight positional stitching without any key-based joining.

Apache Hudi

Apache Hudi's RFC-80 proposes Column Groups, a mechanism to horizontally partition tables where each column group maintains independent base files and log files that can be written and compacted separately. Column groups are defined at table creation time, with each group containing a subset of columns plus the primary key for joining. File naming includes a _cfName suffix to distinguish groups. At write time, data is sorted by primary key then split by column group, with each group written to its own log files independently—updating one column group doesn't touch others. Statistics are maintained per-column-group file, enabling predicate pruning at the group level.

At read time, a RowReader coordinates multiple GroupReader instances (one per column group), each performing sort-merge between its base and log files; the RowReader then performs a sort-merge join on primary key across groups to reconstruct complete rows. Compaction can operate per-column-group (reducing write amplification) or as "full compaction" that merges all groups. Unlike positional stitching approaches, RFC-80 requires primary-key-based joining and upfront column group definition, but leverages sorted data for efficient merging without explicit row IDs.

References

• Iceberg dev list discussion on wide-tables in Apache Iceberg.
• Multifile Parquet support dev list discussion in Apache Arrow.
• Writing very large rowgroups in Parquet dev list discussion in Apache Arrow.

Appendix: Discarded Options

Discarded options

Approach 2: Self-Describing Column Update Files in Parquet

The decision to discard the Parquet-native approach came down to these main arguments:

• The major advantage of handling column updates in the Iceberg layer is that our manifests will always have a complete, self-contained view of all files, including the merged column stats from both base and update files. This is critical for efficient file pruning. In the Parquet approach, Iceberg would only store a reference to the latest logical file, requiring a more complex planning phase to discover the full set of stats needed for pruning.
• The Parquet approach makes it difficult to track file lineage. Because the relationship between a base file and its update file is hidden inside a Parquet footer, it becomes very tricky to determine which physical files belong to the table. This complicates operations like removing orphan files, especially with stacked updates to the same column, and would likely require complex naming conventions to manage.
• The Parquet-native approach would require changes to the Parquet format and its readers. This would involve collaborating with the Parquet community to align on a common goal and tying this feature to their release cycle.
• It would be unfortunate to have such a useful feature tied only to Parquet. By building the column update logic into the table format itself, we create a design that can extend to other formats like ORC in the future.

This approach leverages Parquet's native support for vertical partitioning to enable column-level updates without requiring table format changes.

2.1 Core Idea

Parquet includes native support for vertical partitioning through the `ColumnChunk.file_path` field:

From parquet.thrift

2.2 Writes

A column update operation will write column update files that are self-describing: they physically contain only the updated columns but their footer metadata describes the complete row structure including references to unchanged columns in external files. The latest version file serves as both data and metadata. The table format (like Iceberg) simply references the update file path in its normal DataFile entry—no metadata changes are required.

2.2.1 Column File Creation Example

2.2.2 New column writer - Java pseduo-code

2.3. Reads

Reading self-describing column update files extends Parquet's standard reader to discover and handle columns stored in external files. The reader examines the footer to identify column locations and transparently stitches data from multiple files.

2.3.1 Metadata Resolution

The read process begins by reading the footer from the latest column file to discover column locations. The footer contains complete metadata for all columns in the table schema. Once the footer is loaded, the reader scans all ColumnChunk.file_path fields across row groups to identify which external files are referenced. For each unique external file path discovered, the reader resolves the relative path and prepares to open the file for reading. This discovery process is transparent—files without column updates have all file_path fields set to null, so they work exactly as before without any external file references.

2.3.2 File Pruning

The statistics in the footer for update files reflect the current state of the

  updated columns only. For predicates involving columns stored in external files,

  the reader performs targeted footer reads:

Column predicates that are local are instantly evaluated by leveraging statistics from the primary footer.

External column predicates are evaluated by checking the external file that contains the column (using the file_path).

2.3.3 Column File Resolution

After identifying row groups that survive pruning, the reader builds a column-to-file mapping. For each projected column:

• If file_path = null: read from the current file at the specified offset
• If file_path = "path/to/base.parquet": read from the external file

The reader groups columns by source file to minimize I/O operations, then reads from each file and assembles the pages into a unified PageReadStore. Columns with file_path = null are read from the current file being processed, while columns with external file paths are read from their referenced locations. This grouping strategy ensures that the reader opens each external file only once per row group, caching streams for efficient access across multiple columns from the same source.

For optimal performance on cloud storage, the reader can open external files in parallel once all paths are discovered from the primary footer.

2.3.4 Row Stitching

Since column files are row-aligned with their source files, the reader performs positional stitching. Row N from each column source corresponds to row N in the logical table. The existing RecordReader assembles complete rows by reading values from each column's PageReader without modification.

2.3.5 Reader API changes: Java pseduo-code

Modified ParquetFileReader

Add Multi-File Support to the existing reader class.

Add new multiRowGroupReader()

2.4 Advantages

• This approach delegates the intricacy of managing column files and row stitching to the Parquet level. Iceberg will then record the most recent column file version as a DATA entry within its metadata, requiring no alterations to the existing specification.
• This approach has broader applicability, especially with emerging AI workloads. For instance, the same writer and stitching reader mechanism could be used to split large multi-modal columns across multiple files.
• This capability is table format agnostic and can therefore be extended to include other formats such as Delta Lake and Hudi.

2.5 Disadvantages

• A table format's lack of visibility into the physical files comprising a table makes table management challenging. Tasks like removing orphaned files become difficult unless all versions of the physical file are tracked in the metadata or a specific naming scheme is employed to identify table membership.
• This only applies to Parquet; support for column updates in other file formats, such as ORC, will require separate changes in the respective formats.

Approach 3: Additive column manifests for column updates

This approach was discarded due to the additional planning overhead to calculate the column and file mapping and the need to use a stats override file for every column update.

In this approach, column update files are written as new data files to separate manifests, leaving the base manifests untouched. These column files are structured as standard DATA entries. They include a referenced_data_file pointer to their base file and a list of the columns they contain. This design is focused on minimizing write amplification and is consistent with the V4 single-file-commit principle, where base manifests are immutable, and the final column-to-file mapping is determined during planning.

V4 specification changes for column files

• Extension to the ContentEntry Schema: Add a field `field_ids` to ContentEntry map to root manifest

• Use the existing referenced_file field to refer to the base file for this column file.

Column update representation

3.1 Stale column stats

When new column files are written in an additive fashion, the stats for the updated columns become stale in the base file. To overcome this, every column-level update will also create a corresponding stats override entry which is a new type of entry that contains the stats overrides for the updated columns.

3. 2 Commits

A new commit starts by reading the previous snapshot's root manifest. For column updates, a new root manifest is created with a new leaf data manifest containing the added data entries for the column files along with a stats override entry. The entries in the column update leaf manifest will all have references to a single data file they apply to and a list of field ids this column file contains.

3.2.1 Diagram: Adding or updating columns

3.3 Planning

Query planning with column updates extends Iceberg's existing two-phase planning to a three-phase model.

1. 1. Manifest Entry Resolution (Existing)

Planning starts by reading the root manifest and resolving manifest entries. When a Stats Override (SO) entry is encountered, the planner loads its content_stats and associates them with the target manifest specified in manifest_location. These override stats are held in memory as a map from manifest path to replacement content_stats.

2. File Pruning with Merged Stats (Modified)

For each leaf data manifest, the planner checks if a Stats Override exists for that manifest. If present, the override's content_stats is merged with the manifest's content stats—override values replace base values for matching field IDs, while non-overridden fields retain their original stats. The InclusiveMetricsEvaluator then evaluates the query predicate against these merged stats to determine if the manifest can be pruned entirely. For manifests that pass pruning, individual file entries are evaluated using the same merged stats approach.

3. Column File Resolution (New)

After identifying data files that survive pruning, the planner resolves which column files apply to each base file. For each base file entry, the planner scans the column update manifests for entries where referenced_data_file matches the base file path. Entries are sorted by sequence_number, and for each projected field, the entry with the highest sequence number wins. The result is a mapping of each projected field ID to its source file (either the base file or a column update file). This mapping is attached to the FileScanTask for use during execution.

3.3.1 Diagram:

3.4 Parquet reader changes (in Iceberg)

The FileScanTask will be enhanced with a columnMapping field that contains a mapping from the field_id to the file.

A new ColumnStichingReader will stitch the columns in a batch.The FileScanTask will be improved to include a columnMapping field, which will map the field ID to the corresponding file.

Additionally, a new ColumnStitchingReader will be introduced to efficiently combine columns within a single batch.

Change Detection Scenarios with Column Updates

Change Detection Scenarios with Column Updates

Design Summary

Change detection for column updates follows the same V4 Single File Commits mechanism used for DVs and other manifest mutations. No additional CDC-specific design is needed.                                                                                                              

File-level change detection: The general file_changes() algorithm from Single File Commits identifies REPLACED manifest entries, matches them by  data_file_path to their EXISTING/ADDED counterparts, and diffs the column_files lists to produce column_additions and column_removals. This works identically regardless of whether column files are dense or sparse.

Row-level change detection: Uses the existing _last_updated_sequence_number metadata column (same mechanism as row lineage). The reader infrastructure already supports two modes:

• File-level constant: If the column file does not physically contain _last_updated_sequence_number, the reader uses the file's fileSequenceNumber for all rows (sufficient for simple cases where every row in the file is a new change).                                                                    
• Per-row values: If the column file physically materializes _last_updated_sequence_number as a data column, the reader uses per-row values. This is required during carry-over/merge, where some rows are new changes and others are carried over from a previous update.                                  

  Writer responsibility during carry-over:

•   Rows carried over from a previous column file retain their original _last_updated_sequence_number                                                    
•   Newly updated rows get the current snapshot's sequence number^[p]                                                                                         
•   The writer materializes this as a physical column in the merged file                                                                                  

CDC query: SELECT * FROM table WHERE _last_updated_sequence_number > <checkpoint_seq> returns only rows that were actually changed since the checkpoint excluding carried-over rows.                                                                                                                          

Why this works for both representations:                                                                                                                

• Sparse: In the simple case (no carry-over), every row present in the file is a change at that sequence number — file-level constant is sufficient.
• After merge-on-write carry-over, per-row _last_updated_sequence_number distinguishes new changes from carried-over positions.                          
• Dense: Every row is present in the file, but not all rows are changed. Per-row _last_updated_sequence_number is always required to distinguish real updates from filler/carried-over values.                        

Details

Column updates use the REPLACED status from the V4 Single File Commits design to enable change detection. When a column update modifies the column_files for a base file, the old manifest is marked REPLACED and a new manifest is written with updated entries containing the new column_files list.

A column update is distinguished from a row-level overwrite by the presence of a REPLACED manifest paired with an ADDED manifest where entries share the same data_file_path. In a regular overwrite, old and new entries reference different physical files.

Representation A: Full Row-Aligned Column Files

In this representation, every column file has the same row count as the base file. Row N in the column file corresponds to row N in the base file.

A.1: Full Column Update (all rows) (Add column, update full column)

A table has base file D1 with 1000 rows. A column update rewrites field 3 (salary) for all rows via column file CF1, which also has 1000 rows.

Change Detection algorithm:

1.  M1 is REPLACED, M1' is ADDED
2.  Compute the delta:

1. delta_column_files = M1'.column_files − M1.column_files = [CF1]

3. CF1 has record_count == base file record_count (1000). Every row is an UPDATE.
4. For each row position 0..999:

1. UPDATE_BEFORE: field 3 read from D1 (base file)
2. UPDATE_AFTER: field 3 read from CF1

Successive updates on different fields:

If a second update adds field 4 via CF2, the delta isolates the change:

1.   M1'.column_files  = [CF1(fields:[3])]
2.   M1''.column_files = [CF1(fields:[3]), CF2(fields:[4])]
3.   delta = [CF2]
4.   Only field 4 produces change events.

Successive updates on the same field:

If field 3 is updated again, the old CF1 is replaced by a new CF1' (carry-over semantics). The delta is:

1. M1'.column_files  = [CF1(fields:[3])]
2. M1''.column_files = [CF1'(fields:[3])]
3. delta = [CF1']   (CF1 was replaced, not carried forward)
4. Every row emits an UPDATE. The UPDATE_BEFORE reads field 3 from the old CF1 (via the REPLACED manifest entry), UPDATE_AFTER reads from CF1'.

Interaction with row deletes:

If a DV is also present, deleted positions take precedence:

1. deleted_positions = dv_diff
2. updated_positions = {0..999} − deleted_positions

A.2: Partial Row Update (subset of rows)

Only rows at positions {1, 4, 7} need updated values. However, in the full row-aligned representation, the column file still has 1000 rows and filler values occupy the unchanged positions.

Problem for change detection: The CDC reader sees 1000 values in CF1, all positionally aligned with D1. It has no way to distinguish which values are real updates vs filler values for unchanged rows. The reader would either:

1. Emit all 1000 rows as UPDATEs (incorrect — 997 rows didn't actually change)
2. Compare old vs new values row-by-row to find differences (expensive and unreliable if the update expression produces the same value for some rows)

Possible mitigation:

• Add a companion bitmap or _pos list to the column file to mark real updates — but this negates the simplicity advantage of full row-alignment

Representation B: Sparse Column Files

In this representation, the column file contains only the rows that were updated, identified by their _pos (ordinal position in the base file).

B.1: Full Column Update (all rows)

A table has base file D1 with 1000 rows. A column update rewrites field 3 (salary) for all rows via sparse column file CF1, which contains 1000 rows with  _pos = {0, 1, 2, ..., 999}.

Change Detection algorithm:

1. M1 is REPLACED, M1' is ADDED
2. Compute the delta:
3. delta_column_files = M1'.column_files − M1.column_files = [CF1]
4. CF1 contains _pos values {0, 1, ..., 999} — all positions are present, so every row is an UPDATE.
5. For each _pos:

1. UPDATE_BEFORE: field 3 read from D1 (base file)
2. UPDATE_AFTER: field 3 read from CF1

For the full update case, the _pos column is redundant (it's a contiguous range covering all rows), but the format is consistent with the partial case.

Successive updates on different fields: Same delta logic as A.1 — only new column files produce change events.

Successive updates on the same field:

Same carry-over semantics as A.1 — old CF1 replaced by CF1', delta = [CF1']. UPDATE_BEFORE reads from old CF1, UPDATE_AFTER from CF1'.

Interaction with row deletes:

• deleted_positions = dv_diff
• updated_positions = CF1._pos values − deleted_positions

B.2: Partial Row Update (subset of rows)

Only rows at positions {1, 4, 7} are updated. The sparse column file CF1 contains exactly 3 rows.

Change Detection algorithm:

1. M1 is REPLACED, M1' is ADDED
2. Compute the delta:

1. delta_column_files = [CF1(fields:[3])]
2. CF1 contains _pos values {1, 4, 7}. Only these 3 rows are UPDATEs.

3. Affected_positions = {1, 4, 7}
4. For each affected position:

1. UPDATE_BEFORE: read row at _pos from D1, field 3 from base file
2. UPDATE_AFTER: read row at _pos from D1, field 3 from CF1

5. Rows not in affected_positions are not emitted.

Interaction with row deletes:

•   deleted_positions = dv_diff
•   updated_positions = {1, 4, 7} − deleted_positions

How to support _last_updated_sequence_number and _row_id

Here is the doc with a detailed summary on this.

Requirements for writers:

Assumption 1: We concluded on having “complete” representation for partial updates. This means that we write all the fields of the updated column(s) into the column update file, both the changed and the unchanged values.

Assumption 2: When committing column update into table metadata, we clone the affected manifests while adding column update information to the cloned manifests.

Requirements:

1. Write _last_updated_sequence_number column into the update file

1. For unchanged values write the previous value of _last_updated_sequence_number
2. For changed values write null

2. In column_files.field_ids keep track the ID for _last_updated_sequence_number

1. Reason: The latest _last_updated_sequence_number is tracked by the latest column update file. If storing its ID in the field_ids we don’t need to keep track of the age of the update files to find the latest one.
2. De-duplicte the field IDs: when adding the field ID of _last_updated_sequence_number to the field_ids of the latest column file, we have to remove the same ID from the field_ids of the column file that covered it before.

3. Sequence numbers for TrackedFile when cloning the affected manifests

1. data_sequence_number should stay the same
2. file_sequence_number should be updated to the current snapshot

How to read:

Steps:

1. Do the regular planning to identify the base files affected by the scan
2. Check what field IDs are queried and make a grouping to see which physical file covers what subset of the field IDs

1. Until we find all the required field IDs, go through the filtered base files and look into column_files and check the field_ids member to see what field IDs are covered by that column file
2. _last_updated_sequence_number is no different than any other field from this aspect: we have to find which column file lists its field ID in the field_ids list

1. There should be a single update file for this
2. See de-duplication step on the write side

3. Create a grouping of what file is responsible of what field IDs, or in other words identify the column splits
4. Do the reading and stitching process for the different column splits

1. When doing the grouping of which physical file is responsible to read which queried field ID, handle _last_updated_sequence_number as if was any other regular field ID

1. We have to find which column file lists _last_updated_sequence_number’s field ID in the field_ids list
2. There should be a single update file for this
3. See de-duplication step on the write side

2. The reader dedicated for _last_updated_sequence_number should do the following:

1. Read the dedicated (column update) file’s _last_updated_sequence_number column

1. If there is a value for the row, then use that value
2. If there is a null for the row, then use the file_sequence_number of the base file (TrackedFile)

Implications

• It’s not needed to store sequence_number on the column_file level

• No need to find the latest update file to see which one covers _last_updated_sequence_number
• To get _last_updated_sequence_number when update file contains null, we can use the file_sequence_number from the base file

• Field ID for _last_updated_sequence_number has to be part of the field_ids list for one of the column_files

Column File Representation

Column File Representation

Dense representation

Option 1: Positional matching with null values

The column file is written with exactly base_file.record_count rows. Row N in the column file corresponds to row N in the base file. For positions that have been deleted in the base file's DV, the writer emits a filler value (NULL by default; alternative strategies discussed separately in Section 1.3.3 of the main document). For unchanged positions, the writer carries over the existing value.

At read time, the reader projects the updated field from the column file in place of the base file. Because both files have identical row counts and identical row ordering, the stitched batch is assembled by reference substitution: the column file's column array replaces the base file's column array for the updated field. The DV is then applied to the stitched batch, discarding the deleted positions (including the filler values)

Pros

• Read-path stitching is simplified across all engine implementations (Java, Rust, Python, Go). Since row counts are identical between the base and column files, stitching is reduced to a reference substitution of the column array within the batch. This constant-time operation avoids array materialization and scatter-style APIs.
• Ordinal positions (_pos) are not required for read-path resolution. Position information is implicit via row alignment; while _pos may be retained as advisory metadata for debugging or future optimizations, it is not consulted during standard stitching operations.
• Performance on the read path is predictable. Execution cost is dominated by Parquet decoding and DV application rather than stitching logic.
• The hot path remains efficient by eliminating per-row branching and lookups.
• Incremental storage overhead is bounded. Empirical data shows that NULL fillers at deleted positions incur only 2–3% overhead at a 5% deletion ratio, rising to 4–4.6% at 20% deletions. Including the _pos column using Parquet V2 delta encoding adds less than 0.5%.

Cons

• The writer maintains strict alignment by capturing the base_file.record_count to manage trailing deletions. If rows at the tail of the base file are removed, the writer must emit filler values up to the full record count; otherwise, row count divergence between the base and column files causes silent stitching failures. This record count is retrieved from the manifest's content_stats during planning and broadcast to write tasks as a file_path → record_count map, a pattern validated in the Delta Lake PoC.
• Filler values can introduce skew in Parquet footer statistics. Specifically, NULL fillers inflate the null_value_count, while carry-over values may contaminate lower_bound and upper_bound metrics.
• Correcting null fillers stats is straightforward. Subtracting the delete_count from both null_value_count and value_count—but alternative strategies require more careful handling. Detailed discussion of filler-strategy selection and statistical correction logic is provided in section 1.3.3.
• Incremental storage overhead is incurred for rows eventually discarded at read time. While filler values at deleted positions represent wasted write work, empirical evidence suggests this cost is bounded by the deletion ratio.
• Filler-value selection may introduce unexpected behavior during expression evaluation. For instance, using NULL fillers for a field defined as NOT NULL in the schema requires Parquet schema relaxation. Additionally, some evaluation paths might propagate these fillers before the DV is applied. The primary recommendation (NULL filler) and its associated handling.

Summary of stats pollution concerns

To achieve efficient vectorized reading and avoid joins, readers will insert null values during the read process for positions that are missing in the sparse column files. This approach allows for the efficient stitching of batches from the base file and the sparse column file based on their positions.

Example:

Option 2: (Discarded) Applied Deletes

The column file contains only the rows that are live after applying the base file's DV. A _pos column is required, with one value per row of the column file, identifying that row's ordinal position in the base file. The column file's row count equals base_file.record_count − dv_cardinality.

At read time, the column file contains values for every row that is live after the base file's DV is applied. The reader reads the base file's full batch (including positions deleted by the DV), populates the updated column from the column file using _pos to place each value at its corresponding base position, then applies the DV as a final filter to drop the deleted positions from the output. This requires a scatter operation: for each row position in the base batch, look up whether the column file has a matching _pos and emit the corresponding column file value at that position; positions without a matching _pos correspond to rows that will be filtered out by the DV downstream and can hold any value in the stitched batch. The base file's value for the updated column is never used at read time.

Pros

•  Parquet footer statistics are accurate for live data. No skew, no correction algorithm needed.
•  Smaller files at high deletion ratios.
•  No writer-side record_count broadcast needed. Trailing deletions are implicit.

Cons

• Read-path stitching requires scatter, which is materially harder in non-Java implementations. Vectorized readers (Spark and most current Iceberg readers) populate the updated column at sparse _pos positions in a pre-DV-filter batch. Scatter is essentially free in Java/Spark via virtual dispatch on ColumnVector but requires array materialization in other ecosystems. Detailed per-ecosystem analysis below.
• _pos column is required, not optional. Adds storage cost and an upfront _pos walk on the read path to build the stitching index.

Ecosystem Implementation For  Alignment

Spark / Java

Spark manages in-memory columnar batches via the abstract ColumnVector class, granting the Iceberg reader full control over the path from Parquet decoding to batch population. This architecture enables efficient scatter-style stitching by subclassing ColumnVector and overriding type-specific accessors (e.g., getInt(rowId)). Read operations are redirected to either the base file or column file using a precomputed basePosToCfRow lookup array.

Implementation Example

class StitchingVector extends ColumnVector {

    ColumnVector cfVec; // CF1 column

    int[] basePosToCfRow; // base _pos → CF1 row index, or -1

    ColumnVector baseVec; // D1 column (for unchanged rows)

    @Override public int getInt(int rowId) {

        int cfRow = basePosToCfRow[rowId];

        return cfRow >= 0 ? cfVec.getInt(cfRow) : baseVec.getInt(rowId);

     }

}



Iceberg-rust (arrow-rs)

The iceberg-rust implementation relies on concrete arrow-rs array types where value access compiles to direct memory loads. Unlike the Spark ecosystem, there is no equivalent to the polymorphic dispatch found in ColumnVector; because array types are implemented as structs rather than interfaces, value reads cannot be intercepted via subclassing.

Consequently, generating a scattered column requires allocating a new array through ArrayBuilder and emitting values row-by-row. This results in O(rows) complexity due to materialization, per-row copies, and the absence of SIMD acceleration. While arrow-rs supports gather operations (take), it lacks native scatter primitives suitable for this resolution logic.

Additional overhead occurs at row group boundaries. Since the ParquetRecordBatchReader aligns batches to row groups, any sizing divergence between base and column files necessitates a concat-refill operation. A performant resolution would require an upstream API to force batch alignment without additional memory copies.

PyIceberg (pyArrow)

PyIceberg utilizes native PyArrow array types. Producing a scattered column involves allocating a new array sized to the base batch where column file values are placed at specific _pos coordinates, with remaining slots defaulting to NULL. Although the pc.scatter primitive (introduced in Arrow 23.0.0) executes in C++, the O(rows) cost of allocation and per-position writes remains unavoidable.

iceberg-go

The iceberg-go implementation builds upon arrow-go compute kernels and array.RecordReader. Similar to the Rust implementation, value access is performed through concrete array types without polymorphic dispatch, precluding the use of the ColumnVector override pattern.

To resolve scattered columns, the reader must utilize ArrayBuilder to materialize a new array row-by-row. While arrow-go provides Take (gather) and FilterRecordBatch in its selection kernels, it currently lacks a native Scatter primitive.

(Discarded) Sparse Representation

For partial updates, The sparse column file strategy (Option 2 in Section 1.3.3) stores only the changed rows identified by their ordinal position (_pos) from the base file. This enables partial row updates, a key advantage over the dense/fully-aligned approach where only a small percentage of rows are updated without write amplification for unchanged rows.

In the sparse representation, the presence or absence of a position in the column file carries semantic meaning:                                                                                                                                                    

• Position present in the file (with any value, including NULL): the row was updated. Use the value from the column file.
• Position absent from the file: the row was not updated. Use the value from the base file.

Three scenarios arise when sequential UPDATE operations produce sparse column files for the same base file:  

Scenario 1: No overlap: Different columns, any rows

UPDATE tbl SET col_1 = val_1, col_2 = val_2 WHERE id < 1000;                                                                                                                                                                                                       

UPDATE tbl SET col_3 = val_3, col_4 = val_4 WHERE id >= 1000; 



Column file A contains [_pos, col_1, col_2] for positions 0-999. Column file B contains [_pos, col_3, col_4] for positions 1000-1999. No field_id overlap.

 "column_files": [                                                                                                                                                                                                                                                   

    {"field_ids": [1, 2], "file_path": "col_file_A.parquet", "sequence_number": 3},                                                                                                                                                                                   

    {"field_ids": [3, 4], "file_path": "col_file_B.parquet", "sequence_number": 5}                                                                                                                                                                                   

  ]



Scenario 2: Overlapping field_ids, same or overlapping rows (update-on-update)

UPDATE tbl SET col_1 = val_a, col_2 = val_b WHERE id < 1000;                                                                                                                                                                                                       

UPDATE tbl SET col_1 = val_c, col_2 = val_d WHERE id < 500;



The second update targets col_1 and col_2 which already have a column file entry, and the rows overlap (id < 500 is a subset of id < 1000). Per Section 1.2.4, the writer carries over values from the previous column file and produces a merged file. Positions 0-499 get the new values (val_c, val_d), positions 500-999 retain the old values (val_a, val_b).

"column_files": [

  {

   "field_ids": [1, 2], "file_path": "col_1_2_merged.parquet", "sequence_number": 5

  }

]



Carry-over is straightforward because both files contain only col_2. The writer reads the old file, merges positions (newer value wins for conflicts), and writes the merged result.  

Scenario 3: Overlapping field_ids, different rows, multi-column files

UPDATE tbl SET col_1 = val_a, col_2 = val_b WHERE id < 1000;                                                                                                                                                                                                                                                                                                                                                                                                           

UPDATE tbl SET col_2 = val_c, col_3 = val_d WHERE id >= 1000 AND id < 2000;                                                                                                                                                                                           



The second update overlaps on col_2 but targets entirely different rows. Per the no-overlap invariant, the writer must carry over col_2 from column file A into a new file and remove col_2 from the old entry. However:                                                                                                                                                                                                                                                                                                    

•   Column file A's col_2 covers positions 0–999                                                                                                                                                                                                                      
•   Column file B's col_2 covers positions 1000–1999

If the writer attempts to merge col_2 into column file B (which also contains col_3), the merged file would have positions 0–1999 for col_2 but only positions 1000–1999 for col_3. Parquet requires uniform row counts across columns within a file positions 0–999 would need values for col_3. These cannot be NULL because the reader cannot distinguish "no update" from "intentionally set to NULL" when the position is present in the file. Filling with any sentinel value breaks the sparse representation's semantic contract.

Solution                                                                                                                                                                                                            

To solve this, when overlap is detected, extract the overlapping column into a separate single-column merged file containing the union of positions from both old and new updates. Non-overlapping fields remain in their original multi-column files:

After first update:

{"column_files":[

  {

    "field_ids":[1,2],"file_path":"col_file_A.parquet","sequence_number":3}

  }

  ]

} 

               

  After second update:

 "column_files": [                                                                                                                                                                                                                                                   

    {"field_ids": [1],    "file_path": "col_file_A.parquet",   "sequence_number": 3},

    {"field_ids": [2],    "file_path": "col_2_merged.parquet", "sequence_number": 5},                                                                                                                                                                                 

    {"field_ids": [3],    "file_path": "col_file_B.parquet",   "sequence_number": 5}                                                                                                                                                                                 

  ]     

 

What happened:                                                                                                                                                                                                                                                      

• col_2 was removed from column file A's field_ids ([1, 2] → [1])                                                                                                                                                                                                  
• col_2 was removed from column file B's field_ids ([2, 3] → [3])                                                                                                                                                                                                  
• A new merged file col_2_merged.parquet was created with only [_pos, col_2] for positions 0–1999 (union of both)

Read data files with column files

How to read data files with column files

Background

Currently, the read architecture is capable of reading individual files through the File Format API. With column (update) files, a logical data file may be represented as multiple physical file segments, where each field ID is stored in exactly one vertical split. Reading such a representation requires several additional steps beyond reading a single file:

• Projection splitting: Identify which field IDs from the scan projection are covered by which vertical split.
• Nested readers: Create and manage a dedicated reader for every vertical split involved in the read.
• Row alignment across readers: Each vertical split file may use a different internal layout (e.g. row group misalignment for Parquet files)^[q][r]. While reading by split offset may be sufficient for some splits (for example, the base file), it is not universally applicable. A row alignment algorithm is therefore required to ensure that corresponding rows from different vertical splits can be matched correctly.
• Stitching: Combine the results produced by the individual vertical split readers into a single row representation, making the data appear as if it originated from a single file reader.

Reusability: An important aspect of the implementation is to avoid implementing the same across format models, query engines or different flavours of the readers. The implementation should be reusable as much as possible.

Non-goals:

In V4 we propose to introduce column update files not just for data files but for manifest files too. However, This section focuses on reading data files with column files and not on manifests with column files..

Exposing column files

In the Java implementation, the plan is to expose column files through the DataFile API. See doc and PR. This helps when passing through file information to read or query plan APIs because DataFile already flows through these APIs.

 interface DataFile extends ContentFile<DataFile> {  

     default List<ColumnFile> columnFiles() { return List.of(); }

 }

 class TrackedFileAdapters {

     private static class TrackedDataFile extends ... {

         @Override

         public List<ColumnFile> columnFiles() { return file().columnFiles(); }

     }

 }

Current read implementation in Java

Architecture

(Diagram generated by AI)

FormatModelRegistry: From the user’s perspective this is the central entity for reading data files. After providing the FileFormat, the in-memory row representation (e.g. InternalRow.class, Record.class, etc.) and the particular InputFile to be read, the registry returns a ReadBuilder if there is one registered for the given format and row type.

public final class FormatModelRegistry {  

     public static <D, S> ReadBuilder<D, S> readBuilder(

         FileFormat format, Class<? Extends D> type, InputFile inputFile) { . . . }

 }

FormatModel: This is the abstraction that is responsible for converting between the physical representation of a file to the internal representation of rows. Once a format model is registered to the registry, users can use it via calling:

Readbuilder<Record, ?> b =

    FormatModelRegistry.readBuilder(FileFormat.PARQUET, Record.class, someInputFile);

Note, the example used Parquet as the physical format of the data file and Record as the internal representation.

When creating a format model these inputs are required:

• Class for the internal representation (output type of the reader, input type for the writer)
• Class of the schema type of the internal representation (StructType.class, RowType.class, etc.)
• Writer function
• Reader function

File decryption

An essential part of the read path is when and how we create an InputFile from a ContentFile. InputFile is the input of the FormatModelRegistry (and in turn to the ReadBuilders). If a ContentFile has a non null encryption key_metadata then the derived InputFile is in fact an EncryptedInputFile and contains this information for decryption.

The process of turning ContentFile into InputFile goes through EncryptingFileIO and an EncryptionManager:

public class EncryptingFileIO {  

     public Map<String, InputFile> bulkDecrypt(Iterable<? extends ContentFile<?>>);

     // bulkDecrypt() calls this and then passes the results to EncryptionManager in bulk

     private SimpleEncryptedInputFile wrap(ContentFile<?>);

 }

public interface EncryptionManager {  

     public static InputFile decrypt(EncryptedInputFile);

     // There is one API for bulk decryption too:

     public static Iterable<InputFile> decrypt(Iterable<EncryptedInputFile>);

 }

Here EncryptedInputFile is a wrapper on top of InputFile adding the encryption key metadata on top.

Spark and Flink use the bulk interface on the EncryptionManager API above. The reading process has the below steps:

1. Iterate ScanTasks associated with the reader and collect each referenced file (task().file(), task.deletes())
2. Pass the collected files to EncryptionFileIO.bulkDecrypt(). Within this:

1. Wrap/convert each collected ContentFile into an SimpleEncryptedInputFile
2. Pass them to EncryptionManager.decrypt() and receive an Iterable<InputFile>
3. Create and return a mapping from location to InputFile from the above

3. Cache the mapping and return it to the reader

When reading a ContentFile we follow these steps:

1. Get the InputFile from the above mapping using the location
2. Through FormatModelRegistry create a ReadBuilder using the InputFile

The steps described above are important to understand the architecture for reading a file in the context of column files. For instance a natural abstraction would be to pass a ContentFile to a Composite reader that can handle reading the base file and the column files together including the creation of different readers, splitting the rows together, handling projections, etc. However such a design would result in a one-by-one creation of InputFile from ContentFile that would break the contract of the bulk creation.

WiPSolution proposal

(Diagram generated by AI)

FormatModelRegistry^[s][t]: This class is intact. As described above, this serves as the entry point to the File Format API for reading a single physical data file.

ContentFileReadBuilder: An additional abstraction layer between engine readers and the File Format API. While the engine readers get ScanTasks and in turn ContentFiles as inputs, the File Format API gets an InputFile. Since ContentFile/DataFile contains info about column files, we either do the handling of them on each engine reader or as suggested here, we can introduce another layer below the engine readers to accept ContentFile as an input and do the column file handling. This layer can be reused by any engine reader.

Inputs for this layer:

• ContentFile: The file to be read
• Row representation type: Record, InternalRow, RowData, etc.
• Projection: the list of field IDs requested in the read
• InputFileProvider: A function from location String to InputFile

Responsibilities of this layer:

1.  If input ContentFile doesn’t have column files then return ReadBuilder from FormatModelRegistry as before
2. If input ContentFile has column files then:

1. Using ProjectionSplitter decides which vertical split part is responsible for reading which fields. Some vertical split parts may not be involved. Whatever field is not covered by column files will be assigned to the base file^[u][v].
   E.g. base file: [1, 2], column file 1: [3], column file 2 [4, _last_updated_sequence_number]
2. Using InputFileProvider convert each participating vertical split part (ContentFile.location()) into InputFiles.
3. For each InputFile, create a ReadBuilder using the FormatModelRegistry
4. Wrap the CloseableIterables returned by the ReadBuilders with a RowAlignedStitchingIterable and return that to the user.

(Diagram generated by AI)

ColumnFile is not a ContentFile:

• The current implementation works on ContentFiles. See EncryptingFileIO.bulkDecrypt() in the “File decryption” section.
• Technically for InputFile creation we need 1: location, 2: fileSizeInBytes, keyMetadata. We could have a version of bulkDecrypt() and wrap() in EncryptingFileIO to get these as parameters instead of ContentFile.
• However, for ColumnFiles we have all the information to convert them into ContentFiles. Doing that we can easily pass them through the existing APIs, also could be kept in the same list as the base file and delete files.
• Extend TrackedFileAdapters to adapt ColumnFile as ContentFile/DataFile

• Some fields might come from the base file (recordCount, specId, partition), so we either pass these fields to the adapter or pass the base file itself as below:

class TrackedFileAdapters {

     static DataFile asDataFile(TrackedFile base, ColumnFile file, Map<Integer, PartitionSpec> specsById);

 }

Behaviour of the ContentFileReadBuilder API:

• Most of the API calls are delegated to each vertical split readers, like reuseContainers(), caseSensitive(), etc.
• project(Schema): Field IDs here are given to ProjectionSplitter to determine the vertical split of the read

• filter(): The filter has to be projected to the subset of the field IDs for each of the vertical splits. These projected filters can be pushed down to the respective vertical split readers. The residual of the filter (the part that spans across multiple vertical splits) has to be applied to the stitched rows.
  Note, filter pushdown can mess up the positional alignment of the readers This alignment has to be fixed by the reader.^[w][x][y]
• split(): There is no enforced physical alignment across vertical splits, hence a particular split might make sense for a single reader only, most probably for the base file. At this point we can say that if split is provided, we apply it to the base file’s ReadBuilder.
  Later, we might want to figure out a mechanism to do the split by any of the readers, e.g. if one vertical split is big enough then we can use its split.
• build(): Returns a RowAlignedStitchingIterable

InputFileProvider: This works as a function that gets a String parameter for location and returns an InputFile. In practice when e.g. Spark’s BaseReader.inputFiles() constructs a mapping of locations to InputFiles, it can be used as an InputFileProvider.

Additionally, we can have an implementation of this that is reusable across engine readers in a way where the engine readers provide their ScanTasks as an input and InputFileProvider performs gathering the referenced files, executing bulk decryption, and constructs the location -> InputFile mapping. This way we can avoid InputFile creation (including decryption through EncryptionManager) in ContentFileReadBuilder one-by-one for the input ContentFiles and do them in bulk.

RowStitcher: Each format model has their own implementation on how to stitch vertically split rows together, to act as if it were a single row: InternalRowStitcher, RecordStitcher, etc. This is used by RowAlignedStitchingIterable to stitch the result rows of the vertical split readers together.

FormatModel: Potentially, we can keep the RowStitcher implementations on the FormatModel level. When creating a format model currently we receive the reader function and the writer function that is specific to the model. Similarly, we can receive a model specific stitcher.

The above would introduce some level of redundancy: A FormatModel is identified by file format + internal row representation. However, the stitcher is the same across file formats and only depends on the internal row representation. See (simplified the params to fit):

SparkFormatModels:

AvroFormatModel.create(InternalRow.class, StructType.class, SparkAvroWriter, SparkAvroReader, InternalRowStitcher);

ParquetFormatModel.create(InternalRow.class, StructType.class, SparkParquetWriter, SparkParquetReader, InternalRowStitcher);

OrcFormatModel.create(InternalRow.class, StructType.class, SparkOrcWriter, SparkOrcReader, InternalRowStitcher);

Alternatively, we can introduce a StitcherRegistry that is independent of the file format. Might^[z][aa] be an overkill, though.

Alternatively, we can pass a stitcher to ContentFileReadBuilder as a parameter. The users know the internal format already, should also know the stitcher implementation too. Might be redundant information together with the internal format parameter.

RowAlignedStitchingIterable^[ab]: Contains nested iterators, one for each vertical split participating in the read. The purpose of this compound iterator is to keep the underlying iterators in-line assuming positional alignment, and to apply the stitcher on top of the result rows.

Split reads need some extra care: split(start, length) makes sense to one reader only (probably to the base file’s reader ATM). We have to make the other iterables aligned with the one that jumps to the beginning of the split:

• Include the _pos column to all the readers^[ac][ad]
• Read the first row from the reader that was given the split (e.g. base file’s reader). This gives us the position of the first row in the split
• Iterate the other readers until they reach the position above
• Start getting the results and stitching them together until we reach the end of the reader that was given the split.

Later, this can be more advanced: Each column file has a split_offset field. Technically we can pick whichever we want to be the one that jumps to the beginning of the split and iterate the rest of the readers. This is out of scope ATM.

TODO: vectorized reads

Meeting Notes

Meeting Notes

Sep 8, 2026 

• (Anoop) How sequence number changes with column updates^{[ae][af][ag][ah]}
• (Anoop) Do we need the _pos field? Seems unnecessary now that column files are positionally aligned^{[ai][aj][ak][al]}

• > Concluded on removing the _pos field from the column file, writers are free to add it, but we won't add this to the spec, not the Java implementation

Topics (AI generated)

V4 Spec Clarifications

• position field in files:

• Decision: Remove the position field from the V4 spec.
• Rationale:

• Avoids ambiguity over the source of truth (writer vs. file).
• Prevents field ID conflicts between a data column and a metadata column.

• Note: Implementations can still generate position as a metadata column for internal use but this will not be specified.

• Sequence numbers & equality deletes:

• Sequence Numbers: Column updates will use the base file's data_sequence_number to track the latest row modifications. This is a natural extension of its role as the inherited sequence number.
• Equality Deletes: Equality deletes are not supported for column files.

• Requirement: Any existing equality deletes on a file must be converted to deletion vectors (DVs) before applying a column update.
• Rationale: Ensures consistent state by preventing a scenario where some rows are affected by an equality delete and others are not.

Column File Read Path Architecture

• API Exposure:

• Decision: Expose column files via the DataFile interface, not ContentFile.
• Rationale: ContentFile is an internal abstraction. The public API should expose the DataFile object, which represents a complete data file (base + columns).

• Architecture: Above vs. Below FormatModelRegistry:

• Decision: Build the column file read path above the FormatModelRegistry.
• Rationale:

• Keeps the FormatModelRegistry simple, preserving its role for reading single physical files.
• Centralizes complex, multi-file "stitching" logic in a new, reusable layer.
• Avoids complicating the registry's API with multi-file inputs and engine-specific stitching logic.

• Row Alignment & Vectorized Reads:

• Problem: Different file formats (e.g., Parquet, Avro) have different physical layouts and batching behaviors, making direct byte-offset seeking unreliable for alignment.
• Solution: Use row-offset based seeking to align readers.

• A "primary" reader (e.g., for the base file) uses the byte-offset split from the scan task.
• This reader determines its starting row position.
• Other readers (for column files) then seek to the same row position using file format-specific methods (e.g., Parquet row group metadata, Avro block headers).

• Vectorized Reads: This approach supports vectorized reads.

• Readers produce batches of varying sizes (e.g., Parquet's row-group-aligned batches).
• The stitching layer aligns these batches by row position and combines them into a single, consistent output batch.
• Over-reads from column files are acceptable and ignored if they exceed the row count of the primary reader's batch.

Next Steps

• Gábor:

• Share the read path design doc for offline review.
• Merge the column files metadata struct PR to unblock follow-up work.
• Prepare follow-up PRs:

• Expose column files via the TrackedFileAdapter.
• Update the V4 manifest reader to process column files.
• Define the table-level API for adding column files, requiring DVs for equality deletes.

• Anurag:

• Respond to Anoop on the dev list regarding the position field decision.

Aug 25, 2026 (Cancelled)

• (Anoop) Do we need the _pos field? Seems unnecessary now that column files are positionally aligned
• (Anoop) Can we have field IDs stored sorted within each descriptor, and a canonical order for the descriptor list itself (e.g. sorted by path): this would make duplicate file detection cheap

Aug 11, 2026(recording)

V4 Spec

• Folded column file structures into V4 spec PR (#16025)

Column Files Reviews

• Core: Basic fields and schemas for column files (#16285)

Action Items

• Update the column updates doc clarifying (Anurag)

• Update the metadata columns completely
• Clarify no row group aligning. (No file format specific constructs in metadata layer)

V4 Writers: Stats Schema Handling

• Problem: When rewriting a manifest, the stats file read schema is constructed from the full table schema (e.g., 5,000 fields), while the writer may have only produced stats for a subset (e.g., 100 fields). This leads to inefficient reads and potential data loss. 
• Solution: The stats file read schema will be constructed from the table's current metrics-config (from table properties).

• Why: This ensures only relevant stats are read and prevents carrying forward stale data from previous configs.
• Impact: This is a change from v3, which blindly copied the stats blob. The new approach enables cleaning up manifests by discarding stats for fields no longer tracked.

• Multi-Writer Scenario: Concurrent writers will refresh the table state before commit, ensuring they use the latest metrics-config. The risk of a writer losing new stats from a concurrent commit is considered low and acceptable.

V4 Writers: Commit Operation Intent

• Problem: The operation field in a snapshot's metadata is ambiguous. It can represent either: 

1. Logical Intent: The user's intended action (e.g., MERGE INTO). 
2. Physical Changes: The actual files added/deleted (e.g., only appends occurred). 

• Decision: Snapshot operations will represent logical intent. 

1. Why: This is critical for correct conflict detection and validation. An OVERWRITE operation, even if it results in only appends, requires different validation logic than a simple APPEND.
2. Action: Revert the recent change that automatically converts an OVERWRITE to an APPEND if no files are deleted. 

• API Implications: 

1. The v4 API will likely consolidate operations into a more general UpdateFiles interface.
2. The MergingSnapshotProducer will be refactored to separate v3 and v4 logic, possibly via a new base class or internal utility.

V4 Manifest Format

• Question: Should v4 manifests support both Parquet and Avro, or only Parquet? 
• Decision: V4 manifests will be Parquet-only. 

• Why: 

• Performance: Parquet enables column projection for stats, which significantly improves read performance. Avro does not support this.
• Simplicity: A single format simplifies implementations for integrations and reduces cognitive burden. 
• Consistency: Aligns with the project's focus on read performance. 

• Action: The proposal will be posted to the dev list for final community feedback before being added to the spec PR.

Jul 28, 2026

V4 Upgrade Strategy

The transition from v3 to v4 requires handling legacy manifests without a full metadata rewrite, which is prohibitively expensive for large tables.

• Gradual Migration: Consensus was on an on-demand, incremental migration strategy.
• Mechanism: V4 tables reference legacy V3 manifests, utilizing Metadata Deletion Vectors (MDVs) in the root manifest to track deleted files. When a file is modified, its entry is promoted to V4 format, with V2 positional deletes converted to V4 Deletion Vectors (DVs).

• For Column updates (adding / re-instating the whole column), this is a full rewrite of the metadata into V4.

• Reader/Writer Requirements:

• Writers: New entries must be V4-formatted; legacy entries require full component conversion upon promotion.
• Readers: Legacy readers must apply MDVs; v4 readers utilize format-version routing.

• Change Data Capture (CDC): CDC  remains functional by diffing file entries, identifying column updates via replaced/modified entry pairs. Snapshots maintain support for mixed operations (DV, update, insert).

Action Items

• Amogh: Revise the specification PR to incorporate the finalized v4 entry schema and circulate for community feedback.
• Amogh: Initiate a devlist thread to formalize and document the consensus on V3-to-V4 migration paths.
• All Participants: Review the updated spec PR and participate in the devlist discussion to solidify the final migration framework.

Jul 14, 2026

Data file stats

• Null fillers should ensure the stats are still correct (may not be accurate)
• Updating content stats

Column Updates Metadata representation (dev list thread)

• Exploring the two options on the relationship of base files with column files.

• Option 1: Base File is a Special Entity
• Option 2: Base File is Just Another ColumnFile

• Option 2 is complicated. We need base file to track things like rowid

• Format version, key metadata and file format in column files

Reviews

• Column Updates Project: https://github.com/orgs/apache/projects/619

Jun 30, 2026

Column File representation (dev list thread)

• Push back on leaving the representation to the implementation
• Requires all implementations to implement the super set (low rows only) instead of rows with filler values.
• Argument to mandate rows with rows with filler values in the spec.

Decision:

• Mandate Dense Format: Column update files must use a dense format with filler values for deleted rows. This simplifies reader implementation by enabling direct batch reads and avoiding complex row alignment logic.
• Filler Value Strategy: To preserve stats, required columns will be made optional in the file schema, with filler values written as null. This prevents polluting min/max bounds and leverages existing Parquet/Arrow null-handling.
• Writer Requirements: Writers must write null for rows known to be deleted at write time. This is a "should" requirement to avoid invalidating files if a DV is applied mid-write.

Column Updates Metadata representation (dev list thread)

• Exploring the two options on the relationship of base file with column files.

• Option 1: Base File is a Special Entity
• Option 2: Base File is Just Another ColumnFile

Updates on partition tuples

• The partition tuple in file metadata must always match the file's content. Updates are allowed, including moving a file to an unpartitioned state by unsetting its tuple.

Jun 16, 2026

V4 Metadata: Base File Definition

• The primary blocker for V4 metadata is resolving how to represent the base file.
• Option 1: Base File is a Special Entity

• The current proposal in Gábor's PR.
• The TrackedFile object holds base file metadata (location, size, split offsets) separately from a list of ColumnFile objects.
• Rationale: Keeps the mental model of a base file + updates and uses the base file's location as the unique identifier for the logical file, avoiding the need for a new file ID.

• Option 2: Base File is Just Another ColumnFile

• Treats all files in a logical collection as equal.
• Rationale: Simpler, more consistent data model. The current split-offset-based reader logic is already inefficient and was historically broken, making it a weak argument for the current approach.

V4 Implementation Path

• Blocker: V4 structure serialization on the write path.
• Enabling Change: Steven's upcoming PR to override the apply method in SnapshotProducer.

• This will add a V4-specific path that converts operations into TrackingFile entries and writes a root manifest instead of a manifest list.
• This PR is the key to unblocking all subsequent V4 development.

• Development Target: The generics API.

• Rationale: Enables rapid prototyping and unit testing without the complexities of Spark integration (e.g., waiting for Spark 4.3).
• Dependency: Requires Steven's PR to be merged first.

Column Update Details

• Stats Updates: Updating stats for only modified columns vs. rewriting all stats is an implementation detail, not a spec requirement.

• Rationale: The optimal strategy depends on the number of columns updated and the cost/complexity of managing multiple stat entries.

• Partition Column Updates: Allowed in V4.

• Rationale: V4 plans to drop partition tuples from file metadata. Planning will rely solely on min/max stats for partition transforms, making updates feasible.