Task 8: Dictionary tables and time travel

Warning

This is not authoritative documentation. It describes a plan of work that is not yet implemented and will change as it lands. Once the work is complete this document should be removed; the finished system is described by the Iceberg storage engine spec, the plugin’s user documentation, and the repos themselves.

Repo: https://opendev.org/drizzle/drizzle. Depends on task 4. Floats relative to 5/6/7. Two commits. Goal: Iceberg’s metadata becomes queryable the Drizzle way, and the pinning machinery gains its explicit-pin writer.

Commit 1: DATA_DICTIONARY tables

plugin::TableFunction implementations inside plugin/iceberg (the registry_dictionary pattern), registered only when the engine is active:

  • DATA_DICTIONARY.ICEBERG_SNAPSHOTS(schema, table, snapshot_id, parent_id, committed_at, operation, summary_json) — one row per snapshot per attached table. Enumerating every table’s history on a full scan of this dictionary is catalog-request-heavy; that is inherent and acceptable for a dictionary table (document it) — but implement the standard WHERE-on-schema/table narrowing if the TableFunction interface’s existing usage supports it (check how schema_dictionary handles it; mirror that, invent nothing).

  • DATA_DICTIONARY.ICEBERG_MANIFESTS(...) and DATA_DICTIONARY.ICEBERG_DATA_FILES(schema, table, snapshot_id, file_path, record_count, file_size, partition_json) — straight from iceberg-cpp metadata for the pinned snapshot of the querying session (so a transaction inspecting a table sees the metadata of what it is reading — the pin map is already the source of truth; reuse it).

All values come from the library’s metadata objects; no manifest parsing of our own.

Commit 2: explicit pins (time travel)

A session variable, registered via the module’s variable mechanism:

  • SET @@iceberg_as_of_snapshot = <snapshot-id> scoped by a companion @@iceberg_as_of_table = 'schema.table' — or, cleaner if the variable machinery allows structured values poorly, a single @@iceberg_as_of = 'schema.table@snapshot_id' string parsed by the engine. Pick whichever is least clever; the variable is an interim UX until/unless AS OF grammar happens, and grammar would be a third writer to the same map — say so in a comment, add nothing speculative.

  • Semantics: setting it writes an explicit PinEntry into the session pin map (task 4’s structure, unchanged); the pin survives until the variable is cleared (= ''/= 0) or the session ends — explicit pins are not cleared at transaction end (that is their point); the pin-map clearing logic distinguishes implicit from explicit entries.

  • Validation at set time: table attached, snapshot exists (specific errors).

  • Interaction with writes: an INSERT into a table with an explicit historical pin is refused (appending “into the past” is incoherent; Iceberg would commit onto current anyway — refuse rather than surprise).

Tests

  • Snapshots dictionary lists the fixture table’s known history (seeded count); data-files dictionary row counts sum to the table count.

  • Time travel: pin snapshot N-1 on the multi-snapshot fixture, SELECT shows the old data; clear pin, SELECT shows current; pinned SELECT inside a transaction alongside unpinned tables behaves.

  • Dictionary metadata reflects the session’s pin (query ICEBERG_DATA_FILES under a historical pin → old file list).

  • Refusals: bad snapshot id, unattached table, INSERT under historical pin.

Verification

  • Build green in both switch states (--with-iceberg on/off) per commit; suites green.

  • Docs: an “inspecting and time-traveling Iceberg tables” page with the dictionary schemas and the variable, including the explicit-pin lifetime rule.