Iceberg Storage Engine: Design

Scope and thesis

A Drizzle storage engine plugin (plugin/iceberg) that reads and appends to Apache Iceberg tables through an Iceberg REST catalog, using the native iceberg-cpp library (Arrow C++ underneath for Parquet). Two use cases drive every decision:

  1. SQL access to pipeline output — tables written by Spark/Flink/pyiceberg appear in Drizzle and are queryable, with no import step and no Drizzle-owned metadata.

  2. Cold tier for OLTP data — INSERT INTO iceberg_t SELECT ... FROM hot_t rolls aged records into Iceberg in one well-sized append commit.

The subtractive frame: this engine is a deliberate subset of Iceberg’s surface. REST catalogs only. Primitive column types only. Append-only writes. Snapshot reads. Everything refused is refused loudly with a specific error, not approximated.

Prior-art note: MariaDB’s S3 engine is not the model — it stores Aria-format data in S3 that nothing else can read. Interoperability with the rest of the Iceberg ecosystem is the entire point here; the data and metadata formats are Iceberg’s, and Drizzle is just another engine at the table.

Library and catalog landscape (as of July 2026)

  • iceberg-cpp 0.3.0 (released June 2026; 0.2.0 in January 2026 covered table scan planning with V2 delete support, append + transaction API with snapshot management, a REST catalog client, and an expression system with residual evaluators). Pre-1.0: API churn is expected. Pin an exact version in the CI images and vendor the pin in one place.

  • Arrow C++ is iceberg-cpp’s substrate; Parquet reads surface as Arrow record batches. It is a heavy dependency — pinned and baked into the builder-image lineage (task 1), never detected. The plugin is load_by_default=no and enabled by an explicit build switch (below).

  • Catalogs: the ecosystem consolidated on the Iceberg REST protocol (Polaris is an Apache TLP as of Feb 2026; Nessie and Gravitino speak REST). We support REST only. No Hive metastore, no Glue client, no filesystem/”hadoop” catalog. This one decision deletes most of the configuration surface that plagues Iceberg integrations elsewhere.

Build and dependency policy: container-first, no detection

Binding for every task in this series (established in task 1):

  • The only build command anywhere is podman build, fronted by just targets; Zuul runs the identical targets. Gate/dev parity by construction.

  • Every dependency is pinned exactly — image bases by digest, source deps by tag/sha — in Containerfiles, which are the single pin location. We know what is in the image because we put it there.

  • There are no configure checks for dependencies. Anywhere. Nothing probes for iceberg-cpp or Arrow; the builder image definitionally contains them (FROM iceberg-deps). Plugin inclusion is an explicit build switch (--with-iceberg, default off), which the containerized build always passes. Switch off = plugin excluded, deliberately, by the person building. Wrong pin = loud failure at podman build. There is no “found: no → silently skip” state; that failure mode is deleted, not handled.

  • “Both configurations” throughout the task docs means both switch states, not detection outcomes.

Recorded open puzzle (decide later, before it bites): this engine will not be the last source-built dependency wanting a home in the builder-image lineage — the WiredTiger revival is already staged, and RocksDB-shaped ideas exist. N engines × source-built deps × one builder image is a composition problem: one fat deps image accretes poorly; per-dep build stages composed via multi-stage COPY --from= into the builder is the podman-native shape but needs prefix/soname discipline. Not this series’ problem — the decision point is when the second source-built engine dep lands, and it deserves its own short design note then. Flagged so it is a decision, not an accident.

Identifier model

One Iceberg catalog per server, configured at plugin load (iceberg.catalog-uri, iceberg.warehouse, credential options). Iceberg namespaces surface as Drizzle schemas; tables as tables.

Corner-avoidance rule: internally, every identifier is the full (catalog, namespace, table) triple even though the catalog component is a constant today. The schema↔namespace mapping is one thin translation function. The future multi-catalog home is Drizzle’s own catalog dimension — identifier::Schema already carries an identifier::Catalog (drizzled/identifier/schema.h:47) and the drizzled/catalog/ layer has an engine seam. One Drizzle catalog per Iceberg catalog is the eventual spelling; nothing in this design forecloses it, and nothing in v1 implements it.

Metadata: the catalog is the source of truth

The engine owns its whole namespace the way plugin/function_engine does (function.cc:54-67): it synthesizes message::Table protos on demand and never writes a table-definition file.

  • doGetSchemaIdentifiers / doGetSchemaDefinition (storage_engine.h:362-366) → list namespaces from the REST catalog.

  • doGetTableIdentifiers → list tables in a namespace (the CachedDirectory parameter is ignored, as function_engine ignores it).

  • doGetTableDefinition → fetch the Iceberg table’s current schema, translate to a message::Table proto (return EEXIST on success / ENOENT, per the existing convention).

  • HTON_HAS_SCHEMA_DICTIONARY set, so the engine’s schemas appear alongside local ones.

Tables created by other engines-of-the-ecosystem simply appear. Schema evolution done by Spark simply appears (next doGetTableDefinition reflects it). Drizzle caches table definitions per its normal table-cache rules; a FLUSH TABLES picks up external schema changes — document this rather than building invalidation machinery.

Type mapping

Drizzle FieldType (drizzled/message/table.proto:73-88) ↔ Iceberg primitives:

Iceberg

Drizzle

Notes

boolean

BOOLEAN

int

INTEGER

long

BIGINT

float

DOUBLE

lossless widening; only permitted lossy-direction exception is none — this is exact

double

DOUBLE

decimal(p,s)

DECIMAL

p ≤ Drizzle max; else refuse

date

DATE

time

TIME

microsecond

timestamp

DATETIME

no zone

timestamptz

EPOCH

stored/compared as UTC

string

VARCHAR

binary, fixed(L)

BLOB

uuid

UUID

first-class in Drizzle — free win

Refused, per-table, at ``doGetTableDefinition`` time: struct, list, map, variant, geometry/geography, timestamp_ns (v3). A table containing any unsupported column type returns a specific error naming the column and type. No flattening, no JSON-stringifying, no silently hiding columns — a table is either fully representable or not attached. (Revisit variant when the Arrow C++ variant work lands; it is the connective type the ecosystem is converging on and the likeliest future exception.)

Read path: Planner / Executor split

Snapshot resolution is an explicit planning step that produces a value — not a side effect of opening a cursor. Two components:

  • Planner: (table, snapshot-id, projection[, filter — phase 3]) → ordered vector<FileScanTask>. iceberg-cpp does manifest traversal and V2 delete-file matching. The task list is pinned to the snapshot and immutable for the life of the plan.

  • Executor: one FileScanTask → rows. Arrow record batches from the Parquet reader, packed one row at a time into the Drizzle record buffer via Field::store per the read map.

The Cursor is local glue: doStartTableScan obtains a plan (via the pin — below), rnd_next drains executors task by task, doEndTableScan releases.

This factoring costs nothing today and is exactly the distribution seam for the farm-work-to-peer-drizzles future: the unit of shippable work is (snapshot-id, FileScanTask), which is inherently serializable — the REST spec’s server-side scan planning (plan-id/plan-tasks, Iceberg 1.11) even defines a wire representation for it. Design the Planner so its output can be constructed from a received task list as well as produced by local planning. Build nothing else distributed now.

Row references: position() encodes (task ordinal, row ordinal) into ref; rnd_pos seeks within the pinned plan. References are meaningful only relative to a pinned plan — one of two reasons pinning is per-statement at minimum.

Projection: set HTON_PARTIAL_COLUMN_READ; drive Parquet column selection from Table::read_set (drizzled/table.h:497). With columnar files this is the single biggest read lever and is nearly free.

Engine flags: HTON_ALTER_NOT_SUPPORTED | HTON_TEMPORARY_NOT_SUPPORTED | HTON_SKIP_STORE_LOCK | HTON_PARTIAL_COLUMN_READ | HTON_HAS_SCHEMA_DICTIONARY. No HTON_STATS_RECORDS_IS_EXACT — snapshot summary total-records feeds info()/records() as an estimate (delete files make it approximate). No index API implemented at all; Iceberg has no indexes and Drizzle does not force a fake primary key.

Snapshot pinning: per transaction

Semantics: each transaction sees, per table, the snapshot current at the table’s first touch within that transaction; autocommit degenerates to per-statement. The floor is per-statement and it is a correctness floor: a self-join opens two cursors on one table, and per-cursor resolution lets a concurrent external commit land between the opens, joining two versions of the same table. Going from per-statement to per-transaction is free in the way only immutable storage makes free — pinning is remembering an ID — and yields repeatable read across the transaction at zero cost.

Mechanism: a per-session engine slot via Session::getEngineData(const plugin::MonitoredInTransaction*) (drizzled/session.h:345; innobase stores its trx_t* the same way, ha_innodb.cc:1030) holding map<table-uuid, snapshot-id> plus the write buffers (below). Populated at first touch; cleared in doCommit/doRollback (and at statement end under autocommit via the doStartStatement/doEndStatement hooks, storage_engine.h:143-155).

Read-your-own-writes: none, deliberately. Rows appended within a transaction sit in the commit buffer and belong to no snapshot until commit; a scan in the same transaction does not see them. The roll-off pattern never notices (its SELECT source is the hot table). Documented subset; merging the buffer into scans would be complexity with no user.

Time travel is the same mechanism: an explicit pin instead of pin-to-current. Phase 4 adds the second writer to the same map.

Write path: buffer until commit

doInsertRecord never touches object storage — it appends the row to an in-memory Arrow builder in the session’s engine slot. At doCommit, the engine writes one-or-few Parquet data files sized by Iceberg norms and performs a single append commit through iceberg-cpp’s transaction API (optimistic CAS against the catalog; on conflict, retry the metadata commit — data files are already written and remain valid; bounded retries, then fail the SQL commit).

The engine is a TransactionalStorageEngine (drizzled/plugin/transactional_storage_engine.h): doCommit = write files + catalog commit; doRollback = drop the buffer (rollback is free — nothing is durable before commit); savepoints (doSetSavepoint / doRollbackToSavepoint / doReleaseSavepoint, pure virtuals at lines 181-183) are row-count marks in the buffer.

This shape inherently avoids small-file disease for the roll-off case: one transaction → one commit → few large files. Single-row autocommit inserts work but cost an object-store round trip per commit; the docs say so plainly. start_bulk_insert(ha_rows) is a sizing hint for the builder, nothing more.

Refused: UPDATE and row-level DELETE return HA_ERR_WRONG_COMMAND. iceberg-cpp’s write side is append-only, and neither use case mutates cold data. Position-delete/DV writes are a possible far-future phase, not designed here.

Cross-engine atomicity caveat (roll-off): INSERT INTO iceberg ...; DELETE FROM hot ... in one Drizzle transaction is two engines’ commits, and this engine is deliberately a non-XA TransactionalStorageEngine (task 5: participatesInSqlTransaction true, XA false). There is therefore no two-phase-commit protection for the pair: the server commits the participating engines sequentially with no prepare phase, and a crash or failure between the two commits leaves exactly one side applied. The roll-off safety story rests entirely on the idempotent shape — insert-commit first, verify, then delete from hot in a separate transaction — never on server coordination. A true XA mapping is plausible future work (prepare = write data files and staged manifests, commit = catalog CAS, rollback = abandon staged files) but requires promoting the engine to the XA interfaces and is neither designed nor promised here.

DDL

v1 is attach-only. doCanCreateTable returns false; DROP is refused. Drizzle attaches to tables created elsewhere — both use cases start that way.

CREATE (phase 2 implements; designed now). Drizzle’s grammar already accepts arbitrary engine options: ident_or_text '=' engine_option_value (drizzled/sql_yacc.yy:1098-1105) → parser::buildEngineOption → Engine::Option name/state pairs (drizzled/message/engine.proto:12-21). So this parses today with zero yacc changes:

CREATE TABLE events (...) ENGINE=iceberg
  PARTITION_BY='days(created_at), bucket(16, user_id)'
  SORT_ORDER='created_at desc';

The transform mini-language invents nothing: it is exactly Iceberg’s transform spelling — identity(col) (or bare col), year(col), month(col), day(col)/days(col), hour(col), bucket(N, col), truncate(W, col) — parsed and validated by the engine in doCreateTable: column references checked against the field list, transform/type compatibility checked, result handed to iceberg-cpp’s partition-spec builder. Anyone coming from Spark DDL reads it without a manual. If first-class PARTITION BY (...) grammar is ever wanted, it desugars to this same engine option; proto representation and engine validation don’t move, so deferring the yacc work costs nothing and forecloses nothing.

DROP (phase 2): drop without purge — deregister from the catalog, never delete data files. For attached tables, defaulting to purge would be Drizzle deleting the pipeline’s data because someone tidied their SQL view of it. Purge is unsupported; if ever added, it is behind an explicit engine option and a scary docs paragraph.

RENAME: doRenameTable maps to catalog rename within a namespace; cross-namespace rename refused in v1.

The core gap: predicate pushdown (phase 3)

Drizzle stripped the MySQL handler::cond_push seam (verified: only subquery internals match pushed_cond in the tree). Without it the engine never sees WHERE, so no partition pruning and no Parquet min/max file skipping — the two mechanisms that make Iceberg scans cheap. Phases 1-2 ship with full scans and say so.

Phase 3 adds a seam to core — roughly Cursor::setScanPredicate(Item*) / resetScanPredicate(), optional (default no-op), called by the optimizer with the table-local conjuncts before scan start. The engine translates what it can into iceberg-cpp’s expression system for pruning at plan time; the server still evaluates the full WHERE on returned rows (Iceberg’s residual evaluators exist for exactly this layering), so a partial translation is always correct — translation is pure optimization, never filtering authority. This is a core API change that benefits any smart engine and gets designed once against a real consumer. It is sequenced as its own design-first task, not bloat in the engine’s phase 1.

Observability and time travel (phase 4)

A registry_dictionary-style dictionary plugin (or TableFunctions in the same plugin) exposing DATA_DICTIONARY.ICEBERG_SNAPSHOTS, ICEBERG_MANIFESTS, ICEBERG_DATA_FILES per table, straight from iceberg-cpp metadata — the house style for introspection. Time travel: a session variable (iceberg_snapshot_for or similar) as the pre-grammar interim, writing an explicit pin into the same per-session pin map the transaction logic uses. AS OF grammar, if ever, is a third writer to the same map.

Component layout

plugin/iceberg/
  plugin.ini                 build_conditional on the explicit --with-iceberg switch; load_by_default=no
  iceberg_engine.{h,cc}      StorageEngine subclass: metadata hooks, DDL, flags
  catalog_client.{h,cc}      REST catalog session: config, auth, namespace/table ops
  schema_map.{h,cc}          Iceberg schema <-> message::Table; the refusal list
  planner.{h,cc}             (table, snapshot, projection[, filter]) -> vector<FileScanTask>
  executor.{h,cc}            FileScanTask -> rows (Arrow batch -> record buffer)
  cursor.{h,cc}              Cursor subclass gluing planner/executor; position/rnd_pos
  session_state.{h,cc}       per-session engine slot: pin map + write buffers + savepoint marks
  writer.{h,cc}              Arrow builder -> Parquet files -> append commit
  transforms.{h,cc}          PARTITION_BY / SORT_ORDER mini-language parse + validate (phase 2)
  docs/                      user docs incl. the "what is refused and why" table
  tests/                     drizzle-test cases (need catalog+MinIO fixtures, task 1/2)

Ownership and style

House policy applies unchanged: unique_ptr for ownership, references for never-null working handles, raw pointers only for nullable lookups. iceberg-cpp returns its own shared/unique handles — hold them as vended; do not unwrap. All engine-internal state (planner caches, session slots) is RAII; no manual delete anywhere in the plugin. The Cursor subclass allocates via the engine create() seam exactly as every other engine does.

Task sequence

#

Task

Depends on

Character

1

Task 1: iceberg-cpp spike and CI fixtures — container-first — iceberg-cpp standalone spike + CI fixture images

—

de-risk

2

Task 2: Plugin skeleton and build integration — build integration, config, empty engine

1

scaffolding

3

Task 3: Catalog-backed metadata — namespaces/tables/schema translation

2

metadata

4

Task 4: Read path — planner, executor, cursor, pinning — planner/executor/cursor, projection, pinning

3

the core read

5

Task 5: Transactional buffered append — transactional buffered append

4

the core write

6

Task 6: DDL — CREATE with transforms, DROP without purge, RENAME — CREATE with transforms, DROP-no-purge, RENAME

5

DDL

7

Task 7: Predicate pushdown — a core seam, then pruning — core predicate seam + pruning

4

core + engine

8

Task 8: Dictionary tables and time travel — dictionary tables + explicit pins

4

observability

Critical chain 1 → 2 → 3 → 4; then 5 → 6 and 7, 8 float after 4. Phases: 1-4 = use case 1 delivered read-only; +5 = use case 2 delivered; +6 = self-service DDL; 7-8 = performance and polish.

Verification requirements (all tasks)

  • Full build green per commit, in both switch states (--with-iceberg on/off — excluded is a deliberate build choice, never a detection outcome).

  • drizzle-test green; iceberg suites run against the catalog+MinIO fixtures from task 1, seeded by pyiceberg so interop (not self-consistency) is what’s tested.

  • Valgrind/callgrind/massif CI comparison where a task changes the server (tasks 4, 5, 7): no new leaks vs baseline.

  • Every refusal path has a test asserting the specific error, not just “it failed.”