Warning

This is not authoritative documentation. It describes a plan of work that is not yet implemented and will change as it lands. Once the work is complete this document should be removed; the finished system is described by the Iceberg storage engine spec, the plugin’s user documentation, and the repos themselves.

Task 7: Predicate pushdown — a core seam, then pruning

Repo: https://opendev.org/drizzle/drizzle. Depends on task 4 (read path). Floats relative to 5/6. Design-first: commit 1 of this task is a short design note landed in-tree (docs/ or the plugin’s docs/) because this changes server core, not just the plugin — the seam outlives this consumer.

Context: Drizzle stripped MySQL’s handler::cond_push (verified — only subquery internals match pushed_cond). Without a seam the engine never sees WHERE; no partition pruning, no Parquet min/max file skipping. Those two mechanisms are most of Iceberg’s scan economics; use case 2’s point-ish lookups (“the archived rows for order X”) need them.

The seam (binding shape; the design note argues it)

  • Cursor::setScanPredicate(const Item*) + resetScanPredicate(), virtual, default no-op. The optimizer hands the engine the table-local conjuncts of the WHERE clause before doStartTableScan, and still evaluates the full predicate on every returned row.

  • Translation is pure optimization, never filtering authority. The engine may use any subset of the predicate it understands to return fewer rows; it must never be relied on to filter. This makes partial translation always correct and keeps the server-side evaluation unconditional — Iceberg’s residual evaluators exist for exactly this layering.

  • Call sites in the optimizer: locate where per-table conditions are already split for range analysis and attach there — one clean injection point, not scattered calls. The design note names the exact function(s) with file:line from investigation; do not proceed to commit 2 until it does.

  • EXPLAIN observability: the extra column notes when an engine accepted a scan predicate (e.g. “Using engine condition”), so pruning is visible in plans and testable.

Commits

  1. Design note: seam shape, injection point with citations, semantics (advisory, table-local conjuncts, lifetime of the Item during the scan), why no return-value protocol for “rows filtered” (there isn’t one — server re-evaluates everything), and the compatibility statement (default no-op; zero behavior change for every existing engine).

  2. Core seam: the two virtuals, the optimizer call site, the EXPLAIN marker. No engine implements it yet; every existing test green unchanged — this commit is behavior-invisible by construction.

  3. Engine translation: IcebergCursor::setScanPredicate walks the Item conjuncts and translates what it can into iceberg-cpp expressions: comparisons (=,<,<=,>,>=,!=), IS [NOT] NULL, IN(list), BETWEEN, and conjunctions thereof, over supported column types with constant operands (fold Item::const_item() operands; anything else — functions, subqueries, non-constant — is skipped, not attempted). The translated expression feeds the Planner (its filter parameter exists since task 4’s signature; wire it) → iceberg-cpp scan planning applies partition pruning and file-level stats skipping. Untranslatable conjuncts are simply absent from the filter; correctness is unaffected.

  4. Tests: on the partitioned fixture, a WHERE day-partition-key = const query demonstrably plans fewer FileScanTasks (assert via a planner-level counter exposed through a status variable, and via the callgrind CI job’s instruction counts — this is exactly what perf-tracking exists for); equality on a clustered/sorted column skips files via stats; results identical with the seam disabled (a debug option iceberg.disable-pushdown for A/B in the suite); every skipped- translation shape (function operand, OR at top level if unsupported) returns correct results.

Verification

  • Full tree green after commit 2 with zero result-file changes (the seam is invisible until adopted).

  • Suites green; perf-tracking shows the pruning win on the partitioned fixture (record the numbers in the review).

  • Valgrind: Item lifetime handling correct across scan restarts (range loops) — the reset path is the classic UAF spot; test a join that reopens the scan.