Skip to content

feat: support DataFusion paimon_incremental_query - #511

Open
Pandas886 wants to merge 2 commits into
apache:mainfrom
Pandas886:feat/datafusion-incremental-query
Open

feat: support DataFusion paimon_incremental_query#511
Pandas886 wants to merge 2 commits into
apache:mainfrom
Pandas886:feat/datafusion-incremental-query

Conversation

@Pandas886

@Pandas886 Pandas886 commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #506

This PR adds the DataFusion SQL surface for the batch incremental read APIs now available on main.

SELECT * FROM paimon_incremental_query('t', 0, 2);
SELECT * FROM paimon_incremental_query('t', 0, 2, 'diff');
SELECT rowkind, id, value FROM paimon_incremental_query('t$audit_log', 0, 2, 'changelog');

Brief change log

  • Register TVF paimon_incremental_query in SQLContext
  • 3-arg form defaults to auto; optional 4th arg: auto / delta / changelog / diff
  • Delegate physical planning to core IncrementalScan / AuditLogTable
  • Support $audit_log suffix for rowkind output
  • Support projection and residual filters
  • Resolve qualified table names against the named catalog; unqualified names follow the current catalog
  • Update docs/src/sql.md with usage and examples

Tests

cargo test -p paimon-datafusion --test incremental_query
cargo fmt --all -- --check
cargo clippy -p paimon-datafusion --all-targets -- -D warnings

API and Format

  • New SQL table function only; storage format unchanged
  • Relies on the core incremental APIs already merged into main

Documentation

docs/src/sql.md documents registration, arguments, modes, and examples.

@JingsongLi

Copy link
Copy Markdown
Contributor

The Diff problem of inheriting '# 510'; The non audit mode will first push the SQL projection down to the core, directly triggering the issue of "only comparing projection columns/missing PK". In addition, the catalog section of 'catalog. db. table' is discarded after parsing, and the TVF global registration will be overwritten by the last registered catalog. Multiple catalog scenarios may result in reading the wrong catalog.

@Pandas886
Pandas886 force-pushed the feat/datafusion-incremental-query branch from 9e235b1 to c833e97 Compare July 30, 2026 11:30
@Pandas886

Pandas886 commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Rebased on the latest main and fixed catalog/database resolution. Added coverage for multiple catalogs and USE database. Could you take another look?

@Pandas886
Pandas886 force-pushed the feat/datafusion-incremental-query branch from c833e97 to eb99a81 Compare July 30, 2026 11:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DataFusion: paimon_incremental_query table function

2 participants