Skip to content

Preserve short IN lists for dictionary string columns #26945

Open

Description

@Rich-T-kid

Is your feature request related to a problem or challenge?

When Parquet RLE-to-dictionary conversion is enabled, the expression optimizer rewrites two- and three-value IN lists on dictionary string columns into equality chains. Each equality evaluates the dictionary values and maps its result through the keys separately. A single IN expression can evaluate membership on the dictionary values once and map one result through the keys.

TPC-H Q12 hits this with l_shipmode IN ('MAIL', 'SHIP') and o_orderpriority IN ('1-URGENT', '2-HIGH'). In a local main-versus-patch A/B with DATAFUSION_EXECUTION_PARQUET_ENABLE_RLE_TO_DICTIONARY=true, Q12 improved by 3.1-3.7% across focused and full-suite runs. Q13 and Q22, used as unchanged-plan controls, were flat.

Describe the solution you would like

Retain short literal IN and NOT IN expressions for dictionary columns whose values are Utf8 or Utf8View. Keep the existing rewrite behavior for non-dictionary string columns.

Describe alternatives you have considered

The dictionary physical expression could recognize and fuse equality chains, but preserving the original membership expression is smaller and avoids creating the repeated work in the first place.

Activity

  1. Rich-T-kid commented on Oct 30, 2026

    @Rich-T-kid
    ContributorAuthor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions