Skip to content

[mosaic] Reject TIMESTAMP(0) columns the native writer cannot write - #11235

Open
jlon wants to merge 1 commit into
apache:masterfrom
jlon:mosaic-reject-timestamp-precision-0
Open

jlon wants to merge 1 commit into
apache:masterfrom
jlon:mosaic-reject-timestamp-precision-0

Conversation

@jlon

@jlon jlon commented Nov 18, 2026 •

Copy link
Copy Markdown

Purpose

Close #11228.

MosaicFileFormat.MosaicRowTypeVisitor accepts TIMESTAMP(0) and TIMESTAMP(0) WITH LOCAL TIME ZONE. The Arrow conversion maps precision 0 to the SECOND unit (ArrowFieldTypeConversion.getTimeUnit), and the Mosaic native writer refuses that unit. Verified with the real native library on macOS/aarch64 by writing one row through MosaicFileFormat.createWriterFactory(RowType.of(DataTypes.TIMESTAMP(0))):

java.lang.RuntimeException: writer open failed: unsupported Timestamp unit: Second

The same write with TIMESTAMP(3) succeeds. So the table can be created and every subsequent write fails, naming neither the column nor the format.

This rejects precision 0 while the schema is validated:

Mosaic file format does not support TIMESTAMP with precision 0, use precision 1 to 9

Precisions 1 to 9 are unaffected. Converting precision 0 to milliseconds instead would also work, but that changes ArrowFieldTypeConversion for every Arrow based format and needs matching read/write handling, so rejecting it inside Mosaic is the smaller fix.

Tests

  • MosaicFileFormatTest: testValidateDataFieldsTimestampPrecisionZeroUnsupported, testValidateDataFieldsLocalZonedTimestampPrecisionZeroUnsupported (both fail on master, pass here) and testValidateDataFieldsTimestampPrecisionsOneToNineSupported pinning precisions 1/6/9 plus TIMESTAMP(3) WITH LOCAL TIME ZONE.
  • mvn -pl paimon-mosaic -Pfast-build test → 164 tests pass, including the native round-trip and writer tests.
  • mvn -pl paimon-mosaic spotless:check clean.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Mosaic writer limitation is real. I ran all 163 Mosaic tests on the merge with current master on JDK 11, including native read/write. A separate existing-table upgrade test exposed a read regression. #11241 was consolidated here, and its nested ARRAY/MAP coverage would fit this PR.

* of failing on the first write.
*/
private Void validateTimestampPrecision(int precision) {
if (precision == 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Keep existing Mosaic tables readable after upgrade

validateDataFields also runs from AbstractFileStoreTable.runtimeStore when planning a read. I used master to create a Mosaic table with an integer column, commit row 7, and add a nullable created_at TIMESTAMP(0) column. Master then read (7, NULL) from the original file. With this exact change the same table fails in newScan() with UnsupportedOperationException, before reading that file. This is a live upgrade path for tables that evolved before the first write to the new column. Scope the new validation to new table definitions or writes and add an upgrade read test for the existing schema.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Mosaic accepts TIMESTAMP(0) columns that its native writer cannot write

2 participants