Repository navigation
Conversation
387e959 to
e011356
Compare
JingsongLi
left a comment
There was a problem hiding this comment.
The Mosaic writer limitation is real. I ran all 163 Mosaic tests on the merge with current master on JDK 11, including native read/write. A separate existing-table upgrade test exposed a read regression. #11241 was consolidated here, and its nested ARRAY/MAP coverage would fit this PR.
| * of failing on the first write. | ||
| */ | ||
| private Void validateTimestampPrecision(int precision) { | ||
| if (precision == 0) { |
There was a problem hiding this comment.
[P1] Keep existing Mosaic tables readable after upgrade
validateDataFields also runs from AbstractFileStoreTable.runtimeStore when planning a read. I used master to create a Mosaic table with an integer column, commit row 7, and add a nullable created_at TIMESTAMP(0) column. Master then read (7, NULL) from the original file. With this exact change the same table fails in newScan() with UnsupportedOperationException, before reading that file. This is a live upgrade path for tables that evolved before the first write to the new column. Scope the new validation to new table definitions or writes and add an upgrade read test for the existing schema.
Purpose
Close #11228.
MosaicFileFormat.MosaicRowTypeVisitoracceptsTIMESTAMP(0)andTIMESTAMP(0) WITH LOCAL TIME ZONE. The Arrow conversion maps precision 0 to theSECONDunit (ArrowFieldTypeConversion.getTimeUnit), and the Mosaic native writer refuses that unit. Verified with the real native library on macOS/aarch64 by writing one row throughMosaicFileFormat.createWriterFactory(RowType.of(DataTypes.TIMESTAMP(0))):The same write with
TIMESTAMP(3)succeeds. So the table can be created and every subsequent write fails, naming neither the column nor the format.This rejects precision 0 while the schema is validated:
Precisions 1 to 9 are unaffected. Converting precision 0 to milliseconds instead would also work, but that changes
ArrowFieldTypeConversionfor every Arrow based format and needs matching read/write handling, so rejecting it inside Mosaic is the smaller fix.Tests
MosaicFileFormatTest:testValidateDataFieldsTimestampPrecisionZeroUnsupported,testValidateDataFieldsLocalZonedTimestampPrecisionZeroUnsupported(both fail on master, pass here) andtestValidateDataFieldsTimestampPrecisionsOneToNineSupportedpinning precisions 1/6/9 plusTIMESTAMP(3) WITH LOCAL TIME ZONE.mvn -pl paimon-mosaic -Pfast-build test→ 164 tests pass, including the native round-trip and writer tests.mvn -pl paimon-mosaic spotless:checkclean.