Skip to content

[mosaic] Reject TIMESTAMP(0) when validating Mosaic data fields - #11241

Open
LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/mosaic-timestamp-precision
Open

LuciferYang wants to merge 1 commit into
apache:masterfrom
LuciferYang:fix/mosaic-timestamp-precision

Conversation

@LuciferYang

Copy link
Copy Markdown
Contributor

Purpose

close #11228

Precision 0 maps to the Arrow SECOND unit, which the Mosaic native writer does not accept. A Mosaic table with a TIMESTAMP(0) or TIMESTAMP(0) WITH LOCAL TIME ZONE column can be created, but every write fails with writer open failed: unsupported Timestamp unit: Second. MySQL CDC creates such columns from DATETIME.

MosaicFileFormat.validateDataFields now rejects precision 0, also inside ARRAY and MAP, with the same message style as the other types Mosaic rejects, and suggests TIMESTAMP(3).

This check also runs when a table is loaded. An existing Mosaic table with a TIMESTAMP(0) column could not be written before and now cannot be read either, until the column is changed to TIMESTAMP(3) or file.format is set back. Both ALTERs work on such a table.

Tests

MosaicFileFormatTest:

  • testValidateDataFieldsTimestampPrecisionZeroUnsupported checks the full message for TIMESTAMP(0), ARRAY<TIMESTAMP(0)>, TIMESTAMP(0) WITH LOCAL TIME ZONE and MAP<STRING, TIMESTAMP(0) WITH LOCAL TIME ZONE>. It fails without the change.
  • testValidateDataFieldsTimestampPositivePrecisionSupported checks that precisions 1 and 9 of both types are still accepted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Mosaic accepts TIMESTAMP(0) columns that its native writer cannot write

1 participant