Repository navigation
Description
Search before asking
- I searched in the issues and found nothing similar.
Version
master, commit e78f7366134e (2.0.11-SNAPSHOT), standalone, table model.
Describe the bug and provide the minimal reproduce step
A multi-value IN list on an INT32 or INT64 field converts its literals without the range and decimal handling that = already has. This shows up in two ways:
- On INT32 fields, an out-of-range integer literal is narrowed with an unchecked
(int)cast. It wraps around and matches an unrelated row. - On INT32 and INT64 fields, a decimal literal is cast to
LongLiteral, so the query fails with aClassCastException.
Both the pushed-down scan filter and the projection evaluator are affected. x = a OR x = b is normalized into x IN (a, b), so the OR form returns the same wrong result.
create database castdb;
create table castdb.p(dev string tag, i int32 field);
insert into castdb.p(time, dev, i) values (1,'d1',1),(2,'d1',2),(3,'d1',3),(4,'d1',-2147483648),(5,'d1',2147483647),(6,'d1',null);
create table castdb.q(dev string tag, l int64 field);
insert into castdb.q(time, dev, l) values (1,'d1',1),(2,'d1',2),(3,'d1',3),(4,'d1',null);Wrong rows (INT32)
| Query | Actual rows (time) | Expected |
|---|---|---|
select time, i from castdb.p where i in (1, 4294967298) |
1, 2 | 1 |
select time, i from castdb.p where i in (1, -4294967293) |
1, 3 | 1 |
select time, i from castdb.p where i in (1, 2147483648) |
1, 4 (-2147483648) |
1 |
select time, i from castdb.p where i = 1 or i = 4294967298 |
1, 2 | 1 |
select time, i from castdb.p where i not in (1, 4294967298) |
3, 4, 5 (row 2 missing) | 2, 3, 4, 5 |
explain for the OR query shows the rewritten predicate reaching the scan:
│PushDownPredicate: ("i" IN (1, 4294967298))│
ClassCastException (INT32 and INT64, filter and projection)
These all fail with the same message:
select time, i from castdb.p where i in (2, 2.5)
select time, i from castdb.p where i in (2.0, 2.4)
select time, i from castdb.p where i in (2.5, 3.5)
select time, i from castdb.p where i not in (1, 2.5)
select time, i from castdb.p where i = 2 or i = 2.5
select time, l from castdb.q where l in (2, 2.5)
select time, l from castdb.q where l not in (1, 2.5)
select time, i in (2, 2.5) as m from castdb.p
select time, l in (2, 2.5) as m from castdb.q
Msg: org.apache.iotdb.jdbc.IoTDBSQLException: 301: class org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.DoubleLiteral cannot be cast to class org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.LongLiteral
Filter pushdown path (DataNode log):
java.lang.ClassCastException: class ...DoubleLiteral cannot be cast to class ...LongLiteral
at org.apache.iotdb.db.utils.TypeServices.getLongValue(TypeServices.java:4249)
at org.apache.iotdb.db.utils.TypeServices$Predicate.lambda$static$5(TypeServices.java:3432)
at org.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor.getValue(ConvertPredicateToFilterVisitor.java:364)
at org.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor.constructInSet(ConvertPredicateToFilterVisitor.java:143)
at org.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor.constructInFilter(ConvertPredicateToFilterVisitor.java:136)
at org.apache.iotdb.db.queryengine.plan.relational.analyzer.predicate.ConvertPredicateToFilterVisitor.visitInPredicate(ConvertPredicateToFilterVisitor.java:125)
...
at org.apache.iotdb.db.queryengine.plan.analyze.PredicateUtils.convertPredicateToFilter(PredicateUtils.java:424)
at org.apache.iotdb.db.queryengine.plan.planner.DataNodeTableOperatorGenerator.buildSeriesScanOptions(...)
Expression (projection) path:
java.lang.ClassCastException: class ...DoubleLiteral cannot be cast to class ...LongLiteral
at org.apache.iotdb.calc.utils.TypeServices.lambda$static$136(TypeServices.java:1141)
at org.apache.iotdb.calc.execution.relational.ColumnTransformerBuilder.constructInColumnTransformer(ColumnTransformerBuilder.java:1531)
at org.apache.iotdb.calc.execution.relational.ColumnTransformerBuilder.visitInPredicate(ColumnTransformerBuilder.java:1520)
...
For comparison, the single-value and equality forms already behave as expected on the same data:
| Query | Result |
|---|---|
where i = 4294967298 |
empty |
where i in (4294967298) |
empty |
where i = 2.5 |
empty |
where i in (2.0) |
row 2 |
select time, i in (1, 4294967298) as m |
true only for row 1, null for the NULL row |
What did you expect to see?
x IN (a, b) should give the same result as x = a OR x = b under the existing = semantics for numeric literals on INT32/INT64 fields:
- an integer outside the column range never matches;
- an integral decimal such as
2.0matches2; - a non-integral decimal such as
2.5never matches; NOT INreturns the complement over non-NULL rows.
What did you see instead?
Wrong rows for out-of-range INT32 literals in IN, NOT IN and the equivalent OR form, and a ClassCastException for any decimal literal in a multi-value IN on INT32/INT64, in both the filter and the projection paths.
Anything else?
Root cause
ConvertPredicateToFilterVisitor.visitInPredicatesends a single-value list toconstructCompareFilter. Since fixed the bug that caused incompatible behavior when comparing int64 or int32 column with double literals in query filter conditions #16917 that path handles decimal literals and out-of-range bounds.- A list with two or more values goes to
constructInFilter→constructInSet→getValue. That usesTypeServices.Predicate.RELATIONAL_CONVERT_PREDICATE_VALUE_PARSER_SERVICE, which is(int) ((LongLiteral) v).getParsedValue()for INT32 and((LongLiteral) v).getParsedValue()for INT64. - The analyzer does not insert casts for IN lists. It only checks that the types are comparable, so
DoubleLiteraland oversizedLongLiteralvalues reach this code unchanged. NormalizeOrExpressionRewritermergesx = a OR x = binto the sameInPredicate.- The projection path,
calc-commonsTypeServices.IN_MULTI_COLUMN_TRANSFORMER_SERVICE(used byColumnTransformerBuilder.constructInColumnTransformer), already skips out-of-range INT32 values. It still casts every literal toLongLiteral, though, so decimals fail there too.
Not covered by existing work
- fixed the bug that caused incompatible behavior when comparing int64 or int32 column with double literals in query filter conditions #16917 only changed the comparison path; its diff and tests have no IN cases.
- Fix NULL predicate error in table model without TAG columns #17053 (open) handles
CAST(NULL ...)predicates. - Fix bug while using NOT IN predicate for INT64 type #13515 fixed
NOT INfor INT64 with integer literals in the projection path.
The tree model is not affected in the same way: it rejects these lists with "2.5" cannot be cast to [INT32] instead of returning rows.
How this was verified
- Standalone server: distribution built from master
cd31721c14. The files on this path are identical toe78f7366134e:ConvertPredicateToFilterVisitor, bothTypeServicesclasses,ColumnTransformerBuilder,NormalizeOrExpressionRewriter,ExpressionAnalyzer(apart from an unrelated UDF check), andPredicateUtils. - Unit level: calling
PredicateUtils.convertPredicateToFilterwith anInPredicateagainst thee78f7366134ejars givesmeasurements[0] IN [1, 2]fori IN (1, 4294967298), and aClassCastExceptionfor decimal lists on INT32 and INT64. - Environment: macOS (Darwin 25.6.0, arm64), JDK 27 for the server,
MEMORY_SIZE=2G.
Are you willing to submit a PR?
- I'm willing to submit a PR.
Activity