Repository navigation
fix(batch): raise a clear error for unsupported property types - #2264
steven-asearcy wants to merge 1 commit into
Conversation
|
To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge. |
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| SAST | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
|
I agree with the CLA |
| @pytest.mark.parametrize("value", [Decimal("9.99"), [1.5, Decimal("2.5")], object()]) | ||
| def test_serialize_primitive_raises_for_unsupported_type(value) -> None: | ||
| with pytest.raises(WeaviateInvalidInputError, match="Cannot serialize value of type"): | ||
| _serialize_primitive(value) |
There was a problem hiding this comment.
I think these tests can pass while grpc_object accepts the same Decimal inside a list. They call _serialize_primitive directly, but __translate_properties_from_python_to_grpc picks a branch from a list's first element. [1.5, Decimal("2.5")] goes to the number-array branch, where struct.pack turns the Decimal into 2.5, so grpc_object accepts it, on main too.
Could we make this a parametrized test over _BatchGRPC.grpc_object, covering a scalar, a nested object, a Decimal first and second in a list, and a tuple of UUIDs? One integration test could then cover fixed_size, insert_many and stream().
what the integration test would check
fixed_sizegets 99 good objects and one{"price": Decimal("9.99")}.__send_batchcatches the error, so all 100 objects show up infailed_objectswith the new messageinsert_manywith the same objects raisesWeaviateInvalidInputErrorstream()(1.36+) builds objects in the background loop thread via__generate_stream_requests. That thread stores the error and__check_bg_threads_alivere-raises it in the caller's thread.mock_tests/test_batch.pyalready drivesstream()against a fake gRPC server, so this case could go there too
I didn't find an existing test that sends an unsupported value through any batch path (searched Decimal|Unexpected type|Cannot serialize in test/, mock_tests/ and integration/).
| if value is None or isinstance(value, (str, bool, int, float, dict)): | ||
| return value | ||
| raise WeaviateInvalidInputError(f"Cannot serialize value of type {type(value)} to Weaviate.") |
There was a problem hiding this comment.
I think we could convert Decimal and numpy scalars like np.float32 instead of rejecting them. number is a float64 on the server, so a user who gets this error just writes float(d) and loses the same digits. Something like numbers.Integral -> int() and numbers.Real/Decimal -> float() would do it, still raising for non-finite values and everything else. It'd need to run before a list picks its array branch from the first element, and in __serialize_primitive in data/executor.py too so REST agrees.
If you'd rather keep this PR to the error, doing the conversion as a follow-up works for me too.
how the conversion would behave
float(Decimal("12345678901234567.89"))is1.2345678901234568e+16, which is what the server stores either way- a float-first list already converts today. The number-array branch packs
[1.5, Decimal("2.5")]withstruct.pack, which calls__float__ np.float32isnumbers.Realandnp.int64isnumbers.Integral, whileDecimalis neither, so it needs its own check.boolisIntegraltoo, so the conversion goes after theboolcheckDecimal("1e400")becomesinfandDecimal("NaN")becomesnan, hence the non-finite check[1, Decimal("2")]takes the int-array branch and raises protobuf'sTypeError: 'decimal.Decimal' object cannot be interpreted as an integer, so the conversion has to run before the list's array branch is pickedWeaviateFieldinclasses/types.pywould needDecimaladded for type checkers
closes #2263
batch and
insert_manypassed types they don't handle (Decimal, numpy.float32, etc) straight into protobuf'sStruct.update, which fails withValueError: Unexpected type. now_serialize_primitiveraises the sameWeaviateInvalidInputErrorthatdata.insertalready does.tuples still work, they get turned into lists the same way
Structhandled them before.added unit tests in
test/collection/test_batch.py.pytest testandpytest mock_testspass, and so do the batch and collection integration tests against 1.40.5 usingci/start_weaviate.sh. pre-commit is clean on the changed files apart from pyright, which reports the same 226 errors on main.