Skip to content

fix(batch): raise a clear error for unsupported property types - #2264

Open
steven-asearcy wants to merge 1 commit into
weaviate:mainfrom
steven-asearcy:fix/batch-unsupported-type-error
Open

steven-asearcy wants to merge 1 commit into
weaviate:mainfrom
steven-asearcy:fix/batch-unsupported-type-error

Conversation

@steven-asearcy

@steven-asearcy steven-asearcy commented Nov 18, 2026 •

Copy link
Copy Markdown

closes #2263

batch and insert_many passed types they don't handle (Decimal, numpy.float32, etc) straight into protobuf's Struct.update, which fails with ValueError: Unexpected type. now _serialize_primitive raises the same WeaviateInvalidInputError that data.insert already does.

tuples still work, they get turned into lists the same way Struct handled them before.

added unit tests in test/collection/test_batch.py. pytest test and pytest mock_tests pass, and so do the batch and collection integration tests against 1.40.5 using ci/start_weaviate.sh. pre-commit is clean on the changed files apart from pyright, which reports the same 226 errors on main.

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@steven-asearcy

Copy link
Copy Markdown
Author

I agree with the CLA

Comment on lines +147 to +150
@pytest.mark.parametrize("value", [Decimal("9.99"), [1.5, Decimal("2.5")], object()])
def test_serialize_primitive_raises_for_unsupported_type(value) -> None:
with pytest.raises(WeaviateInvalidInputError, match="Cannot serialize value of type"):
_serialize_primitive(value)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think these tests can pass while grpc_object accepts the same Decimal inside a list. They call _serialize_primitive directly, but __translate_properties_from_python_to_grpc picks a branch from a list's first element. [1.5, Decimal("2.5")] goes to the number-array branch, where struct.pack turns the Decimal into 2.5, so grpc_object accepts it, on main too.

Could we make this a parametrized test over _BatchGRPC.grpc_object, covering a scalar, a nested object, a Decimal first and second in a list, and a tuple of UUIDs? One integration test could then cover fixed_size, insert_many and stream().

what the integration test would check
  • fixed_size gets 99 good objects and one {"price": Decimal("9.99")}. __send_batch catches the error, so all 100 objects show up in failed_objects with the new message
  • insert_many with the same objects raises WeaviateInvalidInputError
  • stream() (1.36+) builds objects in the background loop thread via __generate_stream_requests. That thread stores the error and __check_bg_threads_alive re-raises it in the caller's thread. mock_tests/test_batch.py already drives stream() against a fake gRPC server, so this case could go there too

I didn't find an existing test that sends an unsupported value through any batch path (searched Decimal|Unexpected type|Cannot serialize in test/, mock_tests/ and integration/).

Comment on lines +379 to +381
if value is None or isinstance(value, (str, bool, int, float, dict)):
return value
raise WeaviateInvalidInputError(f"Cannot serialize value of type {type(value)} to Weaviate.")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could convert Decimal and numpy scalars like np.float32 instead of rejecting them. number is a float64 on the server, so a user who gets this error just writes float(d) and loses the same digits. Something like numbers.Integral -> int() and numbers.Real/Decimal -> float() would do it, still raising for non-finite values and everything else. It'd need to run before a list picks its array branch from the first element, and in __serialize_primitive in data/executor.py too so REST agrees.

If you'd rather keep this PR to the error, doing the conversion as a follow-up works for me too.

how the conversion would behave
  • float(Decimal("12345678901234567.89")) is 1.2345678901234568e+16, which is what the server stores either way
  • a float-first list already converts today. The number-array branch packs [1.5, Decimal("2.5")] with struct.pack, which calls __float__
  • np.float32 is numbers.Real and np.int64 is numbers.Integral, while Decimal is neither, so it needs its own check. bool is Integral too, so the conversion goes after the bool check
  • Decimal("1e400") becomes inf and Decimal("NaN") becomes nan, hence the non-finite check
  • [1, Decimal("2")] takes the int-array branch and raises protobuf's TypeError: 'decimal.Decimal' object cannot be interpreted as an integer, so the conversion has to run before the list's array branch is picked
  • WeaviateField in classes/types.py would need Decimal added for type checkers

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

batch insert fails the whole batch with "ValueError: Unexpected type" when a value is a Decimal

3 participants