Skip to content

fix(batch): report a per-object encoding failure instead of failing the whole batch - #2267

Open
shadowmodder wants to merge 4 commits into
weaviate:mainfrom
shadowmodder:shadowmodder/fix/batch-per-object-encoding-errors
Open

shadowmodder wants to merge 4 commits into
weaviate:mainfrom
shadowmodder:shadowmodder/fix/batch-per-object-encoding-errors

Conversation

@shadowmodder

Copy link
Copy Markdown
Contributor

closes #2266

Problem

One object that fails client-side encoding fails every object batched with it, and the error does not say which object it was:

  • _BatchGRPC.grpc_objects built the protobuf message for the whole batch in one list comprehension, so one bad object aborted the request before anything was sent, and __send_batch attached that one error to all 100 objects.
  • In batch.stream(), __generate_stream_requests encoded each object inside the background send loop with no per-object catch, so the error killed the loop and every unsent object was silently dropped with an empty failed_objects.

On current main a Decimal property value reaches the same failure in both modes (the ragged multi-vector trigger arrives with #2039).

Fix

Encode objects one by one and record a failure as that object's own ErrorObject:

  • grpc_batch.py: a new _BatchGRPC.__encode_objects helper encodes per object and collects an error message (naming the object's UUID and the underlying error) keyed by the object's position. objects() sends only the encodable objects; the response callback translates the server's error indices — which are keyed by position within the request — back to the caller's positions past any dropped object, and merges them with the encoding errors. If every object fails, it raises WeaviateInsertManyAllFailedError as before (without sending an empty request). grpc_objects itself is unchanged.
  • sync.py / async_.py: __generate_stream_requests catches a per-object encoding exception, records it via a new __record_encoding_error (an ErrorObject keyed by obj.index in the results and in failed_objects; the object is removed from the uuid cache/lookup so references to it are not sent), and continues with the remaining objects.

Out of scope, as noted in the issue: the stream context manager's __exit__ not re-raising the loop's exception is a separate defect. The error message carries the UUID and the underlying error; it does not thread the failing vector's name through, because encoding failures also arise from properties, where no vector name exists.

Validation

Repro (offline, stub connection / white-box stream generation, Decimal property as the unencodable object on main):

  • Before: objects() raised ValueError: Unexpected type — the whole batch of 100 was lost, the server received nothing.
  • After: server received 99, uuids 99, errors 1 (the bad object's index, message naming its UUID).

Tests (5 new deterministic, server-free regression tests in mock_tests/test_batch.py):

  • test_grpc_objects_one_unencodable_object_does_not_fail_the_batch
  • test_grpc_objects_server_error_is_translated_past_an_encoding_failure
  • test_grpc_objects_all_unencodable_raises_all_failed
  • test_sync_stream_skips_unencodable_object_and_records_error
  • test_async_stream_skips_unencodable_object_and_records_error

All 5 fail with the fix reverted and pass with it. test/collection/test_batch.py: 16 passed. ruff check and ruff format clean on the changed files. The mock-server-backed tests in mock_tests/test_batch.py could not run in my sandbox (their gRPC mock fixture fails at setup identically on unpatched main); CI will exercise them.

AI assistance was used during the creation of this PR.

Signed-off-by: shadowmodder <sudhirvissa@gmail.com>
Signed-off-by: shadowmodder <sudhirvissa@gmail.com>
Signed-off-by: shadowmodder <sudhirvissa@gmail.com>
Signed-off-by: shadowmodder <sudhirvissa@gmail.com>

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

@shadowmodder

Copy link
Copy Markdown
Contributor Author

I agree with the CLA

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

collections/batch: one object that fails client-side encoding fails every object sent with it

2 participants