Repository navigation
Description
Summary
The batch-stream recovery path closes the connection that is shared by every thread of the client and, when its 5 reconnect attempts fail, leaves the whole client permanently closed. After that:
- every query raises
WeaviateClosedClientError, and - every new
collection.batch.stream()blocks for 60 seconds and then raises the misleadingBatch stream was not started within 60 seconds. Please check your connection.
Nothing inside the client ever repairs this; the process needs a new client object. Observed on 4.23.0; the code is unchanged on current main.
How we hit it
A load generator (5 threads, each inserting through its own collection.batch.stream() context, sync client) ran against a 5-node Weaviate 1.39.0 cluster. One server node crashed mid-stream (server bug, crash-restarted). The client never recovered: it spent hours in the state above, while a freshly created client in the same pod connected and started a batch stream instantly over the same endpoints.
py-spy dump of the stuck process: all 5 worker threads sit in _start (sync.py) polling __all_threads_alive() until the 60 s timeout; no batch background threads and no connection event-loop thread exist any more.
The defects
All in weaviate/collections/batch/sync.py (permalinks to main @ 15e4aeb):
-
__reconnect()closes the shared connection. sync.py#L466-L490 callsresult(self.__connection.close("sync"))thenself.__connection.connect(force=True).self.__connectionis the client-wide connection: while one stream recovers, every other thread's operations fail withWeaviateClosedClientError. -
Concurrent recoveries are uncoordinated. Each active batch stream runs its own
recv_wrapper(sync.py#L499-L522); a node crash hangs up several streams at once, so several threads interleaveclose("sync")/connect(force=True)on the same connection. One thread's close races another thread's just-completed reconnect. -
Permanent give-up. After 5 failed attempts
__reconnectsetsself.__bg_exception = eand returns (sync.py#L488-L490) — with the connection left closed. No later operation reopens it. -
_starthides the real error. sync.py#L123-L132 polls__all_threads_alive()for 60 s and never checks__bg_exception. On a closed connection the bg threads die immediately, so everybatch.stream()enter burns 60 s and then reports a connection hint instead of the stored exception.
Reproduction sketch
- Cluster with ≥2 nodes, any collection.
- N threads, each looping
with collection.batch.stream() as b: b.add_object(...)on one shared client. kill -9the weaviate node the gRPC channel is connected to (crash, not graceful shutdown — graceful takes theshutting_downpath instead).- Depending on the race outcome the client either recovers or enters the permanent state above; repeat a few crashes to hit it. Once wedged: queries raise
WeaviateClosedClientError, batch streams time out after 60 s each, forever.
Expected behavior
- A batch stream's recovery must not tear down the connection other threads are using, or must coordinate so that close/connect pairs cannot interleave.
_startshould raise__bg_exceptionas soon as the background threads die instead of waiting 60 s and blaming the network.- If recovery gives up, the client should either keep retrying with backoff or fail every subsequent call fast with the stored root cause — not a mix of 60 s timeouts and closed-client errors.
Environment
- weaviate-python-client 4.23.0 (sync API), python 3.10
- weaviate server 1.39.0, 5 nodes, gRPC through a TLS endpoint
Activity
fix(batch): raise the stored background error from sync _start instea…
fix(batch): surface background stream failures on exit instead of ret…
suryakvk commentedon Nov 13, 2026
Hi @tsmith023 @dirkkul, I'd like to work on defects 1–3 of this issue. #2141 already covers defect 4 (
_startwaiting 60 s before reporting the real error), so I'll stay out of_start/_waitand rebase onto it once it merges.I can reproduce the problem on
main. In bothbatch/sync.pyandbatch/async_.py,__reconnect()callsconnection.close(...)and thenconnection.connect(force=True)on the client-wide connection. That causes three problems:close()marks the connection as closed and tears down the shared httpx client, so queries on other threads fail withWeaviateClosedClientErrorwhile one stream recovers.Proposed fix (private API only, no public signature or overload changes)
connect/v4.py: add a private, coordinated gRPC-only reconnect. The connection gets a lock (threading.Lockfor sync,asyncio.Lockfor async) and an integer counter that goes up each time the gRPC channel is rebuilt. A stream calls the reconnect with the counter value it saw when its stream opened:_connectedare never touched.batch/sync.pyandbatch/async_.py:__recvrecords the counter when the stream opens, and__reconnectcalls the new method instead ofclose()+connect(force=True). The existing node-health wait, backoff and re-adding of cached objects stay as they are.__bg_exception, but leave the client open. New queries and streams can then succeed or fail with the real cause.Tests
asyncio.gather.mock_tests/test where the mock gRPC server aborts a stream partway through a batch withUNAVAILABLE "Socket closed". It asserts that every object arrives exactly once and that a query running at the same time still succeeds.Does this direction work for you? In particular, are you happy with adding a gRPC-only reconnect on the connection, or would you prefer the coordination to live in the batch layer? I'll start once you confirm.