Skip to content

Batch stream recovery closes the shared connection and can leave the client permanently closed #2139

Open

Description

@jfrancoa

Summary

The batch-stream recovery path closes the connection that is shared by every thread of the client and, when its 5 reconnect attempts fail, leaves the whole client permanently closed. After that:

  • every query raises WeaviateClosedClientError, and
  • every new collection.batch.stream() blocks for 60 seconds and then raises the misleading Batch stream was not started within 60 seconds. Please check your connection.

Nothing inside the client ever repairs this; the process needs a new client object. Observed on 4.23.0; the code is unchanged on current main.

How we hit it

A load generator (5 threads, each inserting through its own collection.batch.stream() context, sync client) ran against a 5-node Weaviate 1.39.0 cluster. One server node crashed mid-stream (server bug, crash-restarted). The client never recovered: it spent hours in the state above, while a freshly created client in the same pod connected and started a batch stream instantly over the same endpoints.

py-spy dump of the stuck process: all 5 worker threads sit in _start (sync.py) polling __all_threads_alive() until the 60 s timeout; no batch background threads and no connection event-loop thread exist any more.

The defects

All in weaviate/collections/batch/sync.py (permalinks to main @ 15e4aeb):

  1. __reconnect() closes the shared connection. sync.py#L466-L490 calls result(self.__connection.close("sync")) then self.__connection.connect(force=True). self.__connection is the client-wide connection: while one stream recovers, every other thread's operations fail with WeaviateClosedClientError.

  2. Concurrent recoveries are uncoordinated. Each active batch stream runs its own recv_wrapper (sync.py#L499-L522); a node crash hangs up several streams at once, so several threads interleave close("sync") / connect(force=True) on the same connection. One thread's close races another thread's just-completed reconnect.

  3. Permanent give-up. After 5 failed attempts __reconnect sets self.__bg_exception = e and returns (sync.py#L488-L490) — with the connection left closed. No later operation reopens it.

  4. _start hides the real error. sync.py#L123-L132 polls __all_threads_alive() for 60 s and never checks __bg_exception. On a closed connection the bg threads die immediately, so every batch.stream() enter burns 60 s and then reports a connection hint instead of the stored exception.

Reproduction sketch

  1. Cluster with ≥2 nodes, any collection.
  2. N threads, each looping with collection.batch.stream() as b: b.add_object(...) on one shared client.
  3. kill -9 the weaviate node the gRPC channel is connected to (crash, not graceful shutdown — graceful takes the shutting_down path instead).
  4. Depending on the race outcome the client either recovers or enters the permanent state above; repeat a few crashes to hit it. Once wedged: queries raise WeaviateClosedClientError, batch streams time out after 60 s each, forever.

Expected behavior

  • A batch stream's recovery must not tear down the connection other threads are using, or must coordinate so that close/connect pairs cannot interleave.
  • _start should raise __bg_exception as soon as the background threads die instead of waiting 60 s and blaming the network.
  • If recovery gives up, the client should either keep retrying with backoff or fail every subsequent call fast with the stored root cause — not a mix of 60 s timeouts and closed-client errors.

Environment

  • weaviate-python-client 4.23.0 (sync API), python 3.10
  • weaviate server 1.39.0, 5 nodes, gRPC through a TLS endpoint

Activity

  1. self-assigned this
    on Aug 22, 2026
  2. suryakvk commented on Nov 13, 2026

    @suryakvk

    Hi @tsmith023 @dirkkul, I'd like to work on defects 1–3 of this issue. #2141 already covers defect 4 (_start waiting 60 s before reporting the real error), so I'll stay out of _start / _wait and rebase onto it once it merges.

    I can reproduce the problem on main. In both batch/sync.py and batch/async_.py, __reconnect() calls connection.close(...) and then connection.connect(force=True) on the client-wide connection. That causes three problems:

    • close() marks the connection as closed and tears down the shared httpx client, so queries on other threads fail with WeaviateClosedClientError while one stream recovers.
    • When a node crashes, every active stream hangs up at the same moment. Each one runs its own close/connect, so one stream can close a connection another stream has just reopened.
    • If all 5 attempts fail, the connection is left closed and nothing reopens it.

    Proposed fix (private API only, no public signature or overload changes)

    1. connect/v4.py: add a private, coordinated gRPC-only reconnect. The connection gets a lock (threading.Lock for sync, asyncio.Lock for async) and an integer counter that goes up each time the gRPC channel is rebuilt. A stream calls the reconnect with the counter value it saw when its stream opened:
      • If the counter has changed since then, another stream has already reconnected, so the call does nothing.
      • Otherwise it closes and recreates only the gRPC channel and stub, then increments the counter.
      • The REST client and _connected are never touched.
    2. batch/sync.py and batch/async_.py: __recv records the counter when the stream opens, and __reconnect calls the new method instead of close() + connect(force=True). The existing node-health wait, backoff and re-adding of cached objects stay as they are.
    3. Giving up: after 5 failed attempts, still record the error in __bg_exception, but leave the client open. New queries and streams can then succeed or fail with the real cause.

    Tests

    • Unit tests for the coordinated reconnect:
      • two concurrent callers with the same counter value rebuild the channel exactly once;
      • a caller with an out-of-date counter value does nothing;
      • the REST side stays connected after a reconnect;
      • giving up does not close the client.
    • Async versions of the same tests, run concurrently with asyncio.gather.
    • A mock_tests/ test where the mock gRPC server aborts a stream partway through a batch with UNAVAILABLE "Socket closed". It asserts that every object arrives exactly once and that a query running at the same time still succeeds.

    Does this direction work for you? In particular, are you happy with adding a gRPC-only reconnect on the connection, or would you prefer the coordination to live in the batch layer? I'll start once you confirm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions