Repository navigation
Description
Taskiq version
master (1818f53)
What happened?
MultiBroker.with_result_backend() assigns the same backend instance to the MultiBroker and to every inner broker:
taskiq/taskiq/brokers/multi_broker.py
Lines 262 to 264 in 1818f53
| self.result_backend = result_backend | |
| for inner_broker in self.brokers: | |
| inner_broker.result_backend = result_backend |
MultiBroker.startup() / shutdown() call super() and then every inner broker:
taskiq/taskiq/brokers/multi_broker.py
Lines 224 to 226 in 1818f53
| await super().startup() | |
| for inner_broker in self.brokers: | |
| await inner_broker.startup() |
taskiq/taskiq/brokers/multi_broker.py
Lines 235 to 245 in 1818f53
| for inner_broker in self.brokers: | |
| try: | |
| await inner_broker.shutdown() | |
| except Exception as exc: | |
| logger.warning( | |
| "Cannot shutdown broker %s: %s", | |
| inner_broker, | |
| exc, | |
| exc_info=True, | |
| ) | |
| await super().shutdown() |
and each AsyncBroker starts / stops its own result_backend:
Line 200 in 1818f53
| await self.result_backend.startup() |
Line 221 in 1818f53
| await self.result_backend.shutdown() |
So with N inner brokers the same backend object gets N+1 startup() and N+1 shutdown() calls. Backends that open resources in startup() (connection pools, migrations, etc.) leak them, and the repeated shutdown() can fail.
Reproduction
A backend that behaves like a typical DB backend — opens a pool in startup(), closes it in shutdown():
import asyncio
from taskiq import InMemoryBroker, MultiBroker
from taskiq.result_backends.dummy import DummyResultBackend
class Pool:
created = 0
def __init__(self) -> None:
Pool.created += 1
self.id = Pool.created
self.closed = False
print(f" pool #{self.id} opened")
async def close(self) -> None:
if self.closed:
raise RuntimeError(f"pool #{self.id} already closed")
self.closed = True
print(f" pool #{self.id} closed")
class PoolBackend(DummyResultBackend):
async def startup(self) -> None:
print("startup() called")
self.pool = Pool()
async def shutdown(self) -> None:
print("shutdown() called")
await self.pool.close()
async def main() -> None:
broker = MultiBroker(
routing={"fast": InMemoryBroker(), "slow": InMemoryBroker()},
default_routing_label="fast",
).with_result_backend(PoolBackend())
await broker.startup()
try:
await broker.shutdown()
except RuntimeError as e:
print("!!", e)
print("pools opened:", Pool.created)
asyncio.run(main())Output:
startup() called
pool #1 opened
startup() called
pool #2 opened <- pool #1 is lost, nobody will close it
startup() called
pool #3 opened <- pool #2 is lost too
shutdown() called
pool #3 closed
shutdown() called <- closes the already closed pool #3
Cannot shutdown broker <InMemoryBroker ...>: pool #3 already closed
shutdown() called
!! pool #3 already closed <- raised from broker.shutdown()
pools opened: 3
- Pools Сделать AsyncTaskiqDecoratedTask метаклассом #1 and Make worker CLI #2 are never closed. With e.g.
--workers 4and two inner brokers that's 8 extra pools. - The second and third
shutdown()hit an already closed pool. Failures of inner brokers are logged, but the last one, fromMultiBroker's ownsuper().shutdown(), propagates to the caller. - Side effects in
startup()(e.g. running migrations) run N+1 times.
The existing tests don't catch this because InmemoryResultBackend / DummyResultBackend have no-op startup() / shutdown().
Why the propagation exists, and why it's no longer needed
The propagation comes from the first version of #723, where tasks were registered on inner brokers (@first.task) and find_task() aggregated their registries. Then the client polled the inner broker's backend while the worker saved into the MultiBroker's backend, so they had to be the same object — as stated in the PR description:
with_result_backend()propagates the backend to inner brokers to keep client-side polling and worker-side saving in sync.
During review the design changed (review comment, 4d166eb): tasks are registered only on the MultiBroker, and registration on inner brokers is disabled:
taskiq/taskiq/brokers/multi_broker.py
Line 108 in 1818f53
| inner_broker.task = _task_registration_disabled # type: ignore[method-assign] |
Now every reader of the result backend goes through the MultiBroker:
- kicker —
self.brokeris theMultiBrokerthe task is registered on:Line 176 in 1818f53
result_backend=self.broker.result_backend, - receiver:
taskiq/taskiq/receiver/receiver.py
Line 222 in 1818f53
await self.broker.result_backend.set_result(taskiq_msg.task_id, result) - progress tracker:
taskiq/taskiq/depends/progress_tracker.py
Lines 60 to 67 in 1818f53
await self.context.broker.result_backend.set_progress( self.context.message.task_id, progress, ) async def get_progress(self) -> TaskProgress[_ProgressType] | None: """Get progress.""" return await self.context.broker.result_backend.get_progress(
Inner brokers only publish messages in kick(). The only place their result_backend is used is their own startup() / shutdown() — which is exactly what causes this bug.
Proposed fix
Drop the with_result_backend() override in MultiBroker. The base AsyncBroker.with_result_backend() already sets the backend on the MultiBroker only, so its lifecycle is managed exactly once. This also requires updating test_with_result_backend_propagates_to_inner_brokers and the note in docs/available-components/brokers.md.
I will open a PR with the fix and tests.
Side note
The same N+1 lifecycle happens without MultiBroker if a user shares one backend instance between several regular brokers, since every AsyncBroker assumes it owns its result_backend. That's a broader question (e.g. an explicit ownership flag) and is out of scope here.
Activity
fix: start and stop MultiBroker result backend once (taskiq-python#726)