Skip to content

MultiBroker starts and shuts down a shared result backend N+1 times #726

Open

Description

@zakharkiv-vitalik

Taskiq version

master (1818f53)

What happened?

MultiBroker.with_result_backend() assigns the same backend instance to the MultiBroker and to every inner broker:

self.result_backend = result_backend
for inner_broker in self.brokers:
inner_broker.result_backend = result_backend

MultiBroker.startup() / shutdown() call super() and then every inner broker:

await super().startup()
for inner_broker in self.brokers:
await inner_broker.startup()

for inner_broker in self.brokers:
try:
await inner_broker.shutdown()
except Exception as exc:
logger.warning(
"Cannot shutdown broker %s: %s",
inner_broker,
exc,
exc_info=True,
)
await super().shutdown()

and each AsyncBroker starts / stops its own result_backend:

await self.result_backend.startup()

await self.result_backend.shutdown()

So with N inner brokers the same backend object gets N+1 startup() and N+1 shutdown() calls. Backends that open resources in startup() (connection pools, migrations, etc.) leak them, and the repeated shutdown() can fail.

Reproduction

A backend that behaves like a typical DB backend — opens a pool in startup(), closes it in shutdown():

import asyncio

from taskiq import InMemoryBroker, MultiBroker
from taskiq.result_backends.dummy import DummyResultBackend


class Pool:
    created = 0

    def __init__(self) -> None:
        Pool.created += 1
        self.id = Pool.created
        self.closed = False
        print(f"  pool #{self.id} opened")

    async def close(self) -> None:
        if self.closed:
            raise RuntimeError(f"pool #{self.id} already closed")
        self.closed = True
        print(f"  pool #{self.id} closed")


class PoolBackend(DummyResultBackend):
    async def startup(self) -> None:
        print("startup() called")
        self.pool = Pool()

    async def shutdown(self) -> None:
        print("shutdown() called")
        await self.pool.close()


async def main() -> None:
    broker = MultiBroker(
        routing={"fast": InMemoryBroker(), "slow": InMemoryBroker()},
        default_routing_label="fast",
    ).with_result_backend(PoolBackend())
    await broker.startup()
    try:
        await broker.shutdown()
    except RuntimeError as e:
        print("!!", e)
    print("pools opened:", Pool.created)


asyncio.run(main())

Output:

startup() called
  pool #1 opened
startup() called
  pool #2 opened        <- pool #1 is lost, nobody will close it
startup() called
  pool #3 opened        <- pool #2 is lost too
shutdown() called
  pool #3 closed
shutdown() called       <- closes the already closed pool #3
Cannot shutdown broker <InMemoryBroker ...>: pool #3 already closed
shutdown() called
!! pool #3 already closed   <- raised from broker.shutdown()
pools opened: 3
  1. Pools Сделать AsyncTaskiqDecoratedTask метаклассом #1 and Make worker CLI #2 are never closed. With e.g. --workers 4 and two inner brokers that's 8 extra pools.
  2. The second and third shutdown() hit an already closed pool. Failures of inner brokers are logged, but the last one, from MultiBroker's own super().shutdown(), propagates to the caller.
  3. Side effects in startup() (e.g. running migrations) run N+1 times.

The existing tests don't catch this because InmemoryResultBackend / DummyResultBackend have no-op startup() / shutdown().

Why the propagation exists, and why it's no longer needed

The propagation comes from the first version of #723, where tasks were registered on inner brokers (@first.task) and find_task() aggregated their registries. Then the client polled the inner broker's backend while the worker saved into the MultiBroker's backend, so they had to be the same object — as stated in the PR description:

with_result_backend() propagates the backend to inner brokers to keep client-side polling and worker-side saving in sync.

During review the design changed (review comment, 4d166eb): tasks are registered only on the MultiBroker, and registration on inner brokers is disabled:

inner_broker.task = _task_registration_disabled # type: ignore[method-assign]

Now every reader of the result backend goes through the MultiBroker:

  • kicker — self.broker is the MultiBroker the task is registered on:
    result_backend=self.broker.result_backend,
  • receiver:
    await self.broker.result_backend.set_result(taskiq_msg.task_id, result)
  • progress tracker:
    await self.context.broker.result_backend.set_progress(
    self.context.message.task_id,
    progress,
    )
    async def get_progress(self) -> TaskProgress[_ProgressType] | None:
    """Get progress."""
    return await self.context.broker.result_backend.get_progress(

Inner brokers only publish messages in kick(). The only place their result_backend is used is their own startup() / shutdown() — which is exactly what causes this bug.

Proposed fix

Drop the with_result_backend() override in MultiBroker. The base AsyncBroker.with_result_backend() already sets the backend on the MultiBroker only, so its lifecycle is managed exactly once. This also requires updating test_with_result_backend_propagates_to_inner_brokers and the note in docs/available-components/brokers.md.

I will open a PR with the fix and tests.

Side note

The same N+1 lifecycle happens without MultiBroker if a user shares one backend instance between several regular brokers, since every AsyncBroker assumes it owns its result_backend. That's a broader question (e.g. an explicit ownership flag) and is out of scope here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions