[morgan-wowk]

Problem

The running queue visits container executions round-robin by last_processed_at. A pod that started or finished waits for every other active container execution to be visited first. The Kubernetes launchers already run pod and job informers, so the orchestrator could know right away which container executions changed.

Changes

• ContainerTaskLauncher.pop_changed_container_execution_ids() returns the container executions whose containers changed since the last call. The default returns [].
• Kubernetes launchers:
  • Launched pods carry common_annotations.CONTAINER_EXECUTION_ID_ANNOTATION_KEY, and so do job pod templates. The launcher's own pod_annotations dict is not mutated.
  • MultiNamespaceInformer takes an optional on_change callback. It is registered for ADDED, MODIFIED and DELETED after the informer's initial list, so startup does not mark every existing object.
  • The pod and job informers record each changed object's container execution id. Ids are recorded only from the first pop_changed_container_execution_ids() call on, so a launcher nobody asks keeps no ids.
  • _KubernetesPodOrJobLauncher returns the pod launcher's ids plus the job launcher's.
• internal_process_running_executions_queue() first moves the changed container executions that are still PENDING or RUNNING to the front of the queue. That is one UPDATE ... SET last_processed_at = 1970-01-01.

System impact

• Container executions whose pod or job changed are refreshed on the next sweep, not after a full round-robin cycle.
• Launchers without informers are unaffected.
• Pods and jobs launched before this change carry no id annotation and stay on the round-robin until they end.
• Ended container executions are never moved.

Production test

All three orchestrator PRs ran together in production (https://github.com/Shopify/oasis-backend/pull/843). That test patched the informer in; this PR builds the same behavior into the launchers. Before/after:

• Completion lag (container ended → status committed): p50 128s → 1.0s, p90 182s → 5.2s.
• Start detection (pod running → RUNNING committed): p50 81–250s → 4.8–14s.
• Container executions never seen as RUNNING before they ended: 34–80% → 0–0.4%.

Related PRs

• feat(orchestrator): Sweep again without waiting while work is overdue #426 and feat(orchestrator): Prefetch a finished container's log and outputs off the loop thread #427 are independent of this one. All three merge cleanly in any order, and the combination passes the test suite.
• feat(orchestrator): Prefetch a finished container's log and outputs off the loop thread #427 uses the same 1970-01-01 front-of-queue value and defines the same constant line.
• Deployments that wrap the launcher in a router or proxy need to delegate pop_changed_container_execution_ids(). Otherwise they get the default [].

Testing

• pytest tests: 593 passed.
• New tests/test_orchestrator_changed_container_executions.py covers:
  • a changed container execution is refreshed first
  • an ended one is left alone
  • the default launcher returns []
• New tests/launchers/kubernetes/test_kubernetes_launchers_changed_container_executions.py covers:
  • the pod and job template annotations
  • the launcher's annotations are not mutated
  • each change is reported once
  • objects without an id are ignored
  • nothing is recorded until asked
  • handlers are added after the initial list
  • no handlers are added without on_change

[morgan-wowk]

Interface change

Only the launcher knows which containers changed (its pod and job informers already see every change), while only the orchestrator decides which container executions to refresh next, so a single method on the launcher interface is the smallest way to pass that signal from one to the other. It defaults to [], so launchers that can't tell (Davies, local and Docker launchers, routers that don't delegate) keep the current round-robin with no changes, and it doesn't commit the interface to anything Kubernetes-specific.

[morgan-wowk]

Alternatives that leave ContainerTaskLauncher unchanged:

• Orchestrator kwarg: OrchestratorService_Sql takes an optional function that returns the changed ids, and the deployment passes in the Kubernetes launcher's method.
• Push callback: Kubernetes launchers take an on_container_execution_changed callback that feeds a set the orchestrator owns.
• Optional capability: a separate Protocol, or a getattr check, that only Kubernetes launchers implement.
• Launcher writes to the DB: the informer moves the row to the front itself, but that breaks layering because launchers have no DB session.
• Downstream patch: keep it as a deployment-side monkeypatch, which is fragile because it depends on informer internals.

Why an interface addition?

It's the only option where the launcher reports changes and the orchestrator consumes them through the same typed contract as the rest of the launcher API. The default [] means no other launcher or deployment needs any change, while the alternatives either spread the wiring across each deployment or hide the same method behind a looser contract.

[morgan-wowk]

@Ark-kun Do you want any alternative?