Repository navigation
refactor(remote): make replay and query ports async, drop actor - #4157
Conversation
|
| .begin_replay_attempt(event_id, observed_at) | ||
| .map_err(RemoteReplayApplicationErrorV1::Persistence)?; | ||
| let operation_error = match operation(replay_attempt) { | ||
| let operation_error = match operation(replay_attempt).await { |
There was a problem hiding this comment.
🔴 Cancelled replay strands its attempt
If a caller drops replay during the transaction await, with_replay_attempt skips cleanup. The open spool marker rejects subsequent replays until startup recovery.
Learn more
A replay attempt writes a durable open marker before it submits the project-store transaction. The async transaction can yield, so dropping a direct caller's future at that point drops with_replay_attempt without calling abandon_replay_attempt. Later begin_replay_attempt rejects the still-open marker, even if the project write completed. Only recover_interrupted_replay_attempts releases it on a later recovery pass. HTTP's detached blocking-pool execution keeps that particular caller alive, but the exported async service has no such ownership guarantee.
Example: A client polls service.replay until its project-store commit is pending, then cancels the task. A retry of the same event gets Corruption rather than a duplicate receipt, until the node runs startup recovery.
Recommended fix: Give the attempt operation independent settlement ownership before opening its marker, or install a cancellation-safe cleanup guard that abandons the exact attempt on drop without racing a still-running commit. Test dropping the future while commit is pending, then retry the event without restarting the store.
Was this helpful? React with 👍 or 👎 to provide feedback.
| tracedecay_rusqlite_runtime::native_runtime::on_blocking_pool(async move { | ||
| service | ||
| .execute_controlled(request.request, credential, control) | ||
| .await | ||
| }), |
There was a problem hiding this comment.
🔴 Pending remote requests occupy blocking threads
When project-store dispatches await, on_blocking_pool parks one thread per remote request. Enough pending requests exhaust the bounded pool, delaying unrelated remote operations.
Learn more
The route moves the entire async protocol future onto one blocking-pool thread. on_blocking_pool parks that thread whenever the future yields. Replay, query, and promotion now await project-store operations inside that future, so they retain a thread during reader admission or writer waits rather than freeing it. The pool has a 512-thread ceiling, shared with other synchronous storage work.
Example: If 512 remote queries wait on project-store reads, each holds a parked pool thread. Another remote request cannot begin its synchronous credential or storage work until one of those queries finishes.
Recommended fix: Isolate only synchronous exact-SQL segments on the blocking pool and poll project-store awaits on an async executor, while preserving detached settlement of accepted writes. Cover many pending project-store requests with an independent synchronous operation to demonstrate that the blocking pool remains available.
Was this helpful? React with 👍 or 👎 to provide feedback.
Step 5 of the storage redesign (
docs/plans/storage/README.md).The
tracedecay-remote-replayactor thread is deleted. The remote replay port and the remote exact-observation query port are now async, so neither parks a thread on the writer or on reader admission, and nonative_runtime::block_onremains on those paths. Step 1 (#4118) left the remote query port synchronous and bridged it withblock_on; this removes that bridge. The change cascades through the remote protocol trait family and its implementations.Test plan
bazel build --config=clippyon the touched cratesrustfmt_check_2024on the touched crates