Repository navigation
[Bug]: Single threaded executor causes dead-lock #12
Description
Expected Behavior
When running a durable handler I expect this work with a single threaded executor.
public class SimpleStepExample extends DurableHandler<GreetingRequest, String> {
@Override
protected DurableConfig createConfiguration() {
return DurableConfig.builder()
.withExecutorService(Executors.newSingleThreadExecutor())
.build();
}
@Override
public String handleRequest(GreetingRequest input, DurableContext context) {
// Step 1: Create greeting
var greeting = context.step("create-greeting", String.class, () -> "Hello, " + input.getName());
// Step 2: Transform to uppercase
var uppercase = context.step("to-uppercase", String.class, () -> greeting.toUpperCase());
// Step 3: Add punctuation
var result = context.step("add-punctuation", String.class, () -> uppercase + "!");
return result;
}
}Actual Behavior
What happens instead is that we time out and crash the local SAM container.
SAM_CONTAINER_ID: 96aade58172b46e5918bb97d4eee4dfb67119f45131a41f6fb267e892a0e2ef2
START RequestId: 78ee11b6-3485-4d2e-8c74-4cdbd837b60a Version: $LATEST
2026-01-20 14:00:30.689 [main] DEBUG com.amazonaws.lambda.durable.DurableConfig - Creating default DurableExecutionClient
2026-01-20 14:00:30.987 [main] DEBUG com.amazonaws.lambda.durable.DurableConfig - Default DurableExecutionClient created for region: us-east-1
2026-01-20 14:00:30.988 [main] DEBUG com.amazonaws.lambda.durable.DurableHandler - Raw input from durable handler: {"DurableExecutionArn": "f73c74f2-71c3-44f4-ad46-e1e5390cdf91", "CheckpointToken": "eyJhcm4iOiJmNzNjNzRmMi03MWMzLTQ0ZjQtYWQ0Ni1lMWU1MzkwY2RmOTEiLCJzZXEiOjF9", "InitialExecutionState": {"Operations": [{"Id": "1be5b827-c93e-4014-b5b4-82936ab1c47c", "Type": "EXECUTION", "Status": "STARTED", "StartTimestamp": "2026-01-20 14:00:30.481512+00:00", "ExecutionDetails": {"InputPayload": "{}"}}], "NextMarker": ""}}
2026-01-20 14:00:31.025 [main] DEBUG com.amazonaws.lambda.durable.DurableExecutor - DurableExecution.execute() called
2026-01-20 14:00:31.025 [main] DEBUG com.amazonaws.lambda.durable.DurableExecutor - DurableExecutionArn: f73c74f2-71c3-44f4-ad46-e1e5390cdf91
2026-01-20 14:00:31.025 [main] DEBUG com.amazonaws.lambda.durable.DurableExecutor - CheckpointToken: eyJhcm4iOiJmNzNjNzRmMi03MWMzLTQ0ZjQtYWQ0Ni1lMWU1MzkwY2RmOTEiLCJzZXEiOjF9
2026-01-20 14:00:31.025 [main] DEBUG com.amazonaws.lambda.durable.DurableExecutor - Initial operations count: 1
2026-01-20 14:00:31.026 [main] DEBUG com.amazonaws.lambda.durable.DurableExecutor - EXECUTION operation found: 1be5b827-c93e-4014-b5b4-82936ab1c47c
2026-01-20 14:00:31.027 [pool-1-thread-1] DEBUG com.amazonaws.lambda.durable.execution.ExecutionManager - Registered thread 'pool-1-thread-1' as active. Active threads: 1
2026-01-20 14:00:31.029 [pool-1-thread-1] DEBUG com.amazonaws.lambda.durable.execution.ExecutionManager - Started phaser for operation '1'
2026-01-20 14:00:31.029 [pool-1-thread-1] DEBUG com.amazonaws.lambda.durable.execution.ExecutionManager - Registered thread '1-step' as active. Active threads: 2
2026-01-20 14:00:31.030 [pool-1-thread-1] DEBUG com.amazonaws.lambda.durable.operation.StepOperation - StepOperation.get() attempting to deregister thread: pool-1-thread-1
2026-01-20 14:00:31.030 [pool-1-thread-1] DEBUG com.amazonaws.lambda.durable.execution.ExecutionManager - Deregistered thread 'pool-1-thread-1'. Active threads: 1
2026-01-20 14:00:31.030 [pool-1-thread-1] DEBUG com.amazonaws.lambda.durable.operation.StepOperation - Waiting for operation to finish 1 (Phaser: java.util.concurrent.Phaser@575e2117[phase = 0 parties = 2 arrived = 0])
START RequestId: c7ce3322-8a9a-4196-acaa-0fd9edbbf28c Version: $LATEST
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x3289ac]
goroutine 69 [running]:
github.com/aws/aws-lambda-runtime-interface-emulator/internal/lambda/rapidcore.(*Server).Invoke.func2()
/LambdaRuntimeLocal/internal/lambda/rapidcore/server.go:666 +0xac
created by github.com/aws/aws-lambda-runtime-interface-emulator/internal/lambda/rapidcore.(*Server).Invoke in goroutine 66
/LambdaRuntimeLocal/internal/lambda/rapidcore/server.go:649 +0x17c
Steps to Reproduce
Add this config to any of the examples in the examples folder.
@Override
protected DurableConfig createConfiguration() {
return DurableConfig.builder()
.withExecutorService(Executors.newSingleThreadExecutor())
.build();
}SDK Version
0.5.0-SNAPSHOT
Java Version
17
Is this a regression?
No
Last Working Version
No response
Additional Context
No response
Activity
zhongkechen commentedon Jan 27, 2026
Is this a bug or a requirement (the executor requires more than 1 thread to complete the futures)?
phipag commentedon Jan 28, 2026
It is a good question whether we need to frame this as a requirement. Today, the
ExecutionManagershares the "managedExecutor" with the one that the user configures. At the same time, theCheckpointBatcherhas its own executor to implement the micro-batching strategy in the background for submitting the operation updates to the DAR backend.My question / hypothesis is: What if we configure a shared thread-pool for the ExecutionManager and the CheckpointBatcher that the user cannot overwrite and treat the
managedExecutorthat the user can configure only forctx.step, ... operations?My belief is that this would resolve the dead-lock since the the ExecutionManager resolving futures and advancing the phaser would have its own executor where we keep control that mis-configurations like this cannot happen.
aws-durable-execution-sdk-java/sdk/src/main/java/com/amazonaws/lambda/durable/execution/ExecutionManager.java
Lines 80 to 87 in d06b9f6
What do you think @maschnetwork @zhongkechen ?
phipag commentedon Jan 30, 2026
As discussed, this will be resolved as part of the refactor described in #31 .
phipag commentedon Feb 3, 2026
Tested locally. Resolved by #31