[0z5a]

torchrun sets OMP_NUM_THREADS=1 for every process when --nproc_per_node is larger than 1, unless the variable is already set. DeepSpeedCPUAdam then steps on a single OpenMP thread, and nothing in the log says so. The deepspeed launcher sets the thread count only with --bind_cores_to_rank, which is the path the ZeRO-Offload tutorial documents. Users who launch with torchrun or accelerate get no hint.

Measured effect

Setup: ZeRO-1 with offload_optimizer: cpu, Qwen3-4B in bf16, fp32 gradient accumulation, one GH200 on a 72-core host.

OMP_NUM_THREADS	optimizer step
1	5.6 s
36	0.8 s

That is 7× faster. In a 2-GPU torchrun job on L40S with the default setting, the same step took 6.2 s.

Change

• single_thread_warning() in deepspeed/ops/adam/cpu_adam.py returns a message when OMP_NUM_THREADS is "1" and the rank has at least two cores (CPU affinity divided by LOCAL_WORLD_SIZE).
• DeepSpeedCPUAdam.__init__ logs it once per process, for the first optimizer instance.
• The CPU Adam tuning section of the ZeRO-Offload tutorial gets a note for torchrun users.

It only warns and does not change the thread count:

• an explicit OMP_NUM_THREADS=1 may be intentional;
• the OpenMP runtime may already have read the variable by the time the optimizer is built.

Tests

tests/unit/ops/adam/test_cpu_adam_threads.py is CPU only and does not build the op. It checks four cases:

• warns at the torchrun default when the rank has spare cores;
• silent when the variable is unset or larger than 1;
• silent when the rank has a single core;
• a single process counts all its cores.

yapf, flake8, codespell and the license check pass.