You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Step Over, Step Into, not working when debugging Subprocesses #1876
When debugging a process in python, the program halts on breakpoints yet only the Continue button works. Pressing Step Over or Step into just Continues to the next breakpoint.
I have tried many launch.json configs and the only thing that resolved my issue was downgrading to v2024.10.0.
Can you share your application? This works fine for me with the latest bits.
DebugStepMultiproc.mp4
I used this code here:
importsubprocessimportsysimporttimedefact_busy():
# Simulate a busy operationprint("Subprocess is busy doing something...")
foriinrange(500):
print(f"Working... {i+1}/5")
sys.stdout.flush() # Ensure the output is flushed immediatelytime.sleep(1) # Simulate work with a sleepprint("Subprocess finished working.")
defrun_subprocess():
# Define the command to run as a subprocesscommand= ['python', __file__, 'act_busy']
try:
# Run the subprocess and wait for it to completeresult=subprocess.run(command, capture_output=True, text=True, check=True)
# Print the output of the subprocessprint(result.stdout)
# Print success messageprint("Subprocess completed successfully.")
exceptsubprocess.CalledProcessErrorase:
# Print error message if the subprocess failsprint(f"Subprocess failed with error: {e}")
if__name__=="__main__":
iflen(sys.argv) >1:
act_busy()
else:
run_subprocess()
This issue was closed because it has been stalled for 30 days with no activity. If the issue still persists, please reopen with the information requested. Thanks.
Sounds like this is related to sys monitoring support in Debugpy, which we switched to after 2024.10. Additionally, if it works with 3.11 or earlier (with the latest debugpy), that would also mean it's a sys monitoring issue.
Can confirm that downgrading to Python 3.11 works around this issue and 3.12+ is affected. Happy to test anything out or help in whatever way I can to try and fix this issue. Would love to be able to use the latest Python version on my projects that have subprocesses.
I've started digging into this a big more and I think I can see the cause behind it but not 100% sure. The underlying issues seems to be when the fork subprocess method is used (default for multiprocessing.Process on Linux) and not spawn.
When a stepping action is taken the pydevd_comm.internal_step_in_thread is called.
This assigns the stepping action to info.pydev_step_cmd.
When the new system.monitoring method is calling _pydevd_sys_monitoring._internal_line_event it attempts to get the current thread info which should contain the pydev_step_cmd action requested.
With a breakpoint on line 11 (a = 1) I pressed step over then continue for the main process, then step over when the forked process hit the breakpoint. The logs for these step over actions are
# Main process after pressing step over
0.00s - internal_step_in_thread
TID: 281473609240608
ThreadPyId: 281473602656240
Thread: <_MainThread(MainThread, started 281473609240608)>
InfoPyId: 281473595074368
InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffada5e740>
Info: State:2 Stop:None Cmd: 107 Kill:False
0.00s - _internal_line_event
Frame: <frame at 0xffffad465150, file 'subproc_test.py', line 12, code f>
Line: 12
State: 1
Cmd: 159
TID: 281473609240608
ThreadPyId: 281473588400816
Thread: <_pydevd_sys_monitoring._pydevd_sys_monitoring.ThreadInfo object at 0xffffad4012b0>
InfoPyId: 281473595074368
InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffada5e740>
Info: State:1 Stop:<frame at 0xffffad465150, file 'subproc_test.py', line 12, code f> Cmd: 159 Kill:False
# Forked process after pressing step over
0.00s - internal_step_in_thread
TID: 281473609240608
ThreadPyId: 281473602656240
Thread: <_MainThread(MainThread, started 281473609240608)>
InfoPyId: 281473588257088
InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffad3de140>
Info: State:2 Stop:None Cmd: 107 Kill:False
0.00s - _internal_line_event
Frame: <frame at 0xffffa6b6da40, file 'subproc_test.py', line 12, code f>
Line: 12
State: 1
Cmd: -1
TID: 281473609240608
ThreadPyId: 281473588400816
Thread: <_pydevd_sys_monitoring._pydevd_sys_monitoring.ThreadInfo object at 0xffffad4012b0>
InfoPyId: 281473595074368
InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffada5e740>
Info: State:1 Stop:None Cmd: -1 Kill:False
The forked process' additional_info in the _internal_line_event seems to be a different object to the one processed in internal_step_in_thread. This means it doesn't see the current stepping action and thus continues like Continue was pressed. Interestingly the object address for additional_info in _internal_line_event in Python is 0xffffada5e740 which is the same address as the additional_info in the parent process it was forked from. My guess is that there is some missing cleanup actions that needs to be done during the fork hook. As right now it seems to be sharing the same threading.local() value as the parent process rather than a new one. The internal_step_in_thread doesn't use threading.local() but looks up the thread based on the ID based on the PID and TID.
I'll try and spend a bit more time tomorrow to understand how this value is getting cached and not being cleared but if anyone knows more before I get to it I'll love any pointers.
Good point, it's not immediately clear what the exact equivalent is but I see some of the old code do info = threading.current_thread.additional_info rather than use threading.local().thread_info.additional_info. When I change the following
So the caching of the pre-fork thread data seems to be the problem and using threading.current_thread() to extract the additional_info solves this problem.
The questions I have are
Why does pydevd using the threading.local() TLS store and not threading.current_thread()
Is it better to change the code to use threading.current_thread() or figure out how to clear the TLS cache post fork
I know debugpy just vendor's pydevd but I'm not sure if you have any history behind the sys.monitoring implementation or not/insights on how this should be fixed.
Fabio Zadrozny (@fabioz) did most of the sys monitoring work, I just made some additions. But my understanding is that threading.local() is used so that you get a per thread copy of the data, although that seems unnecessary if the data is stored on the thread itself. In fact it seems like a bug (maybe the bug as you've pointed out)? As another thread can't write to threading.local(), so if you wanted to communicate stuff across threads, you wouldn't use it. I'm not sure how closely Fabio pays attention to debugpy though, but he may have more information.
Certainly using threading.current_thread().additional_info fixes the issue but the sys.monitoring implementation is definitely written with the TLS store threading.local() instead. Hopefully Fabio Zadrozny (@fabioz) can shed some light on why it was changed for that but in the meantime I'll try and see if I can figure out why the post fork setup isn't clearing the TLS or at least the old thread info is still present here. It's definitely the cause behind it and if I can figure out that it'll be a cleaner fix than converting all the sys.monitoring implementations over to a different method.
What is curious is that PyCharm has a similar implementation that uses threading.local() but doesn't seem to suffer from the same problem. I'll have a dig into their fork to see what might be different and see if that sheds some light on it.
Ok figured out why the state wasn't getting reset. The _pydevd_sys_monitoring module when imported in the original process set _thread_local_info = threading.local() creating a TLS for the threads in that process. When a fork occurs this copy of the module will be included in the forked process so the import is never run again and it just inherits the original value from the parent process.
Activity
rchiodo commentedon Mar 26, 2025
Can you share your application? This works fine for me with the latest bits.
DebugStepMultiproc.mp4
I used this code here:
github-actions commentedon Apr 26, 2025
This issue was closed because it has been stalled for 30 days with no activity. If the issue still persists, please reopen with the information requested. Thanks.
SobhanMP commentedon May 5, 2025
It's broken, see the following
rchiodo commentedon May 5, 2025
Seems to work fine for me. Although I'm on windows.
DebugMultiproc.mp4
rchiodo commentedon May 5, 2025
I can reproduce on Linux though.
rchiodo commentedon May 5, 2025
Strange but the subprocess example works fine on Linux. Not sure how multiprocess could affect stepping.
1 remaining item
rchiodo commentedon Jun 2, 2025
Sounds like this is related to sys monitoring support in Debugpy, which we switched to after 2024.10. Additionally, if it works with 3.11 or earlier (with the latest debugpy), that would also mean it's a sys monitoring issue.
jborean93 commentedon Jul 2, 2025
Can confirm that downgrading to Python 3.11 works around this issue and 3.12+ is affected. Happy to test anything out or help in whatever way I can to try and fix this issue. Would love to be able to use the latest Python version on my projects that have subprocesses.
jborean93 commentedon Jul 7, 2025
I've started digging into this a big more and I think I can see the cause behind it but not 100% sure. The underlying issues seems to be when the
forksubprocess method is used (default formultiprocessing.Processon Linux) and notspawn.When a stepping action is taken the
pydevd_comm.internal_step_in_threadis called.debugpy/src/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_comm.py
Lines 714 to 725 in 0d65353
This assigns the stepping action to
info.pydev_step_cmd.When the new
system.monitoringmethod is calling_pydevd_sys_monitoring._internal_line_eventit attempts to get the current thread info which should contain thepydev_step_cmdaction requested.debugpy/src/debugpy/_vendored/pydevd/_pydevd_sys_monitoring/_pydevd_sys_monitoring.py
Lines 1479 to 1486 in 0d65353
I've manually added some extra logging with the following under
internal_step_in_threadAnd the following under
_internal_line_eventI then ran the following code in VSCode
With a breakpoint on line 11 (
a = 1) I pressed step over then continue for the main process, then step over when the forked process hit the breakpoint. The logs for these step over actions areThe forked process'
additional_infoin the_internal_line_eventseems to be a different object to the one processed ininternal_step_in_thread. This means it doesn't see the current stepping action and thus continues like Continue was pressed. Interestingly the object address foradditional_infoin_internal_line_eventin Python is0xffffada5e740which is the same address as theadditional_infoin the parent process it was forked from. My guess is that there is some missing cleanup actions that needs to be done during theforkhook. As right now it seems to be sharing the samethreading.local()value as the parent process rather than a new one. Theinternal_step_in_threaddoesn't usethreading.local()but looks up the thread based on the ID based on the PID and TID.I'll try and spend a bit more time tomorrow to understand how this value is getting cached and not being cleared but if anyone knows more before I get to it I'll love any pointers.
rchiodo commentedon Jul 7, 2025
You might compare this with what happens during 3.11, as that seems to be working. The logic should be similar.
jborean93 commentedon Jul 7, 2025
Good point, it's not immediately clear what the exact equivalent is but I see some of the old code do
info = threading.current_thread.additional_inforather than usethreading.local().thread_info.additional_info. When I change the followingdebugpy/src/debugpy/_vendored/pydevd/_pydevd_sys_monitoring/_pydevd_sys_monitoring.py
Lines 1483 to 1484 in 0d65353
To
So the caching of the pre-fork thread data seems to be the problem and using
threading.current_thread()to extract theadditional_infosolves this problem.The questions I have are
threading.local()TLS store and notthreading.current_thread()threading.current_thread()or figure out how to clear the TLS cache post forkI know debugpy just vendor's pydevd but I'm not sure if you have any history behind the
sys.monitoringimplementation or not/insights on how this should be fixed.rchiodo commentedon Jul 7, 2025
Fabio Zadrozny (@fabioz) did most of the sys monitoring work, I just made some additions. But my understanding is that
threading.local()is used so that you get a per thread copy of the data, although that seems unnecessary if the data is stored on the thread itself. In fact it seems like a bug (maybe the bug as you've pointed out)? As another thread can't write tothreading.local(), so if you wanted to communicate stuff across threads, you wouldn't use it. I'm not sure how closely Fabio pays attention to debugpy though, but he may have more information.jborean93 commentedon Jul 7, 2025
Certainly using
threading.current_thread().additional_infofixes the issue but the sys.monitoring implementation is definitely written with the TLS storethreading.local()instead. Hopefully Fabio Zadrozny (@fabioz) can shed some light on why it was changed for that but in the meantime I'll try and see if I can figure out why the post fork setup isn't clearing the TLS or at least the old thread info is still present here. It's definitely the cause behind it and if I can figure out that it'll be a cleaner fix than converting all the sys.monitoring implementations over to a different method.What is curious is that PyCharm has a similar implementation that uses
threading.local()but doesn't seem to suffer from the same problem. I'll have a dig into their fork to see what might be different and see if that sheds some light on it.jborean93 commentedon Jul 8, 2025
Ok figured out why the state wasn't getting reset. The
_pydevd_sys_monitoringmodule when imported in the original process set_thread_local_info = threading.local()creating a TLS for the threads in that process. When a fork occurs this copy of the module will be included in the forked process so the import is never run again and it just inherits the original value from the parent process.The PR https://github.com/microsoft/debugpy/pull/1921/files adds a reset state similar to what is happening in
_pydevd_frame_evalbut for thesys.monitoringcode.rchiodo commentedon Jul 8, 2025
Nice! Thanks for looking at this.
fix(deps): update backend dependencies (patch) (patch)
Merge branch 'renovate/patch-backend-dependencies' into 'main'
chore(deps): update backend dependencies (patch) (patch)
Merge branch 'renovate/patch-backend-dependencies' into 'main'