Skip to content

Step Over, Step Into, not working when debugging Subprocesses #1876

Closed

Description

When debugging a process in python, the program halts on breakpoints yet only the Continue button works. Pressing Step Over or Step into just Continues to the next breakpoint.

I have tried many launch.json configs and the only thing that resolved my issue was downgrading to v2024.10.0.

here is my config just in case:

    {
        "name": "Python Debugger: Current File",
        "type": "debugpy",
        "request": "launch",
        "program": "app/main.py",
        "console": "integratedTerminal",
        "env": {
            "PYTHONPATH": "${workspaceFolder}"
        },
        "justMyCode": true,
        "subProcess": true,
        "steppingResumesAllThreads": false
    },

Activity

  1. rchiodo commented on Mar 26, 2025

    @rchiodo
    Contributor

    Can you share your application? This works fine for me with the latest bits.

    DebugStepMultiproc.mp4

    I used this code here:

    import subprocess
    import sys
    import time
    
    def act_busy():
        # Simulate a busy operation
        print("Subprocess is busy doing something...")
        for i in range(500):
            print(f"Working... {i + 1}/5")
            sys.stdout.flush()  # Ensure the output is flushed immediately
            time.sleep(1)  # Simulate work with a sleep
    
        print("Subprocess finished working.")
    
    def run_subprocess():
        # Define the command to run as a subprocess
        command = ['python', __file__, 'act_busy']
    
        try:
            # Run the subprocess and wait for it to complete
            result = subprocess.run(command, capture_output=True, text=True, check=True)
            
            # Print the output of the subprocess
            print(result.stdout)
            
            # Print success message
            print("Subprocess completed successfully.")
        except subprocess.CalledProcessError as e:
            # Print error message if the subprocess fails
            print(f"Subprocess failed with error: {e}")
    
    if __name__ == "__main__":
        if len(sys.argv) > 1:
            act_busy()
        else:
            run_subprocess()
  2. github-actions commented on Apr 26, 2025

    @github-actions

    This issue was closed because it has been stalled for 30 days with no activity. If the issue still persists, please reopen with the information requested. Thanks.

  3. SobhanMP commented on May 5, 2025

    @SobhanMP

    It's broken, see the following

    import time
    import multiprocessing as mp
    
    def f():
        for i in range(10):
            time.sleep(1)
            print(i)
    
    if __name__ == "__main__":
        p = mp.Process(target=f)
        p.start()
        p.join()

    Image

  4. rchiodo commented on May 5, 2025

    @rchiodo
    Contributor

    Seems to work fine for me. Although I'm on windows.

    DebugMultiproc.mp4
  5. rchiodo commented on May 5, 2025

    @rchiodo
    Contributor

    I can reproduce on Linux though.

  6. rchiodo commented on May 5, 2025

    @rchiodo
    Contributor

    Strange but the subprocess example works fine on Linux. Not sure how multiprocess could affect stepping.

  7. 1 remaining item

  8. rchiodo commented on Jun 2, 2025

    @rchiodo
    Contributor

    Sounds like this is related to sys monitoring support in Debugpy, which we switched to after 2024.10. Additionally, if it works with 3.11 or earlier (with the latest debugpy), that would also mean it's a sys monitoring issue.

  9. jborean93 commented on Jul 2, 2025

    @jborean93
    Contributor

    Can confirm that downgrading to Python 3.11 works around this issue and 3.12+ is affected. Happy to test anything out or help in whatever way I can to try and fix this issue. Would love to be able to use the latest Python version on my projects that have subprocesses.

  10. jborean93 commented on Jul 7, 2025

    @jborean93
    Contributor

    I've started digging into this a big more and I think I can see the cause behind it but not 100% sure. The underlying issues seems to be when the fork subprocess method is used (default for multiprocessing.Process on Linux) and not spawn.

    When a stepping action is taken the pydevd_comm.internal_step_in_thread is called.

    def internal_step_in_thread(py_db, thread_id, cmd_id, set_additional_thread_info):
    thread_to_step = pydevd_find_thread_by_id(thread_id)
    if thread_to_step is not None:
    info = set_additional_thread_info(thread_to_step)
    info.pydev_original_step_cmd = cmd_id
    info.pydev_step_cmd = cmd_id
    info.pydev_step_stop = None
    info.pydev_state = STATE_RUN
    info.update_stepping_info()
    if py_db.stepping_resumes_all_threads:
    resume_threads("*", except_thread=thread_to_step)

    This assigns the stepping action to info.pydev_step_cmd.

    When the new system.monitoring method is calling _pydevd_sys_monitoring._internal_line_event it attempts to get the current thread info which should contain the pydev_step_cmd action requested.

    def _internal_line_event(func_code_info, frame, line):
    # ENDIF
    # fmt: on
    py_db: object = GlobalDebuggerHolder.global_dbg
    thread_info = _thread_local_info.thread_info
    info = thread_info.additional_info
    step_cmd = info.pydev_step_cmd

    I've manually added some extra logging with the following under internal_step_in_thread

    pydev_log.info(
        "internal_step_in_thread\n  TID: %s\n  ThreadPyId: %s\n  Thread: %r\n  InfoPyId: %s\n  InfoRepr: %r\n  Info: %s",
        threading.get_ident(),
        id(thread_to_step),
        thread_to_step,
        id(info),
        info,
        info)

    And the following under _internal_line_event

    pydev_log.info(
        "j_internal_line_event\n  Frame: %s\n  Line: %s\n  State: %s\n  Cmd: %s\n  TID: %s\n  ThreadPyId: %s\n  Thread: %r\n  InfoPyId: %s\n  InfoRepr: %r\n  Info: %s",
        frame,
        line,
        info.pydev_state,
        step_cmd,
        threading.get_ident(),
        id(thread_info),
        thread_info,
        id(info),
        info,
        info)

    I then ran the following code in VSCode

    import os
    
    import multiprocessing as mp
    
    # mp.set_start_method('spawn', force=True)
    
    def f(test=False):
        if test:
            print(f"Child {os.getpid()}")
    
        a = 1
        return a
    
    
    if __name__ == '__main__':
        print(f"Parent {os.getpid()}")
        f()
    
        p = mp.Process(target=f, args=(True,))
        p.start()
        p.join()

    With a breakpoint on line 11 (a = 1) I pressed step over then continue for the main process, then step over when the forked process hit the breakpoint. The logs for these step over actions are

    # Main process after pressing step over
    0.00s - internal_step_in_thread
      TID: 281473609240608
      ThreadPyId: 281473602656240
      Thread: <_MainThread(MainThread, started 281473609240608)>
      InfoPyId: 281473595074368
      InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffada5e740>
      Info: State:2 Stop:None Cmd: 107 Kill:False
    
    0.00s - _internal_line_event
      Frame: <frame at 0xffffad465150, file 'subproc_test.py', line 12, code f>
      Line: 12
      State: 1
      Cmd: 159
      TID: 281473609240608
      ThreadPyId: 281473588400816
      Thread: <_pydevd_sys_monitoring._pydevd_sys_monitoring.ThreadInfo object at 0xffffad4012b0>
      InfoPyId: 281473595074368
      InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffada5e740>
      Info: State:1 Stop:<frame at 0xffffad465150, file 'subproc_test.py', line 12, code f> Cmd: 159 Kill:False
    
    # Forked process after pressing step over
    0.00s - internal_step_in_thread
      TID: 281473609240608
      ThreadPyId: 281473602656240
      Thread: <_MainThread(MainThread, started 281473609240608)>
      InfoPyId: 281473588257088
      InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffad3de140>
      Info: State:2 Stop:None Cmd: 107 Kill:False
    
    0.00s - _internal_line_event
      Frame: <frame at 0xffffa6b6da40, file 'subproc_test.py', line 12, code f>
      Line: 12
      State: 1
      Cmd: -1
      TID: 281473609240608
      ThreadPyId: 281473588400816
      Thread: <_pydevd_sys_monitoring._pydevd_sys_monitoring.ThreadInfo object at 0xffffad4012b0>
      InfoPyId: 281473595074368
      InfoRepr: <_pydevd_bundle.pydevd_additional_thread_info_regular.PyDBAdditionalThreadInfo object at 0xffffada5e740>
      Info: State:1 Stop:None Cmd: -1 Kill:False
    

    The forked process' additional_info in the _internal_line_event seems to be a different object to the one processed in internal_step_in_thread. This means it doesn't see the current stepping action and thus continues like Continue was pressed. Interestingly the object address for additional_info in _internal_line_event in Python is 0xffffada5e740 which is the same address as the additional_info in the parent process it was forked from. My guess is that there is some missing cleanup actions that needs to be done during the fork hook. As right now it seems to be sharing the same threading.local() value as the parent process rather than a new one. The internal_step_in_thread doesn't use threading.local() but looks up the thread based on the ID based on the PID and TID.

    I'll try and spend a bit more time tomorrow to understand how this value is getting cached and not being cleared but if anyone knows more before I get to it I'll love any pointers.

  11. rchiodo commented on Jul 7, 2025

    @rchiodo
    Contributor

    You might compare this with what happens during 3.11, as that seems to be working. The logic should be similar.

  12. jborean93 commented on Jul 7, 2025

    @jborean93
    Contributor

    Good point, it's not immediately clear what the exact equivalent is but I see some of the old code do info = threading.current_thread.additional_info rather than use threading.local().thread_info.additional_info. When I change the following

    thread_info = _thread_local_info.thread_info
    info = thread_info.additional_info

    To

    t = threading.current_thread()
    info = t.additional_info

    So the caching of the pre-fork thread data seems to be the problem and using threading.current_thread() to extract the additional_info solves this problem.

    The questions I have are

    • Why does pydevd using the threading.local() TLS store and not threading.current_thread()
    • Is it better to change the code to use threading.current_thread() or figure out how to clear the TLS cache post fork

    I know debugpy just vendor's pydevd but I'm not sure if you have any history behind the sys.monitoring implementation or not/insights on how this should be fixed.

  13. rchiodo commented on Jul 7, 2025

    @rchiodo
    Contributor

    Fabio Zadrozny (@fabioz) did most of the sys monitoring work, I just made some additions. But my understanding is that threading.local() is used so that you get a per thread copy of the data, although that seems unnecessary if the data is stored on the thread itself. In fact it seems like a bug (maybe the bug as you've pointed out)? As another thread can't write to threading.local(), so if you wanted to communicate stuff across threads, you wouldn't use it. I'm not sure how closely Fabio pays attention to debugpy though, but he may have more information.

  14. jborean93 commented on Jul 7, 2025

    @jborean93
    Contributor

    Certainly using threading.current_thread().additional_info fixes the issue but the sys.monitoring implementation is definitely written with the TLS store threading.local() instead. Hopefully Fabio Zadrozny (@fabioz) can shed some light on why it was changed for that but in the meantime I'll try and see if I can figure out why the post fork setup isn't clearing the TLS or at least the old thread info is still present here. It's definitely the cause behind it and if I can figure out that it'll be a cleaner fix than converting all the sys.monitoring implementations over to a different method.

    What is curious is that PyCharm has a similar implementation that uses threading.local() but doesn't seem to suffer from the same problem. I'll have a dig into their fork to see what might be different and see if that sheds some light on it.

  15. jborean93 commented on Jul 8, 2025

    @jborean93
    Contributor

    Ok figured out why the state wasn't getting reset. The _pydevd_sys_monitoring module when imported in the original process set _thread_local_info = threading.local() creating a TLS for the threads in that process. When a fork occurs this copy of the module will be included in the forked process so the import is never run again and it just inherits the original value from the parent process.

    The PR https://github.com/microsoft/debugpy/pull/1921/files adds a reset state similar to what is happening in _pydevd_frame_eval but for the sys.monitoring code.

  16. rchiodo commented on Jul 8, 2025

    @rchiodo
    Contributor

    Nice! Thanks for looking at this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions