Repository navigation
Option of defining an environment script for starting engines (with SSHEngineSetLauncher) #1029
Description
We need to adapt the environment before starting the actual ipp engine (through the SSHEngineSetLauncher). Adapting the environment via the environment parameter (in class SSHLauncher) is possible, but it's not very practical, since the launcher needs to know all the engine computer details in advance.
Calling a local environment script before starting the engine would give much more flexibility and would easily allow starting the engine e.g. in a python virtual environment, conda environment, etc.
Although the general idea is quite simple the actual implement is not as easy. The problem: one cannot simply start a shell, next the environment script and then the actual command because the process handle (in python) would point to the shell and not the actual command. Hence, the retrieved process id would be incorrect. Using some system code it would be still possible to retrieve the true process id, but this is especially difficult in windows which is why I tested an alternative solution:
Starting a shell, outputting all environment variable, running the environment script and outputting all environment variable again. A bit of python code analyses the output and collects the environment variable changes (especially added and/or changed env.vars). I have implemented a small test script that works under linux and windows.
The collected changes can then be easily passed to the env parameter of subprocess.Popen function.
Does this sound like an appropriate idea and solution?
As before I'm happy to provide the implementation, tests, documentation, etc.
Activity
minrk commentedon Jun 25, 2026
I think this can largely be addressed with EngineLauncher.engine_cmd, but the UX could probably be smoothed. If we can't find a way to improve the config/API over what we have, at the very least it could be a case for documentation.
For example, you can set
engine_cmdto["bash", "-l", "-c", "python3 -m ipyparallel.engine"]to ensure that a full login shell is launched (this sources .bashrc, /etc/profile.d, etc.), setting loads of environment variables. You can do something similar with more specific control if you have an engine-launching script already on the remote machine:or
where
entrypoint.shis an entrypoint-style script (common in containers) that sets environment variables, etc. and ends withexec "$@"to launch whatever command comes after.If the script is fully local and not yet on the remote machine, you can still do it with
bash -c:minrk commentedon Jun 25, 2026
So one thing that I might start with is: write down what it looks like to do what you're trying to do today with engine_cmd. Then we can see if there are improvements we can make to simplify that.
ottointhesky commentedon Jun 26, 2026
Thanks for your quick reply.
The main reason I came up with the environment parsing concept was that I thought it could be problematic if the started process is not the actual engine but rather a subprocess. I somehow had the impression that Windows and Linux behave differently in such situations, which is why I did a few tests today:
When I kill process A using
process B will stay running. I need to explicitly kill process B as well, to get all processes terminated.
If process B is killed on the other hand, process A will teminate shortly after (since its purpose was only to start B)
Fortunatly, Windows and Linux behave excatly the same. So If we kill the parent process and all subsequent child processes (if requested), it should not matter that we are observing/managing the parent process (process A) and not the child/engine process (process B).
Under Linux ipp uses signals to end processes. I'm not deep into signals but a quick test showed, that signals also only reach the actual process and no subprocesses.
So from my point of view the only changes required (to fullfill our needs) are signals to be send to the process and all child processes. I have NOT done any tests with ipp itself, but does this sounds right for you?
ottointhesky commentedon Jun 29, 2026
Maybe I should have tested a bit more, before writing this issue. The following finding refer to Windows only (I have not tested Linux yet):
Using an entry point script works perfectly for starting cluster using a local or ssh lauchner. But stopping the cluster with
ipcluster stop ....is a problem. But the problem is actually not related to the entry point script but to the deletion of the log files:2026-06-29 11:03:39.995 [IPClusterStart] ERROR | Failed to remove C:\Users\johannes\.ipython\profile_env_test\log\ipengine--1782720504-1.log: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\johannes\\.ipython\\profile_env_test\\log\\ipengine--1782720504-1.log'I have been seeing this error message all the time (and I'm not the first one #648), but I have ignored it so far since it did not effect the work of the cluster (only the shutdown). But interestingly, the shutdown of a ssh cluster behaves differently if an entry point script is used or not. I always get remove error messages, but without an entry point script all python processes get deleted on the ssh machine whereas with entry point script always 2 python processes per engine remain running. This is why I thought that ipp was not designed to start engines with an entry point script...
Can you give me some advice/tools how to trace down the file remove problem...
ottointhesky commentedon Jun 29, 2026
Follow-up: I removed the log file deletion in the
ipclusterscript (to eliminate the error messages) but the 2 python processes on the ssh server kept running (if using an entry point script). So the log file removal error and the remain python processes are two separate problems.Nevertheless, do you have any idea/advice how to tackle those problems?
ottointhesky commentedon Jun 29, 2026
Follow-up 2: it's the engine and the nanny process that will stay running after the
ipcluster stopcommand (until the maximum number of heartbeats misses reached)ottointhesky commentedon Jun 29, 2026
Oh boy, I'm so stupid... Don't even remeber my own code...
It's no surprise that in case of an entry point script the the engine and nanny process keep running under Windows:
ipyparallel/ipyparallel/cluster/shellcmd_receive.py
Lines 201 to 203 in 3b061ac
Hence, to get this combination to work properly,
ShellCommandReceiveWindows.cmd_killneeds to kill all child processes. Using pure windows python code this is not as easy (see comment). There are basically two alternatives:Can you please comment on that (and sorry for my imprecise and lengthy previous messages)
ottointhesky commentedon Jul 2, 2026
I ran some tests on Linux as well. The code behaves identically on Linux and Windows.
On both systems, the engine and nanny processes keep running if they were started through an entry-point script. They eventually exit on their own once the maximum number of missed heartbeats is reached.
From my point of view, stopping the entry-point script process, including all of its child processes, is the rigorous solution. Under Linux, this is easy using
os.killpg, but under Windows it is much more complicated if you do NOT want to rely on the external tooltaskkillor thepsutillibrary, although it is possible.Do you think we should always kill the entire process group, or could that have downsides, like ungracefully terminating some processes?
minrk commentedon Jul 3, 2026
that's why launcher scripts should generally end with
exec "$@". The equivalent syscall in Python is os.execv. This replaces the process instead of launching a subprocess, so the PID of the engine is unchanged. II think it should generally be safe to allow custom commands and signal behavior ought to be the same. This should be included in any docs/examples we add.Re: log deletion, I didn't think about Windows more restrictive open-file rules (you can safely remove open files on non-Windows). I'm afraid I don't know windows debug tools well. We may have to rethink what deletes log files on Windows. On posix, I would use things like
lsofandpsto see what's going on and who is holding onto files (and maybefseventson mac).A test that exercises shutdown and makes sure all processes stop and.
For Windows, I would be quite okay using
taskkill, but we already havepsutilas a dependency, so that is not new.I'm not 100% sure, but I suspect that killing the group is probably appropriate. I think we already learned the same lesson in jupyter client, though our Windows strategy is different there.