Periodic "bad interpreter" when caching virtual environments

Describe the bug
When testing large projects it is convenient to cache dependencies from pypi with a cache key based on the projects setup.py. I've been been using that pattern (yaml shown below) on several projects for > 6 months. It works great, but periodically (every 1-6 weeks) fails with the error:

Once the error occurs once, it will appear consistently until I alter the cache key and force the virtual environment to rebuild.

I think what's happening is that every time the build/setup for the setup-python action changes, it invalidates the link between the virtual environment and the python binary.

Questions:

Is there a recommended method for caching dependencies with setup-python that will not suffer from this issue
Could you recommend file/something that can be hashed and added to the cache key to invalidate the cache when new python builds are pushed?

    steps:

    - name: set up python 3.8
      uses: actions/setup-python@v1
      with:
        python-version: 3.8

    - name: checkout
      uses: actions/checkout@v2

    - name: cache dependencies
      id: xyz-cache
      uses: actions/cache@v2
      with:
        path: venv
        key: ${{ runner.os }}-pip-${{ hashFiles('setup.py') }}-0

    - name: install xyz
      if: steps.sl-cache.outputs.cache-hit != 'true'
      run: |
        python -m venv venv
        source venv/bin/activate
        python -m pip install --U pip setuptools wheel
        pip install -e ".[dev]"

Thanks!

Which version of the action are you using?

v1
v2
Some other tag (such as v2.0.1 or master)

Environment

self-hosted
Linux
Windows
Mac

Python Versions
At least all versions of 3.7 and 3.8 but probably all of them

The folks over at poetry seem to have run into the same issue

They do this:
https://github.com/python-poetry/poetry/blob/master/.github/workflows/main.yml#L55

I'd appreciate a nod if that's the recommended solution

We usually encounter the same problem.

I guess it's connected to some external event, like Github runners update or something similar.
After that, the workflow starts to fail steadily with the same error mentioned above.

We usually fix it by updating the cache key, but it's really annoying.

Made venv cache dependent on exact python and os version.

Update github action poetry cache key

The same issue here.

Platform:

Ubuntu

Version

v3

Python

python3.7
python3.8
python3.9
python3.10

Any news?

@marcosschroh

Hi @marcosschroh,

Just to let you know we are taking a deeper investigation into what is exactly happening here.

Cheers

@marcosschroh

Hi again @marcosschroh,

Do you also use actions/cache for caching as other users? Also which package manager are you using?

Hi,

I am using pip and actions/cache@v3 to cache the dependencies.

@marcosschroh

Hi @marcosschroh, do you mean this workflow?
https://github.com/marcosschroh/dataclasses-avroschema/blob/master/.github/workflows/python-package.yml

As I see, you are using cache key without exact python version. So venv has config file pyvenv.cfg with exact version and python path, for example:

home = /opt/hostedtoolcache/Python/3.10.4/x64/bin
include-system-site-packages = false
version = 3.10.4

Your workflow has only major.minor python version and after update python patch version, for example 3.10.4 -> 3.10.5 your venv cache will break.

You should create key with exact python version, for example:

- uses: actions/setup-python@v4
  id: cp310
  with:
     python-version: "3.10"

- uses: actions/cache@v3
  id: cache  # name for referring later
  with:
    path: .venv
    key: ${{ runner.os }}-cache-${{ steps.cp310.outputs.version }}-${{ hashFiles('requirements.txt') }}

And after update of python patch version this cache will be rebuilt.

gha: fix issue when caching virtual environments

@olirice

Hi @olirice,

Since there is no recent activity on this issue, I am going to close it. Feel free to continue the discussion if you have additional questions.

Cheers

using the output of actions/setup-python@vX that includes the patch version in the cache key sounds great. Thank you for the help

In my case, the error shows in Github action and the root cause is the broken python symbolic link. As a workaround, I just simply re-create it during the process.

Error：
home/runner/work/_temp/810926a2-36c5-4488-ac84-6f3f57713147.sh: /home/runner/work/dataops/dataops/.venv/bin/pytest: /home/runner/work/dataops/dataops/.venv/bin/python: bad interpreter: No such file or directory

Root Cause:
/home/runner/work/dataops/dataops/.venv/bin/python: broken symbolic link to /opt/hostedtoolcache/Python/3.9.15/x64/bin/python3.9

Solution:
Add below scripts into wherever has broken link:
        PYTHON_PATH=$(which python3)
        source .venv/bin/activate
        PYTHON_BROKEN_PATH=$(dirname $(which pip))
        rm $PYTHON_BROKEN_PATH/python
        ln -s $PYTHON_PATH $PYTHON_BROKEN_PATH/python
        export PATH=$PATH:$(dirname $PYTHON_BROKEN_PATH)

Use full python version to avoid bad interpreter errors.

I ran into this. It turned out to be because I was using setup-python to install my Python version when I was doing my cached venv setup.

It was installing into /opt/hostedtoolcache/Python/3.9.19, and I think that used to be a default version on the GH runners, but the default just updated. So, (a) setup-python installed, (b) I created my venv pointing to that version, (c) actions-cache cached the venv symlink but not the installed version, and finally (d) when I restored the cache the symlink was broken because the Python binary wasn't included.

So, the fix was to include /opt/hostedtoolcache/Python/3.9.19 in my cached paths, like:

          path: |
            venv
            ~/.ssl
            .env
            /opt/hostedtoolcache/Python/3.9.19

Note the path must be the same literal value in restores, with no vars as used, as in the cache action. Otherwise it will not match (but will report the same cache key as was generated).

I found debugging with SSH useful along the way.

Bump form-data to fix critical vulnerabilities #182 & #183 (#1163)

chore(deps): update actions/setup-python action to v6 (#11)

⬆️ Update actions/setup-python action to v6 (#21)

Navigation Menu

Repository navigation

Periodic "bad interpreter" when caching virtual environments #182

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Periodic "bad interpreter" when caching virtual environments #182

Description

Activity

olirice commented on Jan 21, 2021

rosik commented on Jul 15, 2021

marcosschroh commented on Apr 5, 2022

panticmilos commented on May 30, 2022

panticmilos commented on May 31, 2022

marcosschroh commented on Jun 8, 2022

vsafonkin commented on Jun 9, 2022

panticmilos commented on Jun 22, 2022

olirice commented on Jun 22, 2022

chance2021 commented on Dec 16, 2022

markfickett commented on Sep 23, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions