Skip to content

Add Python 3 to more build machines #1674

Closed
Closed
Add Python 3 to more build machines#1674
Assignees
refack
Labels
ci-changePSA of configuration changesci-job-requestci-publicci-releaseenhancementstale

Description

@cclauss
cclauss
opened on Jan 30, 2019

Continue the process started in #1644 of putting both Python 2 and Python 3 on various build machines so that we can perform tests and experimental builds on the Python 3.
nodejs/node#25759 (comment)

/R = @rvagg

Activity

  1. cclauss commented on Feb 1, 2019

    @cclauss
    cclauss
    on Feb 1, 2019
    ContributorAuthor

    On those build machines where Python 3 is already installed, we need to do:
    python3 -m pip install --upgrade pipenv # See test failures on nodejs/node#25878

  2. refack commented on Feb 1, 2019

    @refack
    refack
    on Feb 1, 2019
    Contributor

    Need to add to ansible recipes (https://github.com/nodejs/build/search?p=3&q=python&type=&utf8=%E2%9C%93)
    Might be a good chance to refactor common prerequisites.

  3. cclauss commented on Mar 4, 2019

    @cclauss
    cclauss
    on Mar 4, 2019
    ContributorAuthor

    Can we please get this resolves and closed? 303 daze left...

  4. rvagg commented on Mar 13, 2019

    @rvagg
    rvagg
    on Mar 13, 2019
    Member

    some help getting ansible working properly with python3 would be great, I can't run it on my mac anymore for machines that need ccache and have to resort to linux where it uses python2. The line causing problems is https://github.com/nodejs/build/blob/master/ansible/roles/baselayout/tasks/ccache.yml#L13 and there's a similar one in ansible/roles/jenkins-worker/tasks/partials/docker-host.yml.
    Something about jinja2 processing of "{{ ccache_html_content.stdout | regex_findall('ccache-[0-9]+.[0-9]+(?:.[0-9]+)*.tar.gz') | map('regex_replace', 'ccache-') | map('regex_replace', '.tar.gz') | list | latest_version }}" the chaining of the maps seems to be the problem and I don't have the error at hand. If needed I could probably generate the error again.

  5. refack commented on Mar 14, 2019

    @refack
    refack
    on Mar 14, 2019
    Contributor

    @rvagg does #1399 help?

    (BTW this has the new "npm" like pipenv work flow, git fetch, pipenv install, pipenv shell)

  6. cclauss commented on Apr 15, 2019

    @cclauss
    cclauss
    on Apr 15, 2019
    ContributorAuthor

    Should we close this one or is this still an issue?

  7. refack commented on Apr 15, 2019

    @refack
    refack
    on Apr 15, 2019
    Contributor

    We still need Python3 on more machines.

  8. refack
    self-assigned this
    on Apr 15, 2019
  9. refack
    added
    ci-changePSA of configuration changes
    ci-job-request
    ci-public
    ci-release
    enhancement
    on Apr 15, 2019

  10. cclauss commented on Apr 17, 2019

    @cclauss
    cclauss
    on Apr 17, 2019
    ContributorAuthor

    Can we track progress in a table similar to the one below?

    Operating System Architectures Versions Python 2 Python 3
    GNU/Linux x64 kernel >= 3.10, glibc >= 2.17 Tests pass Tests pass
    GNU/Linux x64 kernel >= 3.10, musl >= 1.1.19 Tests pass Tests fail
    GNU/Linux x86 kernel >= 3.10, glibc >= 2.17 Tests pass Not installed
    GNU/Linux arm64 kernel >= 4.5, glibc >= 2.17 Tests pass Not installed
    GNU/Linux armv7 kernel >= 4.14, glibc >= 2.24 Tests pass Not installed
    GNU/Linux armv6 kernel >= 4.14, glibc >= 2.24 Tests pass Not installed
    GNU/Linux ppc64le >=power8 kernel >= 3.13.0, glibc >= 2.19 Tests pass Not installed
    GNU/Linux s390x kernel >= 3.10.0, glibc >= 2.17 Tests pass Not installed
    Windows x64, x86 (WoW64) >= Windows 7/2008 R2/2012 R2 Tests pass Not installed
    Windows x86 (native) >= Windows 7/2008 R2/2012 R2 Tests pass Not installed
    Windows arm64 >= Windows 10 Tests pass Not installed
    macOS x64 >= 10.12 Tests pass Not installed
    SmartOS x64 >= 18 Tests pass Not installed
    AIX ppc64be >=power7 >= 7.1 TL05 Tests pass Not installed
    FreeBSD x64 >= 11 Tests pass Not installed

  11. sam-github commented on May 15, 2019

    @sam-github
    sam-github
    on May 15, 2019
    Contributor

    Is #1674 (comment) needed on the AIX machines? If so, after updating them all to have python3 available, #1775 (comment), I could do it.

  12. 8 remaining items

  13. sam-github commented on May 17, 2019

    @sam-github
    sam-github
    on May 17, 2019
    Contributor

    @cclauss

    % ssh test-osuosl-ubuntu1404-ppc64_be-1 python2 --version
Python 2.7.6
% ssh test-osuosl-ubuntu1404-ppc64_le-1 python2 --version
Python 2.7.6
% ssh test-osuosl-aix61-ppc64_be-1 python2 --version
Python 2.7.11
% ssh test-osuosl-centos7-ppc64_le-1 python2 --version
Python 2.7.5

  14. sam-github commented on May 21, 2019

    @sam-github
    sam-github
    on May 21, 2019
    Contributor

    @cclauss I don't understand why we need to update python2 to a supported version if we have python3. If it was easy, I'd just do it, but its not, so I am pushing back a bit.

    Why should we update Python2 if a supported version of Python3 exists? Especially given python2 is EOL on Jan 1, 2020?

  15. cclauss commented on May 21, 2019

    @cclauss
    cclauss
    on May 21, 2019
    ContributorAuthor

    Let's focus first and foremost of getting a Python >= 3.5 on these machines (and in the $PATH). If it was up to me, I would just take out legacy Python completely and step into the future. However, Node.js is not yet compatible with Python 3. This exercise is to set up the tests on Python 3 but it is not yet clear if those tests will pass and build a workable Node.js.

    Therefor, Python 2 will be a part of Node.js beyond the 1/1/20 date so given that we will be depending on an unsupported codebase, we would like to uplevel the Python 2 to be >= 2.7.9 to encourage our users to avoid the most egregious attack vectors. If it is too much work to fix legacy Python in the near term, lets focus on upgrading Python as you suggest.

    Thanks much for all your hard work.

  16. sam-github commented on May 23, 2019

    @sam-github
    sam-github
    on May 23, 2019
    Contributor

    see: #1807

  17. sam-github commented on May 24, 2019

    @sam-github
    sam-github
    on May 24, 2019
    Contributor

    Folks here, and @nodejs/python , we need more clarity on the expectation for python3. At least, I do.

    Python3 has to be >= 3.5.0.

    When system packages are available, I guess it will be in the system path, as python3.

    Is this the intended setup, in other words, is it a requirement that python3 be in the default PATH?

    And is this a requirement: #1674 (comment) ?

    On test-osuosl-centos7-ppc64--le-1:

    $ ./python3 -m pip install --upgrade pipenv
/opt/python-3.7/bin/python3: No module named pip

    I don't know much about pip, and nothing about pipenv, so it would help if we had a relatively clear statement of what was required.

    Possibly we want node itself to build with python3, if that is even possible. Is there some way I can test that manually? If I have a succinct recipe for what I should do manually on a machine once python3 is installed on it to verify the python3 environment fulfills expectations, that would be helpful.

    EDIT: also see #1807 (comment)

  18. cclauss commented on May 24, 2019

    @cclauss
    cclauss
    on May 24, 2019
    ContributorAuthor

    pip is the package installer for Python. It is the essential bootstrap for installing packages. Without pip you can not install pyenv or any other Python packages so we need pip. You can try python3 -m ensurepip to install it but the preference would be an install which already includes pip.

    python3 --version
python3 -m pip --version

  19. thefourtheye commented on May 27, 2019

    @thefourtheye
    thefourtheye
    on May 27, 2019

    Is this the intended setup, in other words, is it a requirement that python3 be in the default PATH?

    If Python 3 is not there in the default PATH, we may have to update our scripts to detect Python 3 as well, I am afraid. Installing it at /usr/local should not create any problems, right?

    Possibly we want node itself to build with python3, if that is even possible. Is there some way I can test that manually?

    The ultimate goal of this activity is to build and test Node.js with both Python 3 and Python 2.

    If I have a succinct recipe for what I should do manually on a machine once python3 is installed on it to verify the python3 environment fulfills expectations, that would be helpful.

    As far as I am concerned, we just need Python 3 in the default PATH. pip is optional/good-to-have in my opinion, as we do not install many additional packages to test (I could be wrong here, I haven't checked all the scripts we have). @cclauss Do you think pip is necessary for us to proceed at this point?

  20. cclauss commented on May 27, 2019

    @cclauss
    cclauss
    on May 27, 2019
    ContributorAuthor

    A Python install without pip is a painful thing so we should avoid it if we can.

    https://github.com/nodejs/node/blob/master/Makefile#L1316 and below use pip to install flake8 but linting on all platforms is not required.

  21. sam-github commented on Jun 10, 2019

    @sam-github
    sam-github
    on Jun 10, 2019
    Contributor

    For anyone finding their python3 is lacking pip, or ctypes, libffi and zlib (the -dev versions of these) might be missing.

  22. cclauss commented on Jun 11, 2019

    @cclauss
    cclauss
    on Jun 11, 2019
    ContributorAuthor

    See the note above (#1674 (comment)) about ensurepip.

  23. sam-github commented on Jun 11, 2019

    @sam-github
    sam-github
    on Jun 11, 2019
    Contributor

    @cclauss Perhaps I misunderstood that comment, don't you suggest not using ensurepip? Are you suggesting we should use it? Or suggesting that its an alternative to installing python3's dev dependencies before configuring and building python3?

  24. cclauss commented on Sep 2, 2019

    @cclauss
    cclauss
    on Sep 2, 2019
    ContributorAuthor

    Are we done on this one? Can we start testing Python 3.6 on one of the Jenkins test machines? Python 3.7 is pending nodejs/node#29326 but Python 3.6 should be fair game to start testing.

  25. github-actions commented on Jun 29, 2020

    @github-actions
    github-actionsBot
    on Jun 29, 2020

    This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

  26. github-actions
    added
    stale
    on Jun 29, 2020
  27. cclauss
    closed this as completedon Jun 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

ci-changePSA of configuration changesci-job-requestci-publicci-releaseenhancementstale

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions