[go: up one dir, main page]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[vcpkg-get-python-packages] Cache packages via pip precandidate #38760

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

WangWeiLin-MV
Copy link
Contributor
@WangWeiLin-MV WangWeiLin-MV commented May 16, 2024

WIP

Abstract

Download pypi packages by vcpkg_download_distfile, then install packages offline.

Fix #37473 (review). And any other port that depends on pip install.

Changes

  • Bootstrap virtual environment with runtime which has neither standard library venv nor pip (i.e. Embedded Python)
  • Add pip command precandidate to obtain the candidate package filename during local installation
  • Download packages by vcpkg_download_distfile

WIP

@WangWeiLin-MV WangWeiLin-MV added category:new-port The issue is requesting a new library to be added; consider making a PR! info:internal This PR or Issue was filed by the vcpkg team. category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly labels May 16, 2024
@WangWeiLin-MV WangWeiLin-MV mentioned this pull request May 16, 2024
7 tasks
@Neumann-A
Copy link
Contributor

a) this depends on the tool RFC
b) simply no. Doesn't solve the real problem. Simply change vcpkg_get_python_packages so that it is asset cacheable without changing its interface?

@Hoikas
Copy link
Contributor
Hoikas commented May 16, 2024

Python tools need to be the exact same version as the python3 port, or you introduce potential interesting compile failures. AFAIK, the Python 3 port is still on 3.11, but this is using 3.12.

This also seems to be completely unnecessary - we already have the tool package and vcpkg_get_python_packages().

@WangWeiLin-MV
Copy link
Contributor Author
WangWeiLin-MV commented May 17, 2024

The purpose of this port is as a module for finding Python interpreter and packages, and implementing #31748 vcpkg-tool-python3-interpreter, and combine and make vcpkg-get-python-packages cacheable.


Python tools need to be the exact same version as the python3 port, or you introduce potential interesting compile failures. AFAIK, the Python 3 port is still on 3.11, but this is using 3.12.

The ports/python installs python libraries and binary tools, while current script only find or provide a runtime with packages and is not installed as artifacts. I think this will not affect each other.

@Neumann-A
Copy link
Contributor
Neumann-A commented May 17, 2024

Just use the same approach as vcpkg_find_acquire_msys and put it into vcpkg-get-python-packages

@WangWeiLin-MV
Copy link
Contributor Author

Just use the same approach as vcpkg_find_acquire_msys and put it into vcpkg-get-python-packages

vcpkg-get-python-packages does not provide search function. And no optional virtual environment, it makes difficult to integrate for current port purpose.

About same approach as vcpkg_find_acquire_msys, there are some problems to resolve:

For using pypi source dist requires additional build dependencies, I didn't choose this way.

And for pypi binary dist consists of complex dependencies such as https://pypi.org/pypi/pandas/json requires_dist and lots of wheel https://pypi.org/project/pandas/#files depends on platform. The detection function is used here to obtain dependencies.

@Neumann-A
Copy link
Contributor

@WangWeiLin-MV: I am aware of all what your are telling me. I am still saying use the same approach as vcpkg_find_acquire_msys. Just make it download the wheel and declare the runtime dependencies which can be tested using an from package import * after installation.

YOu also want to automate the generation. So you might want to create a powershell/cmake scripts using
"Invoke-WebRequest -Uri \"https://pypi.python.org/pypi/${package_name}/json/\" -Method GET | ConvertFrom-Json | Select-Object -ExpandProperty info | Select-Object -ExpandProperty version"
you can modify the web request to get the url and filename of the wheel.

@WangWeiLin-MV
Copy link
Contributor Author

Thanks to @Neumann-A and @Hoikas, I'll continue the discussion later that for the find_python function and whether it replaces vcpkg_find_acquire_program_python and vcpkg-get-python-packages.

Before this, I will first make vcpkg-get-python-packages cacheable.

@WangWeiLin-MV WangWeiLin-MV force-pushed the port/vcpkg-tool-python-buildtime/init branch from 066e2d2 to 6e49aaa Compare May 17, 2024 06:36
@WangWeiLin-MV WangWeiLin-MV changed the title [vcpkg-tool-python-buildtime] Find Python installation and packages [vcpkg-get-python-packages] Cache packages via pip precandidate May 17, 2024
@WangWeiLin-MV WangWeiLin-MV added category:port-update The issue is with a library, which is requesting update new revision and removed category:new-port The issue is requesting a new library to be added; consider making a PR! labels May 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:port-update The issue is with a library, which is requesting update new revision category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly info:internal This PR or Issue was filed by the vcpkg team.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants