Repository navigation
feat: install PyPI libraries for job tasks and give jobs API access - #96
Merged
Merged
Conversation
- Honour libraries[].pypi: requirements are pip-installed with --target inside the job container, cached on the data volume per set of libraries, and put on PYTHONPATH. repo selects the index. Invalid specs fail the task. - Inject DATABRICKS_HOST and DATABRICKS_TOKEN into every job so WorkspaceClient() works with no arguments. MINILAKE_JOB_API_URL / MINILAKE_JOB_TOKEN override them, and spark_env_vars take precedence. Closes #95
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #95
Problem
Job tasks run in a sibling
apache/sparkcontainer that only ships PySpark, so a script doingfrom databricks.sdk import WorkspaceClientfails withModuleNotFoundError.libraries[].pypiwas accepted but ignored, and the container had no way to call back into the API.Changes
PyPI libraries
libraries[].pypi(package, optionalrepo) is honoured. Requirements are installed withpip install --targetinside the job container, so the interpreter matches the one that runs the script, and put onPYTHONPATHbeforespark-submit.-(a pip option) or an index URL that is not http(s) fails the task. Packages are passed after--.MINILAKE_JOB_EXECUTOR=subprocessgets the same behaviour, installing for thepython3that runs the script.API access from the job
DATABRICKS_HOSTandDATABRICKS_TOKEN, soWorkspaceClient()needs no arguments.MINILAKE_JOB_API_URLif set, otherwise minilake's own address on the network the job container joins (works on the default bridge too, no DNS needed), orhttp://127.0.0.1:<port>in subprocess mode. If it cannot be determined nothing is injected, rather than pointing the job at alocalhostthat is not minilake.MINILAKE_JOB_TOKEN, defaultdapi-minilake.spark_env_varsstill win.The version is not bumped. The changelog entry is under
[Unreleased].Testing
tests/test_jobs.py: a pip-option-looking library fails the task, anddatabricks-sdkis installed from PyPI andWorkspaceClient()works inside a job.tests/test_jobs.py,test_docker_executor.py,test_jobs_submit.pyandtest_app.pypass (79 passed, 2 skipped; the skipped ones needpipin the venv'spython3and pass with the system Python)._own_container_addressverified against real containers on the default bridge and on a user-defined network.Notes
MINILAKE_JOB_API_URLis set). For a bareuv run minilakeusing the Docker executor, setMINILAKE_JOB_API_URLto an address the container can reach.