DVC is a great tool as a data versioning system, but also is great as a build tool for ML experimentation. This repo offers the possibility of using DVC to establish your ML pipeline to be run by Github Actions runners or Gitlab runners.
You can also deploy your own Github runners or your own Gitlab runners with special capabilities like GPUs...
Major benefits of using DVC-CML in your ML projects includes:
- Reproducibility: DVC is always in charge of maintain your experiment tracking all the dependencies, so you don't have to. Additionally your experiment is always running under the same constrains so you don't have to worry about replicating the same environment again.
- Observability: DVC offers you metrics to be tracked. In DVC-action we make those metrics more human friendly and we also offer direct access to other experiments run through the DVC Report offered as checks in Github or Releases in Gitlab.
- Releases: DVC-action tags every experiment that runs with repro generating the report. Aside of that DVC-CML is just a step in your Github Workflow or Gitlab Pipeline that could generate your model releases or deployment according to your business requirements.
- Teaming: Give visibility to your experiments or releases to your teammates working together.
DVC-cml performs in your push or pull requests:
- DVC repro
- Push changes into DVC remote and Git remote
-
- In Github generates a Github check displaying the DVC Report
- In Gitlab generates a Tag/Release displaying the DVC Report
DVC-CML for Github
๐ Knowledge of Github Actions and DVC pipeline is very useful for a fully comprehension.
Example of a simple DVC-CML workflow:
๐ Note the use of the container
name: your-workflow-name
on: [push, pull_request]
jobs:
run:
runs-on: [ubuntu-latest]
container: docker://dvcorg/dvc-cml:latest
steps:
- uses: actions/checkout@v2
- name: dvc_cml_run
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
repo_token: ${{ secrets.GITHUB_TOKEN }}
repro_targets: your_dvc_target.dvc
run: |
# Install your project dependencies.
# An example for Python3:
apt-get install -y python3 python3-pip
pip3 install --upgrade pip
update-alternatives --install /usr/bin/python python $(which python3) 10
update-alternatives --install /usr/bin/pip pip $(which pip3) 10
test -f requirements.txt && pip3 install -r requirements.txt
# Run report:
dvc_cml_runDVC-CML for Gitlab
๐ Knowledge of Gitlab CI/CD Pipeline and DVC pipeline is very useful for a fully comprehension.
Example of a simple DVC-CML workflow in Gitlab:
๐ Some required environment variables like remote credentials and GITLAB_TOKEN are set as CI/CD environment variables in Gitlab's UI
โ ๏ธ tag_prefixshould be set in order to have DVC Reports, i.e. dvc_ . This will generate tags in your repo with the report as release notes
# .gitlab-ci.yml
stages:
- dvc_cml_run
dvc:
stage: dvc_cml_run
image: dvcorg/dvc-cml:latest
variables:
repro_targets: 'eval.dvc'
script:
- pip install tensorflow wget
- dvc_cml_runThis workflow will run every time that you push code or do a Pull/Merge Request.
When triggered DVC-CML will setup the runner and DVC will run the pipelines
specified by repro_targets. Two scenarios may happen:
- DVC repro is up to date and there is nothing to do. This means that the commit that you have done in your code is not related to your DVC pipelines and there is nothing to do.
- DVC pipeline has changed and DVC will run repro, updating the output that may generate (models, data...) in your DVC remote storage and then committing, tagging and pushing the changes in git remote.
Additionally, you may extend your CI/CD Pipeline/Workflow to generate your releases or even deploy automatically your models.
Many CI/CD vendors supports a special comment [ci skip] in the commit avoid run the CI. We support this, ff your commit comment includes the tag the DVC action will skip the CI returning an exit code of 0. We know that ideally the code should be 78 however, at the time of this writing, Github is only accepting 0 or 1 as status codes.
โ ๏ธ In Github Actions they are set viaenv:notinputs:
๐ In Gitlab pipeline they are set via
variables:
| Variable | Type | Required | Default | Info |
|---|---|---|---|---|
repo_token |
string | yes | In Github you can set the default autogenerated GITHUB_TOKEN. In Gitlab you have to generate it. See Tensorflow Mnist in Gitlab example for a complete walkthrough. | |
baseline |
string | no | origin/master | Revision to be compared with current experiment, i.e. origin/master, HEAD~1 or a commit sha. |
repro_targets |
string | no | Dvcfile | Comma delimited array of DVC files. If - is given will skip the process. |
metrics_diff_targets |
string | no | Comma delimited array of metrics files. If not specified will use all the metric files. | |
tag_prefix |
string | no | If set a new tag will be created in the repo with the name ${tag_prefix}${short_sha}. This will enable the "Latest 5 experiments in the branch" list in the report and will enable the DVC Report in Gitlab as a release description. |
|
metrics_format |
string | no | 0[.][0000000] | Metrics format following [numeral.js](http://numeraljs.com/ |
dvc_pull |
string | no | Comma delimited array of targets. If not specified will pull everything. If - is given dvc won't pull |
โ ๏ธ In Gitlab is required that you generate the GITLAB_TOKEN that is analogous to GITHUB_TOKEN. See Tensorflow Mnist in Gitlab example for a complete walkthrough.
Our DVC-CML GPU docker image is an Ubuntu 18.04 that already supports:
- cuda 10.1
- libcudnn 7
- cublas 10
- libinfer 10
- You need to setup properly your nvidia drivers and nvidia-docker in your host
machine.
sudo ubuntu-drivers autoinstall sudo apt-get install nvidia-docker2 sudo systemctl restart docker
- Launch your own runner following your CI vendor instructions.
Github
Repo settings -> Actions -> Add Runner button
Gitlab
Repo settings -> CI/CD -> Runners -> Specific Runners
# Gitlab self-hosted runner with dvc-cml and GPU
gitlab-runner register \
--non-interactive \
--run-untagged="true" \
--locked="false" \
--access-level="not_protected" \
--executor "docker" \
--docker-runtime "nvidia" \
--docker-image "dvcorg/dvc-cml-gpu:latest" \
--url "https://gitlab.com/" \
--tag-list "dvc-cml" \
--registration-token "here_goes_your_gitlab
8384
_runner_token"
gitlab-runner start- Modify your CI pipeline / Workflow to setup your GPU in your DVC job.
Github
# Github
dvc:
runs-on: [self-hosted]
container:
image: docker://dvcorg/dvc-cml-gpu:latest
options: --runtime "nvidia" -e NVIDIA_VISIBLE_DEVICES=allGitlab
# Gitlab
dvc:
tags:
- dvc-cml
stage: dvc_action_run
image: dvcorg/dvc-cml-gpu:latest
variables:
NVIDIA_VISIBLE_DEVICES: all
...- "My runner says: Got permission denied while trying to connect to the Docker daemon socket". You need to add your user to the docker group. Check your OS configuration for further details. Recipe for ubuntu:
sudo groupadd docker
sudo usermod -aG docker ${USER}
su -s ${USER}-
"With Github runners I can't specify custom tags to reach different runners". We know, It's a Github limitation.
-
"I have followed all the steps and I could not make it work". Try to run nvidia-smi in the
runsection in your workflow and see if gpu is available to your docker container.
DVC support different kinds of remote storage. To setup them properly you have to setup credentials (if needed) as Github secrets or Gitlab masked enviroment variables to keep them secure. Additionally in Github you need to add them as env variables in the workflow file.
S3 and S3 compatible storage (Minio, DigitalOcean Spaces, IBM Cloud Object Storage...)
# Github
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_SESSION_TOKEN: ${{ secrets.AWS_SESSION_TOKEN }}๐ AWS_SESSION_TOKEN is optional.
Azure
env:
AZURE_STORAGE_CONNECTION_STRING:
${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}
AZURE_STORAGE_CONTAINER_NAME: ${{ secrets.AZURE_STORAGE_CONTAINER_NAME }}Aliyn
env:
OSS_BUCKET: ${{ secrets.OSS_BUCKET }}
OSS_ACCESS_KEY_ID: ${{ secrets.OSS_ACCESS_KEY_ID }}
OSS_ACCESS_KEY_SECRET: ${{ secrets.OSS_ACCESS_KEY_SECRET }}
OSS_ENDPOINT: ${{ secrets.OSS_ENDPOINT }}Google Storage
โ ๏ธ Normally, GOOGLE_APPLICATION_CREDENTIALS points to the path of the json file that contains the credentials. However in the action this variable CONTAINS the content of the file. Copy that json and add it as a secret.
env:
GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}Google Drive
โ ๏ธ After configuring your Google Drive credentials you will find a json file atyour_project_path/.dvc/tmp/gdrive-user-credentials.json. Copy that json and add it as a secret.
env:
GDRIVE_CREDENTIALS_DATA: ${{ secrets.GDRIVE_CREDENTIALS_DATA }}SSH
โ ๏ธ Not supported yet
HDFS
โ ๏ธ Not supported yet
HTTP
โ ๏ธ Not supported yet