8000 GitHub - ahmedahmedov/cml at gpu_py3 ยท GitHub
[go: up one dir, main page]

Skip to content
ahmedahmedovPublic
forked from iterative/cml

About

CML - Continuous Machine Learning or CI/CD for ML

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

ย 

History

22 Commits

Folders and files

NameName
Last commit message
Last commit date
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 
ย 

Repository files navigation

DVC Github action for continuous delivery for machine learning

  1. Introduction
  2. Usage
  3. How to use GPUs
  4. Working with DVC remotes
  5. Examples

Introduction

DVC is a great tool as a data versioning system, but also is great as a build tool for ML experimentation. This repo offers the possibility of using DVC to establish your ML pipeline to be run by Github Actions runners or Gitlab runners.

You can also deploy your own Github runners or your own Gitlab runners with special capabilities like GPUs...

Major benefits of using DVC-CML in your ML projects includes:

  • Reproducibility: DVC is always in charge of maintain your experiment tracking all the dependencies, so you don't have to. Additionally your experiment is always running under the same constrains so you don't have to worry about replicating the same environment again.
  • Observability: DVC offers you metrics to be tracked. In DVC-action we make those metrics more human friendly and we also offer direct access to other experiments run through the DVC Report offered as checks in Github or Releases in Gitlab.
  • Releases: DVC-action tags every experiment that runs with repro generating the report. Aside of that DVC-CML is just a step in your Github Workflow or Gitlab Pipeline that could generate your model releases or deployment according to your business requirements.
  • Teaming: Give visibility to your experiments or releases to your teammates working together.

DVC-cml performs in your push or pull requests:

  1. DVC repro
  2. Push changes into DVC remote and Git remote
    • In Github generates a Github check displaying the DVC Report
    • In Gitlab generates a Tag/Release displaying the DVC Report

image image

Usage

DVC-CML for Github

๐Ÿ‘€ Knowledge of Github Actions and DVC pipeline is very useful for a fully comprehension.

Example of a simple DVC-CML workflow:

๐Ÿ‘€ Note the use of the container

name: your-workflow-name

on: [push, pull_request]

jobs:
  run:
    runs-on: [ubuntu-latest]
    container: docker://dvcorg/dvc-cml:latest

    steps:
      - uses: actions/checkout@v2

      - name: dvc_cml_run
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        repo_token: ${{ secrets.GITHUB_TOKEN }}
        repro_targets: your_dvc_target.dvc
      run: |
        # Install your project dependencies.
        # An example for Python3:
        apt-get install -y python3 python3-pip
        pip3 install --upgrade pip
        update-alternatives --install /usr/bin/python python $(which python3) 10
        update-alternatives --install /usr/bin/pip pip $(which pip3) 10
        test -f requirements.txt && pip3 install -r requirements.txt
        # Run report:
        dvc_cml_run
DVC-CML for Gitlab

๐Ÿ‘€ Knowledge of Gitlab CI/CD Pipeline and DVC pipeline is very useful for a fully comprehension.

Example of a simple DVC-CML workflow in Gitlab:

๐Ÿ‘€ Some required environment variables like remote credentials and GITLAB_TOKEN are set as CI/CD environment variables in Gitlab's UI

โš ๏ธ tag_prefix should be set in order to have DVC Reports, i.e. dvc_ . This will generate tags in your repo with the report as release notes image

# .gitlab-ci.yml
stages:
  - dvc_cml_run

dvc:
  stage: dvc_cml_run
  image: dvcorg/dvc-cml:latest
  variables:
    repro_targets: 'eval.dvc'
  script:
    - pip install tensorflow wget
    - dvc_cml_run

This workflow will run every time that you push code or do a Pull/Merge Request. When triggered DVC-CML will setup the runner and DVC will run the pipelines specified by repro_targets. Two scenarios may happen:

  1. DVC repro is up to date and there is nothing to do. This means that the commit that you have done in your code is not related to your DVC pipelines and there is nothing to do.
  2. DVC pipeline has changed and DVC will run repro, updating the output that may generate (models, data...) in your DVC remote storage and then committing, tagging and pushing the changes in git remote.

Additionally, you may extend your CI/CD Pipeline/Workflow to generate your releases or even deploy automatically your models.

Support for [ci skip] comment in Github

Many CI/CD vendors supports a special comment [ci skip] in the commit avoid run the CI. We support this, ff your commit comment includes the tag the DVC action will skip the CI returning an exit code of 0. We know that ideally the code should be 78 however, at the time of this writing, Github is only accepting 0 or 1 as status codes.

Variables

โš ๏ธ In Github Actions they are set via env: not inputs:

๐Ÿ‘€ In Gitlab pipeline they are set via variables:

Variable Type Required Default Info
repo_token string yes In Github you can set the default autogenerated GITHUB_TOKEN. In Gitlab you have to generate it. See Tensorflow Mnist in Gitlab example for a complete walkthrough.
baseline string no origin/master Revision to be compared with current experiment, i.e. origin/master, HEAD~1 or a commit sha.
repro_targets string no Dvcfile Comma delimited array of DVC files. If - is given will skip the process.
metrics_diff_targets string no Comma delimited array of metrics files. If not specified will use all the metric files.
tag_prefix string no If set a new tag will be created in the repo with the name ${tag_prefix}${short_sha}. This will enable the "Latest 5 experiments in the branch" list in the report and will enable the DVC Report in Gitlab as a release description.
metrics_format string no 0[.][0000000] Metrics format following [numeral.js](http://numeraljs.com/
dvc_pull string no Comma delimited array of targets. If not specified will pull everything. If - is given dvc won't pull

โš ๏ธ In Gitlab is required that you generate the GITLAB_TOKEN that is analogous to GITHUB_TOKEN. See Tensorflow Mnist in Gitlab example for a complete walkthrough.

How to use GPUs

Our DVC-CML GPU docker image is an Ubuntu 18.04 that already supports:

  • cuda 10.1
  • libcudnn 7
  • cublas 10
  • libinfer 10

Setup

  1. You need to setup properly your nvidia drivers and nvidia-docker in your host machine.
     sudo ubuntu-drivers autoinstall
     sudo apt-get install nvidia-docker2
     sudo systemctl restart docker
  2. Launch your own runner following your CI vendor instructions.
Github

Repo settings -> Actions -> Add Runner button

Gitlab

Repo settings -> CI/CD -> Runners -> Specific Runners

# Gitlab self-hosted runner with dvc-cml and GPU
gitlab-runner register \
    --non-interactive \
    --run-untagged="true" \
    --locked="false" \
    --access-level="not_protected" \
    --executor "docker" \
    --docker-runtime "nvidia" \
    --docker-image "dvcorg/dvc-cml-gpu:latest" \
    --url "https://gitlab.com/" \
    --tag-list "dvc-cml" \
    --registration-token "here_goes_your_gitlab
8384
_runner_token"

gitlab-runner start
  1. Modify your CI pipeline / Workflow to setup your GPU in your DVC job.
Github
# Github
dvc:
  runs-on: [self-hosted]
  container:
    image: docker://dvcorg/dvc-cml-gpu:latest
    options: --runtime "nvidia" -e NVIDIA_VISIBLE_DEVICES=all
Gitlab
# Gitlab
dvc:
 tags:
   - dvc-cml
 stage: dvc_action_run
 image: dvcorg/dvc-cml-gpu:latest

 variables:
   NVIDIA_VISIBLE_DEVICES: all
   ...

Pitfalls

  • "My runner says: Got permission denied while trying to connect to the Docker daemon socket". You need to add your user to the docker group. Check your OS configuration for further details. Recipe for ubuntu:
sudo groupadd docker
sudo usermod -aG docker ${USER}
su -s ${USER}
  • "With Github runners I can't specify custom tags to reach different runners". We know, It's a Github limitation.

  • "I have followed all the steps and I could not make it work". Try to run nvidia-smi in the run section in your workflow and see if gpu is available to your docker container. image

Working with DVC remotes

DVC support different kinds of remote storage. To setup them properly you have to setup credentials (if needed) as Github secrets or Gitlab masked enviroment variables to keep them secure. Additionally in Github you need to add them as env variables in the workflow file.

S3 and S3 compatible storage (Minio, DigitalOcean Spaces, IBM Cloud Object Storage...)
# Github
env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_SESSION_TOKEN: ${{ secrets.AWS_SESSION_TOKEN }}

๐Ÿ‘‰ AWS_SESSION_TOKEN is optional.

Azure
env:
  AZURE_STORAGE_CONNECTION_STRING:
    ${{ secrets.AZURE_STORAGE_CONNECTION_STRING }}
  AZURE_STORAGE_CONTAINER_NAME: ${{ secrets.AZURE_STORAGE_CONTAINER_NAME }}
Aliyn
env:
  OSS_BUCKET: ${{ secrets.OSS_BUCKET }}
  OSS_ACCESS_KEY_ID: ${{ secrets.OSS_ACCESS_KEY_ID }}
  OSS_ACCESS_KEY_SECRET: ${{ secrets.OSS_ACCESS_KEY_SECRET }}
  OSS_ENDPOINT: ${{ secrets.OSS_ENDPOINT }}
Google Storage

โš ๏ธ Normally, GOOGLE_APPLICATION_CREDENTIALS points to the path of the json file that contains the credentials. However in the action this variable CONTAINS the content of the file. Copy that json and add it as a secret.

env:
  GOOGLE_APPLICATION_CREDENTIALS: ${{ secrets.GOOGLE_APPLICATION_CREDENTIALS }}
Google Drive

โš ๏ธ After configuring your Google Drive credentials you will find a json file at your_project_path/.dvc/tmp/gdrive-user-credentials.json. Copy that json and add it as a secret.

env:
  GDRIVE_CREDENTIALS_DATA: ${{ secrets.GDRIVE_CREDENTIALS_DATA }}
SSH

โš ๏ธ Not supported yet

HDFS

โš ๏ธ Not supported yet

HTTP

โš ๏ธ Not supported yet

Examples

About

CML - Continuous Machine Learning or CI/CD for ML

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0