8000 ci: run the backend tests in parallel, start every CI job at once by adeelahmad · Pull Request #328 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 67 additions & 0 deletions .github/actions/affected/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
name: Affected
description: >-
Sets up pnpm, Node and Nx, and asks Nx whether the change affects a project. A pull request is compared with the main
it merges into, a push to main with main's last green CI run; a run by hand affects every project. Each CI job asks
for itself, so none of them waits for another job (and for a free runner a second time) before it starts.
inputs:
project:
description: The project to ask about; empty asks about every project but backend and frontend
required: false
default: ""
cache-dependency-path:
description: The lockfiles that key the pnpm cache
required: false
default: pnpm-lock.yaml
outputs:
run:
description: "'true' when the project (or, with no project, any other project) is affected, or Nx couldn't say"
value: ${{ steps.affected.outputs.run }}
projects:
description: The affected projects, as a JSON list; empty when Nx couldn't say
value: ${{ steps.affected.outputs.projects }}
others:
description: The affected projects but backend and frontend, as a JSON list; empty when Nx couldn't say
value: ${{ steps.affected.outputs.others }}
runs:
using: composite
steps:
- uses: pnpm/action-setup@v6
with:
# the pnpm version comes from packageManager in package.json (the same in the frontend's)
run_install: false
- uses: actions/setup-node@v7
with:
node-version: "22"
# on a self-hosted runner pnpm's store stays on disk; saving it to GitHub's cache took minutes
cache: ${{ runner.environment == 'github-hosted' && 'pnpm' || '' }}
cache-dependency-path: ${{ inputs.cache-dependency-path }}
- name: Install Nx
shell: bash
working-directory: ${{ github.workspace }}
run: pnpm install --frozen-lockfile
- name: The commits to compare
if: github.event_name != 'workflow_dispatch'
uses: nrwl/nx-set-shas@v5
- name: Is it affected?
id: affected
shell: bash
working-directory: ${{ github.workspace }}
env:
PROJECT: ${{ inputs.project }}
run: |
# when Nx can't say, the job runs: a wrong answer here must never skip a check
set -o pipefail
if [ "$GITHUB_EVENT_NAME" = workflow_dispatch ]; then which=(); else which=(--affected); fi
if ! projects=$(pnpm exec nx show projects "${which[@]}" --json | jq -ce 'arrays'); then
echo "::warning::Nx couldn't list the affected projects, so this job runs"
echo "run=true" >> "$GITHUB_OUTPUT"
exit 0
fi
others=$(jq -c 'map(select(. != "backend" and . != "frontend"))' <<<"$projects")
if [ -n "$PROJECT" ]; then
run=$(jq --arg p "$PROJECT" 'any(.[]; . == $p)' <<<"$projects")
else
run=$(jq 'length > 0' <<<"$others")
fi
echo "Affected: $projects"
{ echo "run=$run"; echo "projects=$projects"; echo "others=$others"; } >> "$GITHUB_OUTPUT"
52 changes: 52 additions & 0 deletions .github/actions/backend-setup/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Backend setup
description: >-
What the backend's tests need on the runner: ffmpeg, tesseract, poppler and LibreOffice Writer (kept between runs in
GitHub's cache), uv with Python 3.12, and the backend's packages.
runs:
using: composite
steps:
# the packages are kept between runs: the mirror can be slow (LibreOffice is 170 MB)
# GitHub's cache only on GitHub's runners; a self-hosted one keeps ~/apt-archives on its disk
- name: Restore system packages
if: runner.environment == 'github-hosted'
id: apt-cache
uses: actions/cache/restore@v6
with:
path: ~/apt-archives
key: apt-${{ runner.os }}-${{ hashFiles('.github/actions/backend-setup/action.yml') }}
restore-keys: apt-${{ runner.os }}-
- name: System packages (ffmpeg, tesseract, poppler, LibreOffice Writer)
if: runner.os == 'Linux'
shell: bash
run: |
# a self-hosted runner that has them already needs no sudo
have=yes; for t in ffmpeg tesseract pdftoppm soffice; do command -v "$t" >/dev/null || have=""; done
if [ -n "$have" ]; then echo "already installed"; exit 0; fi
mkdir -p ~/apt-archives/partial
APT=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o "Dir::Cache::Archives=$HOME/apt-archives" -o APT::Keep-Downloaded-Packages=true)
# a stalled mirror fails the job instead of holding it for hours
timeout 25m sudo apt-get "${APT[@]}" update
timeout 25m sudo apt-get "${APT[@]}" install -y --no-install-recommends ffmpeg tesseract-ocr poppler-utils libreoffice-writer-nogui
sudo chown -R "$(id -u)" ~/apt-archives
- name: Keep system packages
if: runner.environment == 'github-hosted' && steps.apt-cache.outputs.cache-hit != 'true'
uses: actions/cache/save@v6
with:
path: ~/apt-archives
key: apt-${{ runner.os }}-${{ hashFiles('.github/actions/backend-setup/action.yml') }}
- name: System packages on macOS (ffmpeg, tesseract, poppler; LibreOffice tests skip without it)
if: runner.os == 'macOS'
shell: bash
env:
HOMEBREW_NO_AUTO_UPDATE: "1"
run: for p in ffmpeg tesseract poppler; do brew list "$p" >/dev/null 2>&1 || brew install "$p"; done

- uses: astral-sh/setup-uv@v7
with:
# a cache that outlives the job on the self-hosted runner (the default is in the job's temp folder)
cache-local-path: ${{ runner.environment == 'self-hosted' && format('{0}/uv-cache', runner.tool_cache) || '' }}
python-version: "3.12"
- name: Install
shell: bash
working-directory: ${{ github.workspace }}/fastapi_backend
run: uv sync --frozen
Loading
Loading
0