Automated progress tracking for the Mwmbl search engine, displayed at progress.mwmbl.org (GitHub Pages).
┌─────────────────────────────────────────────────────────────┐
│ GitHub Actions (daily at 02:00 UTC) │
│ 1. Runs collect-metrics.js │
│ 2. Fetches: GitHub API (commits, blog posts) │
│ + mwmbl API (pages crawled) │
│ 3. Merges with manual-metrics.json (from main branch) │
│ 4. Writes metrics.json → pushes to `data` branch │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ GitHub Pages (gh-pages branch) │
│ - index.html + app.js (static, never changes) │
│ - On load: fetches metrics.json from `data` branch │
│ - Renders progress table client-side │
└─────────────────────────────────────────────────────────────┘
Branches:
main— site code +manual-metrics.json(config, rarely changes)data—metrics.json(auto-updated daily, force-pushed)gh-pages— built site (deployed via GitHub Pages)
- Git commits — across mwmbl, mwmbl_rank, front-end, book repos
- Pages crawled/day — from mwmbl internal API
- NDCG score — from rankeval data (placeholder, needs implementation)
- Blog posts — auto from mwmbl/blog repo content/ folder
- YouTube videos — manual (in manual-metrics.json)
- Volunteers — manual (Firefox extension stats)
- Employees — manual
- Incorporation milestones — manual (UK nonprofit, UK charity, US nonprofit)
- Affiliated organisations — manual
- Book commits — auto from book repo
# Install dependencies
npm install
# Run collector (requires env vars)
GITHUB_TOKEN=xxx MWMBL_API_KEY=xxx npm run collect
# View metrics.json
cat metrics.json
# Serve locally (any static server)
npx serve .
# or
python3 -m http.server 8000The page auto-detects local mode when running on localhost, 127.0.0.1, or file:// protocol. It will fetch metrics.json and manual-metrics.json from the local filesystem instead of GitHub.
To test locally:
- Run a local server:
python3 -m http.server 8000 - Open
http://localhost:8000— it will use localmetrics.jsonautomatically
To force production mode locally:
- Add
?local=0to URL:http://localhost:8000?local=0
To force local mode on production:
- Add
?local=1to URL, or - Run in console:
progressDebug.toggleLocal()
Console debugging:
progressDebug.isLocal() // true/false
progressDebug.getUrls() // { data, manual, mode }
progressDebug.toggleLocal() // Switch mode & reloadA sample metrics.json is included in the repo with example data matching the book.mwmbl.org progress page (June 2025 snapshot).
gh repo create mwmbl/progress --public --source=. --push- Settings → Pages → Source: "Deploy from a branch"
- Branch:
gh-pages// (root)
| Secret | Description |
|---|---|
GITHUB_TOKEN |
Auto-provided by GitHub (no action needed) |
MWMBL_API_KEY |
API key for mwmbl internal API (crawler stats) |
MWMBL_API_URL |
Optional: mwmbl API base URL (default: https://api.mwmbl.org) |
The gh-pages branch needs the static files. Either:
- Push
index.html+app.jstogh-pagesbranch manually, or - Add a deploy workflow (see below)
Edit manual-metrics.json on main branch for:
- YouTube video count
- Volunteer count
- Employee count
- Incorporation status
- Affiliated organisations
Create .github/workflows/deploy.yml:
name: Deploy to GitHub Pages
on:
push:
branches: [main]
workflow_dispatch:
permissions:
contents: read
pages: write
id-token: write
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/configure-pages@v4
- uses: actions/upload-pages-artifact@v3
with:
path: .
- uses: actions/deploy-pages@v4- Edit
manual-metrics.jsononmainbranch - Commit & push
- Next daily run (or manual trigger) will pick up new values
GET /repos/{owner}/{repo}/stats/commit_activity— commit countsGET /repos/{owner}/{repo}/contents/{path}— blog post countGET /api/v1/crawler/stats/— mwmbl internal (requires API key)
- Goals/points: Edit
manual-metrics.json→goalssection - Repos tracked: Edit
REPOSarray incollect-metrics.js - Schedule: Edit cron in
.github/workflows/collect-metrics.yml