8000 Add Knulli/RG35XX SP packaging with pygame-ce squashfs runtime by jmahmood · Pull Request #27 · jmahmood/CardBrick · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,7 @@ test_cache/
CLAUDE.md
venv-syncd
workspace

# Knulli packaging outputs (deploy/knulli)
deploy/knulli/dist/
deploy/knulli/runtime-build/build/
151 changes: 151 additions & 0 deletions deploy/knulli/CardBrick.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
#!/bin/bash
# CardBrick — Knulli / RG35XX SP launch script (pygame-ce squashfs runtime).
#
# Lives at /userdata/roms/ports/CardBrick.sh next to the CardBrick/
# folder (see PACKAGING.md). Knulli's Ports menu runs it as root.
# Runtime model derived from https://github.com/viniciusfs/pygame-ce-runtime:
# a self-contained ARM64 Python venv in a squashfs image is loop-mounted,
# and the app runs on that interpreter. pygame-ce inside the runtime is
# linked against the FIRMWARE's SDL2, so Knulli's mali/kmsdrm video
# drivers work.
#
# Usage (also over SSH):
# ./CardBrick.sh # normal launch: study, fullscreen
# ./CardBrick.sh --smoke-test # non-interactive deployment check
# ./CardBrick.sh --input-diagnostic # controller test & calibration
# ./CardBrick.sh <any main.py args> # passthrough
#
# The app/runtime tree is treated as read-only; ALL mutable state goes
# to $CARD_BRICK_DATA_DIR (default /userdata/saves/cardbrick).

set -u # undefined vars are bugs; no set -e — we want to log failures

SCRIPT_DIR="$(dirname "$(readlink -f "$0")")"
GAMEDIR="${SCRIPT_DIR}/CardBrick"
APP_DIR="${GAMEDIR}/cardbrick-py"

# ------------------------------------------------------------- data dir
CARD_BRICK_DATA_DIR="${CARD_BRICK_DATA_DIR:-/userdata/saves/cardbrick}"
export CARD_BRICK_DATA_DIR
LOG_DIR="${CARD_BRICK_DATA_DIR}/logs"
LAUNCH_LOG="${LOG_DIR}/launch.log"
mkdir -p "$LOG_DIR" || {
echo "FATAL: cannot create ${LOG_DIR} — is /userdata writable?" >&2
exit 1
}

log() { echo "$*" >> "$LAUNCH_LOG"; }
fail() {
log "FATAL: $*"
echo "FATAL: $*" >&2
exit 1
}

log "=== CardBrick launch $(date) ==="

[ -d "$APP_DIR" ] || fail "app directory not found: $APP_DIR"

# -------------------------------------------------------------- runtime
# Newest squashfs wins, so a runtime upgrade is just one file copy.
RUNTIME_SQUASHFS="$(ls -1t "${GAMEDIR}/runtime/"pygame-ce_*.squashfs 2>/dev/null | head -n 1)"
RUNTIME_EXTRACTED="${GAMEDIR}/runtime/extracted" # unsquashed fallback
MOUNTPOINT="${CARDBRICK_RUNTIME_MOUNT:-/tmp/cardbrick-runtime}"

MOUNTED="" # how we mounted, for cleanup: "", "loop" or "fuse"
RUNTIME_DIR=""

cleanup() {
cd /
case "$MOUNTED" in
loop) umount "$MOUNTPOINT" 2>/dev/null ;;
fuse) fusermount -u "$MOUNTPOINT" 2>/dev/null \
|| umount "$MOUNTPOINT" 2>/dev/null ;;
esac
}
trap cleanup EXIT INT TERM

if [ -n "$RUNTIME_SQUASHFS" ]; then
mkdir -p "$MOUNTPOINT"
# A stale mount from a crashed run would shadow the new runtime.
umount "$MOUNTPOINT" 2>/dev/null
if mount -o loop,ro "$RUNTIME_SQUASHFS" "$MOUNTPOINT" 2>>"$LAUNCH_LOG"; then
MOUNTED="loop"
RUNTIME_DIR="$MOUNTPOINT"
elif command -v squashfuse >/dev/null 2>&1 \
&& squashfuse "$RUNTIME_SQUASHFS" "$MOUNTPOINT" 2>>"$LAUNCH_LOG"; then
MOUNTED="fuse"
RUNTIME_DIR="$MOUNTPOINT"
else
log "WARN: could not mount $RUNTIME_SQUASHFS"
fi
fi

if [ -z "$RUNTIME_DIR" ] && [ -x "${RUNTIME_EXTRACTED}/bin/python3" ]; then
log "using extracted runtime fallback: $RUNTIME_EXTRACTED"
RUNTIME_DIR="$RUNTIME_EXTRACTED"
fi

if [ -n "$RUNTIME_DIR" ]; then
PYTHON="${RUNTIME_DIR}/bin/python3"
[ -x "$PYTHON" ] || fail "runtime mounted but ${PYTHON} is missing/not executable"
# stdlib dir, e.g. .../lib/python3.12 (version-agnostic glob)
PY_LIB="$(ls -d "${RUNTIME_DIR}/lib/"python3.* 2>/dev/null | head -n 1)"
[ -n "$PY_LIB" ] || fail "no lib/python3.* inside runtime ${RUNTIME_DIR}"
export PYTHONHOME="$RUNTIME_DIR"
export PYTHONPATH="${PY_LIB}:${PY_LIB}/site-packages"
# libpython3.X.so.1.0 lives in runtime/lib (venv python is a copy of
# a --enable-shared build)
export LD_LIBRARY_PATH="${RUNTIME_DIR}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
else
# Last resort: firmware python3 (needs pygame available system-wide).
log "WARN: no usable bundled runtime; falling back to system python3"
PYTHON="python3"
command -v python3 >/dev/null 2>&1 || fail "no bundled runtime and no system python3"
fi

# The squashfs (and possibly the whole SD card) is read-only — keep
# .pyc files out of the app tree.
export PYTHONPYCACHEPREFIX="${CARD_BRICK_DATA_DIR}/.pycache"

# ------------------------------------------------------------------ SDL
# `mali` is what the upstream runtime validated on Knulli / Allwinner
# H700 (RG35XX family). If the screen stays black try kmsdrm:
# SDL_VIDEODRIVER=kmsdrm ./CardBrick.sh
export SDL_VIDEODRIVER="${SDL_VIDEODRIVER:-mali}"
export SDL_AUDIODRIVER="${SDL_AUDIODRIVER:-alsa}"

# ---------------------------------------------------------------- launch
{
echo "GAMEDIR=${GAMEDIR}"
echo "RUNTIME_SQUASHFS=${RUNTIME_SQUASHFS:-<none>}"
echo "RUNTIME_DIR=${RUNTIME_DIR:-<system python>}"
echo "PYTHON=${PYTHON}"
echo "PYTHONHOME=${PYTHONHOME:-<unset>}"
echo "PYTHONPATH=${PYTHONPATH:-<unset>}"
echo "SDL_VIDEODRIVER=${SDL_VIDEODRIVER}"
echo "SDL_AUDIODRIVER=${SDL_AUDIODRIVER}"
echo "CARD_BRICK_DATA_DIR=${CARD_BRICK_DATA_DIR}"
"$PYTHON" --version 2>&1
} >> "$LAUNCH_LOG"

cd "$APP_DIR" || fail "cannot cd to ${APP_DIR}"

# First boot on this device: run the smoke test once and keep its output.
if [ ! -f "${CARD_BRICK_DATA_DIR}/.smoke-ok" ] && [ "$#" -eq 0 ]; then
if "$PYTHON" main.py --knulli --smoke-test >> "$LAUNCH_LOG" 2>&1; then
touch "${CARD_BRICK_DATA_DIR}/.smoke-ok"
log "first-run smoke test PASSED"
else
log "first-run smoke test FAILED — see above; starting anyway"
fi
fi

if [ "$#" -gt 0 ]; then
# Passthrough mode (e.g. --smoke-test over SSH). Show output on the
# terminal AND keep it in the log.
"$PYTHON" main.py --knulli "$@" 2>&1 | tee -a "$LAUNCH_LOG"
exit_code=${PIPESTATUS[0]}
exit "$exit_code"
fi

"$PYTHON" main.py --knulli study --fullscreen >> "$LAUNCH_LOG" 2>&1
49 changes: 49 additions & 0 deletions deploy/knulli/Makefile
8000
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# CardBrick — Knulli / RG35XX SP packaging (pygame-ce squashfs runtime).
# Run from deploy/knulli/. See PACKAGING.md for the full walkthrough.
#
# make runtime build the ARM64 squashfs runtime (Docker, slow, once)
# make package stage dist/ + zip (needs the runtime, or
# RUNTIME_SQUASHFS=/path/to/file.squashfs)
# make package-fast app-only package, no runtime (quick iteration)
# make validate check the staged package before deploying
# make deploy-sd copy to a mounted SD card [SD=/media/me/SHARE]
# make deploy-ssh push over SSH [HOST=root@192.168.1.42]
# make smoke-ssh deploy over SSH + run the on-device smoke test
# make clean remove dist/ (keeps the built runtime)
# make distclean remove dist/ AND the built runtime

SD ?=
HOST ?=

HOSTFLAG = $(if $(HOST),--host $(HOST),)

.PHONY: runtime package package-fast validate deploy-sd deploy-ssh smoke-ssh clean distclean

runtime:
scripts/build_runtime.sh

package:
scripts/build_package.sh
scripts/validate_package.sh

package-fast:
scripts/build_package.sh --no-runtime
scripts/validate_package.sh

validate:
scripts/validate_package.sh

deploy-sd:
scripts/deploy_sdcard.sh $(SD)

deploy-ssh:
scripts/deploy_ssh.sh $(HOSTFLAG)

smoke-ssh:
scripts/deploy_ssh.sh $(HOSTFLAG) --smoke

clean:
rm -rf dist

distclean: clean
rm -rf runtime-build/build
181 changes: 181 additions & 0 deletions deploy/knulli/PACKAGING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
# Packaging CardBrick for Knulli / RG35XX SP (pygame-ce runtime)

This is the supported way to put CardBrick on a Knulli handheld: the
app ships unchanged, next to a self-contained ARM64 Python runtime
built with the approach from
[viniciusfs/pygame-ce-runtime](https://github.com/viniciusfs/pygame-ce-runtime)
— a portable Python 3.12 virtual environment with pygame-ce, packed
into a single SquashFS image that the launch script loop-mounts at
start and unmounts at exit. The device needs **no preinstalled Python,
no pip, no network**.

Why a source-built runtime instead of pip wheels: the manylinux
pygame-ce wheel bundles its own SDL2, which knows nothing about the
handheld's display. The runtime builder compiles pygame-ce against
SDL2 *headers* so it links `libSDL2-2.0.so.0` dynamically — on the
device that resolves to Knulli's firmware SDL2, which has the
`mali`/`kmsdrm` video drivers the RG35XX family actually needs. The
Python 3.12.8 + pygame-ce 2.5.6 pair is what upstream validated on an
RG35XX-H running Knulli (same Allwinner H700 family as the RG35XX SP).

Everything below runs from `deploy/knulli/` on a PC.

```
deploy/knulli/
├── Makefile # make runtime / package / validate / deploy-*
├── CardBrick.sh # on-device launch script (goes to roms/ports/)
├── runtime-build/ # Docker ARM64 runtime builder
│ ├── build-config # PYTHON_VERSION / PYGAME_VERSION pins
│ ├── docker-compose.yml
│ ├── build-runtime.sh # runs inside the container
│ └── requirements.txt # extra deps baked into the runtime (fsrs, …)
├── scripts/
│ ├── build_runtime.sh # host wrapper for the Docker build
│ ├── build_package.sh # stage dist/ + zip
│ ├── validate_package.sh # pre-deploy checks ([PASS]/[WARN]/[FAIL])
│ ├── deploy_sdcard.sh # copy to a mounted SD card
│ └── deploy_ssh.sh # push over SSH (+ on-device smoke test)
└── dist/ # build output (gitignored)
```

## 0. Prerequisites (PC)

- **Docker + docker compose** — only for building the runtime image.
On an x86 PC, ARM64 emulation must be enabled once:

```sh
docker run --privileged --rm tonistiigi/binfmt --install arm64
```

- `zip` (for the SD-card archive), `python3` (validation),
optionally `squashfs-tools` (deeper validation) and `rsync`
(faster SD copies). None of these are needed on the device.

No Docker? You can point the packager at any compatible prebuilt
runtime file instead:
`RUNTIME_SQUASHFS=/path/to/pygame-ce_2.5.6_python_3.12.8.squashfs make package`.

## 1. Build the runtime (once, and after version bumps)

```sh
cd deploy/knulli
make runtime
```

This starts a Debian ARM64 container, compiles pygame-ce 2.5.6 from
source, installs CardBrick's runtime deps (`runtime-build/requirements.txt`),
and packs the venv into
`runtime-build/build/pygame-ce_2.5.6_python_3.12.8.squashfs` (plus a
`runtime-manifest.txt` recording exactly what went in). Under
emulation on an x86 host this takes a while — it is a one-time cost;
the artifact is reused by every subsequent package build.

Version pins live in `runtime-build/build-config`. Optional
`runtime-build/pre-install.sh` / `post-install.sh` hooks run inside
the container before/after the venv is assembled.

## 2. Build + validate the package

```sh
make package # = build_package.sh + validate_package.sh
```

Produces:

- `dist/CardBrick.sh` and `dist/CardBrick/` — exactly what lands in
`/userdata/roms/ports/` on the device (app code minus tests/caches,
the runtime squashfs, `VERSION`, `BUILD_INFO`, and a
`PACKAGE_MANIFEST.sha256` covering every file);
- `dist/CardBrick-knulli-v<version>.zip` — the same tree zipped, for
manual SD-card installs.

`make validate` re-runs the checks alone: launch-script syntax,
required files, no stray caches, host-side `py_compile` of every
source, squashfs integrity and contents (python3/libpython/pygame/fsrs
present), checksum manifest, zip integrity.

During development, `make package-fast` builds an app-only package
(no runtime inside) — pair it with `deploy_ssh.sh`, which never
re-uploads a runtime the device already has.

## 3a. Deploy by SD card

Either unzip `dist/CardBrick-knulli-v*.zip` into the card's
`roms/ports/` yourself, or let the script find the card:

```sh
make deploy-sd # auto-detect mounted SHARE partition
make deploy-sd SD=/media/$USER/SHARE
scripts/deploy_sdcard.sh /Volumes/SHARE --clean # wipe old app dir first
```

The script locates `roms/ports/` (SHARE root or bare roms partition),
copies `CardBrick.sh` + `CardBrick/`, and runs `sync` before telling
you it is safe to eject. On the device: if the Ports menu doesn't show
CardBrick, Start menu → Games Settings → **Update Gamelists** (or
reboot).

## 3b. Deploy over SSH

Knulli's SSH defaults: user `root`, password `linux`. Find the IP in
the device's main menu (Network). For scripted deploys install a key
once: `ssh-copy-id root@<ip>`.

```sh
make deploy-ssh HOST=root@192.168.1.42
make smoke-ssh HOST=root@192.168.1.42 # deploy + on-device smoke test
scripts/deploy_ssh.sh --host root@192.168.1.42 --no-runtime # code only
```

The script uploads with tar-over-ssh (no rsync needed on the device),
skips the runtime upload when the on-device checksum already matches,
verifies `PACKAGE_MANIFEST.sha256` on the device, pokes
EmulationStation to reload the games list, and with `--smoke` runs
the app's own `--smoke-test` on real hardware — one `[PASS]/[FAIL]`
line per subsystem (display, font, DB, scheduler, controller, audio).

## 4. On the device

```
/userdata/roms/ports/CardBrick.sh # Ports-menu entry
/userdata/roms/ports/CardBrick/
├── cardbrick-py/ # the app, read-only
├── runtime/pygame-ce_*.squashfs # mounted at /tmp/cardbrick-runtime
├── VERSION BUILD_INFO PACKAGE_MANIFEST.sha256

/userdata/saves/cardbrick/ # ALL mutable data (auto-created)
├── cardbrick.db settings.json input_mapping.json media/
└── logs/launch.log + cardbrick.log # start here when debugging
```

The launch script mounts the newest `runtime/*.squashfs`, exports
`PYTHONHOME`/`PYTHONPATH`/`LD_LIBRARY_PATH` into it, keeps `.pyc`
files in the data dir (`PYTHONPYCACHEPREFIX` — the app tree stays
read-only), runs a one-time smoke test on first boot, and unmounts on
exit. Run it manually over SSH for diagnostics:

```sh
bash /userdata/roms/ports/CardBrick.sh --smoke-test
bash /userdata/roms/ports/CardBrick.sh --input-diagnostic
```

## 5. Troubleshooting

| Symptom | Likely cause / fix |
|---|---|
| Black screen, app in log | Video driver. Default is `mali` (upstream-validated on Knulli/H700). Try `SDL_VIDEODRIVER=kmsdrm bash CardBrick.sh`; the winner can be exported permanently by editing `CardBrick.sh`. |
| `could not mount … squashfs` in launch.log | Rare on Knulli (ports run as root). Fallbacks: install nothing — just unsquash on the PC and ship `CardBrick/runtime/extracted/` instead. |
| `ImportError`/ABI errors at pygame init | Runtime/firmware mismatch — rebuild the runtime; don't mix wheels into it. The smoke test catches this at the `pygame init` step. |
| Smoke test WARN on joystick/audio over SSH | Normal when ES owns the controller or audio is busy; verify from the Ports menu launch. |
| Ports menu doesn't list CardBrick | Update Gamelists (Start → Games Settings) or reboot. |
| Wrong/laggy buttons | Parent Mode → Controller test & setup; hold any button ~3 s to force calibration. |

## Relationship to the older vendored-wheels docs

`README.md` / `package_layout.md` / `launch_cardbrick_spanish.sh` in
this folder describe the earlier, runtime-less option (firmware
python3 + `pip install --target vendor`). It still works as a
fallback when Docker is unavailable, but the squashfs runtime above is
the recommended path — it pins the interpreter, survives firmware
Python changes, and matches a combination already proven on this
hardware family. `smoke_test_checklist.md` applies to both.
Loading
Loading
0