8000 feat(packaging): self-contained Synology package (SPK) that runs Lens in Container Manager by adeelahmad · Pull Request #11 · adeelahmad/lense · GitHub
[go: up one dir, main page]

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
59 changes: 59 additions & 0 deletions .github/workflows/synology.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
name: Synology package

# Publishing a release builds both packages and attaches them to it: through the release event, or, for a release the
# Release workflow publishes with GITHUB_TOKEN (which fires no release event), through workflow_dispatch with `tag`.
# Run by hand without a tag, it attaches the .spk files to the run instead.
on:
release:
types: [published]
workflow_dispatch:
inputs:
tag:
description: "Release tag to build and attach the packages to (the Release workflow passes it); blank: build this ref"
default: ""
target:
description: "Backend image: lean, or full (LibreOffice and Chromium; a much bigger package)"
default: lean
type: choice
options: [lean, full]
build:
description: "Package build number, such as 0002 (blank: the run number, so each build upgrades the last)"
default: ""

jobs:
spk:
name: SPK (${{ matrix.arch }})
runs-on: ubuntu-latest
permissions:
contents: write # to attach the packages to the release
strategy:
matrix:
arch: [x86_64, armv8]
steps:
- uses: actions/checkout@v4
with:
ref: ${{ inputs.tag || github.ref }}
- uses: docker/setup-qemu-action@v3
if: matrix.arch == 'armv8'
- uses: docker/setup-buildx-action@v3
with:
driver: docker
- name: Build
env:
ARCH: ${{ matrix.arch }}
LENS_TARGET: ${{ inputs.target || 'lean' }}
INPUT_BUILD: ${{ inputs.build }}
RUN_NUMBER: ${{ github.run_number }}
# a release numbers its packages by run, so each one installs as an upgrade over the last
run: BUILD=${INPUT_BUILD:-$(printf '%04d' "$RUN_NUMBER")} packaging/synology/build.sh
- uses: actions/upload-artifact@v4
with:
name: lens-synology-${{ matrix.arch }}
path: dist/synology/*.spk
compression-level: 0
- name: Attach to the release
if: github.event_name == 'release' || inputs.tag != ''
env:
GH_TOKEN: ${{ github.token }}
TAG: ${{ github.event.release.tag_name || inputs.tag }}
run: gh release upload "$TAG" dist/synology/*.spk --clobber --repo "${{ github.repository }}"
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,14 @@ The backend (`fastapi_backend`) and the frontend (`nextjs-frontend`) are version

## Unreleased

- **Synology package.** `packaging/synology/build.sh` builds a self-contained `.spk` for DSM 7.2.1+ (x86_64, or
64-bit ARM with `ARCH=armv8`): Manual Install it in Package Center, answer a short wizard (port, address, setup
code), and Lens runs. The package carries the API/worker, web app and SurrealDB images and hands them to Container
Manager as a project through DSM's `docker-project` resource, so nothing is pulled and no container is set up by
hand. Data, the database and the generated secrets live in a `lens` shared folder that upgrades and uninstalls
leave alone; an upgrade removes the previous version's images. Mail can be set in the wizard, and any `MAIL_*` or
`LENS_*` line in the folder's `lens.conf` reaches the API and the worker. Publishing a GitHub release builds the packages for both architectures and attaches them to it
(the **Synology package** workflow, which also runs by hand from the Actions tab; packaging/synology/README.md).
- **Notifications to chat and webhooks.** A namespace's owners can have Lens tell a chat room or another app when a
run finishes or fails, a batch run finishes, or something is added: on the namespace's page under Notifications,
with a test button and a list of what each target was sent (docs/notifications.md).
Expand Down
16 changes: 16 additions & 0 deletions packaging/synology/INFO.in
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package="lens"
version="@VERSION@"
os_min_ver="7.2-69057"
arch="@ARCH@"
displayname="Lens"
description="Lens: transcribe, search and ask questions about your recordings and documents. Runs in Container Manager; everything it needs is inside the package."
maintainer="Adeel Ahmad"
maintainer_url="https://github.com/adeelahmad/lense"
support_url="https://github.com/adeelahmad/lense/issues"
thirdparty="yes"
install_dep_packages="ContainerManager>=20.10.23-1432"
startable="yes"
ctl_stop="yes"
dsmuidir="ui"
dsmappname="com.lens.app"
silent_upgrade="no"
88 changes: 88 additions & 0 deletions packaging/synology/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Lens for Synology DSM

A self-contained Synology package (`.spk`). Install it in Package Center and Lens runs: the package carries the
images for the API and worker, the web app and SurrealDB, and has Container Manager run them as a project called
`lens`. Nothing is pulled from a registry and nothing is set up by hand.

Needs DSM 7.2.1 or later with Container Manager (the package installs it if it's missing), on an x86_64 model or a
64-bit ARM model that Container Manager supports.

## Build

On any machine with Docker (and buildx, which Docker Desktop and current Docker Engine include):

```bash
packaging/synology/build.sh # x86_64, most Plus and Value models
ARCH=armv8 packaging/synology/build.sh # 64-bit ARM models (cross-builds: needs QEMU/binfmt on x86)
LENS_TARGET=full packaging/synology/build.sh # with LibreOffice and Chromium, to read Office files and web pages
```

The package lands in `dist/synology/lens-<version>-<build>-<arch>.spk`.

Releases carry them already: publishing a GitHub release runs the **Synology package** workflow, which builds both
architectures and attaches the `.spk` files to the release (numbered by run, so each installs as an upgrade over the
last). The workflow can also be run by hand from the Actions tab, for the `full` image or a chosen build number.

To find your NAS's architecture: Control Panel > Info Center shows the CPU; Intel and AMD are `x86_64`, Realtek
RTD1296/RTD1619 and Marvell Armada 37xx are `armv8`.

## Install

1. Package Center > **Manual Install** > choose the `.spk` > Next. Accept the third-party package warning.
2. In the wizard, pick the web port (3000 unless something else uses it), the address people will open Lens at
(leave it blank for `http://<NAS address>:<port>`), and a setup code. The mail server for password resets is an
optional second page.
3. When it's done, open Lens from the DSM main menu (or the address above), create the first admin account with the
setup code, and follow the setup steps (namespace, model provider, storage; each can be skipped).

The package creates a shared folder `lens` (or reuses one with that name):

| Folder | What's in it |
| --- | --- |
| `lens/db` | The SurrealDB database |
| `lens/archive` | Uploads, media and everything Lens derives from them |
| `lens/audio/podcasts`, `lens/audio/interviews` | Recordings Lens can scan into the two starting namespaces |
| `lens/lens.conf` | Secrets and the wizard's answers, kept with the data so a reinstall still reads it |

To change a setting later, edit `lens/lens.conf` and Manual Install the same `.spk` again. Any `MAIL_*` or `LENS_*` line
in it reaches the API and the worker as it is, so the variables in [Configuration](../../docs/configuration.md) work
there too: `LENS_LLM_BASE_URL` for a model server on your network, or `LENS_ADMIN_EMAIL`, `LENS_ADMIN_PASSWORD`,
`LENS_NAMESPACE` and `LENS_SETUP_WIZARD=off` to answer the first-run setup ahead of time
([First-run setup](../../docs/configuration.md#first-run-setup)).

Back up the whole `lens` folder (Hyper Backup) with Lens stopped. Uninstalling the package leaves the folder alone.

## Upgrade

Build with a higher `BUILD` (or a new Lens version) and Manual Install the new `.spk` over the old one. The data, the
secrets and the wizard's answers are kept; Container Manager loads the new images and recreates the containers.
The previous version's Lens images are removed once the new containers are up.

```bash
BUILD=0002 packaging/synology/build.sh
```

## How it works

The package uses DSM's `docker-project` resource (Container Manager 1432+, DSM 7.2.1+): `conf/resource` points
Container Manager at `compose/compose.yaml` inside the package and at the bundled image tarball, which it loads before
creating the project. Container Manager starts and stops the project with the package. The package's own scripts run
as an unprivileged package user, as DSM 7 requires; they only write the compose file.

- `scripts/postinst` resolves the `lens` shared folder, writes `lens.conf` (secrets generated once, never
overwritten) and renders `compose/compose.yaml` and the DSM menu entry (`ui/config`) from their templates.
- `package/compose/compose.yaml.in` is `docker-compose.prod.yml` with built images instead of build contexts, the
shared folder instead of named volumes, and a one-shot `permissions` container that hands `lens/archive` to the
API's user (uid 1000).
- The API uses the image's `docker/archive.yaml`, so it starts with the `podcasts` and `interviews` namespaces.
- A one-shot `cleanup` container (the Docker CLI, bundled) removes Lens images of earlier versions after an upgrade.
Its command is fixed in the compose file: it removes images carrying the `net.lens.package=synology` label that
`build.sh` puts on the Lens images, other than the current version's. It is given the Docker socket to do that,
which is as powerful as root on the NAS, and exits when done.

## Troubleshooting

- **Logs:** Container Manager > Project > `lens` > the container > Log. The API's log names the setup code too.
- **Port in use:** reinstall and pick another port; the data stays.
- **Behind a reverse proxy (HTTPS):** give its address in the wizard, so sign-in redirects and emailed links use it;
Control Panel > Login Portal > Advanced > Reverse Proxy can forward it to `localhost:<port>`.
101 changes: 101 additions & 0 deletions packaging/synology/WIZARD_UIFILES/install_uifile
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
[
{
"step_title": "Lens",
"items": [
{
"type": "textfield",
"desc": "Lens keeps its database and archive in the shared folder \"lens\", which this package creates (or reuses). Lens can scan recordings you put in its audio/podcasts and audio/interviews folders.",
"subitems": [
{
"key": "wizard_port",
"desc": "Web port",
"defaultValue": "3000",
"validator": {
"allowBlank": false,
"regex": {
"expr": "/^[1-9][0-9]{2,4}$/",
"errorText": "A port number, such as 3000"
}
}
},
{
"key": "wizard_url",
"desc": "Address people will open Lens at (blank: http://this NAS's address:port)",
"emptyText": "https://lens.example.org",
"validator": {
"allowBlank": true,
"regex": {
"expr": "/^https?:\\/\\/[A-Za-z0-9.\\-]+(:[0-9]+)?\\/?$/",
"errorText": "Like http://192.168.1.10:3000 or https://lens.example.org"
}
}
}
]
},
{
"type": "password",
"desc": "Choose a setup code. You enter it in the web app to create the first admin account.",
"subitems": [
{
"key": "wizard_setup_code",
"desc": "Setup code",
"validator": {
"allowBlank": false,
"minLength": 6,
"regex": {
"expr": "/^[A-Za-z0-9_\\-]+$/",
"errorText": "At least 6 letters, digits, - or _"
}
}
}
]
}
]
},
{
"step_title": "Email (optional)",
"items": [
{
"type": "textfield",
"desc": "Lens sends password resets and share notices by email. Leave these blank to skip; you can add them to lens.conf later.",
"subitems": [
{
"key": "wizard_mail_server",
"desc": "SMTP server",
"emptyText": "smtp.example.org"
},
{
"key": "wizard_mail_port",
"desc": "Port",
"defaultValue": "587",
"validator": {
"allowBlank": true,
"regex": {
"expr": "/^[0-9]{1,5}$/",
"errorText": "A port number, such as 587"
}
}
},
{
"key": "wizard_mail_username",
"desc": "User name"
},
{
"key": "wizard_mail_from",
"desc": "From address",
"emptyText": "lens@example.org"
}
]
},
{
"type": "password",
"subitems": [
{
"key": "wizard_mail_password",
"desc": "Password"
}
]
}
]
}
]
73 changes: 73 additions & 0 deletions packaging/synology/build.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
#!/bin/sh
# Builds a self-contained Synology package (SPK) for Lens: the API/worker image, the web app image and SurrealDB are
# saved inside it, so the NAS needs no registry access and nobody sets up containers by hand.
#
# packaging/synology/build.sh # x86_64 (most Plus models), lean image
# ARCH=armv8 packaging/synology/build.sh # 64-bit ARM models (cross-builds with QEMU/binfmt)
# LENS_TARGET=full packaging/synology/build.sh # with LibreOffice and Chromium (a much bigger package)
# SKIP_BUILD=1 BUILD=0002 packaging/synology/build.sh # package lens-backend:<version> and lens-frontend:<version>
# # images already built for that architecture (label them
# # net.lens.package=synology so upgrades remove them)
#
# Needs Docker with buildx. Writes dist/synology/lens-<version>-<arch>.spk.
set -eu

HERE=$(cd "$(dirname "$0")" && pwd)
ROOT=$(cd "$HERE/../.." && pwd)
ARCH=${ARCH:-x86_64}
LENS_TARGET=${LENS_TARGET:-lean}
EXTRAS=${EXTRAS:-}
BUILD=${BUILD:-0001}
SURREAL_IMAGE=surrealdb/surrealdb:v3.2.4
DOCKER_CLI_IMAGE=docker:27-cli
APP_VERSION=$(sed -n 's/^version = "\(.*\)"/\1/p' "$ROOT/fastapi_backend/pyproject.toml" | head -n 1)
VERSION="$APP_VERSION-$BUILD"

case "$ARCH" in
x86_64) PLATFORM=linux/amd64 ;;
armv8) PLATFORM=linux/arm64 ;;
*) echo "ARCH must be x86_64 or armv8" >&2; exit 1 ;;
esac

OUT=$ROOT/dist/synology
WORK=$OUT/work-$ARCH
rm -rf "$WORK"
mkdir -p "$WORK/spk" "$WORK/package/ui/images"

if [ -z "${SKIP_BUILD:-}" ]; then
echo "==> Building images for $PLATFORM ($LENS_TARGET)"
docker buildx build --platform "$PLATFORM" --load --label net.lens.package=synology --target "$LENS_TARGET" --build-arg EXTRAS="$EXTRAS" \
-t "lens-backend:$VERSION" "$ROOT/fastapi_backend"
docker buildx build --platform "$PLATFORM" --load --label net.lens.package=synology -f "$ROOT/nextjs-frontend/Dockerfile.prod" \
-t "lens-frontend:$VERSION" "$ROOT/nextjs-frontend"
fi
docker pull --platform "$PLATFORM" "$SURREAL_IMAGE"
docker pull --platform "$PLATFORM" "$DOCKER_CLI_IMAGE"

echo "==> Saving images into the package"
SAVE_PLATFORM=
docker save --help 2>/dev/null | grep -q -- '--platform' && SAVE_PLATFORM="--platform $PLATFORM"
# shellcheck disable=SC2086
docker save $SAVE_PLATFORM "lens-backend:$VERSION" "lens-frontend:$VERSION" "$SURREAL_IMAGE" "$DOCKER_CLI_IMAGE" \
| gzip -1 > "$WORK/package/lens-images.tar.gz"

echo "==> Assembling the SPK"
cp -R "$HERE/package/compose" "$WORK/package/"
cp "$HERE/package/ui/config.in" "$WORK/package/ui/"
cp "$HERE"/icons/lens_*.png "$WORK/package/ui/images/"
# Package Center needs some ui/config before postinst writes the real one (with the chosen port).
sed 's/@PORT@/3000/' "$HERE/package/ui/config.in" > "$WORK/package/ui/config"
(cd "$WORK/package" && tar czf "$WORK/spk/package.tgz" .)

cp -R "$HERE/conf" "$HERE/scripts" "$HERE/WIZARD_UIFILES" "$WORK/spk/"
chmod 755 "$WORK/spk/scripts/"*
cp "$HERE/icons/lens_64.png" "$WORK/spk/PACKAGE_ICON.PNG"
cp "$HERE/icons/lens_256.png" "$WORK/spk/PACKAGE_ICON_256.PNG"
sed -e "s/@VERSION@/$VERSION/" -e "s/@ARCH@/$ARCH/" "$HERE/INFO.in" > "$WORK/spk/INFO"
echo "checksum=\"$(md5sum "$WORK/spk/package.tgz" | cut -d' ' -f1)\"" >> "$WORK/spk/INFO"
echo "extractsize=\"$(du -sk "$WORK/package" | cut -f1)\"" >> "$WORK/spk/INFO"

SPK=$OUT/lens-$VERSION-$ARCH.spk
(cd "$WORK/spk" && tar cf "$SPK" INFO package.tgz scripts conf WIZARD_UIFILES PACKAGE_ICON.PNG PACKAGE_ICON_256.PNG)
rm -rf "$WORK"
echo "==> $SPK ($(du -h "$SPK" | cut -f1))"
5 changes: 5 additions & 0 deletions packaging/synology/conf/privilege
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"defaults": {
"run-as": "package"
}
}
20 changes: 20 additions & 0 deletions packaging/synology/conf/resource
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"data-share": {
"shares": [
{
"name": "lens",
"permission": { "rw": ["lens"] }
}
]
},
"docker-project": {
"preload-image": "lens-images.tar.gz",
"projects": [
{
"name": "lens",
"path": "compose",
"build_params": { "force_pull": false, "build": false, "force_recreate": true }
}
]
}
}
Binary file added packaging/synology/icons/lens_16.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/synology/icons/lens_24.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/synology/icons/lens_256.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/synology/icons/lens_32.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/synology/icons/lens_48.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/synology/icons/lens_64.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added packaging/synology/icons/lens_72.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading
0