we need E2E/Integration tests

The application has grown to a point where it is becoming difficult to keep the entire context in my head. We now integrate with several operating systems, multiple Bitcoin networks, and different scenarios where Bitcoin Core may or may not be present.

Reviewing PRs has also become increasingly cumbersome because we still rely heavily on manual testing. It is often unclear whether a change is unintentionally breaking behavior in another part of the system.

The workflow itself is quite complex. We depend on Docker, and there are many failure scenarios to account for, such as services being unavailable and ensuring the UI reacts as expected in those situations.

I do not yet know how to approach this or have time to properly plan it right now. I am only writing this down as a reminder to revisit later.

I've used tools like Cypress and Playwright in the past for E2E testing to validate critical user flows and UI behavior, such as:

onboarding/setup
error banners
enabled/disabled actions
recovery after failures

One open question is how much we should rely on mocks.

For some scenarios, mocking could be useful (RPC responses from Core, captured container logs used by diagnostics, ASIC devices such as Bitaxe, etc.). This would reduce CI resource usage and speed up tests. The downside is the maintenance burden, since mocks need to evolve together with the real applications.

My current inclination would be to prefer real dependencies whenever possible (real Core instance and real managed containers), and only use mocks for scenarios that are difficult or expensive to reproduce. The tests would be slower, but they would provide much higher confidence that integrations still behave correctly.

We could leverage Playwright's own test harnesses and potentially build a small abstraction around environment setup, allowing tests to define scenarios such as:

testEnv.stopCore();
testEnv.startCore();
testEnv.dropContainer();
testEnv.disableDocker();

A lot of the testing does not even require third party services. The wizard process, form validation, and general UI behavior can likely be covered with straightforward Playwright scripts.

Some initial high value scenarios to automate:

Are we generating the correct configuration files from the wizard flow?
Are diagnostics correctly surfacing the large error banners?
Does the UI properly react when Docker is unavailable at startup?
Are forms validating inputs correctly and surfacing user friendly errors (e.g. authority pubkey validation, addresses etc)?
Does the UI react correctly when Core or managed containers become unavailable?
The different scenarios are correctly spawning up the correct containers?

These tests are intentionally higher level. The goal is not to validate every network message or internal interaction, but rather to verify that the application reacts correctly to changes in the environment and presents the expected state to the user.

An example of bug related to how configurations are generated which should ideally be detected by our CI: #202 (comment)

Another example of problem we should check on CI: getumbrel/umbrel-apps#5933 (comment)

#234 added a new sv2-app-config job in our CI workflow which checks that config generator generates config files which work with the real docker images of both JDC and tProxy:

sv2-ui/.github/workflows/ci.yml

Lines 94 to 111 in 1c81ce5

    
           sv2-app-config: 
        
             name: SV2 Apps Config Compatibility 
        
             runs-on: ubuntu-latest 
        
             needs: test 
        
             steps: 
        
               - uses: actions/checkout@v4 
        
               - name: Setup Node.js 
        
                 uses: actions/setup-node@v4 
        
                 with: 
        
                   node-version: '24' 
        
                   cache: 'npm' 
        
               - name: Install dependencies 
        
                 run: npm ci 
        
               - name: Smoke-test generated configs against selected sv2-apps images 
        
                 run: npm run check:sv2-app-config

Things that will probably break the Umbrel Setup if changed, or at least give trouble to make the transitions from one version to another:

DOCKER_VOLUME_NAME
DOCKER_NETWORK_NAME
extra_hosts: sv2-translator:host-gateway, sv2-jdc:host-gateway
DEFAULT_BITCOIN_PATHS.umbrel
The loopback binding of internal containers. See bind monitoring API ports 9091/9092 to loopback only #251 (comment)
Umbrel starting to provide HTTPS could potentially cause problems with the session cookie (just a hunch). I tested with Umbrel 2.0, and so far there have been no issues.
Using the Umbrel proxy would require a refactor and would probably break something silently.
Adding a static CORS allowlist will break it, although Loupe suggested doing so anyway.
Random weird things related to Docker-in-Docker (DinD) that I don't fully understand or know how to solve. For example: Troubleshooting Umbrel Pro Setup #189 (comment)

And these are just the known issues. There are probably many others that I'm not aware of and that are catch only with real executions

@lucasbalieiro

Hey @lucasbalieiro, I gave your list as input to some agents, and I asked them to give me a more complete list of things which could break the Umbrel deployment.

It would be good to get your feedback on it.

Here's the analysis:

Building on @lucasbalieiro's list above, this is an inventory of the changes a sv2-ui PR could make that would break Umbrel. It's meant as the spec for an Umbrel CI job, and as a review checklist until that job exists.

It comes from reading every file in the Umbrel package (compose file, entrypoint.sh, hooks/pre-start, manifest), Umbrel's app gateway, and every sv2-ui code path that depends on them. Code links point to main at 07b4017. Nothing here was run on a real Umbrel. Items marked (unverified) are inferred from the code and need a check on a device.

How Umbrel runs sv2-ui (needed to follow the list)

sv2-ui runs as a normal Umbrel app container. Umbrel's gateway proxies umbrel.local:8077 to sv2-ui_sv2-ui_1:8080, with Umbrel's own login disabled (PROXY_AUTH_ADD: "false").
A privileged Docker-in-Docker sidecar (docker:28.3.0-dind) runs with network_mode: host, --iptables=false and --userland-proxy=true. Its data dir is mounted over sv2-ui's /var/run, which is how sv2-ui finds /var/run/docker.sock.
sv2-ui creates the Translator and JDC inside that DinD daemon, not in Umbrel's Docker. Because DinD shares the host network namespace, their published ports open directly on the Umbrel host.
Every time the DinD sidecar starts, entrypoint.sh makes sure sv2-network exists, adds FORWARD/MASQUERADE rules for its current bridge and subnet, and makes sure the sv2-config volume exists as a bind to /app/data/config. That is the same host folder sv2-ui sees at /app/data/config.
sv2-ui is not on sv2-network. It reaches the monitoring APIs as http://sv2-translator:9092 and http://sv2-jdc:9091, which the package's extra_hosts maps to host-gateway, so the traffic goes through the published ports.
The only paths shared with nested containers are the Bitcoin data dir, mounted read-only at /root/.bitcoin, and /app/data/config.
DinD's data root survives restarts and updates. The pre-start hook only clears the pid file, the socket and the exec root.

The image and its release

Running the container as a non-root user. ~ is expanded with os.homedir() because Umbrel doesn't set HOST_HOME (docker.ts#L36), so ~/.bitcoin stops matching DinD's /root/.bitcoin mount and JD mode breaks. Access to the DinD socket would likely fail too.
Changing PORT=8080, or listening only on 127.0.0.1. Umbrel's gateway connects to port 8080 from outside the container. (Dockerfile#L51, index.ts#L1073)
Moving CONFIG_DIR away from /app/data/config. It's the only folder Umbrel persists, and DinD's sv2-config volume is bound to the same host folder. (Dockerfile#L52)
Dropping NODE_ENV=production. getContainerUrl() then targets localhost instead of the container names that extra_hosts resolves. (index.ts#L950)
Changing the start command so SIGTERM no longer reaches Node, for example npm start instead of node under tini. Umbrel's stop would skip shutdown(), so JDC would no longer be stopped before the Translator, which can crash Bitcoin Core. See also item 33. (index.ts#L1110, docker.ts#L962)
Switching to a base image without a shell. Umbrel's app terminal stops working, and the FAQ's Umbrel password-recovery steps rely on it.
Renaming the image or publishing fewer platforms. The package pins stratumv2/sv2-ui by multi-arch digest, and Raspberry Pi Umbrels need linux/arm64. (docker-release.yaml#L64)

Inputs Umbrel provides

Renaming HOST_OS or STRATUM_HOST, or the umbrel OS value. The wizard stops preselecting Umbrel and shows miners the wrong host. (index.ts#L622)
Depending on DOCKER_HOST, DOCKER_SOCKET_PATH or HOST_HOME. Umbrel sets none of them.
Changing the socket lookup so /var/run/docker.sock isn't found, or writing files under /var/run. Umbrel replaces the whole /var/run with DinD's data dir. (constants.ts#L25)
Requiring any new env var, mount, port or extra_hosts entry. It only works once the Umbrel package adds it, so it must ship as a coordinated package update.

Resources inside the DinD daemon

Renaming sv2-network, sv2-config, sv2-translator or sv2-jdc. All four are hardcoded in the package's entrypoint.sh, healthcheck and extra_hosts. (constants.ts#L18)
Deleting or recreating sv2-network, or creating it with different options. entrypoint.sh adds firewall rules for the network's bridge and subnet each time the DinD sidecar starts. A network recreated while sv2-ui is running has no rules until the next app restart or Umbrel reboot, so containers on it lose outbound access and pool and telemetry connections fail. What prevents this today is that ensureNetwork() only creates the network when it's missing. (docker.ts#L701)
Attaching any container to a new network. It gets no firewall rules either. Today's helpers use bridge (DinD's dind0, which has rules), host or none.
Mounting a new volume into the mining containers instead of sv2-config. A volume that sv2-ui creates in DinD starts empty, and only sv2-config is bound to sv2-ui's folder. (docker.ts#L833)
Adding bind mounts to mining or helper containers. Their "host" is DinD's own filesystem. Only /root/.bitcoin and /app/data/config are shared, so any other path is missing or isn't the file you expect.
Adding a new container that sv2-ui must talk to. It needs a published port and a new extra_hosts entry in the package (item 11).
Making connectSv2UiToNetwork() mandatory. On Umbrel, sv2-ui lives in the outer daemon and is never found, so the step silently returns today. (docker.ts#L718)
Using Docker Engine API features newer than 28.3, the version the package pins.
Using helper images without linux/arm64. Helper images such as node:24-alpine are pulled inside DinD. (images.ts#L3)

Ports

Binding published ports to 127.0.0.1. With --iptables=false, published ports are served by docker-proxy in the host namespace, and sv2-ui's traffic arrives at the bridge gateway address, not loopback. This is what bind monitoring API ports 9091/9092 to loopback only #251 ran into. (docker.ts#L846, docker.ts#L898)
Unpublishing 9091 or 9092, or reaching the monitoring APIs another way, such as container IPs or Docker DNS. Container name plus published port is the only path from sv2-ui into DinD.
Adding or changing published ports. They open on the Umbrel host itself and can clash with other apps or Umbrel services. Changing 34255 or 34265 also disconnects every miner that's already configured. Transmission's Umbrel app already uses 9091 as its web port, so JD mode likely clashes with it today (unverified).

Persistence and updates

Writing state that must survive a restart outside CONFIG_DIR. Umbrel recreates the container on every update.
Changing the state.json format without still reading every older format. Umbrel users can skip several versions in one update.
Releasing with sv2-apps image tags instead of digests. main uses moving :main tags and only the release branch pins digests (images.ts#L9). pullImage() only pulls missing images and DinD keeps its images across updates, so a release cut from the wrong branch would leave Umbrel users on whatever image they pulled first, for good. (docker.ts#L752)
Bumping the pinned sv2-apps images without checking them. The current images run as root, and three things depend on it. jdc.toml sets no data_dir, so JDC's ~/.bitcoin only matches the socket mount at /root/.bitcoin when it runs as root. Umbrel's Bitcoin app runs as uid 1000, so a non-root JDC may not be able to open Core's socket. And sv2-ui keeps the config folder at 0700 with 0600 files, so a non-root Translator or JDC couldn't read its own config through sv2-config. A new image can also drop arm64 or a Core IPC version. (docker.ts#L319, config-dir.ts#L27)
Moving or renaming credential.json. The FAQ's Umbrel recovery steps hardcode /app/data/config/credential.json.
Rejecting a config folder owned by another user, for example while fixing config-files: credentials and configs are never stored in a directory other local accounts can write #310. umbrelOS creates the folder, so it's probably owned by uid 1000 while sv2-ui runs as root (unverified). (config-dir.ts#L18)

Start, stop and crash behavior

Breaking the startup reconcile or its retry loop. Mining containers use RestartPolicy: no, so this is the only thing that restarts mining after an Umbrel reboot or update. It checks every 30 seconds and, after a failure, backs off from 1 minute up to 5 minutes. (index.ts#L1006)
Persisting "stopped" on shutdown. Mining would stay off after every reboot.
Making shutdown take longer than 60 seconds. Umbrel's stop_grace_period is 1m. After that sv2-ui is killed, and dockerd stops JDC and the Translator in arbitrary order.
Removing or changing StopSignal: 'SIGINT' on the Translator and JDC. The graceful JDC shutdown that protects Bitcoin Core relies on it. It's also the fallback for item 32: if sv2-ui is killed, dockerd stops the containers with that signal. This breaks every deployment, not only Umbrel. (docker.ts#L841, docker.ts#L893)
Exiting with code 0 on a fatal error. The package uses restart: on-failure, so a clean exit leaves sv2-ui down.
Crashing or giving up for good when Docker is briefly unreachable. The DinD sidecar can restart on its own.

Browser access through Umbrel's gateway

On umbrelOS 2.0, the gateway keeps the original Host, sets X-Forwarded-Proto, X-Forwarded-Host and X-Forwarded-For, has no timeout by default, and only strips Umbrel's own cookies.

Allowing only certain hostnames or origins, for example while fixing http-boundary: a state-changing request is accepted only for a host the instance is served at #320. Users reach Umbrel through umbrel.local, a custom device name, a LAN IP and a Tor .onion address.
Comparing origin schemes without using X-Forwarded-Proto (http-boundary: an Origin on the other scheme of the same hostname is rejected #325). The fallback check builds the request origin with a hardcoded http://, so a scheme comparison on top of it would reject HTTPS access on Umbrel. (request-origin.ts#L47)
Changing trust proxy. All Umbrel clients would share the gateway's IP, so per-client login limits (auth: failed attempts from one client do not lock out another #314) would lock everyone out at once. (index.ts#L73)
Forcing Secure cookies, a __Host- cookie name, HSTS or upgrade-insecure-requests. Umbrel is usually reached over plain HTTP, so login would silently fail. (sessions.ts#L72)
Serving the UI under a sub-path, or hardcoding hosts or ports in the frontend. Umbrel serves the app at the root of its own port.

Bitcoin Core

Changing DEFAULT_BITCOIN_PATHS.umbrel or the socket file naming. Both must match DinD's /root/.bitcoin mount and where Umbrel's Bitcoin app creates node.sock. (constants.ts#L7, constants.ts#L48)
Changing the RPC discovery probes or how they read credentials. On Umbrel the probes run inside DinD and reach Bitcoin's RPC port through the host network. Nobody has checked which transport and credential source actually succeed on Umbrel (unverified). (docker.ts#L263)
Writing into the Bitcoin data dir. DinD mounts it read-only.

Soft limits on slower hardware

These aren't contract breaks like the items above. A CI job will only catch them if it runs with Pi-like memory and CPU limits, or on an arm64 runner. A GitHub runner with 16 GB of RAM won't.

Growing memory use significantly, for example through a large dependency, in-memory log buffering or more aggressive polling. Many Umbrels are Raspberry Pi 4 or 5 boards that also run Bitcoin Core.
Tightening timeouts. Shorter values than today's Docker call timeout (10 s), log-read abort (2 s) and monitoring fetch timeout (5 s) can make status and logs flaky on a busy Pi. (docker.ts#L26, docker.ts#L674, index.ts#L971)

Open audit issues whose fix touches this list

http-boundary: a monitoring proxy response never renders as a page on the UI origin #308: unpublishing the monitoring ports would break item 22.
config-files: credentials and configs are never stored in a directory other local accounts can write #310: a stricter config folder check can hit item 29.
config-files: the mining containers cannot read the administrator credential #313: hiding credential.json from the mining containers must not use a new volume (item 15), and the file must stay in CONFIG_DIR (item 24).
auth: failed attempts from one client do not lock out another #314: per-client login limits depend on trust proxy (item 38).
http-boundary: a state-changing request is accepted only for a host the instance is served at #320: host validation must accept every way users reach Umbrel (item 36).
http-boundary: an Origin on the other scheme of the same hostname is rejected #325: the scheme comparison must use X-Forwarded-Proto (item 37).
docker: each deployment runs its JDC with its own Noise authority keypair #350 and allow the JDC to generate an exclusive key-pair #241: a generated JDC key pair must be stored in CONFIG_DIR (item 24).

It would be good to get your feedback on it.

It is definitely more verbose and pedantic.

I don't have any additional feedback to add. Some of the items just repeat what I said, some make good points, and others might be a bit of a long shot, but they may still be worth keeping on the list, especially since Mr. AI even took the time to read the

Umbrel's app gateway,

which I haven't read myself.

Navigation Menu

Repository navigation

we need E2E/Integration tests #164

The image and its release

Inputs Umbrel provides

Resources inside the DinD daemon

Ports

Persistence and updates

Start, stop and crash behavior

Browser access through Umbrel's gateway

Bitcoin Core

Soft limits on slower hardware

Open audit issues whose fix touches this list

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	sv2-app-config:
	name: SV2 Apps Config Compatibility
	runs-on: ubuntu-latest
	needs: test
	steps:
	- uses: actions/checkout@v4

	- name: Setup Node.js
	uses: actions/setup-node@v4
	with:
	node-version: '24'
	cache: 'npm'

	- name: Install dependencies
	run: npm ci

	- name: Smoke-test generated configs against selected sv2-apps images
	run: npm run check:sv2-app-config

we need E2E/Integration tests #164

Description

Activity

lucasbalieiro commented on Jul 14, 2026

GitGab19 commented on Jul 27, 2026

GitGab19 commented on Jul 29, 2026

GitGab19 commented on Jul 30, 2026

lucasbalieiro commented on Oct 6, 2026

GitGab19 commented on Oct 7, 2026

The image and its release

Inputs Umbrel provides

Resources inside the DinD daemon

Ports

Persistence and updates

Start, stop and crash behavior

Browser access through Umbrel's gateway

Bitcoin Core

Soft limits on slower hardware

Open audit issues whose fix touches this list

lucasbalieiro commented on Oct 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions