Repository navigation
agent: image retention counts tags instead of images, and rmi by ID fails on multi-tagged images #109
Description
Summary
jiji-agent's continuous image retention applies retain: N to tag rows instead of distinct images, then removes images by ID with a plain rmi. When one image ID carries several tags (any redeploy that builds byte-identical content reuses the ID and only adds a tag), the removal is refused and retention never makes progress for that repo.
Symptom
jiji network diagnostics / jiji server upgrade reports the host as SKIP with a climbing image_retention failure counter:
SKIP <host> ... image_retention failures=8706 last_error=<service>: could not
remove image 'cb5950e3c44c': Error: unable to delete image
"cb5950e3c44c..." by ID with more than one tag
([localhost:31270/<repo>:1787887154
localhost:31270/<repo>:1787887065
localhost:31270/<repo>:1787878680]): please force removal
The host's image cache never shrinks to retain: N. Nothing else breaks.
Root cause
crates/jiji-agent/src/image_retention.rs:
list_imagesrunsimages --format '{{.ID}} {{.Repository}}:{{.Tag}}' --filter reference=<repo>. Docker and Podman return one row per tag, so one image with three tags is three rows with the same ID.prune_repo_with_binarydoes.skip(retain_n)over those rows, soretain_ncounts tags. A row for an ID that also has a newer, retained tag can land past the cutoff.- The in-use check compares by
repo:tag. If the running container uses one tag of an image, rows for that image's other tags are not recognized as in use. remove_imagerunsrmi <id>without-f, which both engines refuse for an image with more than one tag.
Reproduced on Docker (Podman behaves the same, per the error above):
$ docker images --format '{{.ID}} {{.Repository}}:{{.Tag}}' --filter reference=probe
1b982297ddd4 probe:v1
1b982297ddd4 probe:v2
1b982297ddd4 probe:v3
$ docker rmi 1b982297ddd4
Error response from daemon: conflict: unable to delete 1b982297ddd4 (must be forced) - image is referenced in multiple repositories
The sharpest case: a single in-use image with stale tags (repeated redeploys of unchanged content). Every pass tries rmi <id> on the image the running container uses, and fails.
Proposed fix
- Group rows by image ID, keeping the engine's newest-first order (the documented assumption
prune_repoalready relies on). - Apply
retain_nto distinct images. - Treat an image as in use if any of its tags is referenced by a container.
- For images past the cutoff, remove each tag with
rmi <repo>:<tag>rather thanrmi -f <id>: removing a tag can never delete an image a container still runs, and the image goes away with its last tag.
Note that a stale tag on a retained image (inside the window) is not removed by this; it only stops being counted as an extra image.
Workaround
Remove stale tags by name, never by ID, for an image a container is using:
podman images --filter reference=<repo>
podman rmi <repo>:<stale-tag>
Related: retry behavior (#110), the same gap in jiji service prune (#111), test coverage (#112).
Activity