Repository navigation
Implement the Storage section on the Control page - #258
yakutovicha wants to merge 5 commits into
Conversation
Replace the StorageWidget placeholder with a disk-usage breakdown for the loaded profile and a storage-maintenance runner. - Breakdown: file repository (du -sb), database (pg_database_size for psql_dos; part of the repository directory for sqlite_dos), installed apps and the home filesystem. Each row fails independently to an inline error; du is bounded by a timeout. - Maintenance runs storage.maintain() on a background thread, as a dry run by default. Full maintenance is refused while the daemon runs, since it needs exclusive access to the profile. The aiida.storage log is captured at INFO level and shown, and real runs report the repository space reclaimed. - All controls, including refresh, are disabled while maintenance runs, and refresh() is a no-op meanwhile, so revisiting the tab cannot measure the repository mid-run. Add unit tests for the repository path, the du wrapper, the log capture and the maintenance flows (daemon guard, dry run, reclaimed space, failure), plus a refresh and a dry run against the test profile's real storage.
du -sb reports apparent size. Packing a disk-objectstore's small loose files frees whole filesystem blocks while barely changing the apparent size, which can even grow because of the pack index. So the Storage tab understated the repository's footprint and reported "Reclaimed 0.0 B" after a run that freed most of it. Count allocated blocks with du -s --block-size=1 instead, which also matches the physical usage shown for the home filesystem. test_du_bytes now compares against st_blocks * 512 instead of the written byte count.
AiiDA writes repository_uri with Path.as_uri(), which percent-encodes the path, but _repository_path only stripped the file:// prefix. For a repository path containing spaces or non-ASCII characters, the Storage tab showed a "does not exist" error and silently skipped the reclaimed-space report. Decode it with urlparse and unquote, as AiiDA itself does. test_repository_path_psql_dos now uses a percent-encoded URI.
test_storage_do_refresh accepted the installed-apps row either way, because the apps directory may not exist in the test environment, so the inline error of a failing row was never asserted. Point AIIDALAB_APPS at a missing directory instead: exactly that row must show the error, and the profile, repository, database and home rows must not.
Full maintenance locks the profile, which fails while any other process that has loaded its storage is alive: daemon workers, but also the kernels of other open AiiDAlab apps. The Control page's own kernel does not count. The checkbox label only mentioned the daemon, so users would stop it and still hit the lock error. Name both in the label and in the daemon-guard warning.
|
I haven't tested or looked at the (AI) code, just some quick thoughts based on the PR description and Edan's screenshot:
Should we instead spawn a separate background process? It is so easy to accidentally close a browser tab. |
Fixes #253
Replaces the
StorageWidgetplaceholder.Disk usage breakdown for the loaded profile: file repository, database (
pg_database_sizeforcore.psql_dos; part of the repository directory forcore.sqlite_dos), installed apps and the home filesystem.du -s --block-size=1), not apparent size: packing frees whole blocks without shrinking the apparent size, so only blocks show what maintenance reclaims.duis bounded by a timeout.Storage maintenance runs
storage.maintain()on a background thread:aiida.storagelog is captured at INFO level and shown; a real run reports the repository space reclaimed.refresh()is a no-op meanwhile, so revisiting the tab can't measure the repository mid-run.Notes for review:
get_session()returns AiiDA's own thread-local session.repository_uriis percent-encoded by AiiDA, so it is decoded like AiiDA does (paths with spaces work).Tests
Unit tests for the repository path (including a percent-encoded URI), the
duwrapper, the log capture, per-row errors in the breakdown and the maintenance flows (daemon guard, dry run, reclaimed space, failure), plus a refresh and a dry run against the test profile's real storage.This is the first of a stack of PRs implementing the remaining Control page sections (#248).
🤖 Generated with Claude Code