8000 Implement the Storage section on the Control page by yakutovicha · Pull Request #258 · aiidalab/aiidalab-home · GitHub
[go: up one dir, main page]

Skip to content

Implement the Storage section on the Control page - #258

Open
yakutovicha wants to merge 5 commits into
mainfrom
feature/storage-tab
Open

yakutovicha wants to merge 5 commits into
mainfrom
feature/storage-tab

Conversation

@yakutovicha
@yakutovicha yakutovicha commented Sep 28, 2026 •
Copy link
Copy Markdown
Member

Fixes #253

Replaces the StorageWidget placeholder.

Disk usage breakdown for the loaded profile: file repository, database (pg_database_size for core.psql_dos; part of the repository directory for core.sqlite_dos), installed apps and the home filesystem.

  • Sizes are allocated disk blocks (du -s --block-size=1), not apparent size: packing frees whole blocks without shrinking the apparent size, so only blocks show what maintenance reclaims.
  • Each row fails independently to an inline error, and du is bounded by a timeout.

Storage maintenance runs storage.maintain() on a background thread:

  • Dry run by default.
  • Full maintenance needs exclusive access to the profile: it is refused while the daemon runs, and its label also asks to close other AiiDAlab apps, whose kernels hold the profile too.
  • The aiida.storage log is captured at INFO level and shown; a real run reports the repository space reclaimed.
  • All controls, including Refresh, are disabled while it runs, and refresh() is a no-op meanwhile, so revisiting the tab can't measure the repository mid-run.

Notes for review:

  • Maintenance runs in the kernel, so closing the page mid-run interrupts it. disk-objectstore writes and fsyncs new packs before committing and only then deletes old data, so an interruption leaves reclaimable leftovers, not lost objects; a stale profile lock from a dead PID is cleared by AiiDA.
  • The database-size query does not close its session: get_session() returns AiiDA's own thread-local session.
  • The psql_dos repository_uri is percent-encoded by AiiDA, so it is decoded like AiiDA does (paths with spaces work).

Tests

Unit tests for the repository path (including a percent-encoded URI), the du wrapper, the log capture, per-row errors in the breakdown and the maintenance flows (daemon guard, dry run, reclaimed space, failure), plus a refresh and a dry run against the test profile's real storage.

This is the first of a stack of PRs implementing the remaining Control page sections (#248).

🤖 Generated with Claude Code

Replace the StorageWidget placeholder with a disk-usage breakdown for
the loaded profile and a storage-maintenance runner.

- Breakdown: file repository (du -sb), database (pg_database_size for
  psql_dos; part of the repository directory for sqlite_dos),
  installed apps and the home filesystem. Each row fails
  independently to an inline error; du is bounded by a timeout.
- Maintenance runs storage.maintain() on a background thread, as a
  dry run by default. Full maintenance is refused while the daemon
  runs, since it needs exclusive access to the profile. The
  aiida.storage log is captured at INFO level and shown, and real
  runs report the repository space reclaimed.
- All controls, including refresh, are disabled while maintenance
  runs, and refresh() is a no-op meanwhile, so revisiting the tab
  cannot measure the repository mid-run.

Add unit tests for the repository path, the du wrapper, the log
capture and the maintenance flows (daemon guard, dry run, reclaimed
space, failure), plus a refresh and a dry run against the test
profile's real storage.
du -sb reports apparent size. Packing a disk-objectstore's small
loose files frees whole filesystem blocks while barely changing the
apparent size, which can even grow because of the pack index. So
the Storage tab understated the repository's footprint and reported
"Reclaimed 0.0 B" after a run that freed most of it. Count
allocated blocks with du -s --block-size=1 instead, which also
matches the physical usage shown for the home filesystem.

test_du_bytes now compares against st_blocks * 512 instead of the
written byte count.
AiiDA writes repository_uri with Path.as_uri(), which percent-encodes
the path, but _repository_path only stripped the file:// prefix. For
a repository path containing spaces or non-ASCII characters, the
Storage tab showed a "does not exist" error and silently skipped the
reclaimed-space report. Decode it with urlparse and unquote, as AiiDA
itself does.

test_repository_path_psql_dos now uses a percent-encoded URI.
test_storage_do_refresh accepted the installed-apps row either way,
because the apps directory may not exist in the test environment, so
the inline error of a failing row was never asserted. Point
AIIDALAB_APPS at a missing directory instead: exactly that row must
show the error, and the profile, repository, database and home rows
must not.
Full maintenance locks the profile, which fails while any other
process that has loaded its storage is alive: daemon workers, but
also the kernels of other open AiiDAlab apps. The Control page's own
kernel does not count. The checkbox label only mentioned the daemon,
so users would stop it and still hit the lock error. Name both in
the label and in the daemon-guard warning.
@yakutovicha
yakutovicha added this pull request to stack #263 September 29, 2026 07:24
@yakutovicha
yakutovicha marked this pull request as ready for review September 29, 2026 07:25
@edan-bainglass
edan-bainglass commented Sep 29, 2026 •
Copy link
Copy Markdown
Member

Seems fine. We've already discussed the WSL issue suggesting I have a TB of storage (I don't 🙂). One comment is that the maintenance log just stays there. Not ideal. Maybe the refresh button should also clear the log?


image

@danielhollas
Copy link
Copy Markdown
Contributor

I haven't tested or looked at the (AI) code, just some quick thoughts based on the PR description and Edan's screenshot:

  1. Can we split this PR into two? The storage info part can be merged quickly, while we might want to be more careful with the Maintanence part as it is actively modifying the AiiDA DB. (this is also in service of not blocking release of Python 3.12 image on this).

  2. This part makes me nervous and will need to be tested thoroughly (and verify AI claims here)

Maintenance runs in the kernel, so closing the page mid-run interrupts it. disk-objectstore writes and fsyncs new packs before committing and only then deletes old data, so an interruption leaves reclaimable leftovers, not lost objects; a stale profile lock from a dead PID is cleared by AiiDA.

  1. The "Maintanence" header should be more specific, e.g. AiiDA storage maintanence, and there should be an expanded description explaining what it actually does (and potential risks etc)

Should we instead spawn a separate background process? It is so easy to accidentally close a browser tab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Control page: implement the Storage section

3 participants

0