Repository navigation
Adding a model path parameter #16
Description
Several Biaflows containers are making use of AI models for segmentation. Instead of including the models into the Docker image, it is more convenient to store models permanently outside the container inside a folder on the HPC. This would also make it possible to store different (fine-tuned) models other than the default model loaded by the particular AI tool.
Proposal
-
Create an additional parameter in slurm-config.ini named
slurm_aimodels_path. In the example slurm-config.inislurm_aimodels_path=my-scratch/models. There is already a parameter "slurm_model_paths" so I would suggest to use aimodels . -
At
biomero/biomero/slurm_client.py
Lines 2032 to 2038 in 15acc42
Add
"AIMODEL_PATH": f"{self.slurm_aimodels_path}",
So we create a env variable AIMODEL_PATH and pass this to the slurm job template. -
At the job_template.sh add a
--bindparameter to singularity run
e.g.--bind "$AIMODEL_PATH:/tmp/models" -
For every container that wants/needs to make use of this we add a line to the
DockerFileto tell the tool where to save the models. For example, microsam uses this env variable MICROSAM_CACHEDIR to indicate where to store the models. Cellpose does this similarly.
ENV MICROSAM_CACHEDIR=/tmp/models/microsam_cache
- With this implementation if no AIMODEL_PATH is set when running the container with singularity it will be saved in the /tmp folder on the HPC and likely not cleaned up, maybe this is not ideal? Use home or scratch folder instead? When running with docker it requires downloading the model again every time, unless the /tmp folder is mounted.
Activity
TorecLuik commentedon Jul 17, 2025
ok this is a bit related to the Cellpose model path issue.
TorecLuik/W_NucleiSegmentation-Cellpose#1
the actual issue there is singularity: the models are actually built into the docker image (which you suggest not doing), but singularity by default overrides the home dir with that of the singularity user (slurm user in this case). At that point, it loses all the prebuilt containers and tries to download them from the broken endpoints.
At this point I'm not sure who to blame or fix, but it sounds related to this issue.
However, models are not a biomero property, unless we mean storing (custom) models in OMERO and then sending them to HPC for a specific biaflow. We can't adjust all biaflows, because we want to reuse existing biaflows from other people. So what is generic here? Also, we'd need subdirectories per biaflow right? Can't use a celllpose model for stardist.
maartenpaul commentedon Aug 11, 2025
I see you fixed the issue for the Cellpose container (TorecLuik/W_NucleiSegmentation-Cellpose#1), by changing the model storage path outside the HOME directory :) .
Would it be an idea to always add at least one model to a defined path in the container (e.g. /opt/cellpose/models , /opt/microsam/models), which we then add to the repository and copy (like is done for CellPose).
When people want to make use of external/custom models they need to explicitly define a
AI model pathfor that container, which then binds that folder to the container, and uses the models in a folder at the HPC. Difficulty remains that this is only possible for the biaflows containers we maintain and where we know where the models are stored in the container, so we can overwrite by binding that folder.TorecLuik commentedon Jan 13, 2026
On the backlog, in 2 separate ideas:
26. add a way to add custom models (from biomero) to a wf (e.g. your own model to run with cellpose)
57. add some generic model path on slurm where downloaded models can stay so they don't need to be redownloaded all the time (e.g. the cellpose downloaded pretrained models)
Add persistent AI model path on SLURM (issue #16)
TorecLuik commentedon Jul 2, 2026
Is released / will be released per v2.7/v2.8 -> (as user) attach OMERO attachments to your workflow if they are declared in descriptor (only possible via Bilayers descriptor/config.yaml - not via BiaFlows)
So the model has to be attached to some container (project/dataset/screen/plate/image) in OMERO, perhaps as a result of running the training workflow
This does not tackle a generic path where we store 'persistent' attachments of some sort. Don't know if that's needed either tbh - Slurm is supposed to be not permanent storage, while OMERO is.