Repository navigation
issue with cellpose #17
Description
It appears, cellpose is currently not correctly installing together with devbio-napari:
On @kevinyamauchi's computer there is an error, see details conda-forge/devbio-napari-feedstock#17 (comment)
On mine the installation works:
mamba create --name devbio-napari-env python=3.9 devbio-napari -c conda-forge
And then I receive an error when running cellpose from the tools menu on blobs.gif:
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
File ~\miniconda3\envs\devbio-napari-env4\lib\site-packages\napari_tools_menu\__init__.py:71, in ToolsMenu.make_sub_sub_menu.<locals>.func(whatever=False)
66 break
67 # cannot look for param.kind == param.VAR_KEYWORD because
68 # QWidget allows **kwargs but errs on unknown keyword arguments
69
70 # instantiate the widget
---> 71 wdg = action(**kwargs)
action = <function widget_wrapper at 0x000002004BBB48B0>
kwargs = {}
72 dw = napari_viewer.window.add_dock_widget(wdg, name=title)
73 if dw is not None:
74 # workaround for https://github.com/napari/napari/issues/4348
File ~\miniconda3\envs\devbio-napari-env4\lib\site-packages\cellpose_napari\_dock_widget.py:50, in widget_wrapper()
48 from napari.qt.threading import thread_worker
49 try:
---> 50 from torch import no_grad
51 except ImportError:
52 def no_grad():
File ~\miniconda3\envs\devbio-napari-env4\lib\site-packages\torch\__init__.py:124
122 err = ctypes.WinError(last_error)
123 err.strerror += f' Error loading "{dll}" or one of its dependencies.'
--> 124 raise err
err = OSError(22, 'The operating system cannot run %1.', None, 182)
125 elif res is not None:
126 is_loaded = True
OSError: [WinError 182] The operating system cannot run %1. Error loading "C:\Users\rober\miniconda3\envs\devbio-napari-env4\lib\site-packages\torch\lib\caffe2_detectron_ops.dll" or one of its dependencies.
Activity
haesleinhuepf commentedon Sep 17, 2022
I propose removing cellpose from devbio-napari if there is no easy solution to this.
kevinyamauchi commentedon Sep 19, 2022
That would work for me for now.
That being said, I would like to try and figure out the best way to allow a pytorch dependency as that is going to be increasingly "required" for quality segmentations and other tasks. pytorch has been my preferred deep learning library for some time, but with the move to transfer the library to the linux foundation, I am even more inclined to depend on pytorch over tensorflow.
In general, I think it could make sense to define version constraints versions for key dependencies that are likely to cause conflicts in the
devbio-napariecosystem and explicitly install them. Then plugin developers who wish to be included in thedevbio-napari"distribution" know what they need to follow.Some potential candidates:
haesleinhuepf commentedon Sep 19, 2022
Yes! That goes very well in line with our local discussions. CC @thawn @zoccoler @jo-mueller
kevinyamauchi commentedon Sep 19, 2022
Cool, glad we're thinking along the same lines!
To get the ball rolling, I propose the following (I haven't tested if this conflicts with any existing dependencies):
torch == 1.8.2 (this is the long term service version - seems reasonable?)actually it looks like the LTS is compatible with MacOS and requires python <= 3.8It could be worth keeping an eye on the BioImage Model Zoo, as they might specify pytorch versions and it would be nice to be in line with that project, as I imagine there will be a bunch of useful models available there in the coming months/years.
kevinyamauchi commentedon Sep 19, 2022
Actually, I just ran into this issue when trying to install
pytorch1.12.1on my mac. Maybe it's best to use1.11.0or maybe even older.pytorch/pytorch#78490 (comment)
haesleinhuepf commentedon Oct 2, 2022
I've just tried to make devbio-napari work together with cellpose and failed. I've tried installation commands similar to those on a pretty emptyish windows computer:
When starting cellpose from within napari, I receive this error:
This is the
conda list:I will remove cellpose-napari until this has been figured out.
haesleinhuepf commentedon Oct 2, 2022
Related issue:
removed cellpose
kevinyamauchi commentedon Oct 8, 2022
Now that conda-forge/devbio-napari-feedstock#17 has merged and @psobolewskiPhD has tested it, I think we can revisit this.
Based on this comment, I think we can specify a torch dependency and ask the user to add the
pytorchchannel to their installation command.When
conda-forgeadds support for specifying additional channels, we can add thepytorchchannel to the recipe and remove the-c pytorchfrom the suggested installation command.I would suggest that we start with something like
pytorch 1.11.0. The1.12series doesn't install properly on Mac for me: #17 (comment)psobolewskiPhD commentedon Oct 8, 2022
FYI: on my arm64 macOS 12.6
mamba create --name test-env2 python=3.9 cellpose-napariJustWorks with+ pytorch 1.12.1 cpu_py39hf1faf6a_1being installed.Note: On my M1, 1.12.x is markedly faster than before because it's using Apple Accelerate library and the AMX for BLAS.
haesleinhuepf commentedon Oct 8, 2022
Do you remember how you installed that? That's the key problem ;-)
kevinyamauchi commentedon Oct 8, 2022
Thanks!
Yeah, it would be great to use torch
1.12.xfor the acceleration on M1/M2! However, currently with1.12.1there are issues with Intel Macs 🤦 : pytorch/pytorch#78490 (comment)For now, I think there are a couple of paths forward to including pytorch deps with
devbio-napari(open to other suggestions!)pytorch==1.11.0as pytorch1.12.1currently doesn't work on Intel Macs. However, this would create a conflict for M1/M2 Mac folks that want to use1.12.1to take advantage of the acceleration.devbio-napari. The advantage is that the user would be able to install the best version for their platform. The downside is that many users won't know how to figure out what the best version is for their platform.psobolewskiPhD commentedon Oct 8, 2022
@kevinyamauchi What python env are you seeing cellpose issues on Intel macOS?
It runs fine—not very fast, but fine—for me from conda-forge in rosetta.
I just did
mamba install cellpose-napariin an env with just napari. (Just conda-forge channel BTW.)pytorch 1.12.1 cpu_py39h9b0ea23_0 conda-forgeEdit: i can't reproduce the issue in that thread either.
kevinyamauchi commentedon Oct 8, 2022
I do the following:
conda create -n test-pytorch python=3.9 numpy pytorch -c pytorch -c conda-forge conda activate test-pytorch python -c "import numpy;import torch"and I get the same error as linked in the issue above:
psobolewskiPhD commentedon Oct 8, 2022
why not just:
conda create -n test-pytorch python=3.9 pytorch -c conda-forgeI think using the different channels mixing numpy and pytorch from different channels is problematic.
kevinyamauchi commentedon Oct 8, 2022
Personally, I've always installed from the pytorch channel, as those are the official instructions on https://pytorch.org/. In the context of
devbio-napari, as far as I know theconda-forgepytorchdoesn't have a build for windows, so I think thepytorchchannel is preferred.I haven't had issues mixing numpy from
conda-forgeandpytorchfrom thepytorchchannel before1.12.0- I think this is supposed to be supported given the discussion in the issue I linked above.That being said, I tried installing pytorch from
conda-forgeon my Intel Mac as you suggested and it works!kevinyamauchi commentedon Oct 8, 2022
What if the installation instructions were:
I think this makes
conda-forgethe higher priority channel so that it will installpytorchfromconda-forgeand if that fails (e.g., Windows), it will fall back to thepytorchchannel. We should verify that the order channels are provided is indeed guaranteed to set the priority of the channels.This works on my Intel Mac. I'd be curious if it also works on Windows (cc @haesleinhuepf ).
kevinyamauchi commentedon Oct 8, 2022
Okay, I just checked the docs. It looks like the channels are indeed searched in the order they are given. Thus, I think providing both
conda-forgeandpytorchchannels might be a good solution (assuming it works on Windows). Thanks for the tip aboutconda-forgechannel, @psobolewskiPhD !https://docs.conda.io/projects/conda/en/latest/commands/install.html#Channel%20Customization
psobolewskiPhD commentedon Oct 8, 2022
Why specify numpy? It's a depend, it will come along?
kevinyamauchi commentedon Oct 8, 2022
Sorry for being unclear. I was just trying to create a minimal environment for testing pytorch installation. The bug with pytorch
1.12.1pops up when trying to import numpy and then pytorch.edit: did you mean that
numpyis a dependency ofpytorch? If you installpytorchfrom thepytorchchannel, it doesn't install numpy. So I think you need to specifynumpyfor the windows case since it will have to install from thepytorchchannel (don't have a windows machine to test at the moment).psobolewskiPhD commentedon Oct 8, 2022
I meant in terms of devbio, because napari, cellpose, etc. definitely use numpy. But i get it now.
romainGuiet commentedon Feb 13, 2023
Hi @kevinyamauchi , Hi @psobolewskiPhD , Hi @haesleinhuepf ,
I came here because of the cellpose title and I'm not sure if you're still looking for a solution...
I tried on my Windows machine to create an env with :
conda create -n test-pytorch python=3.9 numpy pytorch -c conda-forge -c pytorchthen
conda activate test-pytorchand finally
python -c "import numpy;import torch"and I got no error so far...
and also tried
python -c "import torch;print(torch.cuda.is_available())"which returnsFalseas expectedconda listreturns :is there something else to try ? I'll be happy to see cellpose added to devbio !
Cheers,
Romain
haesleinhuepf commentedon May 1, 2023
The issue might be resolved with this one:
If so, we can add cellpose again to devbio-napari 🌞
kevinyamauchi commentedon May 1, 2023
Super nice! Thanks, @haesleinhuepf !