Repository navigation
PyTorch v1.0.0 multi-GPU compatibility issue #13
Description
Currently, we cannot run the multi-GPU training on PyTorch v1.0.0 due to a strange null gradient issue.
Currently, we cannot run the multi-GPU training on PyTorch v1.0.0 due to a strange null gradient issue.
Activity
candlewill commentedon Dec 21, 2018
Oh my God. I have trained on the multi-GPU version for one week with all of my four GPUs. In the
params/flowavenet/dir, only one checkpoint was generated.Thanks for pointing out this.
L0SG commentedon Dec 21, 2018
Oops, sorry about the delayed issue post in this repo. Filed the report to the PyTorch repo about two weeks ago, so please stick to v0.4.1 until the issue is resolved.
L0SG commentedon Feb 12, 2019
Update: the issue still persists in the latest 1.0.1 release.
feat: Distributed Training implemented with Apex
L0SG commentedon Apr 23, 2019
Note:
DistributedDataParallelimplementation from @1ytic circumvents the multi-GPU issue, so please usetrain_apex.pyof the master branch until the issue fromDataParallel(fromtrain.py) is resolved.L0SG commentedon Oct 10, 2019
Update: the issue was fixed with the 1.2.0 release. We'll keep this issue open for a while for a future reference.