8000 Expand documentation - distributed training · Issue #2 · gpoesia/minimo · GitHub
[go: up one dir, main page]

Skip to content
  • Expand documentation - distributed training #2

    Open

    Description

    @maharajamihir

    Hey Gabriel,

    Cool project! I am interested in working with it. Could you expand on the documentation on performing distributed training with this? Training on multiple GPUs on a single node would be a great start.

    Mihir

    Activity

    1. gpoesia commented on Oct 30, 2024

      @gpoesia
      Owner

      Thanks for the interest, Mihir! Sorry I've been a bit busy with deadlines, but I'll aim to explain the setup better this week!

      I'll warn that unfortunately the current distributed setup still needs a bit of babysitting - I set it up very manually, with a Redis server serving as a job queue, Celery workers that I spawn by hand (celery -A worker worker --concurrency=1 -n redis-server-ip-address:port), and the main process that loads the queue with jobs (it does conjecturing), and collects results and does fine-tuning (this process is bootstrap.py - set the REDIS environment variable to redis-server:port like in the worker, and run with DISTRIBUTED=1). It's likely due to me not knowing how to set it up properly, but Redis would eventually fill up and not evict keys (despite my attempts at using LRU and limiting memory), and so stop to take new jobs, so I'd have to restart it (but bootstrap.py checkpoints everything, so you can pick up from the last iteration if you point it to a checkpoint).

      In the longer run, I'm intending to revamp the whole distributed setup with Ray -- it does what I did by hand in a much cleaner way, and looks like it won't be a lot of work to switch. But it will take me a month or so before I switch back to this!

    2. emergenz commented on Oct 30, 2024

      @emergenz
      Contributor

      Nice. This seems to work for me for single-node setups. I added a PR with docs + launchers in #5

    3. added a commit that references this issue on Jun 8, 2025
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      No labels
      No labels

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions

        0