docker

Leggi il leggimi in Italiano.

Colibrì Guide - Local Inference Engine

A simple guide to running Colibrì, a local inference engine based on GLM 5.2, without needing programming knowledge. If you already have Docker installed, you are well on your way.

📋 Table of Contents

What is Colibrì?

Colibrì is an application that allows you to run an artificial intelligence model (GLM 5.2) directly on your computer, without connecting to external servers. It is also possible to run it in Docker, which isolates the application from the rest of the system.

Important note: The model is very large. Expect to wait several minutes for an answer to a simple question, especially with low RAM. At the end of this readme, you will see the result on my PC (without a discrete graphics card), and I reach 0.01 tokens per second.

Requirements

Hardware

RAM Memory	Works?	Notes
< 16 GB	❌ No	Insufficient memory
24 GB	⚠️ Maybe	Possible, needs testing
32 GB	✅ Yes	The minimum (but see memory section for Windows)
48+ GB	✅ Yes	Better

Additionally: a fast SSD is essential. Colibrì uses the disk as additional memory. Having an NVidia graphics card is even better.

Software

Docker Desktop (Windows, Mac, Linux) — download here
Python (only if you want to download the model yourself)
Windows: python.org or Microsoft Store
Linux: apt-get install python3 python3-pip
Mac: python.org or Homebrew

No build environment is needed. Everything happens inside the Docker container.

How to get started

Step 1: Download the model

The GLM 5.2 model is approximately 360 GB. Choose one of these methods:

Method A: Using Python (recommended)

Install the library for Hugging Face:

python -m pip install -U huggingface_hub[cli]

On Linux, use python3 instead of python. 2. Download the model (open the terminal in the folder where you want to save it):

hf_download mastouri/GLM-5.2-colibri-int4-g64-with-int8-mtp --local-dir .

Example: if you want to save it in C:\LLM\models\glm-5.2 (Windows):

Open PowerShell in that folder
Copy and paste the command above
Wait (a long time)

Method B: Without Python (only if necessary)

If you are on Windows and cannot get it to work with Python:

Download manually from Hugging Face
Unzip into a folder (e.g., C:\LLM\models\glm-5.2)

Step 2: Download the Colibrì Dockerfile

Go to: https://github.com/JustVugg/colibri/blob/main/docker/Dockerfile
Click the Download button (⬇️ icon) in the top right
Save the file in a folder (e.g., C:\LLM\Colibrì)

Step 3: Build the Docker image

Open the terminal (PowerShell on Windows, Terminal on Mac/Linux) in the folder where you saved the Dockerfile and type:

Windows:

docker build -t colibri-i .

Linux/Mac:

sudo docker build -t colibri-i .

Wait for it to finish (a few minutes). If everything goes well, you will see: Successfully tagged colibri-i:latest

This builds the code on GitHub, not the code on your disk. The Dockerfile clones the repository inside the image — that is what makes this path work with a single downloaded file and no checkout. If you are developing and want your local changes in the image, use Dockerfile.slim instead (see Slim image below); it copies c/ from the build context.

If you want to receive repository updates: First delete the old image with docker rmi colibri-i and rebuild.

Step 4: Start Colibrì

Open the terminal and type the command below (replace C:\LLM\models\glm-5.2 with the actual path on your PC):

Windows (PowerShell):

$MODEL_PATH="C:\LLM\models\glm-5.2"
docker run --rm -it --name colibri-c `
  -v "$MODEL_PATH`:/app/glm-5.2" `
  -e COLI_MODEL=/app/glm-5.2 `
  colibri-i ./coli chat

Mac/Linux (Terminal/Bash):

MODEL_PATH="/path/to/glm-5.2"
docker run --rm -it --name colibri-c \
  -v "$MODEL_PATH:/app/glm-5.2" \
  -e COLI_MODEL=/app/glm-5.2 \
  colibri-i ./coli chat

Example for Linux:

MODEL_PATH="/home/user/LLM/glm-5.2"
docker run --rm -it --name colibri-c \
  -v "$MODEL_PATH:/app/glm-5.2" \
  -e COLI_MODEL=/app/glm-5.2 \
  colibri-i ./coli chat

What does that command mean?

Part	Explanation
`docker run`	Starts a container
`--rm`	Deletes the container when you close it
`-it`	Interactive mode (you can write and read)
`-v "PATH:/app/glm-5.2"`	Mounts your model inside the container
`-e COLI_MODEL=/app/glm-5.2`	Tells Colibrì where to find the model
`colibri-i`	Name of the Docker image
`./coli chat`	Starts Colibrì in chat mode

Using Colibrì

Once started, you will see a prompt like this:

  ──────────────────────────────────────────────────────────
  type and press Enter · Ctrl-C stops the answer · :more continues · :reset clears memory · :q exits

Useful commands:

Write a question + Enter → Receive the answer
Ctrl + C → Stop the answer
:reset → Clear conversation memory
:q → Exit

Usage example:

› How many inhabitants does China have?

China is currently the most populous country in the world. 
The population is approximately 1.41 billion people.

The model understands Italian, English, Chinese, and other languages, although it is optimized for English and Chinese.

Enter the container

If you want to explore the container as if it were a normal Linux machine:

docker run --rm -it --name colibri-c \
  -v "MODEL_PATH:/app/glm-5.2" \
  -e COLI_MODEL=/app/glm-5.2 \
  colibri-i /bin/bash

Now you are inside Linux. Type exit to leave.

Troubleshooting

❌ "Docker not found"

Cause: Docker is not installed or the terminal does not recognize it.

Solution:

Reinstall Docker Desktop
Restart your computer
Open a new terminal and try again

❌ "Out of memory" or container closes immediately

Cause: Your computer does not have enough RAM, or on Windows, WSL is using less memory than available.

Solution for Windows (WSL):

Open PowerShell and check the memory available to WSL:

wsl
cat /proc/meminfo | grep MemTotal
exit

Divide the number by 1,073,741,824 (which is 1024³) to get it in GB. 2. If WSL uses less than what you have, create a configuration file:

Open a text editor (Notepad is fine)
Copy this:

[wsl2]
memory=24GB
processors=12
swap=16GB

Save the file with the name: .wslconfig (with the dot)
Place it in: C:\Users\YourUsername\

Restart WSL from PowerShell:

wsl --shutdown
wsl

Check again:

# cat /proc/meminfo | grep MemTotal
# exit

Solution for Mac/Linux: Increase the RAM available to Docker from the Docker Desktop settings, or add more RAM to the computer.

❌ The answer is very slow

Possible causes:

The disk is slow
You have low RAM
Colibrì is using the disk as additional memory (normal)

How to check disk speed:

Windows (PowerShell as administrator):

winsat disk -drive C

Change C with your disk letter.

Linux/Mac (Terminal):

sudo hdparm -Tt /dev/sda

Change /dev/sda with your disk (check with lsblk on Linux).

A modern NVMe SSD reaches 15 GB/sec. If yours is under 2-3 GB/sec, it is slow.

❌ "Permission denied" on Linux

Cause: Docker requires administrator permissions.

Solution - Option 1 (quick):

sudo docker build -t colibri-i .
sudo docker run ... (as above, with sudo in front)

Solution - Option 2 (permanent):

sudo usermod -aG docker $USER
# Restart the computer
docker run ... (without sudo)

❌ "Image not found" or error during build

Cause: The Dockerfile is corrupted or not in the right folder.

Solution:

Verify that the Dockerfile is in the folder where you open the terminal:

ls Dockerfile  # Mac/Linux
dir Dockerfile # Windows

Redownload the Dockerfile from the GitHub repository
Delete the old image: docker rmi colibri-i
Retry the build

❌ "hf_download: command not found"

Cause: The Hugging Face library is not installed correctly.

Solution:

pip install -U huggingface_hub[cli]
# or on Linux/Mac:
pip3 install -U huggingface_hub[cli]

Then retry the hf_download command.

❌ The model does not download (timeout or network errors)

Causes: Slow or unstable connection, Hugging Face temporarily unavailable.

Solution:

Wait and retry the hf_download command
If it continues, download manually from here
Unzip the ZIP file into the desired folder

Technical notes

Why is the disk important?

Colibrì uses the disk as "additional virtual RAM" (paging). A fast disk is crucial for decent performance.

NVMe SSD (recommended): 1-15 GB/sec
SATA SSD: 0.5-1 GB/sec
Rotational hard disk: 0.05-0.1 GB/sec ❌ (too slow)

If your disk is slow, the answers will be very slow even with a lot of RAM.

Recommended default configuration for WLS on Windows

If you have exactly 32 GB of RAM and are using Windows, it is very likely that WLS by default is set to consume no more than 16 GB of RAM. We need to increase this limit Troubleshooting . In my case I adopted this configuration:

[wsl2]
memory=24GB
processors=12
swap=16GB

That is, in my case, I left 8 GB of RAM and 4 CPUs to Windows and gave 24 GB and 12 processors to WSL + Linux.

Frequently Asked Questions

Q: What if I have less than 32 GB of RAM?

A: It is likely that it will not work well. You can try if you have 24 GB, but it is not guaranteed.

Q: Can I increase the response speed?

A: Yes, partly:

Use a fast NVMe SSD
Increase RAM
Reduce the complexity of questions
Use :reset to clear memory and lighten the load

Q: Can I use Colibrì without Docker?

A: Colibrì was born that way, but this guide assumes Docker. To build from source, see the GitHub repository.

Q: How much internet connection do I need after downloading the model?

A: Zero. Colibrì works completely offline.

Support and contributions

If you find errors or have suggestions for improving this guide, open an issue or a pull request on the Colibrì GitHub repository.

Have fun! 🐦

Testing on a low resource PC

In the first case, I asked a question in Italian; in the second, in Japanese; and in the third, I repeated the question in Japanese but requested an answer in Italian.

PS C:\quack\llm\colibri\docker> docker run --rm -it --name colibri-c -v "C:\quack\llm\models\glm-5.2:/app/glm-5.2" -e COLI_MODEL=/app/glm-5.2 colibri-i ./coli chat

     ▄▀▀▀▄  ▄        colibrì v1.0
  ▄▄▄▄▀▀▀▀▄▀▀        tiny engine, immense model
      ▀▀▀▀▀▀▀        GLM-5.2 · 744B MoE · int4 · streaming CPU
        ▀▀▀▀         chat · glm-5.2 · ram -GB · topp off
          ▀
  ──────────────────────────────────────────────────────────
  type and press Enter · Ctrl-C stops the answer · :more continues · :reset clears memory · :q exits

  ╭────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ › Quanti abitanti ha la Cina?                                                                  │
  ╰────────────────────────────────────────────────────────────────────────────────────────────────╯

  ◆ colibrì
  La Cina è attualmente il paese più popoloso al mondo (sebbene, secondo alcune stime recenti, sia stata ormai superata dall'India).

  La popolazione totale della Repubblica Popolare Cinese è di circa 1,41 miliardi di abitanti (dati del 2020-2022 circa).
  └─ 76 tok · 0.04 tok/s · hit 3% · RSS 15.9 GB · 2012s

  ╭────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ › 漫画「ワンピース」の主人公の名前を教えてください。名前だけで、それ以上のコメントはありません。                                              │
  ╰────────────────────────────────────────────────────────────────────────────────────────────────╯

  ◆ colibrì
  ルフィ
  └─ 2 tok · 0.01 tok/s · hit 1% · RSS 16.7 GB · 260s

  ╭────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ › 漫画「ワンピース」の主人公の名前を教えてください。名前だけで、それ以上のコメントはありません。イタリア語で返信                                      │
  ╰────────────────────────────────────────────────────────────────────────────────────────────────╯

  ◆ colibrì
  Il nome del protagonista di One Piece è Monkey D. Luffy.
  └─ 14 tok · 0.02 tok/s · hit 2% · RSS 17.3 GB · 593s

  ╭────────────────────────────────────────────────────────────────────────────────────────────────╮
  │ ›

Slim image (developers / production)

The guide above uses Dockerfile, which clones the repo inside the image — one file to download, nothing else on your machine. For CI, servers, or anyone who already has the repo checked out, Dockerfile.slim is a multi-stage build that produces a ~115 MB image (vs. the full toolchain the clone-in-image keeps around), runs as a non-root user, and compiles the engine with a portable ISA (x86-64-v3) so the image runs on any modern x86-64 host instead of only the machine that built it.

Build it from the repo root (it needs the c/ sources as context):

git clone https://github.com/JustVugg/colibri.git && cd colibri
docker build -t colibri -f docker/Dockerfile.slim .

Run — the model is bind-mounted at /model, never baked in:

docker run --rm -v /nvme/glm52_i4:/model colibri info
docker run --rm -it -v /nvme/glm52_i4:/model colibri chat --ram 24
docker run --rm -p 5000:5000 -v /nvme/glm52_i4:/model colibri \
    serve --host 0.0.0.0 --model-id glm-5.2 --ram 24

Reaching it from another machine (or coli web). Binding --host 0.0.0.0 is not enough: the server has a DNS-rebinding guard that only trusts the Host header for loopback and its own bind address, so a browser pointed at http://<server-ip>:<port> gets 403 Host header not allowed. Behind Docker's port mapping the container cannot know the address the browser will send, so tell it what to accept — the exact name/IP, or * when the bind is already deliberately public:

docker run --rm -p 36873:8000 -e COLI_ALLOW_INSECURE_BIND=1 \
    -v /nvme/glm52_i4:/model colibri \
    web --host 0.0.0.0 --allowed-host '*'

--allowed-host '*' disables the guard (it warns at startup); prefer an explicit --allowed-host my-server.lan when you know the name. This is a LAN-exposure decision, hence opt-in alongside COLI_ALLOW_INSECURE_BIND=1.

The entrypoint is coli, so the first argument is the subcommand (info, chat, serve, run, plan, doctor) — no ./coli prefix. The container runs as UID 1000, so the model directory must be readable by that user (a directory you own already is; a root-owned one needs chmod -R a+rX or a matching --user).

One-command server (docker compose)

MODEL_DIR=/nvme/glm52_i4 COLI_RAM=24 docker compose -f docker/docker-compose.yml up

Serves the OpenAI-compatible API on localhost:5000. The converter (which needs PyTorch) is intentionally not in this image — convert from a source checkout, then point MODEL_DIR at the result.

[source: 1]

Navigation Menu

Repository navigation

docker

docker

README.md

Colibrì Guide - Local Inference Engine

📋 Table of Contents

What is Colibrì?

Requirements

Hardware

Software

How to get started

Step 1: Download the model

Method A: Using Python (recommended)

Method B: Without Python (only if necessary)

Step 2: Download the Colibrì Dockerfile

Step 3: Build the Docker image

Step 4: Start Colibrì

What does that command mean?

Using Colibrì

Enter the container

Troubleshooting

❌ "Docker not found"

❌ "Out of memory" or container closes immediately

❌ The answer is very slow

❌ "Permission denied" on Linux

❌ "Image not found" or error during build

❌ "hf_download: command not found"

❌ The model does not download (timeout or network errors)

Technical notes

Why is the disk important?

Recommended default configuration for WLS on Windows

Frequently Asked Questions

Support and contributions

Testing on a low resource PC

Slim image (developers / production)

One-command server (docker compose)

Name		Name	Last commit message	Last commit date
parent directory ..
Dockerfile		Dockerfile
Dockerfile.slim		Dockerfile.slim
README.IT.md		README.IT.md
README.md		README.md
docker-compose.yml		docker-compose.yml

FilesExpand file tree

docker

Directory actions

More options

Directory actions

More options

Latest commit

History

docker

Folders and files

parent directory

README.md

Colibrì Guide - Local Inference Engine

📋 Table of Contents

What is Colibrì?

Requirements

Hardware

Software

How to get started

Step 1: Download the model

Method A: Using Python (recommended)

Method B: Without Python (only if necessary)

Step 2: Download the Colibrì Dockerfile

Step 3: Build the Docker image

Step 4: Start Colibrì

What does that command mean?

Using Colibrì

Enter the container

Troubleshooting

❌ "Docker not found"

❌ "Out of memory" or container closes immediately

❌ The answer is very slow

❌ "Permission denied" on Linux

❌ "Image not found" or error during build

❌ "hf_download: command not found"

❌ The model does not download (timeout or network errors)

Technical notes

Why is the disk important?

Recommended default configuration for WLS on Windows

Frequently Asked Questions

Support and contributions

Testing on a low resource PC

Slim image (developers / production)

One-command server (docker compose)