8000 GitHub - raoqu/getmodel: Native C++ CLI for downloading HuggingFace and ModelScope AI models with automatic source selection, terminal progress, and resumable downloads. · GitHub
[go: up one dir, main page]

Skip to content
raoquPublic

About

Native C++ CLI for downloading HuggingFace and ModelScope AI models with automatic source selection, terminal progress, and resumable downloads.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

Repository files navigation

getmodel

中文说明

getmodel is a small native C++ command-line downloader for AI model repositories. It supports HuggingFace and ModelScope URLs, automatically chooses the better available source, and resumes interrupted downloads.

Features

  • Download public HuggingFace model repositories.
  • Download public ModelScope model repositories.
  • Download a full repository, a single repository directory, or a single repository file.
  • Pass one URL or both mirror URLs:
getmodel [--target <path>] [--force] [--retry <n>] [--delay HH:MM:SS] <huggingface_url> [modelscope_url]
getmodel [--target <path>] [--force] [--retry <n>] [--delay HH:MM:SS] <modelscope_url>
  • When two URLs are provided, each repository is checked with a 3 second timeout. One source may be unavailable; getmodel only fails when both sources are unavailable.
  • If both sources are reachable, getmodel probes download speed for up to 3 seconds and selects the faster source.
  • Interrupted downloads resume from .part files.
  • Network failures during file download are retried 5 times by default. Use --retry <n> to change the retry count; --retry 0 retries forever, switching from a 2 second delay to a 5 second delay after 10 retries.
  • Use --delay HH:MM:SS to wait for a relative duration before contacting remote sources and starting downloads. The wait displays a countdown.
  • The selected source is stored in models/<repo>/.getmodel/state.txt, so resumed downloads prefer the same source.
  • Completed downloads write models/<repo>/.getmodel/manifest.txt; later non-force runs verify local completeness first and print only the model path without probing networks.
  • --force verifies existing files with remote SHA256 when available and downloads only changed files.
  • Progress is displayed with a terminal-friendly ASCII progress bar.

Build

Requirements:

  • CMake 3.16+
  • C++17 compiler
  • libcurl development package

Build with the helper script:

./build.sh

Or build manually:

cmake -S . -B build
cmake --build build

The executable is written to build/getmodel and copied to ./getmodel.

Usage

Download from one source:

./getmodel https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct

Download with HuggingFace and ModelScope mirrors:

./getmodel \
  https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct \
  https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct

Supported URL forms include:

https://huggingface.co/owner/model
https://huggingface.co/owner/model/tree/main
https://huggingface.co/owner/model/tree/main/path/to/dir
https://huggingface.co/owner/model/blob/main/path/to/file
https://huggingface.co/owner/model/resolve/main/path/to/file
https://modelscope.cn/models/owner/model
https://modelscope.cn/models/owner/model/files
https://modelscope.cn/models/owner/model/tree/master/path/to/dir
https://modelscope.cn/models/owner/model/file/view/master/path/to/file

For example, download one quantized directory:

./getmodel https://huggingface.co/unsloth/GLM-5.2-GGUF/tree/main/UD-IQ1_M

Or download one file:

./getmodel "https://modelscope.cn/models/unsloth/GLM-5.2-GGUF/file/view/master/configuration.json?status=1"

Choose a target directory:

./getmodel --target ./Qwen2.5-0.5B-Instruct https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct

Update an existing directory by hash:

./getmodel --force --target ./Qwen2.5-0.5B-Instruct \
  https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct \
  https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct

Set download retry behavior:

./getmodel --retry 10 https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct
./getmodel --retry 0 https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct

Delay the download start:

./getmodel --delay 00:05:00 https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct

By default, files are written under:

models/<owner>-<model>

Resume Behavior

Files are downloaded as filename.part and renamed only after the expected size is complete. Running the same command again continues partial files with HTTP range requests.

If a previous run selected a source and partial files still exist, getmodel reads .getmodel/state.txt and prefers that source for continuation.

After a successful complete download, getmodel writes .getmodel/manifest.txt. On later non-force runs, if the state and manifest match the requested model and all files are complete, getmodel returns immediately and prints only the model path.

With --force, existing complete files are checked against remote SHA256 when the provider exposes one. Matching files are skipped; changed files are downloaded again. Files without remote SHA256 fall back to the normal complete-size check.

C++ API

The download workflow is available as a function:

#include "getmodel.h"

DownloadModel({
    "https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct",
    "https://modelscope.cn/models/Qwen/Qwen2.5-0.5B-Instruct",
}, "./Qwen2.5-0.5B-Instruct", true);

Signature:

void DownloadModel(const std::vector<std::string>& urls,
                   const std::filesystem::path& target_path = {},
                   bool force_update = false,
                   int retry_count = 5,
                   long long delay_seconds = 0);

Authentication

Optional environment variables:

  • HF_TOKEN or HUGGINGFACE_TOKEN for HuggingFace
  • MODELSCOPE_SDK_TOKEN for ModelScope

Clean

./clean.sh

This removes the build directory and downloaded models/ directory.

About

Native C++ CLI for downloading HuggingFace and ModelScope AI models with automatic source selection, terminal progress, and resumable downloads.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages

0