Repository navigation
Tags
Tags
v1.2.1 - reject binary input Binary files (ELF, PDF, image, archive) and UTF-16 text now produce a clear error instead of silently returning tens of millions of garbage tokens. Detection is a NUL-byte scan over the first 8 KB, matching the long-standing convention of grep / file(1) / git.
v1.2.0 - lax fallback + 2026-05 catalog from live APIs Key change: openai and google now tokenize ALL their catalog models locally via prefix/alias fallback. OPENAI_API_KEY and GEMINI_API_KEY are now optional escape hatches rather than commonly needed. Catalog refreshed by directly querying each provider's /v1/models endpoint as of 2026-05. OpenAI 5.1/5.2/5.4 series, Anthropic 4-1/ 4-5 entries, and Gemini's correct -preview-suffixed 3.x IDs are all in. Anthropic still always uses the remote count_tokens API because no public Claude tokenizer exists.
v1.1.0 - OpenAI hybrid fallback + 2026-05 catalog refresh - New OpenAI fallback for models not in tiktoken-go (e.g. gpt-5.5): hits POST /v1/responses/input_tokens with OPENAI_API_KEY. - Catalog updated to 2026-05: Claude Opus 4.7, Sonnet 4.6, Gemini 3.1 Pro / Flash-Lite, Gemini 3 Flash, GPT-5.5 / Pro. - OPENAI_API_KEY policy changed from 'unused' to 'optional, used for new-model fallback'.
release(v1.0.1): musl-static linux builds + install.sh bootstrap The v1.0.0 linux artifacts linked against ubuntu-24.04's GLIBC 2.39, so they failed to run on RHEL 9 / Rocky 9 / Amazon Linux 2023 (GLIBC 2.34) and other distros with older runtimes. Changes: - Makefile: STATIC=1 mode for linux (musl + libstdc++ statically linked, netgo + osusergo to drop the cgo resolvers, -s -w to strip symbols). TOKENIZERS_TRIPLET is now overridable; default detection unchanged. - release.yml: linux jobs build inside golang:1.24-alpine via docker, pulling daulet/tokenizers' linux-musl-* prebuilts. macOS jobs stay native. Resulting linux binaries run on any glibc. - install.sh: one-liner bootstrap that detects host, fetches the matching archive from the latest release, verifies SHA-256, and installs into \$INSTALL_DIR (default ~/.local/bin). curl or wget; sha256sum or shasum. - README: quick install section and build-from-source clarifications. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
v1.0.0 - first release Multi-provider LLM token counter CLI. Providers: - OpenAI (local tiktoken-go, no API key) - Anthropic Claude (REST API, ANTHROPIC_API_KEY required) - Google Gemini (local tokenizer + auto remote fallback) - HuggingFace (daulet/tokenizers, any <org>/<repo>) Features: - stdout = single integer; stderr + exit 1 on error - shell tab completion (bash/zsh/fish/powershell) - HF org autocomplete via Hub API with 24h cache - Makefile install/uninstall via symlink (default ~/.local/bin)