8000 Tags · antegral/tok · GitHub
[go: up one dir, main page]

Skip to content

Tags

Tags

v1.2.2

Toggle v1.2.2's commit message

Verified

This tag was signed with the committer’s verified signature.
antegral SeongUk Moon
Release v1.2.2

v1.2.1

Toggle v1.2.1's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
v1.2.1 - reject binary input

Binary files (ELF, PDF, image, archive) and UTF-16 text now produce
a clear error instead of silently returning tens of millions of
garbage tokens. Detection is a NUL-byte scan over the first 8 KB,
matching the long-standing convention of grep / file(1) / git.

v1.2.0

Toggle v1.2.0's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
v1.2.0 - lax fallback + 2026-05 catalog from live APIs

Key change: openai and google now tokenize ALL their catalog models
locally via prefix/alias fallback. OPENAI_API_KEY and GEMINI_API_KEY
are now optional escape hatches rather than commonly needed.

Catalog refreshed by directly querying each provider's /v1/models
endpoint as of 2026-05. OpenAI 5.1/5.2/5.4 series, Anthropic 4-1/
4-5 entries, and Gemini's correct -preview-suffixed 3.x IDs are
all in. Anthropic still always uses the remote count_tokens API
because no public Claude tokenizer exists.

v1.1.0

Toggle v1.1.0's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
v1.1.0 - OpenAI hybrid fallback + 2026-05 catalog refresh

- New OpenAI fallback for models not in tiktoken-go (e.g. gpt-5.5):
  hits POST /v1/responses/input_tokens with OPENAI_API_KEY.
- Catalog updated to 2026-05: Claude Opus 4.7, Sonnet 4.6,
  Gemini 3.1 Pro / Flash-Lite, Gemini 3 Flash, GPT-5.5 / Pro.
- OPENAI_API_KEY policy changed from 'unused' to 'optional, used
  for new-model fallback'.

v1.0.1

Toggle v1.0.1's commit message

Verified

This commit was signed with the committer’s verified signature.
antegral SeongUk Moon
release(v1.0.1): musl-static linux builds + install.sh bootstrap

The v1.0.0 linux artifacts linked against ubuntu-24.04's GLIBC 2.39, so
they failed to run on RHEL 9 / Rocky 9 / Amazon Linux 2023 (GLIBC 2.34)
and other distros with older runtimes.

Changes:
- Makefile: STATIC=1 mode for linux (musl + libstdc++ statically linked,
  netgo + osusergo to drop the cgo resolvers, -s -w to strip symbols).
  TOKENIZERS_TRIPLET is now overridable; default detection unchanged.
- release.yml: linux jobs build inside golang:1.24-alpine via docker,
  pulling daulet/tokenizers' linux-musl-* prebuilts. macOS jobs stay
  native. Resulting linux binaries run on any glibc.
- install.sh: one-liner bootstrap that detects host, fetches the matching
  archive from the latest release, verifies SHA-256, and installs into
  \$INSTALL_DIR (default ~/.local/bin). curl or wget; sha256sum or shasum.
- README: quick install section and build-from-source clarifications.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

v1.0.0

Toggle v1.0.0's commit message

Unverified

This tag is not signed, but one or more authors requires that any tag attributed to them is signed.
v1.0.0 - first release

Multi-provider LLM token counter CLI.

Providers:
- OpenAI (local tiktoken-go, no API key)
- Anthropic Claude (REST API, ANTHROPIC_API_KEY required)
- Google Gemini (local tokenizer + auto remote fallback)
- HuggingFace (daulet/tokenizers, any <org>/<repo>)

Features:
- stdout = single integer; stderr + exit 1 on error
- shell tab completion (bash/zsh/fish/powershell)
- HF org autocomplete via Hub API with 24h cache
- Makefile install/uninstall via symlink (default ~/.local/bin)
0