Repository navigation
Cache reliability improvements, FIFOSemaphore with context, and downloader bug fixes - #62
Merged
Merged
Conversation
8000
Added support for Context in the FIFOSemaphore; Fixes #60
- Avoid re-downloading model revision info when it is already cached by default. Documented update polling via `DownloadInfo(true)`.
- Short-circuit revision lookup when `revision` is already a 40-character commit hash.
- Removed unused `revisionHashRefreshed` field.
- Fixed `LockedDownload` to preserve existing cached files until a forced re-download successfully completes and atomically replaces them, avoiding cache deletion on network failure (issue #59).
janpfeifer
added a commit
that referenced
this pull request
Nov 5, 2026
* Add GetTensorRaw and ToGGMLQuantType for native GGML quantized weights - Add Model.GetTensorRaw() to load raw tensor bytes without dequantization, enabling quantized weights to stay in native format - Add TensorType.ToGGMLQuantType() mapping GGUF tensor types to GoMLX GGMLQuantType (Q4_0, Q8_0, IQ4_NL, Q2_K–Q6_K) - Update go.mod dependencies Depends on: ajroetker/gomlx#6 * Add multi-file GGUF support, FlexToken type, and dependency updates - Multi-file GGUF: support multimodal models (e.g. LLaVA) that split tensors across multiple GGUF files via LoadAll/LoadFiles/ListGGUFFiles. Tensors are looked up across all files transparently. - Pair extra files with their readers in a single extraEntry struct to prevent parallel-slice divergence and enable per-file lazy reader init. - Fix Close() to close all readers instead of leaking on first error. - Deduplicate .gguf filename filtering into a shared ggufFileNames helper. - Add FlexToken type to handle HuggingFace token config fields that can be either a plain string or an object with a "content" field. - Update go-xla, golang.org/x/sys, k8s.io/klog dependencies. * Add ExtraFiles accessor for additional GGUF files * Address PR review: rename Get→Read, eliminate double lookups, clean up FlexToken - Rename GetTensor→ReadTensor, GetTensorRaw→ReadTensorBytes per reviewer convention (Read prefix for I/O methods) - Merge findTensorFile+fileForIndex into findTensor to avoid double map lookup in GetTensorInfo and readerForTensor - Return error instead of silently swallowing invalid JSON in FlexToken.UnmarshalJSON - Remove unused FlexToken.String() method - Replace 7 copy-pasted config fallback blocks in resolveSpecialTokens with table-driven loop * Added the `models/transformer` package for loading HuggingFace transformer models. (Tested with tencent/KaLM-Embedding-Gemma3-12B-2511) * Cleanup, improved documentation. * Updated GoMLX dependency. * Updated CHANGELOG. * Updated dependency to GoMLX v0.27.2. * - Added `datasets` package to read HuggingFace datasets - Refactor `hub.Repo` functionality so it can be shared with `datasets` * Upper-case acronyms (initialism) * Added struct generator for datasets. * Renamed "version" -> "subset". * Added support to retrieve files information. Added dataset pretty printing. * Added downloading files API. Added API to iterate over parquet files. * Updated CHANGELOG. * - Package `transformers`: added `Model.GetTokenizer / SetTokenizer` methods. - Updated README.md and added some examples. - Updated go.mod dependencies. * Updated go.sum. * Fixed Model.LoadContext to return an error. * Fixed IterParquetFromDataset. * Broken attempt to fix parquet-go handling of list elements. * Fixed the ParquetFixListSchema function. * Fixed test of parquet parsing using local example parquet file. * - Added `ParquetFixListSchema` for fixing list schema parsed from Go struct (a bug? in parquet-go where it hard-codes the group/element node names in lists). - Fixed `cmd/generate_dataset_structs` to use the schema from the files themselves (as opposed from the Dataset info) * Enhance README with details on model conversion and datasets Updated README to clarify model conversion and dataset handling. * Ignore IDE files. * Removed debug message. * Reintroduced debug messages at klog level 1. * Added tokenizers/bucket to tokenize and bucketize sentences in a streaming fashion. * Updated changelog. * Fixed comments for Bucketizer.Run. * New tokenizer api, support for special tokens masks, bug fixes * Updated to latest * Adjusted to GoMLX transformer API updates: more logic moved to the GoMLX transformer package. * Updated GoMLX dependency. * - Added support for configured task prompts (via `task_prompts.json`) * refactor: replace task_prompts.json with SentenceTransformerConfig prompts and update prompt building logic * Added support for layer types (and support for sliding_attention configuration), for RoPE scaling, for separate configuration for RoPE for the sliding attention. * Added similarity function, based on HuggingFace json configuration. * Updated dependencies to matching changes in GoMLX. * Added support for reading tensors directly to a backend in packages `safetensors` and `transformer` * Packages `gguf` and `safetensors` now load tenso 8000 rs directly to backend (using shared buffers if available), if a backend is given. The `transformer.LoadContext()` now take an optional (nil-lable) backend, so variables are loaded directly to backend device. * Fixed version dependency to GoMLX. * Added `Config() *Config` method to the Tokenizer API. * - Packages `hftokenizer` and `sentencepiece` - Added support to `AddBosToken` and `AddEosToken`. - Added post-processing to the `sentencepiece` tokenizer. * Updated CHANGELOG. * Added KaLM-Gemma3 example. * Added MSMARCO dataset example. * Updated REAME.md and CHANGELOG. Added CI and .github directory. * Fixed CI workflow. * go mod tidy. * Skip MSMarco test if -test.short is set. * Updated warning message. * Revise experimental status and coverage in README Updated the README to clarify the experimental status and expand on the coverage of models, tokenizers, and datasets. * Support Batching and Padding (Masking) for Sentence Embeddings (#34) * Fixed ApplySentencePooling to handle masks. * Cosmetic. * Fixed gather of last token, when pooling for sentence. * Added batch+padding tests for KaLM-Gema3 model. * Updated GoMLX dependency. * Updated CHANGELOG. * Updated GoMLX dependency. * Enhance safetensors performance (#35) - Re-implemented IterTensors and IterTensorsFromRepo with github.com/edsrzf/mmap-go (to avoid one kernel buffer copy to normal memory) - IterTensorsFromRepo has its own implementation with a parallelized pipeline (including parallelization of transfers to the GPU, to ensure XLA will optimize them in a queue) for loading model throughput. - Loading of KaLM-Gemma3 12B model (with 22 GB of weights) now takes ~1.5s where previously it took ~8.5s (and slightly faster than PyTorch's 1.72s). * Buket non-padding tokens count (#36) * Added number of non-pad tokens to bucket. * Improved humanize internal package. * Added "two-bits bucketing" strategy. (#37) * Added `bucket` example. * Update README.md Added info about `bucket` at the intro section. * Replace internal humanize with gomlx/pkg/support/humanize (#38) * Updated `humanize` to use GoMLX pkg/support/humanize instead. * Updated GoMLX version. * Added datasets.CreateParquetReader (#39) * Adding IterParquetFromDatasetAt (#40) * Add support for BERT architecture and BAAI-bge-small-en-v1.5 model (#41) * Refactoring embedding testing functions to internal/testing. * Adding (but not working yet) BAAI/bge-small-en-v1.5 model under examples/. Added test of embeddings. * Added handling of chinese characters in hftokenizer. * Added support for BERT architecture and BAAI/bge-small-en-v1.5 model. * Added benchmark_embed to test/measure the speed of embedding models. * Added bucket.WithBatchPadding and use it in benchmark_embed. * Updated documentation. * Updated GoMLX dependency to version with BERT transformer support. * Added -num_embedders flag. * Fixed flag name. * go mod tidy * Removed dependency to github.com/gomlx/exceptions. * Updated GoMLX dependency. * Updated dependency to GoMLX v0.27.3; Updated CHANGELOG. (#42) * Updated to migration of backends to github.com/gomlx/compute (#43) * Moved `shapes`, `dtypes`, `xslices` and `bfloat16` to new gomlx/compute repo. * Replaced package `float16` with `compute` version. * Fixed to new `float16.FromFloat32()`. * Updated backends -> compute. * Moved to use testutil.TestOfficialBackends. * Updated deps. * Updated dependencies to `humanize` to use the one in github.com/gomlx/compute. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Updated GitHub workflows to use go 1.26 * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Added -warmup and -shapes flag to MSMARCO benchmark program. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Added -cpuprof to benchmark_embed tool. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Updated to upcoming GoMLX v0.28 API changes (#44) * Remove pkg/ subdirectory from GoMLX exported packages. * Remove pkg/ subdirectory from GoMLX exported packages. * Convert GoMLX's context to model. * Fixed broken renamed standard `context`. * Reverted wrong renaming of standard `context`. * Fixed renaming from GoMLX context to model. * Reverted changes to standard `context`. * Converting to new model.Store design. * Fix GoMLX v0.28 renaming. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Renaming for GoMLX v0.28. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Adjusted to GoMLX v0.28 redesign. * GoMLX v0.28 updates. * Added default backends. * Added LoadStoreFiltered. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Enhance README overview formatting for clarity Emphasized key features in the overview section. * Update README to clarify datasets description Removed the word 'simplified' from the datasets description. * Humanize log messages. (#45) * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Feat: Add Size field to FileInfo and introduce cmd/hubinfo CLI (#47) * - Added `?blobs=true` to info request. - Added many of the missing fields in RepoInfo and FileInfo. - Added test parsing of the RepoInfo structure. * Added Hub.IterFileInfos * Added cmd/hubinfo to quickly check on the info about a HuggingFace repo. * List UsedStorage by default. * Update README.md * Add file sizes to model files in README Updated file sizes for various model-related files in README. * Refactor ONNX model usage in README.md Updated ONNX model handling and GoMLX integration in README. * Improve support for large datasets, on-demand downloads, and CLI styling (#48) * Include error message returned from the server. * Tool `github.com/gomlx/go-huggingface/cmd/generate_dataset_structs` - Added coloring of the output; - Handle large datasets: pick one parquet file to extract the info; * Fixed handling of large datasets, requiring crawling of files. * Fixed removing the .lock file on failures. * Fixed iteration on large datasets. * Download one parquet file at a time, in IterParquetFromDataset (preventing it to download all files when only one is needed). * Updated README.md. * Fixed typos, and small updates. * Updated CHANGELOG. * Revise README for HuggingFace model details Updated README.md to reflect changes in HuggingFace model support and usage instructions. * Fixed Transformer models demo. Updated to GoMLX v0.28 API and added details. * Fixes to Tokenizer section in README Reorganize and update the section on tokenizer classes for clarity and consistency. * Add Dataset.DownloadAll to pre-download parquet files (#49) * Added Dataset.DownloadAll * Added ./cmd/dataset_download tool to manage dataset downloads. * gofmt fix. * Update to use new GoMLX generate/kvcache API and add Gemma4 demo (#50) * Added support for missign Gemma tokenization to hftokenizer. * Read RoPE config from corresponding JSON file if available. * Added Gemma4 4B-it demo. * Added gemma debug test. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Skip Gemma tests in -short mode. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Skip tests that hit HuggingFace Hub if -short: the CI tests hit their rate limits with no HuggingFace token set. * Switch the check for tests hitting HuggingFace Hub: now they are skipped when HF_TOKEN is not set. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update go-huggingface to conform to the new gomlx transformer CallOptions API * Added -naive version to generation. * Added klog.InitFlags to allow enabling logging. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Move temporary file download logic to downloader.Download and modernize sync.WaitGroup usage (#51) * Modernized `sync.WaitGroup` usage. * Moved logic that handles downloading to a temporary file first to `donwloader.Download` (before it was in `downloader.LockedDownload`) * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Add SAM2 image segmentation model (#52) * Added support package models/image and function `PreprocessGraph`. * Added test image. * Initial SAM2 implemnetation. * Cleaned up test and API. * Clean ups. * Fixed and documented some of the abbreviations used. * Added a Go API for image segmentation. Added a demo. * Renamed file to segmenter.go * Added package documentation and a README.md. * Updated dep to GoMLX. * Renamed SegmentationOptions. * Update GoMLX dependencies for github.com/gomlx/go-huggingface * Updated CHANGELOG. * Updated GoMLX dependency. * Updated dependencies. * Revise sponsorship link and add support information Updated sponsorship link and added support section for donations. * Add documentation badge to README * feat(hub): add local-directory mode for offline model loading (#53) * feat(hub): add local-directory mode for offline model loading Add hub.NewLocal(dir) / Repo.WithLocalDir(dir) so a *hub.Repo can read files straight from a plain local directory instead of downloading from HuggingFace Hub, with no network access. DownloadInfo scans the directory and DownloadFile(s) resolve paths within it; every existing consumer of *hub.Repo works unchanged. * Delete docs/local-model-loading-plan.md Removing artifact of the AI generation. * Update CHANGELOG to remove unnecessary text Removed design rationale reference from changelog entry. --------- Co-authored-by: Jan <janpfeifer@users.noreply.github.com> * Add repository Save(), DeleteCache(), FetchFiles(), and embedded repository (hub.NewEmbed) support (#54) * ### 1. Save() and SaveCtx() in local.go • func (r *Repo) Save(dirPath string, linkOnly bool) error • func (r *Repo) SaveCtx(ctx context.Context, dirPath string, linkOnly bool) error • Resolves/downloads all files in the repository into the Hugging Face cache structure and copies or hard-links (os.Link) them into dirPath. • Always overwrites existing files in dirPath. • Fails with an error if r.IsLocal() is true. ### 2. DeleteCache() in delete.go • func (r *Repo) DeleteCache() error • Deletes the repository's cache directory (e.g., ~/.cache/huggingface/hub/models--...). • Documented that calling any Repo method after DeleteCache() will require files to be re-downloaded from Hugging Face Hub. • Fails with an error if r.IsLocal() is true. ### 3. Command-line Flags in main.go Added three flags to hubinfo: • -save=<target_dir>: Saves the repository files to target_dir. • -link_only: Uses hard links instead of copying files when -save is specified. • -delete_cache: Deletes the downloaded cache directory for the repository (executed after -save if both are specified). * Added `-local` flag to `hubinfo` CLI tool. Improved documentation. * Added new functionality to README.md. * ### 1. hub Package Additions (embed.go) • hub.NewEmbed(fsys fs.FS, subDir string) *Repo & Repo.WithEmbedFS(fsys, subDir): Allows backing a *hub.Repo with any fs.FS (such as embed.FS). • Repo.IsEmbed() bool: Checks if the repository is in embedded-filesystem mode (alongside IsLocal()). • Repo.Open(name) & Repo.ReadFile(name): Stream reading methods that work across all repository modes (embedded, local, and remote) directly from memory/FS without disk extraction. • Fallback for DownloadFile(s): If legacy functions requesting disk file paths are called on an embedded repository, the files are extracted to a temporary directory under os. TempDir()/go-huggingface-embed/<ID>/ and the local disk path is returned. • Detailed docstrings added to NewEmbed explaining semantics, side-effects, and stream reading usage. ### 2. Downstream Libraries Updated • tokenizers (tokenizers.go & hftokenizer.go): Updated GetConfig() and hftokenizer.New() to use repo.ReadFile() directly. • models/transformer (model.go): Updated model configuration loader to use repo.ReadFile(). • models/safetensors (header.go): Updated parseHeader to read safetensors headers directly via repo.Open(). ### 3. Documentation & Changelog • Added "Embedded Models (//go:embed)" sub-section to README.md with a complete usage example. • Updated ## Unreleased section of CHANGELOG.md describing all new features. * Added FetchFile(s) to pre-download files (no-op for local or embed repos). Added comments to avoid DownloadFile(s), and instead use Open(), ReadFile() or FetchFile(s) instead. * Updated KaLM-Gemma3 to use new API to access files. * Updated README.md. * fix: BertNormalizer/Lowercase offset tracking under-counts multi-byte UTF-8 output (#55) applyNormalizerWithSpans returns offsets[normalizedBytePos] = origBytePos, one entry per BYTE of the normalized string, since callers (encodeCore, preTokenizeWithSpans, tokenizeWordWithSpans) index it as a direct byte-slice offset map. The "BertNormalizer" case's accent-stripping/lowercasing branch appended exactly one offset entry per RUNE (`for range s`) while writing the rune's full byte length via `result.WriteString(s)`. For any character that passes through unchanged as a multi-byte UTF-8 sequence — every accented Latin character (á, é, í, ó, ú, ñ, ã, ç, ă, â, î, ș, ț, ü, ö, ä, ...) on a cased model that neither strips accents nor lowercases (e.g. bert-base-cased-derived NER models, whose tokenizer.json ships strip_accents=null, lowercase=false) — this under-filled offsets by one entry relative to len(result). The same bug was present in the HandleChineseChars branch's `result.WriteRune(r)` call (CJK characters are typically 3 bytes, appending only 1 offset entry), and in the "Lowercase" case's `for range lowerRunes` loop (bounds-checked so it didn't panic, but left trailing offset entries at their zero value for any multi-byte lowercased character). The deficit compounds across the string. Once len(offsets) falls behind len(normalized text), downstream indexing into offsets by byte position runs past its length, producing "slice bounds out of range" panics. Reproduced directly against the real optimum/bert-base-NER tokenizer.json (strip_accents=null, lowercase=false, handle_chinese_chars=true) via github.com/knights-analytics/hugot: every tested Romanian, Spanish, Portuguese, and CJK/Cyrillic sentence triggered the panic in production (baditaflorin/go_named_entity_recognizer), silently downgrading every non-English request to a far weaker fallback engine. Fix: iterate by BYTE COUNT (`for range len(s)`, `for range runeLen`) instead of rune count when appending/filling offset entries, so offsets always has exactly one entry per byte of the transformed output. Adds hftokenizer_offsets_test.go: unit tests reproducing the deficit for accented Latin text, CJK text, a full Romanian/Spanish/Portuguese sentence, and the Lowercase branch, asserting len(offsets) == len(normalized) and that every entry resolves to the correct source rune. Verified these tests fail against the pre-fix code and pass after. * Support dynamic shapes in sentence pooling and update GomLX dependencies (#56) * Updated test to use new `Exec.Compile` (and removed the old call to `Exec.PreCompile`) * Added using dynamic shapes for MSMarco benchmark tests, if the Backend support it. * Updated dependencies. * Better handling dynamic shapes. * Moving from Capbilities.DynamicAxes bool -> DynamicShapes as an enum. * Updated dependencies. * Updated dependencies. * Updated dependencies. * safetensors: bounds-check tensor data_offsets before slicing (#57) A safetensors file header is untrusted input, and each tensor's data_offsets come straight from it. Both the single-tensor reader (ReadTensor) and the iterator sliced mmapBuf[offset:end] using those offsets: ReadTensor only checked that (end-start) matched the shape's byte size, and the iterator did no check at all. Neither verified the offsets fell within the mapped data. A crafted header could therefore declare data_offsets pointing far beyond the buffer, or a negative start, and crash the reader with a slice-out-of-range panic while loading an untrusted model. Validate that the offsets are non-negative, ordered, and within the data section before slicing, returning an error otherwise. Adds a regression test. * Updated dependencies. * Updated GitHub action to go1.27.x * Updated dependencies. * Updated dependencies. * Add -dynamic flag for benchmark (#58) * Added -dynamic flag for benchmark. * Updated dependency * Updated dependencies. * Updated dependencies. * Dimension dtype now depends on Backend's DynamicDimDType. * Updated dependencies. * Updated dependencies. * Updated dependencies. * Cache reliability improvements, FIFOSemaphore with context, and downloader bug fixes (#62) * Fixed report of false error, described in #61 * Replaced Semaphore with a FIFOSemaphore; Added support for Context in the FIFOSemaphore; Fixes #60 * - 20Cache and offline reliability improvements in `hub`: - Avoid re-downloading model revision info when it is already cached by default. Documented update polling via `DownloadInfo(true)`. - Short-circuit revision lookup when `revision` is already a 40-character commit hash. - Removed unused `revisionHashRefreshed` field. - Fixed `LockedDownload` to preserve existing cached files until a forced re-download successfully completes and atomically replaces them, avoiding cache deletion on network failure (issue #59). * Updated CHANGELOG with other bug fixes. * Added .agents/AGENTS.md * fix: accept string or array for cardData.license (#63) * fix(hub): accept string or array for cardData.license Hugging Face Hub may return license as a JSON array; unmarshaling into a string field broke DownloadInfo for datasets like bigcode/the-stack. Signed-off-by: Sobha Cheruku <scheruku@redhat.com> * test: skip live Hub license test in short mode * fix: update CardData.License as []string --------- Signed-off-by: Sobha Cheruku <scheruku@redhat.com> * default an empty pretokenize split regex to all chars (#64) * Auto-detect causal mask from model configuration (fixes #65) (#66) * Added package documentation. Auto-detect `CausalMask` for the model. * Removed `-use_causal_mask` flag, that was previously being ignored and now was setting the causal mask to the wrong value. * Updated dependencies. * Fix token byte spans drifting with Replace normalizer and refactor normalizer (fixes #67) (#68) * Created issue #67 test. * Refactored the normalizer out of `hftokenizer.go` into `normalizer.go` Large cleanups and simplification of the code, in particular the handling of the offsets from the normalize string to the original string. * Refactored tests into their corresponding files. * Added saving/loading HF transformer models to/from a checkpoint. (#69) * Updated dependencies. * Updated dependencies. * - `hub`: Resolve symlinks via `filepath.EvalSymlinks` in `Repo.Save` / `Repo.SaveCtx` when `linkOnly=true` so hard links point to the underlying blob files rather than copying relative snapshot symlinks. (#70) * Updated dependencies. * Updated dependencies. * Add native GoMLX support for Mistral AI's Timestral time-series foundation models (models/timestral) (#71) * Updated dependencies. * Added `timestral` times series foundational model support. * Updated dependencies. * models/safetensors: Add option to load tensors via standard file I/O without mmap (#72) * Make loading files with MMap optional. * More memory debugging options; added -mmap flag. * Removed the inserted GC used for debugging. * Updated dependencies. * Updated dependencies. * models/timestral: Fuse attention gate Sigmoid and FusedSwiGLU (#73) * - `models/timestral`: - Fused attention `_gate` projection (`Dense` + `Sigmoid`) into a single `nn.Dense(..., activation.TypeSigmoid)` (`FusedDense` with `ActivationSigmoid`). - Swapped `w12` `[gate; up]` halves to `[up; gate]` at weight-load time and replaced manual `Slice` + `Silu` + `Mul` in `FusedSwiGLU` with `activation.SwiGLU(w12)`. - `examples/timestral`: Format elapsed durations using `humanize.Duration()`. * Updated dependencies. * Updated dependencies. * - `examples/msmarco/benchmark_embed`: Added support for ONNX models via `github.com/gomlx/onnx-gomlx` when `.safetensors` files are not available in the model repository (e.g. `KnightsAnalytics/all-MiniLM-L6-v2`). - `models/transformer`: - Added `Model.SentenceEmbeddingFromLastHiddenState(lastHiddenState, seqLen)` to apply post-Transformer sentence embedding modules (pooling and L2 normalization) to an existing hidden-state tensor. * Updated dependencies. * Added metrics to MSMarco bencharmker. * Updated dependencies. --------- Signed-off-by: Sobha Cheruku <scheruku@redhat.com> Co-authored-by: Jan Pfeifer <pfeifer@gmail.com> Co-authored-by: Jan <janpfeifer@users.noreply.github.com> Co-authored-by: Rob Keevil <rob@knightsanalytics.com> Co-authored-by: Niexiawei <1135215907@qq.com> Co-authored-by: Florin Badita-Nistor <baditaflorin@gmail.com> Co-authored-by: Vladimir Babin <vovababin@gmail.com> Co-authored-by: Sobha Cheruku <scheruku@redhat.com> Co-authored-by: Rob Keevil <robkeevil@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary of Changes
This PR bundles several downloader bug fixes, offline caching improvements in
hub, and AI agent documentation:Fix Downloader Error Reporting (Fixes A non-EOF read error during Download() is wrapped around a stale nil, so the function returns success on a real failure #61):
internal/downloader/downloader.goto wrapreadErrinstead of nil/shadowederrwhen reading the response body fails.FIFOSemaphore with Context Support (Fixes Semaphore's cond.Signal on Release can be won by a fresh Acquire instead of an already-waiting goroutine, stalling concurrent downloads #60):
FIFOSemaphoreininternal/downloaderto guarantee strict FIFO waiter ordering and enable cancellation viacontext.Context.FIFOSemaphorecovering basic acquisition, strict FIFO order, cancellation, resizing, and drain behavior.Hub Cache & Offline Reliability Improvements (Fixes A warm cache still needs the network on every process, and an offline refresh failure deletes the cached revision info #59):
DownloadInfo(true)).revisionHashRefreshedfield.LockedDownloadto preserve existing cached files during forced re-downloads until the download succeeds and atomically replaces the file, preventing cache loss on transient network errors.Documentation:
docs/CHANGELOG.mdwith all recent fixes and improvements..agents/AGENTS.mdas an architectural and engineering reference for AI developer agents.