Repository navigation
Tags
Tags
- emily now requires Nx 1.0, and Axon 0.9 / Bumblebee 0.8 for the opt…
…ional
Axon and Bumblebee integrations. To stay on Nx 0.12, pin
`{:emily, "~> 1.3.0"}`.
- The native compiler (`native: true`) now compiles `Nx.block`s that
`Emily.Backend` has no native kernel for, including every Axon 0.9
activation (GeLU, softmax, ReLU, SiLU and the rest). It lowers the block's
default implementation, which is exactly what the Evaluator runs, so the
results are identical. Without this, every Axon or Bumblebee model would
fall back to the Evaluator under Axon 0.9.
- Under `native: true`, a hook passed in the `:hooks` option (for example
by `Nx.Serving` when streaming Bumblebee text generation) was silently
skipped: the defn compiled natively and the hook never ran. A defn with
a hooked `io_call` now falls back to the Evaluator, which runs the hook.
Named `io_call`s with no hook registered still compile natively.
- Installing emily from Livebook (`Mix.install`) no longer crashes while downloading the precompiled NIF with "Can't set long node name!". The download ran on a named peer VM, which fails to boot under a longnames-distributed parent when the hostname is not fully qualified. The peer now starts without distribution.
- `Emily.Quantization.Transform` now ships with the package. Previously it was test-only code. `quantize/3` rewrites every `:dense` layer of an Axon or Bumblebee model to run through `Emily.Quantization.Layers.quantized_dense/4` and replaces its kernel in the `Axon.ModelState` with an `Emily.QuantizedWeight`. `quantize_dense_layers/2` and `quantize_model_state/3` do each half separately. Options are `:bits`, `:group_size`, `:transpose`, `:mode` and `:except`, which keeps matching layers as float `:dense` layers. The module is defined only when Axon is available. The quantized weights are evaluated before `quantize/3` returns, so the dense kernels are freed as soon as the caller drops the original model state instead of staying resident until the first forward pass. The `qwen3_quantized` livebook now calls `quantize/3` instead of defining its own transform. - Under `native: true`, `Nx.cumulative_sum/2`, `cumulative_product/2`, `cumulative_max/2` and `cumulative_min/2` on any axis but the last now read their own operand. They previously used the function's first input instead, which gave a silently wrong result when the shapes matched and an MLX `[slice]` error when they did not.
- `Emily.QuantizedWeight.from_dense/2` accepts `bits: 1` for `"affine"` quantization, which MLX 0.32.4 added. Each element dequantizes to its group's minimum or maximum, packed 32 to a `u32`. 1-bit weights work with `Emily.Quantization.quantized_matmul/2`, `quantized_matmul_defn/2` and `dequantize_defn/1`, and `Emily.Quantization.defn_supported_bits/0` now includes `1`. On the GPU, MLX widens every group's range to include zero, so a group whose values are all negative dequantizes to its minimum or `0`. - Updated the pinned MLX to 0.32.4. Emily picks up its upstream fixes without API changes. They cover strided and sliced views, reductions over very large arrays, FFTs of size one, empty unsigned sums and products on Metal, and a deadlock during eval cleanup. Split-K quantized matmul now keeps its partial sums in float32. The release also speeds up quantized matvec when the output size isn't a multiple of 8, GEMMs with a thin N, and vector SDPA decode, and uses less memory for SDPA with head dim 512 on chips before M5.
- Compiling emily as a dependency no longer warns that `:public_key.cacerts_get/0` and `:public_key.pkix_verify_hostname_match_fun/1` are undefined. The warnings came from `mix emily.checksums`, a maintainer-only release task that shipped in the hex package. The package now leaves out the Mix tasks that only work in a checkout of the repository (`emily.checksums`, `bench.native` and `clang.tidy`); `mix emily.doctor` is still included.
- `Emily.Fast.scaled_dot_product_attention/4` and
`scaled_dot_product_attention_with_mask/5` accept `force_fused: true`.
MLX then always uses one of its fused attention kernels instead of
choosing per shape, and raises `ArgumentError` if no fused kernel
supports the shape. This keeps attention memory bounded for
long-context prefill with head dims 192 and 256 (for example Gemma),
where MLX otherwise builds the full attention-score matrix, at some
cost in prefill throughput. Off by default; it works in both eager
and natively compiled code.
- `Emily.Fast.rope/3` and `rope_with_freqs/4` accept a `{batch}` offset
as well as a scalar, so each entry along the leading axis starts at its
own position, for example in batched decode.
- `Emily.Bumblebee.FastKernels.apply/2` accepts `force_fused_sdpa: true`,
which passes `force_fused: true` to every rewritten attention layer.
- `Emily.Fast.einsum/2` can now be called inside `defn`. Under
`Emily.Compiler`, evaluated or natively compiled, it still runs MLX's
`mx::einsum`. Other backends, such as `Nx.BinaryBackend` and EXLA, run a
composed `Nx.dot/6` fallback that handles diagonals, ellipses,
broadcasting and implicit output. Equations that `mx::einsum` would
reject raise `ArgumentError` on every backend, naming the problem.
- `Emily.Fast` kernels can now be called from inside a caller's own
`defn` or `defnp`, as the module documentation shows. Before, they only
worked as the top-level function passed to `Nx.Defn.jit`, and raised
"cannot invoke ... inside defn because it was not defined with defn"
when nested (#205).
- `Emily.Bumblebee.FastKernels.apply/1` now rewrites the attention and
RoPE layers of Bumblebee models. Before, it only rewrote RMSNorm and
LayerNorm: it never recognised Bumblebee's attention and RoPE layers,
so the fused SDPA and RoPE kernels never ran. Decoder models such as
Qwen3 now get the fused kernels, whose advantage grows with context
length. The newly active paths are checked against stock Bumblebee,
including the `:linear`, `:dynamic`, `:longrope` and `:llama3` RoPE
scaling strategies, padded and batched position ids, and causal decode
against a KV cache. Attention is fused even in models whose checkpoint
sets a nonzero attention dropout rate, such as DistilBERT and other
BERT-family models, because dropout does nothing at inference (#221).
**If you train a rewritten model, it no longer applies attention
dropout.** The fused attention layer has none, whereas the stock
attention layers that `apply/1` used to leave in place did. Train with
the stock model, and apply `FastKernels` for inference only.
- `Emily.Fast.rope_with_freqs/4` rotated by the wrong angles on the
Emily backend: MLX expects periods, not the inverse frequencies the
function takes. It now matches its documented contract.
- The defn fallback of `Emily.Fast.scaled_dot_product_attention/4` (used
on non-Emily backends) applied an inverted causal mask, letting each
query attend to later keys instead of earlier ones.
- The defn fallback of `Emily.Fast.rope/3` and `rope_with_freqs/4` (used
on non-Emily backends) added the offset after applying `:scale`; it now
scales the whole position, as MLX does.
- Emily now compiles without warnings on Elixir 1.20. Its type checker
flagged an unreachable clause in `mix emily.doctor` (#195).
- `Emily.Fast.einsum/2` no longer requires every operand to be on
`Emily.Backend`. Operands on other backends run the composed
fallback, or are moved to Emily when mixed with Emily tensors.
- Updated the pinned MLX to 0.32.3. This is a maintenance bump that picks
up upstream correctness fixes (NaN propagation in arg reductions,
quantized matmul with a quantized dim that is not a multiple of 32,
scans and sorts over zero-size axes, quantizing sliced arrays) and
faster GQA decode attention, all transparently. No API changes.
- Building MLX from source (a git checkout of Emily) now needs Xcode 16
or later. MLX 0.32.1 started using a Metal standard library
type that Xcode 15 does not ship. The precompiled NIFs are unaffected
and still support macOS 14.
First stable release. Emily runs `Nx` computations on Apple Silicon through MLX — set it as the default Nx backend and Bumblebee models run on the Metal GPU with no further integration. With 1.0 the public API is stable and follows semantic versioning from here; there are no breaking changes from 0.7.x, so existing code keeps working. The headline this cycle is quantized inference — 4-bit layers now stream through the fused `mx::quantized_matmul` kernel, making native quantized generation ~13× faster end-to-end and, for the first time, faster than dense (details below). - The precompiled NIF now declares an explicit minimum macOS per variant — macOS 14 for the AOT build, macOS 26.2 for the JIT build — instead of inheriting whateve 8001 r macOS the release runner happened to be on. Published artifacts now have a deterministic macOS compatibility floor, and CI asserts it on every build. - Updated the pinned MLX to 0.32.0. This is a maintenance bump that also picks up faster small-batch quantized matvec (`qmv_wide`) — accelerating the fused quantized path — and broader fused SDPA coverage (asymmetric Q/V head dims), both transparently. No API changes. - Quantized dense layers now use the fused `mx::quantized_matmul` kernel instead of dequantizing the full weight to bf16 and running a dense matmul. The packed low-bit weights are streamed directly, so a decode step no longer re-dequantizes the entire model on every token. On a 4-bit Qwen3-0.6B this makes native quantized generation roughly 13× faster end-to-end — and quantized inference is now *faster* than dense, as it should be, rather than slower.
- The README performance section now compares Emily against both benchmark baselines — EXLA (host CPU) and EMLX (the older MLX-backed Nx backend on the Metal GPU) — instead of EXLA alone, and its rule-of-thumb figures (ViT-base, DistilBERT) are reconciled with the current benchmark report. - The benchmark report's environment block now records the Emily version the numbers were produced on (0.7.0) and drops a misleading run timestamp. - The `MAINTAINING.md` release runbook is corrected: `mix publisho` is no longer described as pushing (it only commits and tags), and the obsolete manual draft-promotion step is dropped — `release-nif.yml` now publishes the release automatically once the NIFs are built.
PreviousNext