• PEFT: QLoRA support, LoRA/QLora for Mixture-of-Experts (MoE) dense layer
• State Space Models & Hybrid Architecture support (Mamba2 and NV-Mamba2-hybrid)
• Support Nemotron, Minitron, Gemma2, Qwen, RAG
• Custom Tokenizer training in NeMo
• Update the Auto-Configurator for EP, CP and FSDP

• NeVA: Add SOTA LLM backbone support (Mixtral/LLaMA3) and suite of model parallelism support (PP/EP)
• Support Language Instructed Temporal-Localization Assistant (LITA) on top of video NeVA

• SpeechLM and SALM
• Adapters for Canary Customization
• Pytorch allocator in PyTorch 2.2 improves training speed up to 30% for all ASR models
• Cuda Graphs for Transducer Inference
• Replaced webdataset with Lhotse - gives up to 2x speedup
• Transcription Improvements - Speedup and QoL Changes
• ASR Prompt Formatter for multimodal Canary

• In framework PyTriton deployment with backends: - PyTorch - vLLM - TRT-LLM update to 0.10
• TRT-LLM C++ runtime

• Megatron Core RETRO

  • Pre-training
  • Zero-shot Evaluation

• Pretraining, conversion, evaluation, SFT, and PEFT for:

  • Mixtral 8X22B
  • Llama 3
  • SpaceGemma

• Embedding Models Fine Tuning

  • Mistral
  • BERT

• BERT models

  • Context Parallel
  • Distributed checkpoint

• Video capabilities with NeVa

• Distributed Checkpointing

  • Torch native backend
  • Parallel read/write
  • Async write

• Multimodal LLM (LLAVA/NeVA)

  • Pipeline Parallelism support
  • Sequence packing support

• Integration of Export & Deploy Modules into NeMo Framework container
  • Upgrade to TRT-LLM 0.9

• AED Multi Task Models (Canary) - Multi-Task Multi-Lingual Speech Recognition / Speech Translation model
• Multimodal Domain - Speech LLM supporting SALM Model
• Parakeet-tdt_ctc-1.1b Model - RTFx of > 1500 (can transcribe 1500 seconds of audio in 1 second)
• Audio Codec 16kHz Small - NeMo Neural Audio Codec for discretizing speech for use in LLMs
  • mel_codec_22khz_medium
  • mel_codec_44khz_medium

• Transcribe() upgrade - Enables one line transcribe with files, tensors, data loaders
• Frame looping algorithm for RNNT faster decoding - Improves Real Time Factor (RTF) by 2-3x
• Cuda Graphs + Label-Looping algorithm for RNN-T and TDT Decoding - Transducer Greedy decoding at over 1500x RTFx, on par with CTC Non-Autoregressive models
• Semi Sorted Batching support - External User contribution that speeds up training by 15-30%.

• Context biasing for CTC word stamping - Improve accuracy for custom vocabulary and pronunciation
  • Longform Inference
  • Longform inference support for AED models
• Transcription of multi-channel audio for AED models

• Upgraded webdataset - Speech and LLM / Multimodal unified container

• Announcement - https://developer.nvidia.com/blog/unlock-your-llm-coding-potential-with-starcoder2/
• AI Foundation Model Inference - https://catalog.ngc.nvidia.com/orgs/nvidia/teams/ai-foundation/models/starcoder2-15b
• https://huggingface.co/bigcode/starcoder2-15b

Announcement - https://nvidia.github.io/NeMo/blogs/2024/2024-02-canary/

• https://huggingface.co/nvidia/canary-1b

• Falcon
• Code Llama
• StarCoder
• GPT perf improvements
• Context parallelism
• Mistral
• Mixtral (without expert parallelism)
• Mcore GPT Dataset integration

• CLIP
• Stable Diffusion (supporting LoRA)
• Imagen
• ControlNet (for SD)
• Instruct pix2pix (for SD)
• LLAVA
• NeVA
• DreamFusion++
• NSFW filtering

• Lhotse Dataloading support #7880
• Canary: Multi task multi lingual ASR #8242
• LongForm Audio for Diarization #7737
• Faster algorithm for RNN-T Greedy #7926
• Cache-Aware streaming notebook #8296

Previously, the RNNT metric was stateful while the CTC one was not (r1.22.0, r1.23.0)

Therefore this calculation in the RNNT joint for fused operation worked properly. However with the unification of metrics in r1.23.0, a bug was introduced where only the last sub-batch of metrics calculates the scores and does not accumulate. This is patched via NVIDIA#8587 and will be fixed in the next release.

Workaround: Explicitly disable fused batch size during inference using the following command

from omegaconf import open_dict
model = ...
decoding_cfg = model.cfg.decoding
with open_dict(decoding_cfg):
  decoding_cfg.fused_batch_size = -1
model.change_decoding_strategy(decoding_cfg)

Note: This bug does not affect scores calculated via model.transcribe() (since it does not calculate metrics during inference, just text), or using the transcribe_speech.py or speech_to_text_eval.py in examples/asr.

For additional information regarding NeMo containers, please visit: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo

docker pull nvcr.io/nvidia/nemo:24.01.speech