8000 GitHub - seyhunak/awesome-ai: A curated list of Enterprise AI, LLMs, GenAI, AI Agents, MCP, governance, security, MLOps, and production-ready AI tools. · GitHub
[go: up one dir, main page]

Skip to content

About

A curated list of Enterprise AI, LLMs, GenAI, AI Agents, MCP, governance, security, MLOps, and production-ready AI tools.

Resources

Code of conduct

Contributing

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

🧠 Awesome AI Awesome

A curated list of Enterprise AI, LLMs, GenAI, AI Agents, MCP, governance, security, MLOps, and production-ready AI tools.

RAG, vector databases, orchestration frameworks, and enterprise AI resources.

GitHub stars GitHub forks GitHub watchers GitHub contributors

Link Check License: MIT PRs Welcome Last commit Commit activity Issues Pull requests Repo size Code of Conduct

GitHub followers Website Made with Markdown


📖 About

This list is opinionated and production-first. Entries are selected for teams that have to ship, operate, secure, and govern AI systems — not for demos. Where a category has an obvious default, it is marked with ⭐.

Legend: ⭐ widely adopted default · 🔓 open source · 💰 commercial / paid tier · ☁️ managed service · 🆓 free


📑 Table of Contents

Build Measure, Secure & Operate
🤖 Foundation Models (LLMs, VLMs, SLMs) 📊 Benchmarks
💬 Prompt Engineering 📐 Metrics
🧠 RAG 🛡️ AI Security & Guardrails
🔍 Embeddings & Vector Databases 🏢 Enterprise AI
🤝 AI Agents ☁️ Cloud AI (Azure, AWS, GCP)
🔌 MCP (Model Context Protocol) 📦 MLOps & LLMOps
🎓 Agent Skills 🚀 Deployment
🛠️ Agent Frameworks 📚 Courses & Learning
🏗️ AI & ML Frameworks 🎥 Videos & Talks
⚡ AI SDKs 📰 Newsletters & Blogs
🧰 Developer Tooling 🌍 Open Source Projects
🧪 Evaluation & Observability 💼 Real-world Case Studies

Contributing · Author · License


🤖 Foundation Models (LLMs, VLMs, SLMs)

Frontier & Proprietary Models

Model Family Provider Highlights Access
Claude ⭐ Anthropic Claude 5 family (Fable, Opus, Sonnet) + Haiku — strong reasoning, coding, long context, agentic tool use API · Console
GPT OpenAI Broad general-purpose family with reasoning variants API
Gemini Google DeepMind Natively multimodal, very long context, tight GCP integration API
Grok xAI Real-time-leaning general models API
Command Cohere Enterprise/RAG-oriented, strong multilingual + citations API
Nova Amazon Cost-tiered family, native to Bedrock Bedrock
Mistral Large Mistral AI EU-based provider, open + commercial mix API

Open-Weight Models

Model Family Org Notes License
Llama ⭐ Meta The de-facto open baseline; huge fine-tune/tooling ecosystem Llama Community
Qwen ⭐ Alibaba Dense + MoE, very strong multilingual & coding, many sizes Apache-2.0 (most)
DeepSeek DeepSeek MoE + reasoning models with strong price/performance MIT (most)
Mistral / Mixtral Mistral AI Efficient dense and MoE models Apache-2.0
Gemma Google Lightweight open siblings of Gemini Gemma Terms
Phi Microsoft Small models trained on curated "textbook" data MIT
GLM Z.ai (Zhipu) Strong bilingual + agentic/coding variants MIT / Apache-2.0
Kimi Moonshot AI Very large MoE, long-context and agentic focus Modified MIT
Granite IBM Enterprise-governed, indemnified, watsonx-native Apache-2.0
OLMo Ai2 Fully open: weights, data, code, logs — best for research Apache-2.0
Falcon TII Multilingual open family Apache-2.0
SmolLM Hugging Face Genuinely small (135M–1.7B), on-device oriented Apache-2.0

Vision-Language & Multimodal (VLMs)

Project Description
Qwen-VL Leading open VLM family — documents, charts, video, GUI grounding
InternVL Open multimodal models competitive with proprietary VLMs
Pixtral Mistral's open multimodal model
Molmo Ai2 open VLM with pointing/grounding capability
Whisper Robust open speech-to-text across many languages
CLIP / OpenCLIP Foundational image-text embedding models

Model Access, Routing & Local Runtimes

Tool Description
Hugging Face Hub ⭐ 🔓 The model registry of record — weights, datasets, Spaces, leaderboards
OpenRouter ☁️ One API across hundreds of models with routing, fallbacks and price transparency
Ollama ⭐ 🔓 Dead-simple local model runner with a familiar CLI/API
LM Studio 🆓 Desktop GUI for running and serving local models
llama.cpp 🔓 The C/C++ inference engine underpinning much of local AI (GGUF)
MLX 🔓 Apple-silicon array framework for fast on-device training/inference
Together AI · Fireworks · Groq · Cerebras ☁️ High-throughput / low-latency hosted open-model inference
Replicate ☁️ Run and fine-tune open models via API, pay-per-second
Artificial Analysis 🆓 Independent benchmarks of quality, speed and price across providers
LMArena 🆓 Crowd-sourced pairwise model comparison leaderboard

⬆ back to top


💬 Prompt Engineering

Guides & References

Resource Description
Anthropic Prompt Engineering Guide ⭐ The most practical vendor guide — clarity, examples, XML structure, thinking, chaining
Anthropic Prompt Library Ready-made, well-structured prompts across dozens of tasks
OpenAI Prompt Engineering Vendor guidance and reasoning-model prompting patterns
Google Prompting Guidance Gemini-specific strategies and multimodal prompting
Prompt Engineering Guide 🆓 Comprehensive, paper-backed technique catalogue
Learn Prompting 🆓 Structured course from basics to adversarial prompting
Anthropic Prompt Engineering Interactive Tutorial Hands-on notebook course, chapter by chapter
Anthropic Cookbook ⭐ Runnable recipes: tool use, RAG, vision, classification, evals

Core Techniques

Technique What it does Reference
Few-shot prompting Demonstrations steer format and behavior more reliably than instructions alone Paper
Chain-of-Thought Ask for intermediate reasoning before the answer Paper
Self-consistency Sample multiple reasoning paths, take the majority answer Paper
ReAct Interleave reasoning with tool calls — the basis of most agents Paper
Reflexion / self-critique Let the model review and revise its own output Paper
Tree of Thoughts Explore and score branching reasoning paths Paper
Prompt caching Cache long, stable prefixes to cut cost and latency dramatically Docs
Structured output Constrain generation to a schema instead of parsing prose Guide

Tooling

Tool Description
DSPy ⭐ 🔓 Program — don't prompt. Declarative modules with automatic prompt/weight optimization
promptfoo ⭐ 🔓 Test, compare and red-team prompts with assertions in CI
Instructor 🔓 Pydantic-typed structured outputs with validation and retries
BAML 🔓 A dedicated language for prompts as typed functions, with tests
Outlines 🔓 Guaranteed-valid structured generation (JSON, regex, grammars)
Guidance 🔓 Constrained generation with interleaved control flow
TextGrad 🔓 Backpropagate natural-language "gradients" to optimize prompts
Priompt 🔓 Priority-based JSX prompt composition for context budgeting

⬆ back to top


🧠 RAG

Rule of thumb: retrieval quality dominates generation quality. Invest in parsing, chunking and reranking before swapping models.

Frameworks

Tool Description
LlamaIndex ⭐ 🔓 The most complete data framework for RAG — connectors, indexes, query engines, workflows
LangChain ⭐ 🔓 Ubiquitous building blocks and integrations across the entire stack
Haystack 🔓 Production-minded, explicitly-wired pipelines from deepset
RAGFlow 🔓 Deep document understanding RAG engine with a full UI
R2R 🔓 RAG-as-a-service with ingestion, graphs, auth and observability built in
txtai 🔓 All-in-one embeddings database for semantic search and RAG
Cognita 🔓 Modular, config-driven RAG framework for production teams
Verba 🔓 Open-source RAG application built on Weaviate
Dify ⭐ 🔓 Visual LLM app platform with RAG pipeline, agents and observability

Document Parsing & Ingestion

Tool Description
Docling ⭐ 🔓 IBM's high-fidelity document converter — layout, tables, formulas → structured output
Unstructured 🔓 Preprocess 25+ file types into clean, chunk-ready elements
MinerU 🔓 High-quality PDF → Markdown/JSON extraction, strong on scientific docs
Marker 🔓 Fast, accurate PDF-to-Markdown with equation and table support
LlamaParse 💰 ☁️ Managed parser tuned for complex/tabular documents
Firecrawl ⭐ 🔓 Crawl and scrape sites into clean LLM-ready markdown
Crawl4AI 🔓 Async, LLM-friendly web crawler and scraper
Jina Reader 🆓 Prefix any URL with r.jina.ai to get markdown
Chonkie 🔓 Fast, no-nonsense chunking library with many strategies

Retrieval & Reranking

Tool Description
Cohere Rerank ⭐ ☁️ The default cross-encoder reranker — often the single biggest quality win
BGE Reranker 🔓 Strong open reranker family from BAAI
Jina Reranker ☁️ Multilingual and long-context reranking models
RAGatouille 🔓 Late-interaction (ColBERT) retrieval made usable
rerankers 🔓 One tiny API over every reranking method
rank_bm25 🔓 Classic lexical retrieval — still essential in hybrid search

Advanced RAG Patterns

Pattern Why it matters Reference
Hybrid search (BM25 + dense) Lexical recall covers what embeddings miss (IDs, names, rare terms) Guide
Contextual Retrieval Prepend chunk-specific context before embedding; large recall gains Anthropic
GraphRAG Build a knowledge graph for global, corpus-wide questions Microsoft
LightRAG Simpler, cheaper graph-augmented retrieval HKUDS
RAPTOR Recursive clustering + summarization into a retrieval tree Stanford
Self-RAG Model decides when to retrieve and critiques what it retrieved Paper
Corrective RAG Grade retrieved docs, fall back to web search when weak Paper
Neo4j GraphRAG Official graph + vector retrieval package Neo4j

RAG Evaluation

Tool Description
Ragas ⭐ 🔓 The standard RAG metrics suite — faithfulness, relevancy, context precision/recall
DeepEval 🔓 Pytest-style LLM evaluation with RAG-specific metrics
TruLens 🔓 Feedback functions and the "RAG triad" for tracking quality
FlashRAG 🔓 Research toolkit to reproduce and compare RAG methods

⬆ back to top


🔍 Embeddings & Vector Databases

Embedding Models

Model Provider Notes
Voyage AI ⭐ ☁️ Voyage Anthropic-recommended; strong general + domain-specific (code, finance, legal) models
OpenAI Embeddings ☁️ OpenAI Solid general-purpose default with adjustable dimensions
Cohere Embed ☁️ Cohere Excellent multilingual and compressed (int8/binary) embeddings
BGE ⭐ 🔓 BAAI The leading open embedding family, many sizes and languages
Jina Embeddings 🔓 ☁️ Jina AI Long-context and multimodal open embeddings
Nomic Embed 🔓 Nomic Fully open (weights + data + code) long-context embeddings
E5 / multilingual-e5 🔓 Microsoft Strong, well-studied open baselines
Sentence-Transformers ⭐ 🔓 UKP Lab The library for using and fine-tuning embedding models
Model2Vec 🔓 Minish Lab Distill encoders into static embeddings — orders of magnitude faster
MTEB Leaderboard 🆓 HF Compare embedding models before you commit

Vector Databases

Database Model Best for License
pgvector ⭐ 🔓 Postgres extension Teams already on Postgres — start here before adding infrastructure PostgreSQL
Qdrant ⭐ 🔓 ☁️ Standalone / cloud Rich filtering, quantization, excellent performance-per-euro Apache-2.0
Weaviate 🔓 ☁️ Standalone / cloud Built-in hybrid search, modules and multi-tenancy BSD-3
Milvus 🔓 ☁️ Distributed Billion-scale workloads and GPU indexing Apache-2.0
Chroma 🔓 Embedded / server Prototyping and local-first apps Apache-2.0
LanceDB 🔓 ☁️ Embedded / serverless Multimodal, object-storage-native, zero-copy versioning Apache-2.0
Pinecone 💰 ☁️ Managed Fully managed serverless vector search with minimal ops Proprietary
Turbopuffer 💰 ☁️ Managed Object-storage-backed search at very low cost per vector Proprietary
Vespa 🔓 ☁️ Distributed Serious hybrid ranking and real-time recommendation at scale Apache-2.0
Elasticsearch / OpenSearch Search engine Adding vectors to an existing search deployment Elastic v2 / Apache-2.0
Redis Vector 🔓 ☁️ In-memory Lowest-latency lookups, semantic caching RSALv2/SSPL
MongoDB Atlas Vector Search 💰 ☁️ Document DB Vectors next to operational documents Proprietary
Supabase Vector 🔓 ☁️ Postgres platform pgvector with auth, storage and edge functions attached Apache-2.0
ClickHouse 🔓 ☁️ OLAP Analytics + vector filtering over huge tables Apache-2.0

ANN Libraries (bring your own store)

Library Description
FAISS ⭐ 🔓 Meta's similarity-search library — the reference implementation for ANN
hnswlib 🔓 Fast, minimal header-only HNSW
USearch 🔓 Compact, multi-language engine with many metrics and quantization
ScaNN 🔓 Google's high-recall, high-throughput ANN
Annoy 🔓 Memory-mapped, read-only indexes from Spotify
DiskANN 🔓 SSD-resident indexes for datasets far larger than RAM

⬆ back to top


🤝 AI Agents

Coding Agents

Agent Description
Claude Code ⭐ 💰 Anthropic's agentic coding tool — terminal, IDE, desktop and web, with hooks, subagents, skills and MCP
OpenAI Codex 💰 OpenAI's coding agent CLI and cloud environment
Gemini CLI 🔓 Google's open-source terminal agent
Cursor 💰 AI-native editor with deep codebase context and background agents
Cline 🔓 Autonomous coding agent inside VS Code, bring-your-own-model
Aider 🔓 Pair programming in the terminal, git-commit-native
OpenHands 🔓 Full agent platform that codes, browses and runs commands in a sandbox
Goose 🔓 Block's extensible on-machine agent, MCP-first
Continue 🔓 Build and run custom autocomplete/chat/agent flows in your IDE
SWE-agent 🔓 The research agent that popularized agent-computer interfaces
Plandex 🔓 Terminal agent designed for large, multi-file, long-running tasks
Devin 💰 Cognition's autonomous software engineer

Browser & Computer-Use Agents

Tool Description
browser-use ⭐ 🔓 Let agents drive a real browser — the most widely used option
Stagehand 🔓 Playwright plus act/extract/observe primitives; deterministic where it matters
Playwright MCP 🔓 Accessibility-tree browser control exposed over MCP
Skyvern 🔓 LLM + vision workflow automation for browser tasks
Computer Use (Claude) 💰 Screen-, mouse- and keyboard-level control reference implementation
E2B 🔓 ☁️ Secure cloud sandboxes for running agent-generated code
Daytona 🔓 ☁️ Fast, isolated runtimes for agent workloads

Autonomous & Multi-Agent Systems

Project Description
MetaGPT 🔓 Multi-agent software company simulation with SOP-driven roles
AutoGPT 🔓 The original autonomous agent, now a workflow platform
Open Interpreter 🔓 Natural-language interface to run code locally
Letta 🔓 Stateful agents with long-term memory (formerly MemGPT)
CAMEL 🔓 Research framework for multi-agent communication at scale
Suna 🔓 Open-source generalist assistant with browser, files and shell

Agent Benchmarks

See Benchmarks → Agents & Tool Use for SWE-bench, Terminal-Bench, τ-bench, BFCL, GAIA, OSWorld and WebArena, and Metrics → Agent Metrics for what to measure on your own traffic.

⬆ back to top


🔌 MCP (Model Context Protocol)

MCP is the open standard for connecting AI applications to tools, data and prompts — "USB-C for AI apps". It is now the default integration layer across Anthropic, OpenAI, Google and Microsoft tooling.

Specification & Official Resources

Resource Description
modelcontextprotocol.io ⭐ Official documentation, concepts and tutorials
Specification The normative protocol spec, versioned by date
Reference Servers ⭐ Official server implementations and a large community index
MCP Registry Official community registry of available servers
MCP Inspector ⭐ Interactive developer tool for testing and debugging servers
Anthropic MCP docs Connecting MCP servers to Claude apps and the API

SDKs

Language SDK
Python python-sdk ⭐
TypeScript typescript-sdk ⭐
Java java-sdk
Kotlin kotlin-sdk
C# / .NET csharp-sdk
Go go-sdk
Rust rust-sdk
Swift swift-sdk
Ruby ruby-sdk

Frameworks & Infrastructure

Tool Description
FastMCP ⭐ 🔓 The fast, Pythonic way to build MCP servers and clients
mcp-use 🔓 Connect any LLM to any MCP server in a few lines
mcp-agent 🔓 Build agents using MCP with composable workflow patterns
Cloudflare Remote MCP ☁️ Deploy authenticated remote MCP servers on Workers
Docker MCP Toolkit 🔓 Containerized, signed MCP servers with a managed gateway
MCP Gateway (IBM) 🔓 Federate, secure and observe many MCP servers behind one endpoint

Directories & Ecosystem

Directory Description
Smithery Large registry with one-click install and hosted deployment
Glama MCP Indexed directory with security and quality scoring
PulseMCP Servers, clients and a weekly ecosystem newsletter
mcpservers.org Curated, human-reviewed server listing
Awesome MCP Servers The community's own curated list

Enterprise Considerations

Topic Guidance
Authorization Remote servers should implement the OAuth 2.1 authorization spec — never ship static shared secrets
Transport Prefer Streamable HTTP for remote servers; stdio for local-only tools
Tool poisoning Treat third-party tool descriptions as untrusted input — they enter the model's context
Least privilege Scope each server to one system with the narrowest credentials that work
Auditability Log every tool invocation with arguments, identity and outcome
Vetting Pin versions and review source before installing community servers

⬆ back to top


🎓 Agent Skills

If MCP gives an agent tools and data, Agent Skills give it procedural knowledge — how this team does code review, ships a release, or formats a report. A skill is a folder with a SKILL.md file; agents load the name and description at startup and pull in the full instructions only when a task matches.

This section deliberately favors skills written and used by the teams that maintain the product they describe, over bulk-generated skill dumps. A skill encoding a real team's hard-won conventions is worth a hundred generated from a framework's README.

The Standard

Resource Description
agentskills.io ⭐ Home of the open standard — overview, quickstart and client showcase
Specification ⭐ The normative SKILL.md format: frontmatter, structure, progressive disclosure
agentskills/agentskills Where the standard is developed in the open
anthropics/skills ⭐ 🔓 Anthropic's reference implementation and production-grade example skills
Equipping agents for the real world ⭐ The design rationale, and how Anthropic uses skills in its own products
Claude Code skills docs Authoring, installing and scoping skills in practice
Claude platform docs Skills across the API, Agent SDK and Claude apps

Skills Shipped by the Teams That Own the Domain

The strongest signal that a skill is real: the people who maintain the framework, platform or product also maintain its skills, and ship them in the repo.

Skills Maintained by What they encode
Laravel Boost ⭐ 🔓 Laravel core team Laravel conventions and best practices, so agents write idiomatic Laravel rather than generic PHP
Anthropic document skills ⭐ Anthropic The pdf, docx, pptx and xlsx skills that power Claude's real document editing
Spring AI 🔓 Spring team Bringing skills to Spring AI agents in the Java ecosystem
Google AI Edge Gallery 🔓 Google AI Edge Skills shipped alongside an on-device LLM application
Goose 🔓 Block Context engineering with skills in an open agent runtime
Pulumi Neo Pulumi Infrastructure workflows constrained by org policy and approvals
Databricks Genie Code Databricks Data engineering and analytics procedures inside the lakehouse
Snowflake Cortex Code Snowflake Extending an in-platform data agent with team procedures
Qodo Qodo A written-up account of using skills to auto-fix PR issues
Letta 🔓 Letta Skills combined with persistent agent memory
OpenHands 🔓 OpenHands Skills for cloud coding agents at scale
Firebender Firebender Android-specific build, emulator and test workflows

Methodology Collections

Skills that encode a whole way of working, not a single task.

Collection Author What it encodes
Superpowers ⭐ 🔓 Jesse Vincent A full development methodology — TDD red/green/refactor, systematic debugging, brainstorming before building, verification before completion, subagent-driven development, worktree management
Netresearch marketplace 🔓 Netresearch An agency's working skills for TYPO3, PHP, Go, Docker, Jira, security and docs
skill-creator ⭐ Anthropic The skill for writing skills — start here before authoring your own
mcp-builder Anthropic Generating well-formed MCP servers, itself packaged as a skill

Directories & Curated Lists

Directory Notes
heilcheng/awesome-agent-skills ⭐ Explicitly curates skills used by real engineering teams over bulk-generated ones
VoltAgent/awesome-agent-skills Large community-maintained collection across many agents
ComposioHQ/awesome-claude-skills Broad productivity-oriented catalogue
travisvn/awesome-claude-skills Curated skills, resources and authoring tools
agent-skills topic The raw firehose — useful for discovery, apply your own judgment

Vetting a skill before you install it: a skill is instructions your agent will follow and scripts it may execute. Read SKILL.md in full, check what scripts/ actually does, prefer skills maintained by the owners of the domain, and pin a commit rather than tracking a moving branch.

Client Support

The format is portable; the same SKILL.md works across a growing set of agents.

Client Skills documentation
Claude Code ⭐ Terminal, IDE, desktop and web
OpenAI Codex Codex CLI and cloud
GitHub Copilot Across Copilot agents
VS Code Editor-native customization
Cursor Skills as codebase context
Gemini CLI 🔓 Google's open terminal agent
Goose 🔓 Block's extensible agent
OpenCode 🔓 Terminal, IDE and desktop
Amp Sourcegraph's coding agent
Roo Code 🔓 Multi-mode agent team in your editor
Factory Droids across IDE and CI/CD
Kiro Spec-driven development
Junie JetBrains IDE agent
Tabnine Enterprise-controlled deployments

Writing Skills That Actually Get Used

Practice Why
The description is the trigger ⭐ It is all the agent sees until activation. Write when to use it, not what it is — vague descriptions never fire
Keep SKILL.md short It loads into context wholesale. Push detail into references/ and link to it
Use progressive disclosure ⭐ Name/description at startup, instructions on activation, referenced files on demand. Design for all three stages
Prefer scripts over prose Deterministic steps belong in scripts/, not in instructions the model may paraphrase
Encode judgment, not just steps The valuable part is why your team does it this way and when to deviate
Version-control with the code A skill describing your repo's conventions belongs in your repo, reviewed like any other change
Test it like code Run real tasks against it. A skill that never activates is worse than none — it creates false confidence
One skill, one job Composable skills beat a monolith; the agent loads only what the task needs

⬆ back to top


🛠️ Agent Frameworks

General-Purpose Orchestration

Framework Language Distinctive strength
LangGraph ⭐ 🔓 Python, JS Graph-based, durable, stateful agents with human-in-the-loop and time travel
CrewAI ⭐ 🔓 Python Role-playing crews and deterministic flows; standalone of LangChain
AutoGen 🔓 Python, .NET Event-driven multi-agent conversations from Microsoft Research
AG2 🔓 Python Community continuation of the original AutoGen line
Microsoft Agent Framework 🔓 Python, .NET Convergence of AutoGen + Semantic Kernel for production agents
Semantic Kernel 🔓 .NET, Python, Java Enterprise .NET-first orchestration with planners and plugins
PydanticAI ⭐ 🔓 Python Type-safe agents that feel like FastAPI; excellent validation and testing
Mastra ⭐ 🔓 TypeScript Batteries-included TS framework — agents, workflows, RAG, evals, memory
OpenAI Agents SDK 🔓 Python, JS Minimal primitives: agents, handoffs, guardrails, sessions, tracing
Claude Agent SDK ⭐ 🔓 Python, TS Build agents on the same harness that powers Claude Code
Google ADK 🔓 Python, Java Code-first agent development, Vertex AI Agent Engine deployment
Strands Agents 🔓 Python AWS's model-driven SDK, integrates with Bedrock AgentCore
Agno 🔓 Python High-performance runtime with built-in memory, knowledge and UI
smolagents 🔓 Python Minimal agents that think in code; ~1k lines of core logic
LlamaIndex Workflows 🔓 Python, TS Event-driven, async-first orchestration tightly coupled to retrieval
Atomic Agents 🔓 Python Small, composable, strictly-typed agent building blocks
Cloudflare Agents SDK 🔓 TypeScript Durable, stateful agents on Workers with WebSockets and scheduling

Visual & Low-Code Builders

Tool Description
Dify ⭐ 🔓 Full LLMOps platform: visual workflows, RAG, agents, observability
n8n ⭐ 🔓 Workflow automation with native AI/agent nodes and 400+ integrations
Flowise 🔓 Drag-and-drop LLM app builder
Langflow 🔓 Visual builder for agents and RAG flows, deployable as an API
Rivet 🔓 Visual graph IDE for debugging complex prompt chains

Durable Execution (the unglamorous half of production agents)

Tool Description
Temporal ⭐ 🔓 ☁️ Durable workflow engine — the reference answer for long-running, retryable agents
Inngest 🔓 ☁️ Event-driven durable functions with steps, retries and concurrency control
Restate 🔓 ☁️ Durable execution with low-latency journaling
Cloudflare Workflows ☁️ Durable multi-step execution at the edge
Prefect 🔓 ☁️ Pythonic orchestration that adapts well to AI pipelines

⬆ back to top


🏗️ AI & ML Frameworks

Agent frameworks orchestrate models. These frameworks are the layer models are built, trained and optimized in.

Deep Learning Frameworks

Framework Org Notes
PyTorch ⭐ 🔓 PyTorch Foundation The framework nearly all modern AI research and open models are written in
JAX 🔓 Google Composable function transforms + XLA; dominant for large-scale TPU training
TensorFlow 🔓 Google Mature production ecosystem; still widespread in established deployments
Keras 🔓 Keras team High-level API that now runs on JAX, PyTorch or TensorFlow
Flax NNX 🔓 Google The neural network library most JAX research builds on
MLX 🔓 Apple Array framework designed for Apple silicon unified memory
tinygrad 🔓 tiny corp Radically minimal framework — excellent for understanding the stack
Candle 🔓 Hugging Face Minimalist Rust ML framework for serverless and embedded inference
Burn 🔓 Tracel AI Rust deep learning framework with pluggable backends
ONNX 🔓 LF AI Open interchange format for moving models between frameworks

Model & Training Libraries

Library Description
Transformers ⭐ 🔓 The model-definition library of record for text, vision, audio and multimodal
Diffusers ⭐ 🔓 State-of-the-art diffusion models for image, audio and video generation
Sentence-Transformers 🔓 Train and serve embedding and reranking models
Accelerate 🔓 Run the same training script on any distributed configuration
Lightning ⭐ 🔓 Remove training boilerplate without giving up PyTorch control
Composer 🔓 MosaicML's speed-focused training library with algorithmic optimizations
timm 🔓 The reference collection of vision backbones and training recipes

Distributed & Large-Scale Training

Framework Description
DeepSpeed ⭐ 🔓 ZeRO sharding, offload and pipeline parallelism for very large models
Megatron-LM ⭐ 🔓 NVIDIA's tensor/pipeline/sequence parallelism reference implementation
PyTorch FSDP 🔓 Native fully-sharded data parallel training
torchtitan 🔓 PyTorch-native reference for pretraining at scale with 4D parallelism
Ray Train 🔓 Distributed training orchestration across heterogeneous clusters
Colossal-AI 🔓 Unified parallelism toolkit for large model training
Nanotron 🔓 Minimal, readable 3D-parallel pretraining library
MaxText 🔓 High-performance JAX/TPU LLM training reference
Levanter 🔓 Legible, scalable, bitwise-reproducible JAX training

Reinforcement Learning & Post-Training

Library Description
TRL ⭐ 🔓 SFT, DPO, GRPO and PPO trainers that integrate with Transformers
verl ⭐ 🔓 Production-grade RL library for LLM post-training (HybridFlow)
OpenRLHF 🔓 Scalable RLHF built on Ray, vLLM and DeepSpeed
TorchRL 🔓 PyTorch-native RL primitives and environments
Gymnasium ⭐ 🔓 The standard RL environment API (maintained fork of OpenAI Gym)
PettingZoo 🔓 Gymnasium for multi-agent environments
Stable-Baselines3 🔓 Reliable, well-tested implementations of standard RL algorithms
CleanRL 🔓 Single-file RL implementations built for readability
Ray RLlib 🔓 Scalable RL for production workloads

Classical ML & Structured Data

Library Description
scikit-learn ⭐ 🔓 The baseline that still solves most tabular problems — try it before an LLM
XGBoost ⭐ 🔓 Gradient boosting that remains state of the art on tabular data
LightGBM 🔓 Fast, memory-efficient gradient boosting
CatBoost 🔓 Gradient boosting with excellent categorical feature handling
statsmodels 🔓 Statistical models and hypothesis testing with proper inference
PyTorch Geometric 🔓 Graph neural networks on PyTorch
DGL 🔓 Deep graph library with multi-backend support
Prophet 🔓 Interpretable forecasting for business time series
Darts 🔓 Unified time series forecasting and anomaly detection

⬆ back to top


⚡ AI SDKs

Provider SDKs

Provider SDKs
Anthropic Python ⭐ · TypeScript · Java · Go · Ruby · PHP · C#
OpenAI Python · Node · Go · Java
Google google-genai Python · JS
Mistral Python · TS
Cohere Python · TS

Cross-Provider & Application SDKs

SDK Description
Vercel AI SDK ⭐ 🔓 The TypeScript standard for AI apps — unified provider API, streaming, tools, generative UI
LiteLLM ⭐ 🔓 Call 100+ LLMs with the OpenAI format; proxy adds keys, budgets, routing and logs
aisuite 🔓 Thin, uniform interface across providers for easy comparison
Instructor 🔓 Structured, validated outputs across providers
Mirascope 🔓 LLM calls as typed Python functions
Portkey Gateway 🔓 ☁️ Fast AI gateway with routing, fallbacks, caching and guardrails
Chainlit 🔓 Build and ship conversational AI UIs in pure Python
Gradio ⭐ 🔓 Turn any model into a shareable web demo in a few lines
Streamlit 🔓 Data and AI apps in Python, no frontend required
assistant-ui 🔓 Composable React primitives for chat and agent UIs
CopilotKit 🔓 Drop in-app copilots and agentic UX into React apps

⬆ back to top


🧰 Developer Tooling

Tokenization & Context Budgeting

Tool Description
tiktoken ⭐ 🔓 Fast BPE tokenizer — the standard way to count tokens before you send them
Tokenizers 🔓 Rust-backed tokenization used across the open ecosystem
Token counting API 🆓 Count tokens exactly as the provider will, including tools and images
tokencost 🔓 Estimate USD cost of prompts and completions across providers
ttok 🔓 Count and truncate text to a token budget from the shell

Playgrounds & Prompt IDEs

Tool Description
Anthropic Console ⭐ Workbench with prompt generator, improver, test cases and evaluations
Google AI Studio 🆓 Fast Gemini prototyping with multimodal input and code export
OpenAI Playground Model and parameter experimentation with function calling
Langfuse Prompt Management 🔓 Version, label and deploy prompts without a code release
Latitude 🔓 Open-source prompt engineering platform with evals and versioning
Agenta 🔓 Prompt playground, versioning and evaluation for teams
PromptLayer 💰 Prompt registry and observability aimed at non-engineer collaborators

Data Labeling & Curation

Tool Description
Argilla ⭐ 🔓 Collaboration tool for AI engineers who care about data quality
Label Studio ⭐ 🔓 Multi-type labeling for text, image, audio, video and time series
doccano 🔓 Lightweight open-source text annotation
Cleanlab 🔓 Automatically find label errors and data issues in your datasets
Lilac 🔓 Explore, cluster and clean unstructured datasets before training
Nomic Atlas 🔓 Interactively visualize and inspect millions of embeddings

Synthetic Data Generation

Tool Description
distilabel ⭐ 🔓 Synthetic data and AI feedback pipelines for reliable dataset creation
Curator 🔓 Bespoke Labs' library for post-training and structured synthetic data
DataDreamer 🔓 Reproducible synthetic data and prompting workflows for research
NeMo Curator 🔓 GPU-accelerat 10BC8 ed curation, deduplication and filtering at scale
SDV 🔓 Synthetic tabular and relational data with privacy-preserving evaluation

Quantization & Model Optimization

Tool Description
bitsandbytes ⭐ 🔓 8-bit and 4-bit quantization primitives used throughout fine-tuning
GGUF ⭐ The dominant quantization format for local inference (produced by llama.cpp)
llm-compressor 🔓 Produce quantized, sparse checkpoints ready to serve on vLLM
AutoAWQ 🔓 Activation-aware weight quantization for 4-bit inference
GPTQModel 🔓 Maintained GPTQ quantization toolkit
torchao 🔓 Native PyTorch quantization and sparsity for training and inference
Optimum 🔓 Export and optimize Transformers for ONNX, OpenVINO and accelerators
ExLlamaV2 🔓 Fast quantized inference on consumer GPUs

Benchmarking & Load Testing

Tool Description
GuideLLM ⭐ 🔓 Evaluate real-world latency, throughput and capacity of an LLM deployment
vLLM benchmarks 🔓 Reference throughput and serving benchmark scripts
LLMPerf 🔓 Load-test and compare LLM API providers on latency and consistency
k6 🔓 General-purpose load testing that adapts well to streaming endpoints
nvitop 🔓 Interactive GPU process monitoring — the htop of NVIDIA devices

Terminal & Agent Utilities

Tool Description
llm ⭐ 🔓 Simon Willison's CLI for prompting, logging and templating across models
files-to-prompt 🔓 Concatenate a directory into a single, well-structured prompt
Repomix ⭐ 🔓 Pack an entire repository into one AI-friendly file
gitingest 🔓 Turn any Git repo into a digestible text digest via URL
aichat 🔓 All-in-one CLI with shell assistant, RAG and function calling
Context7 🔓 Up-to-date library documentation served to coding agents over MCP
strip-tags 🔓 Strip HTML down to the text an LLM actually needs

⬆ back to top


🧪 Evaluation & Observability

If you ship without evals, you don't have a product — you have a demo. Instrument first, then iterate.

Tracing & Observability Platforms

Platform Description
Langfuse ⭐ 🔓 ☁️ Open-source LLM engineering platform — tracing, evals, prompt management, self-hostable
LangSmith 💰 ☁️ Deep tracing, datasets and evaluation; best-in-class for LangChain/LangGraph
Arize Phoenix 🔓 Open-source, OpenTelemetry-native tracing and evaluation notebook-first
Braintrust 💰 ☁️ Eval-centric workflow with playgrounds and CI integration
W&B Weave 🔓 ☁️ Tracing and evaluation inside the Weights & Biases ecosystem
Opik 🔓 ☁️ Comet's open-source tracing, evaluation and guardrails
Helicone 🔓 ☁️ One-line proxy for logging, caching, rate limiting and cost tracking
OpenLLMetry 🔓 OpenTelemetry extensions for LLM apps — vendor-neutral by design
OpenTelemetry GenAI 🆓 The emerging standard semantic conventions for AI telemetry
Datadog LLM Observability 💰 ☁️ LLM traces alongside your existing APM and infrastructure monitoring

Evaluation Frameworks

Tool Description
promptfoo ⭐ 🔓 Declarative evals and red-teaming that run in CI
DeepEval ⭐ 🔓 Pytest-like unit testing for LLM outputs with 40+ metrics
Ragas 🔓 Purpose-built RAG and agent metrics
Inspect AI ⭐ 🔓 The UK AI Security Institute's rigorous evaluation framework
lm-evaluation-harness 🔓 The standard harness for academic benchmark reporting
HELM 🔓 Stanford's holistic, multi-metric model evaluation
OpenAI Evals 🔓 Framework and registry of benchmark evals
Giskard 🔓 Automated vulnerability and quality scanning for ML and LLM apps
Evidently 🔓 Data drift, quality and LLM evaluation with rich reports
MLflow LLM Evaluate 🔓 Evaluation integrated with experiment tracking and the model registry

⬆ back to top


📊 Benchmarks

Public benchmarks tell you which models to shortlist. They do not tell you whether your system works — only a golden dataset drawn from your own traffic does that. Treat leaderboards as a filter, never as an acceptance test.

Reasoning & Knowledge

Benchmark Measures
MMLU-Pro ⭐ Harder, less saturated successor to MMLU across 14 domains
GPQA ⭐ Graduate-level physics, chemistry and biology questions that resist search
Humanity's Last Exam Expert-written questions at the frontier of human knowledge
ARC-AGI ⭐ Abstract visual reasoning designed to resist memorization
BIG-bench Hard The subset of BIG-bench where models historically underperformed humans
MuSR Multi-step soft reasoning over natural language narratives
DROP Discrete reasoning over paragraphs — arithmetic, counting, sorting

Mathematics

Benchmark Measures
AIME ⭐ Competition math; the current default frontier reasoning check
MATH 12.5k competition problems with step-by-step solutions
GSM8K Grade-school word problems — largely saturated, still a smoke test
FrontierMath Unpublished research-level problems designed to stay unsaturated
MathArena Evaluation on competitions held after model training cutoffs

Code & Software Engineering

Benchmark Measures
SWE-bench ⭐ Resolving real GitHub issues in real repositories — the industry reference
SWE-bench Verified ⭐ Human-validated 500-problem subset; the number most vendors report
Terminal-Bench ⭐ End-to-end task completion in a real terminal environment
LiveCodeBench Contamination-free competitive programming, continuously refreshed
BigCodeBench Realistic tasks requiring compositional use of many libraries
HumanEval The classic function-synthesis benchmark — saturated, but ubiquitous
MBPP Entry-level Python programming problems
Aider Polyglot Editing existing code correctly across six languages
Commit0 Writing entire libraries from scratch against a test suite

Agents & Tool Use

Benchmark Measures
τ-bench / τ²-bench ⭐ Tool use plus policy adherence in realistic customer dialogues
BFCL ⭐ Berkeley Function Calling Leaderboard — the standard tool-calling test
GAIA General assistant tasks needing tools, browsing and multi-step reasoning
OSWorld Open-ended computer-use tasks in a real operating system
WebArena Autonomous agents on realistic self-hosted websites
AgentBench Agent capability across eight distinct environments
MLE-bench Agents competing on real Kaggle machine learning tasks

Long Context & Retrieval

Benchmark Measures
RULER ⭐ What a model's effective context length actually is, beyond the spec sheet
LongBench Bilingual, multi-task long-context understanding
Needle in a Haystack Retrieval of a single fact at varying depth and context length
BEIR ⭐ Zero-shot retrieval generalization across 18 datasets
MTEB ⭐ Massive text embedding benchmark across tasks and languages
LoCoMo Very long-term conversational memory

Multimodal

Benchmark Measures
MMMU ⭐ College-level multimodal understanding across 30 subjects
MathVista Mathematical reasoning in visual contexts
DocVQA Question answering over document images — core to enterprise RAG
ChartQA Reasoning over charts and plots
Video-MME Comprehensive video understanding evaluation
ScreenSpot GUI element grounding — the basis of computer-use agents

Instruction Following, Safety & Truthfulness

Benchmark Measures
IFEval ⭐ Verifiable instruction following ("write exactly 3 bullets in JSON")
SimpleQA ⭐ Short-form factuality and, crucially, calibrated abstention
TruthfulQA Resistance to reproducing common human misconceptions
HaluEval Hallucination recognition across QA, dialogue and summarization
HarmBench Standardized red-teaming and refusal robustness
AgentHarm Harmfulness of agentic behavior, not just generated text
AIR-Bench 2024 Safety evaluated against 314 risk categories drawn from real regulations

Multilingual

Benchmark Measures
Global-MMLU Culturally-aware MMLU across 42 languages
MGSM Grade-school math reasoning in 10+ languages
FLORES-200 Machine translation across 200 languages
XTREME Cross-lingual transfer across 40 languages and 9 tasks
Belebele Reading comprehension parallel across 122 language variants

Live Leaderboards

Leaderboard Focus
LMArena ⭐ Human preference via blind pairwise voting
Artificial Analysis ⭐ Quality vs. price vs. latency across providers
LiveBench Contamination-resistant, monthly-refreshed questions
Open LLM Leaderboard Standardized open-model benchmarks
SWE-bench Leaderboard Real-world software engineering task resolution
MTEB Leaderboard Embedding model quality across tasks and languages
BFCL Leaderboard Function/tool calling accuracy
Epoch AI Benchmarking Hub Independent tracking of capability trends over time

Reading Benchmark Results Honestly

Pitfall What to do
Contamination Test sets leak into training data. Prefer continuously-refreshed benchmarks (LiveBench, LiveCodeBench, MathArena)
Saturation A benchmark above ~90% has stopped discriminating. Move up a tier
Prompt sensitivity Scores swing several points on formatting alone. Compare only same-harness numbers
Self-reported numbers Vendors pick favorable configurations. Prefer independent runs
Pass@k inflation pass@10 is not pass@1. Check which is quoted
Averaging across subsets A strong average can hide a domain that matters to you. Read the breakdown
Cost and latency omitted A 2-point gain for 10× the cost is usually a bad trade

⬆ back to top


📐 Metrics

Pick metrics before you build. A metric you added after seeing results is a story, not a measurement.

Retrieval Metrics

Metric Definition Use when
Recall@k Fraction of all relevant documents that appear in the top k The ceiling on your RAG quality — measure this first
Precision@k Fraction of the top k that are relevant Context window is tight and noise hurts
MRR Mean of 1/rank of the first relevant result Exactly one right answer exists (lookup, navigation)
nDCG@k Gain discounted by log rank, normalized to the ideal ordering Graded relevance and ranking order both matter
MAP Mean of average precision across queries Multiple relevant documents, order-sensitive
Hit Rate@k Share of queries with ≥1 relevant result in top k A blunt but readable executive-facing number

In practice: optimize retrieval for Recall@k with a generous k, then let a reranker fix precision. Improving the generator cannot recover a document retrieval never returned.

RAG Metrics

Metric Question it answers Typical source
Faithfulness / Groundedness ⭐ Is every claim supported by retrieved context? Ragas
Answer Relevancy Does the answer actually address the question asked? Ragas, DeepEval
Context Precision Are the retrieved chunks relevant and well-ranked? Ragas
Context Recall Was all information needed for the ground truth retrieved? Ragas
Noise Sensitivity Does irrelevant retrieved context degrade the answer? Ragas
Citation Accuracy Do the cited sources genuinely support the claims? Custom / LLM judge
Refusal Correctness ⭐ Does it decline when the context truly lacks the answer? Custom

Generation Quality Metrics

Metric Type Notes
LLM-as-Judge ⭐ Model-graded The workhorse for open-ended output. Requires its own validation against human labels
Pairwise preference ⭐ Model/human More reliable than absolute scoring — comparison beats calibration
BERTScore Embedding Semantic similarity to a reference; better than n-gram overlap
ROUGE Lexical Summarization overlap. Weak signal, but cheap and stable
BLEU / chrF Lexical Translation. Still standard in MT, poor for open generation
Exact Match / F1 Lexical Extractive QA and structured field extraction
Perplexity Probabilistic Model fit during training — not a product quality metric
Schema validity ⭐ Deterministic Percentage of outputs that parse and validate. Cheap and unambiguous

On LLM judges: validate the judge before you trust it. Measure judge-vs-human agreement (Cohen's κ) on a labeled sample, control position and verbosity bias, and never judge with the same model and prompt that produced the output.

Classification & Extraction Metrics

Metric Formula Use when
Precision TP / (TP + FP) False positives are expensive (spam filters, auto-actions)
Recall TP / (TP + FN) Misses are expensive (safety, compliance, fraud)
F1 2PR / (P + R) You need one balanced number
Fβ Weighted harmonic mean Recall matters β× more than precision
PR-AUC ⭐ Area under precision-recall Imbalanced classes — preferred over ROC-AUC here
Cohen's κ Agreement above chance Comparing model to human, or judge to human
ECE Expected calibration error Confidence scores drive downstream routing or escalation

Agent Metrics

Metric What it captures
Task Success Rate ⭐ Did the agent achieve the goal, verified by an outcome check — not self-report
Tool Selection Accuracy Right tool chosen for the step
Tool Call Validity Arguments parse and satisfy the schema
Steps to Completion Efficiency; sudden growth signals looping or confusion
Cost per Task ⭐ Total tokens across every turn and subagent — the number finance asks about
Loop / Stall Rate Runs hitting the step ceiling without terminating
Recovery Rate Runs that hit an error and still finished successfully
Human Intervention Rate ⭐ How often a person had to step in — the real autonomy measure
Trajectory Match Did it follow an acceptable path, not just reach an acceptable end

Serving & Production Metrics

Metric Definition Why it matters
TTFT ⭐ Time to first token Dominates perceived latency in streaming UIs
TPOT / ITL Time per output token (inter-token latency) Determines whether text renders faster than reading speed
E2E Latency Total request time The number that matters for non-streaming and batch
Throughput Output tokens/sec, requests/sec Capacity planning and cost per unit of work
Goodput ⭐ Throughput of requests that met their SLO Prevents "fast but useless" scaling wins
Queue Time Wait before scheduling Early warning of saturation
Cache Hit Rate ⭐ Prompt-cache hits / total Usually the biggest single cost lever
Cost per Request/User ⭐ Spend attributed by tenant and feature Unit economics; measure before scaling
Error / Refusal Rate 4xx, 5xx, overloads, refusals Split these — a refusal is a product signal, not an outage

Report percentiles, not averages. p50 hides the experience; p95 and p99 are what users complain about. Track TTFT and E2E separately — they fail for different reasons.

Metric Selection by Use Case

Use case Primary metric Guardrail metrics
RAG question answering Faithfulness Context Recall, refusal correctness, cost/query
Search & retrieval nDCG@10 Recall@50, p95 latency
Summarization LLM-judge quality Faithfulness, length adherence
Structured extraction Field-level F1 Schema validity rate, cost/document
Classification PR-AUC or F1 Calibration, per-class recall
Coding agent Task success rate Test pass rate, cost/task, human intervention
Customer support agent Resolution rate Escalation rate, policy violations, CSAT
Content generation Pairwise preference Safety violation rate, brand adherence

⬆ back to top


🛡️ AI Security & Guardrails

Standards & Threat Models

Resource Description
OWASP Top 10 for LLM Applications ⭐ The canonical risk list — prompt injection, data leakage, supply chain, excessive agency
OWASP Agentic AI Threats & Mitigations Threat taxonomy specific to autonomous agents
MITRE ATLAS ⭐ Adversarial tactics and techniques against AI systems, ATT&CK-style
NIST AI RMF ⭐ Govern / Map / Measure / Manage — the framework most enterprise programs anchor to
NIST Adversarial ML Taxonomy Standard vocabulary for attacks and mitigations
Google SAIF Secure AI Framework with a practical risk self-assessment
CSA AI Controls Matrix Control framework for securing AI in the cloud

Guardrails & Runtime Defense

Tool Description
Guardrails AI ⭐ 🔓 Input/output validators ("guards") with a shared hub of checks
NeMo Guardrails 🔓 Programmable rails for topic, safety and execution control
LLM Guard 🔓 Comprehensive scanner suite — injection, PII, toxicity, secrets
Llama Guard / Purple Llama 🔓 Meta's safety classifiers and cybersecurity evals for LLM I/O
Granite Guardian 🔓 IBM's risk detection models for prompts, responses and RAG groundedness
ShieldGemma 🔓 Google's open content-safety classifiers
Presidio ⭐ 🔓 PII detection, anonymization and redaction for text and images
Rebuff 🔓 Multi-layered prompt-injection detection with canary tokens
Bedrock Guardrails ☁️ Managed policy, PII and contextual-grounding filters
Azure AI Content Safety ☁️ Content filtering plus prompt-shield detection

Red Teaming & Testing

Tool Description
garak ⭐ 🔓 LLM vulnerability scanner — the nmap of prompt attacks
PyRIT ⭐ 🔓 Microsoft's Python Risk Identification Toolkit for generative AI
promptfoo red team 🔓 Automated adversarial test generation wired into CI
Adversarial Robustness Toolbox 🔓 Attacks and defenses across the full ML stack
Counterfit 🔓 CLI for automating AI security assessments
HarmBench 🔓 Standardized evaluation of red-teaming and refusal robustness

Agent Security Patterns

Pattern Practice
Untrusted content isolation Anything retrieved, scraped or returned by a tool is untrusted input — never let it grant new authority
Sandboxed execution Run generated code in E2B, Daytona, gVisor or Firecracker — never on the host
Human-in-the-loop gates Require approval for irreversible, outward-facing or high-blast-radius actions
Least-privilege credentials Scoped, short-lived tokens per tool; no ambient production credentials
Egress control Allowlist network destinations to contain exfiltration via tool calls
Dual LLM / CaMeL Separate the planner from the data-handling model so injected text can't change the plan (paper)
Full audit trail Log prompts, tool calls, arguments, identities and outcomes for replay

⬆ back to top


🏢 Enterprise AI

Governance & Compliance

Framework Scope
EU AI Act ⭐ Risk-tiered legal obligations for AI systems and GPAI models in the EU
ISO/IEC 42001 ⭐ Certifiable AI management system standard — the "ISO 27001 of AI"
ISO/IEC 23894 AI-specific risk management guidance
NIST AI RMF + GenAI Profile Voluntary framework with a generative-AI-specific companion
Model Cards Standard documentation for model intent, performance and limitations
Datasheets for Datasets Provenance and consent documentation for training data
MLCommons AILuminate Industry-standard AI safety benchmarking

Enterprise Platforms & Assistants

Platform Description
Claude for Enterprise ⭐ SSO/SCIM, audit logs, expanded context, data-retention controls; Claude Code seats
Microsoft 365 Copilot AI across Office, Graph-grounded, with Copilot Studio for custom agents
Google Gemini Enterprise Gemini across Workspace and Cloud with enterprise controls
ChatGPT Enterprise Managed ChatGPT with admin, compliance and connector controls
Glean Enterprise search and assistants over permission-aware company knowledge
Writer Full-stack enterprise generative AI with owned models and graph-based RAG
Databricks Mosaic AI Build, serve and govern models next to your lakehouse data
Snowflake Cortex LLM functions and agents executed inside the data platform
Palantir AIP Ontology-grounded AI with tight operational guardrails
Salesforce Agentforce Agents grounded in CRM data and business rules
ServiceNow AI Agents Agentic automation across ITSM and enterprise workflows

Adoption Checklist

Stage Key questions
Use case selection Is there a measurable baseline? What is the cost of being wrong? Is a human already reviewing this output?
Data readiness Where does ground truth live? Are permissions enforced at retrieval time, not just in the UI?
Build vs. buy Does this differentiate us? Total cost including evals, on-call and model upgrades?
Security review Threat-modelled against OWASP LLM Top 10 and MITRE ATLAS? Injection paths enumerated?
Evaluation Golden dataset committed to version control? Regression gate in CI before deploy?
Observability Traces, cost per request/user, and a feedback signal captured in production?
Governance Model inventory, owner, risk tier, documented human oversight, retention policy?
Change management Who is accountable when the model changes? Is there a rollback and a pinned version?

⬆ back to top


☁️ Cloud AI (Azure, AWS, GCP)

Microsoft Azure

Service Purpose
Azure AI Foundry ⭐ Unified platform to build, evaluate and deploy AI apps and agents
Azure OpenAI Service OpenAI models with Azure networking, identity and compliance
Azure AI Search Hybrid + vector retrieval with integrated ingestion skills
Azure AI Foundry Agent Service Managed agent runtime with tools, threads and tracing
Copilot Studio Low-code custom copilots and agents over Microsoft data
Azure Machine Learning Classic MLOps: training, registry, endpoints, pipelines

Amazon Web Services

Service Purpose
Amazon Bedrock ⭐ Serverless access to many model providers with guardrails and evaluations
Bedrock AgentCore Secure, managed runtime for agents at scale (memory, identity, tools, browser)
Bedrock Knowledge Bases Managed RAG over your data sources
Amazon SageMaker AI Train, tune and host models with full MLOps tooling
Amazon Q Business and developer assistants across AWS and enterprise data
Amazon OpenSearch Serverless Managed vector engine for RAG workloads

Google Cloud

Service Purpose
Vertex AI ⭐ End-to-end platform for models, tuning, evaluation and serving
Gemini API Direct developer access to Gemini models
Vertex AI Agent Builder / Engine Build, deploy and scale agents with ADK
Vertex AI Search Google-quality retrieval over enterprise data
Vertex AI Vector Search Billion-scale ANN service built on ScaNN
BigQuery ML Generate embeddings and call LLMs directly in SQL

Cross-Cloud & Alternatives

Provider Purpose
Cloudflare AI ⭐ Workers AI, AI Gateway, Vectorize and Agents SDK at the edge
NVIDIA NIM Containerized, optimized inference microservices for any environment
IBM watsonx Governed enterprise AI with indemnified Granite models
Oracle OCI Generative AI Managed GenAI with dedicated clusters
Modal · RunPod · Baseten Serverless GPU compute for custom model workloads

⬆ back to top


📦 MLOps & LLMOps

Experiment Tracking & Registries

Tool Description
MLflow ⭐ 🔓 Tracking, projects, models and registry — with first-class GenAI support
Weights & Biases 💰 ☁️ Experiment tracking, sweeps, artifacts and model management
ClearML 🔓 End-to-end MLOps: experiments, orchestration, data and serving
Neptune 💰 ☁️ Tracking built for very long, large-scale training runs
DVC 🔓 Git-native data and model versioning
LakeFS 🔓 Git-like branching and commits over object storage

Pipelines & Orchestration

Tool Description
ZenML ⭐ 🔓 Portable MLOps + LLMOps pipelines that run on any stack
Kubeflow 🔓 The Kubernetes-native ML toolkit
Metaflow 🔓 Netflix's human-centric framework for real-life ML/AI projects
Flyte 🔓 Strongly-typed, reproducible, Kubernetes-native workflows
Dagster 🔓 Asset-oriented orchestration with strong data-quality semantics
Airflow 🔓 The ubiquitous scheduler, now with AI/LLM provider packages
Ray ⭐ 🔓 Distributed compute underpinning much of modern training and serving
SkyPilot 🔓 Run jobs on the cheapest available GPUs across clouds
Feast 🔓 Open-source feature store for online/offline consistency

Fine-Tuning & Post-Training

Tool Description
Unsloth ⭐ 🔓 Dramatically faster, lower-memory fine-tuning for popular open models
LLaMA-Factory ⭐ 🔓 Unified fine-tuning for 100+ models with a web UI
Axolotl 🔓 Config-driven fine-tuning covering the full method matrix
TRL ⭐ 🔓 SFT, DPO, GRPO and PPO from Hugging Face
PEFT 🔓 LoRA/QLoRA and other parameter-efficient methods
torchtune 🔓 Native PyTorch library for LLM post-training
DeepSpeed 🔓 ZeRO optimization for training very large models
Liger Kernel 🔓 Triton kernels that cut training memory substantially
verl 🔓 Production RL library for LLM post-training

Cost & Gateway Management

Tool Description
LiteLLM Proxy ⭐ 🔓 Virtual keys, budgets, rate limits, fallbacks and spend tracking
Portkey 🔓 ☁️ AI gateway with caching, retries, guardrails and analytics
Cloudflare AI Gateway ☁️ Caching, rate limiting and observability in front of any provider
Kong AI Gateway 🔓 ☁️ Enterprise API gateway extended with AI-specific plugins
Envoy AI Gateway 🔓 CNCF-aligned gateway for LLM/AI traffic
Prompt caching Often the single largest cost lever — cache stable prefixes
Batch APIs ~50% discounts for asynchronous, non-latency-sensitive work

⬆ back to top


🚀 Deployment

High-Throughput Inference Servers

Engine Description
vLLM ⭐ 🔓 The default open serving engine — PagedAttention, continuous batching, OpenAI-compatible API
SGLang ⭐ 🔓 Fast serving with RadixAttention prefix caching; excellent for agentic/structured workloads
TensorRT-LLM 🔓 NVIDIA's maximum-performance compiled inference
Text Generation Inference 🔓 Hugging Face's production serving stack
LMDeploy 🔓 Efficient compression, deployment and serving toolkit
Triton Inference Server 🔓 Multi-framework, multi-model serving with dynamic batching
Ray Serve 🔓 Compose multi-model, multi-stage inference graphs
BentoML 🔓 Package and ship any model as a production service
KServe 🔓 Kubernetes-native, standards-based model serving

Edge, Local & On-Device

Tool Description
llama.cpp ⭐ 🔓 CPU/GPU inference anywhere, from laptops to Raspberry Pi
Ollama ⭐ 🔓 The friendliest way to run and serve local models
ONNX Runtime 🔓 Cross-platform accelerated inference including browsers
ExecuTorch 🔓 PyTorch on mobile and embedded devices
MLC LLM 🔓 Compile and deploy LLMs to almost any hardware backend
OpenVINO 🔓 Intel-optimized inference for CPU/iGPU/NPU
LocalAI 🔓 Drop-in OpenAI-compatible API you can self-host
Xinference 🔓 Serve LLMs, embeddings and rerankers with one command

Serverless & Managed Inference

Provider Description
Modal ⭐ ☁️ Python-native serverless GPUs with fast cold starts
Baseten ☁️ Dedicated deployments with autoscaling and observability
Together AI · Fireworks AI ☁️ Fast hosted open models plus fine-tuning
Groq · Cerebras · SambaNova ☁️ Custom silicon for very low-latency inference
Replicate ☁️ Pay-per-second hosting for open and custom models
HF Inference Endpoints ☁️ One-click dedicated endpoints from the Hub

Production Checklist

  • Version pinning — pin model IDs; never let a silent upgrade change behavior
  • Streaming — stream tokens; long non-streaming calls will hit gateway timeouts
  • Timeouts & retries — exponential backoff with jitter, and a hard request budget
  • Fallback chain — a secondary provider or smaller model for outages and overload
  • Prompt caching — cache stable system prompts and long shared context
  • Cost attribution — tag every request with tenant, feature and user
  • Rate limiting — per-user and per-tenant quotas before the provider enforces its own
  • Structured logging — request/response, latency, tokens, cache hits, tool calls
  • Eval gate in CI — block deploys that regress your golden dataset
  • Canary rollout — shift traffic gradually on any model or prompt change
  • Kill switch — disable a feature or agent without a redeploy
  • Feedback capture — thumbs, corrections and downstream outcomes into your eval set

⬆ back to top


📚 Courses & Learning

Course Provider Level
Anthropic Academy ⭐ 🆓 Anthropic Beginner → Advanced
Claude Code in Action 🆓 Anthropic Intermediate
Prompt Engineering Interactive Tutorial ⭐ 🆓 Anthropic Beginner
DeepLearning.AI Short Courses ⭐ 🆓 DeepLearning.AI All levels
Neural Networks: Zero to Hero ⭐ 🆓 Andrej Karpathy Intermediate
Hugging Face LLM Course 🆓 Hugging Face Beginner → Intermediate
Hugging Face Agents Course ⭐ 🆓 Hugging Face Intermediate
Hugging Face MCP Course 🆓 Hugging Face Intermediate
Generative AI for Beginners ⭐ 🆓 Microsoft Beginner
AI Agents for Beginners 🆓 Microsoft Beginner
MCP for Beginners 🆓 Microsoft Beginner
Stanford CS336: Language Modeling from Scratch 🆓 Stanford Advanced
Stanford CS224N: NLP with Deep Learning 🆓 Stanford Advanced
LangChain Academy 🆓 LangChain Intermediate
Google Cloud Generative AI Path 🆓 Google Beginner → Intermediate
Practical Deep Learning for Coders 🆓 fast.ai Beginner → Advanced
smol-course 🆓 Hugging Face Intermediate

Books

Book Author Focus
AI Engineering ⭐ Chip Huyen Building applications on foundation models
Designing Machine Learning Systems ⭐ Chip Huyen Production ML system design
Build a Large Language Model (From Scratch) Sebastian Raschka Implementing an LLM end to end
Hands-On Large Language Models Alammar & Grootendorst Visual, practical LLM engineering
Prompt Engineering for LLMs Berryman & Ziegler Prompt design as an engineering discipline
Designing Data-Intensive Applications Martin Kleppmann The systems foundation AI infrastructure rests on

⬆ back to top


🎥 Videos & Talks

Talk Speaker Why watch
Deep Dive into LLMs like ChatGPT ⭐ Andrej Karpathy The best single explanation of how modern LLMs actually work
Intro to Large Language Models ⭐ Andrej Karpathy The one-hour mental model everyone should share with their team
Let's build GPT: from scratch, in code ⭐ Andrej Karpathy Build a transformer line by line
Software Is Changing (Again) Andrej Karpathy "Software 3.0" and what it means for engineering teams
But what is a GPT? / Attention explained ⭐ 3Blue1Brown The clearest visual intuition for transformers and attention
Anthropic YouTube Anthropic Model launches, research explainers and Claude Code deep dives
AI Engineer Conference ⭐ AI Engineer The largest archive of practitioner talks on shipping AI
Latent Space swyx & Alessio Long-form interviews with the people building the stack
Stanford MLSys Seminar Stanford Systems-level view of ML infrastructure
Hugging Face YouTube Hugging Face Open-model tooling walkthroughs

⬆ back to top


📰 Newsletters & Blogs

Engineering & Research Blogs

Blog Author / Org Focus
Anthropic Engineering ⭐ Anthropic Agent design, context engineering, Claude Code internals
Anthropic Research ⭐ Anthropic Interpretability, alignment and safety research
Simon Willison's Weblog ⭐ Simon Willison The most reliable running commentary on what actually changed
Lil'Log ⭐ Lilian Weng Definitive deep dives on agents, hallucination and RL
Interconnects Nathan Lambert Post-training, RLHF and open-model policy
Ahead of AI Sebastian Raschka Careful technical explainers and paper reviews
Chip Huyen Chip Huyen AI engineering and production system design
Eugene Yan ⭐ Eugene Yan Patterns for building LLM systems that work
Hamel Husain ⭐ Hamel Husain Evals, fine-tuning and hard-won practitioner lessons
Google DeepMind Blog DeepMind Research and model announcements
Hugging Face Blog Hugging Face Open-model releases and practical tutorials
Answer.AI Jeremy Howard et al. Practical R&D from the fast.ai lineage
Netflix Tech Blog · Uber Engineering · Airbnb Engineering Various Real production ML/AI architecture writeups

Newsletters

Newsletter Cadence Focus
The Batch ⭐ Weekly Andrew Ng's balanced industry roundup
Import AI ⭐ Weekly Jack Clark on research, policy and capability trends
TLDR AI Daily Five-minute daily digest
Ben's Bites Daily Product- and startup-oriented AI news
Last Week in AI Weekly Summaries plus a podcast
AI Snake Oil Occasional Princeton researchers on separating hype from evidence
Latent Space Weekly AI engineering practice and interviews
PulseMCP Weekly MCP ecosystem news and new servers
Data Machina Weekly Dense technical link roundup

⬆ back to top


🌍 Open Source Projects

Core libraries — PyTorch, JAX, Transformers, scikit-learn and friends — live in AI & ML Frameworks. This section covers the applications and utilities built on top of them.

Chat UIs & Self-Hosted Apps

Project Description
Open WebUI ⭐ Feature-rich, offline-capable self-hosted AI interface
LibreChat ⭐ Multi-provider ChatGPT-style app with agents, RAG and auth
AnythingLLM All-in-one desktop/docker app for documents and agents
Jan Offline-first, open-source ChatGPT alternative
Lobe Chat Modern multi-modal chat framework with a plugin ecosystem
Khoj Self-hostable AI second brain over your own documents
Perplexica Open-source AI answer engine
SearXNG Privacy-respecting metasearch — the retrieval backbone for many OSS agents

Speech, Vision & Creative

Project Description
Whisper ⭐ Robust multilingual speech recognition
faster-whisper CTranslate2 reimplementation, several times faster
WhisperX Word-level timestamps and speaker diarization
Kokoro Small, high-quality open TTS
F5-TTS Fast, natural zero-shot voice cloning
Piper Fast local neural TTS for edge devices
ComfyUI ⭐ Node-based generative pipeline engine with a huge ecosystem
Ultralytics YOLO Real-time detection, segmentation and tracking
Segment Anything Promptable segmentation foundation model

Data & Utilities

Project Description
Polars ⭐ Blazing-fast DataFrames for AI data preparation
DuckDB ⭐ In-process analytical SQL — ideal for eval and dataset work
Pydantic ⭐ Validation layer underpinning most Python AI tooling
Datasets Load, stream and process datasets of any size efficiently
datatrove Large-scale text data processing pipelines for pretraining corpora
Apache Arrow The columnar memory format most AI data tooling interchanges through

⬆ back to top


💼 Real-world Case Studies

Curated Collections

Collection Description
ZenML LLMOps Database ⭐ 500+ documented production GenAI deployments, summarized and searchable
Anthropic Customer Stories How enterprises deploy Claude, with reported outcomes
Google Cloud: Real-world gen AI use cases Hundreds of named customer implementations
Microsoft AI Transformation Stories Cross-industry deployment writeups
AWS Generative AI Customer Stories Bedrock and SageMaker production references
Evidently AI LLM Case Studies Practitioner-focused breakdowns of shipped LLM apps

Notable Engineering Writeups

Company Use case Takeaway
LinkedIn ⭐ Generative AI product Honest account of the last 20% — evals, latency and quality plateaus
Uber QueryGPT: natural language → SQL Multi-agent decomposition beat a single large prompt
Uber Platform evolution Migrating an established ML platform to serve GenAI
Airbnb Conversational AI platform Constrain agents with workflows rather than trusting free-form autonomy
Airbnb Migrating 3.5k test files with LLMs Retry loops with expanded context beat prompt perfection
DoorDash Support automation RAG plus an LLM Judge to control hallucination in customer-facing replies
Discord Shipping GenAI features A pragmatic prototype → eval → productionize loop
GitHub Copilot Workspace Designing agentic developer UX around review and correction
Grab Data classification at scale LLMs for governance and metadata, not just chat
Klarna Customer service assistant Widely-cited deployment numbers — and a useful lesson in later rebalancing toward humans
Bloomberg BloombergGPT Domain-specific pretraining, and why most teams should not repeat it
Replit Agent self-testing at scale REPL-base 8CCE d verification instead of expensive computer-use checking
Anthropic ⭐ Multi-agent research system Orchestrator–worker patterns, and where multi-agent genuinely pays off
Anthropic ⭐ Building effective agents Start simple: most "agent" problems are solved by workflows

⬆ back to top


🤝 Contributing

Contributions are very welcome — this list is only as good as the people maintaining it.

Before opening a PR, please check that your entry:

  1. ✅ Is production-relevant — something a team would actually deploy, not a weekend demo
  2. ✅ Is actively maintained — recent commits, releases or documentation updates
  3. ✅ Is not already listed under another section
  4. ✅ Uses the canonical link — the project's repository or official documentation
  5. ✅ Follows the existing table format and has a one-line, non-marketing description
  6. ✅ Is placed in the most specific applicable section

How to contribute:

# 1. Fork and clone
git clone https://github.com/<your-username>/awesome-ai.git
cd awesome-ai

# 2. Create a branch
git checkout -b add-awesome-tool

# 3. Edit README.md, then commit
git commit -am "Add <Tool> to <Section>"

# 4. Push and open a pull request
git push origin add-awesome-tool

Found a dead link, a project that has been archived, or a description that is no longer accurate? Open an issue — corrections are as valuable as additions.

⬆ back to top


⭐ Star History

Star History Chart

If this list is useful to you, please consider starring it — it helps other engineers find it.


👤 Author

Seyhun Akyürek

Website GitHub X LinkedIn

GitHub followers GitHub stars


📄 License

CC0

Released under the MIT License.

Copyright © 2026 Seyhun Akyürek

All trademarks, product names and logos belong to their respective owners.
Listing here does not imply endorsement by, or affiliation with, any listed project or vendor.



⬆ back to top

Made with ❤️ for the AI engineering community

About

A curated list of Enterprise AI, LLMs, GenAI, AI Agents, MCP, governance, security, MLOps, and production-ready AI tools.

Resources

Code of conduct

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

0