Ankur Singhania

🔧 How RLHF Is Built (Step-by-Step) 1️⃣ Pretrained Model Start with a large language model trained on massive data. Good at patterns — not yet aligned with human preference. 2️⃣ Supervised Fine-Tuning (SFT) Humans create example responses the model should learn to imitate. 3️⃣ Human Preference Data For each prompt, generate several responses. Humans rank them from best to worst. 4️⃣ Train a Reward Model This model learns to predict which answer humans prefer. 5️⃣ Reinforcement Learning (e.g., PPO) The main model is optimized to maximize the reward score while staying close to the original knowledge. The result is behavior that aligns with human expectations. 🌍 Real-Life Company Example: OpenAI’s ChatGPT -OpenAI uses RLHF to train ChatGPT so that its responses are more helpful, safe, and aligned with human preferences — not just statistically likely text. RLHF is the reason ChatGPT feels conversational, context-aware, and user-friendly rather than purely predictive. This isn’t just theory — it’s what makes modern assistants like ChatGPT useful in products across customer support, education, and enterprise workflows. ❤️ Why RLHF Matters Without RLHF: -Models are smart but can be tone-deaf. -They may produce harmful or unhelpful responses. With RLHF: -Responses feel natural and safe. -Behavior reflects human expectations. -User experience improves dramatically. RLHF doesn’t make models smarter — it makes them aligned. And in real-world AI products, alignment wins. #AI #MachineLearning #LLM #RLHF #Product #AIAlignment

4

Ever wondered how we fine-tune billion-parameter models using just consumer-grade hardware? When it comes to fine tuning of the transformers, as the parameters scaled, updating weights has become humongous task. In fact, techniques to fine tune efficiently were developed parallelly as models kept scaling. LoRA - Low Rank Adaptation proposed in 2021 paved the way to easier fine tuning without having to update all the weights / parameters. Imagine, a 200 * 100 weight matrix needs to be updated. The earlier methods focussed on changing all parameters or added extra adapters which needs to be trained - both never solved the latency issues. But LoRA - a smart technique keeps the original matrix intact. I chooses r (number of dimensions) - let's say 2 in this case to come up with A - 200 * 2 matrix, B - 2 * 100 matrix. We just keep updating A & B, once updated multiply A & B to get a 200* 100 matrix which is added to original weight matrix. So trainable parameters here are 200*2 + 2 * 100 = 600 The simple hack here is create a weight update matrix of original matrix size from two low ranked submatrices. This is based on earlier idea / observation that large models can still learn efficiently despite a random projection to a smaller subspace. This effectively reduced parameters update from 20,000 to 600 - 97% drop in parameters to tune. The larger the model, the more redundant its parameter space tends to be — which makes low-rank adaptation surprisingly powerful.

11

4 Comments

Data Scientists obsess over model accuracy. AI Engineers obsess over milliseconds and GPU memory. Because once an LLM hits production, the problem changes. Now you care about: → Latency → Throughput → KV cache → Batching → GPU utilization → Routing → Cost per request This is Inference Engineering. And one of the first things to understand is that LLM inference has 2 very different workloads: Prefill: process the prompt and build the KV cache. Decode: generate tokens one at a time. Prefill is often compute-heavy. Decode can become memory-bandwidth heavy. That single distinction explains a lot of modern LLM infrastructure: → FlashAttention reduces memory movement → PagedAttention improves KV-cache management → Continuous batching keeps GPUs busy → Prefix caching avoids repeated work → GQA/MQA reduce KV memory You don't need to become a CUDA expert. But if you're moving from Data Scientist → AI Engineer, you need to understand what happens after: prompt → model → response Because the model determines what gets generated. Inference engineering determines: How fast. How cheaply. And at what scale. That’s a very different skill set than traditional Data Science. 🚨 Want to learn how to build + ship AI and Data Science projects (that businesses actually want in 2026)? On September 23rd, I am hosting a free workshop to help you get started with AI + DS projects in Python (free). Register here (500 seats): https://lnkd.in/gGKsiqKi 

2

1 Comment

Use Neo4j’s query language Cypher to manage and retrieve data stored in knowledge graphs. Write knowledge graph queries that find and format text data to provide more relevant context to LLMs for Retrieval Augmented Generation. Build a question-answering system using Neo4j and LangChain to chat with a knowledge graph of structured text documents.

4

Quantization in Large Language Models (LLMs) As LLMs scale, intelligence is no longer the bottleneck — memory, latency, and cost are. Quantization turns out to be one of the most powerful (and misunderstood) techniques enabling real-world deployment of large models. In this blog, I break down: • Why LLMs are surprisingly quantization-friendly • INT8 vs INT4 vs NF4 — what actually works in practice • How modern architectures absorb low-bit noise • Where quantization fails and how to design around it Quantization is no longer an optimization step — it’s foundational infrastructure for production LLM systems. Link: https://lnkd.in/gqGje75G #LLMs #AIEngineering #Quantization #MachineLearning #GenAI #MLOps

9

2 Comments

𝐏𝐫𝐨𝐛𝐥𝐞𝐦 : Models trained in PyTorch / TensorFlow are too slow, heavy, and generic to run efficiently in production. 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧 : AI inference runtimes / deployment toolchains take trained models and make them fast, small, and hardware-efficient. BITS IIT Bombay

2

Kharagpur Data Analytics Group, IIT Kharagpur is partnering with NYU Tandon and Pathway to bring you a sharp conversation on what comes after Transformers, straight from the people shaping the next wave of AI. Especially for you if you’re: → 2nd/3rd year exploring research internships → Final year planning MS/PhD applications abroad This is part of a larger push to connect Indian AI talent with top US research labs, and we made sure IIT KGP has a seat at the table. Speakers: → Martín Farach-Colton (Chair CSE, NYU Tandon) → Julian Togelius (Professor NYU, Author Artificial General Intelligence) → Adrian Kosowski (CSO, Pathway | PhD at 20, 100+ papers) → Zuzanna Stamirowska (CEO, Pathway, Moderator) Date: Feb 6, 2026 | 9:30 PM IST Register: https://lnkd.in/dQMwHfCi Limited registrations with exclusive merch for select participants!

93

One interesting thing about learning #machinelearning with #R is that there is no single “best” resource. Different resources work for different stages of learning. After sharing an introductory resource earlier, I thought it might also help to share something more hands-on. Hands-On Machine Learning with R: https://lnkd.in/d6zjNWzH This guide focuses on implementing machine learning techniques in R through practical examples, covering areas such as: • Tree-based models • Boosting methods • Clustering techniques • Neural networks • Ensemble models For #statisticians, #biostatisticians, and #dataanalysts working in R, it can be a helpful way to build practical machine learning skills step by step. I’d love to hear from my network: What resources would you recommend for learning machine learning with R? #Biostatistics #MachineLearning #RStats #DataScience #OpenScience #StatisticalLearning

40

4 Comments

Deepseek Moment for India When the DeepSeek models were launched, the global AI community took notice. They quickly emerged as a strong contender alongside models from OpenAI and Anthropic, showing that new players could challenge the status quo in foundation models. Now, it feels like India is having a similar moment. The Sarvam team has introduced Indic open LLMs - Sarvam 30B and Sarvam 105B, marking an exciting milestone for the Indian AI ecosystem. Building powerful open models tailored for Indic languages is a huge step toward making AI more inclusive and locally relevant. Kudos to the team for pushing the boundaries and contributing to the open AI movement. They have also launched an arena platform where users can test and compare models, which is a great way for the community to explore the capabilities of these models firsthand. Really excited to experiment with these models and see how they can be applied in real-world use cases. To Try: https://lnkd.in/gbYBpsbW To Learn More about Sarvam Models: https://lnkd.in/gBHdsDWc #India #AI #LLMs #GenAI

11