Langfuse

Evals are the most important part of building AI systems, but there’s not a lot out there about building them well. So I hosted a fireside chat on evals with Liam Bush (LangChain), Lotte Verheyden (Langfuse), Braden Holstege (Mercor), Emmanuel Turlay (CoreWeave), and Soumya Mohan (Galileo) to walk me through the tricks of the trade - how they actually work in production, where teams get them wrong, and where they're headed over the next 12 months. We get into: -online vs offline evals -why running evals can cost you more than running the actual agent -what nobody can see after the agent makes a tool call -why understanding accounting rules might be harder than AGI -and much, much, more… Some of my favorite parts: 1/ Running your evals can cost more than running your agent. "We deployed an agent where we had a 100% sample rate initially, and it 4x’d the budget, because we're spending 3x as much tokens running the live evals as we're actually running the agent." Mercor’s team caught this on one of their own deployments. If you're running live evals on every user input, you're paying for a second agent whose entire job is watching the first one - that gets expensive, really fast. Over time, they reduce the sampling rate to reduce the cost, especially as the agent improves. 2/ “Ship WITHOUT evals.” "I can't believe I have to say that, because I would say the opposite to people a few years ago. But for an agent that is not safety critical or not regulated, just launch and let it fail and learn from that, and build your evals from there." You can guess how users are going to use your product all you want, but you won’t actually know until you launch and observe them using it. Broad online evals catch the stuff you'd never have tested for, and you convert that into offline evals afterward. He did add one caveat - don't let it fail for too long or your users will churn. 3/ Never ask an LLM to rate anything 1 to 10. "It's tempting to ask the LLM judge to score on a scale of zero to ten. This is really bad. You should probably not do that at all." Just like its hard for humans to know the difference between a 5 and a 6, its hard for models too. Classifiers are cheaper and more accurate. Example: if you're grading politeness on call transcripts, your options should be insufficiently polite, sufficiently polite, and extremely polite: "I don't want my agent to be extremely polite. That's actually a bad case. I don't want overly wordy responses, or the user will be frustrated if something is not working and the LLM is like, 'you're absolutely right' - because we all hate that response." A 1-10 scale gives that response a 10. Categories force you to decide what you actually want, which is the harder and more useful exercise. Link to full episode in the comments. — Follow me Basil Chatha for everything AI Agents. And thanks to AngelList for making this possible!

Max Deichmann , Co-founder and CTO of Langfuse, and Joel Foo from GovTech Singapore will be speaking at our Langfuse User Meetup this Friday, in collaboration with AWS. Join us by registering at the link below! Rana Banerji Seyan Thurairajah Surya Srinivasan Shirley H. Derek Chia Maruthi Prithivirajan Lokananthan SiShuo Y. Derk van Ogtrop https://lnkd.in/dD-W__BY

Over the last 5 days X went completely wild over TypeSafe AI's new AI model called Jev. They claim it to be 20 to 200x faster and 40 to 400x cheaper than frontier models like Fable 5.1 and GPT 5.6. Everyone is super excited, even though the model can not write a single sentence. Instead, it is really good at picking a category, taking a true/false decision, or assigning a score. This makes it very valuable for AI pipeline steps that answer questions like: - What is the topic of this email? - What research category is this paper from? - Does the user demand re-routing to a human? - Does this output fulfill my quality criterion X? Jev is an example for a model trained through Reinforcement Learning for Calibrated Decisions (RLCD). It is not a secret sauce that will solve AI cost and latency, but is part of an emerging category of models optimized for specific tasks, compromising on other dimensions to optimize cost and latency. Check out our write-up, link in comments 👇