Justin Muller

Judging the (LLM as ) Judges: LLM judges are widely used to evaluate AI outputs, yet their reliability is rarely tested systematically. Small validation checks against human raters provide limited assurance about how these systems perform under real-world variations. The open source library, Judge Reliability Harness (JRH), introduces automated reliability tests with human-in-the-loop review and standardized reporting to make LLM evaluation more robust, transparent, and reproducible. https://lnkd.in/guJR3PEf

6

2 Comments

😎 Super interesting — especially as we met a firm at ALPMA: Australasian Legal Practice Management Association who has rolled out MikeOSS + Claude 🛠️ to build: 📚 A research tool connected to NZ law (which is really what you need to reduce hallucinations — because as you know, AI is all about good inputs for good results) …a slew of other tools that lawyers dreamt up. So the firm is implementing AI chiefly to automate specific processes — and likely harnessing firm knowledge). 💸 AI is charged by token, as disbursements. So they’ve effectively bypassed all the tools which wrap an user friendly interface and legal workflows around an LLM — eliminating PMS + AI, or all the AI tools MikeOSS had the audacity to claim it could replace 😎 It’s the first time I’ve seen it play out in a firm. Plus, they’re of course navigating all the issues of change-resistant lawyers of a certain age — who are profitable doing things as they always have, even as the firm is innovating efficiencies all around them. Anyway, home time! But definitely a highlight of ALPMA this year. Final little plug for Logic Firm — if you can charge out AI use at $10/mo (approx.) to clients, 👉 and the AI is connected via MCP to Logic Firm (which only costs $199/mo) 👉…well, you’re not paying for your AI use (since your passing it on as a very reasonably priced disbursement) 👉 nor are you bearing the significant unknown risks posed by AI stuffed into the black box of someone else’s PMS. That’s it! Signing off. Happy weekend, LI world!

5

Over the last six months we have been building a completely different approach to AI at OneBit AI, one that runs fully offline on CPUs, enables real agentic workflows, and rethinks the math behind intelligence instead of just scaling models, and now seeing a team of four Caltech professors at PrismML emerge from stealth working on a strikingly similar direction is a strong signal that this is where the world is headed, except we have already been building and deploying it from the ground up with a working system and research backing it, which makes the timing of conviction incredibly important because by the time this becomes obvious the opportunity will already be gone, and for the VCs who understand deep tech cycles and want to be early rather than right in hindsight, we are still actively opening up conversations to fund and accelerate our research before it is too late. #deeptech #vc #investment #ai

22

We’re excited to announce the availability of Sora 2, OpenAI’s next-generation multimodal video generation model, now in Public Preview on Azure AI Foundry. Sora 2 lets creators, developers and businesses integrate generative AI video into apps and agentic workflows, transforming text or image prompts into lifelike, story-driven videos with synchronized audio, advanced creative controls, and robust safety features empowering end-users to turn ideas into dynamic content seamlessly. Why it matters Realistic Video Generation: Powered by advanced simulation and physics for lifelike results. Enhanced creative control: Detailed prompt understanding for studio shots, scene details and camera angles. Synchronized Audio in Major Languages: Immersive storytelling across most global languages. Enterprise-Grade Security & Compliance: Built on Azure’s trusted infrastructure with C2PA provenance, input/output moderation, and robust governance frameworks. Seamless Azure Integration: Unified access to Sora 2 and other generative models via intuitive APIs and the Playground for rapid experimentation and prototyping.

7

Jensen Huang just gave the clearest framework I've seen for understanding where AI actually lives. He calls it the Five-Layer Cake. 🍰 Energy → Chips → Infrastructure → Models → Applications Everyone's fighting over the top layer. The applications. The interfaces. The demos. But Jensen's point cuts deeper. The real bottleneck isn't the model. It's everything underneath it. China can build a hospital in a weekend. It takes the US 3 years to stand up a data center. That gap isn't a technology problem. It's an infrastructure problem. At gisax.io, this is exactly how we think about building AI systems for our clients. Anyone can wrap an API and call it an AI product. We're more interested in the data pipelines, the infrastructure decisions, and the architectural foundations that make those products actually work at scale six months after the demo. The frosting gets the attention. The layers underneath determine whether the whole thing holds. Which layer are you building in? (ai infrastructure artificial intelligence ai systems data infrastructure machine learning systems ai architecture scalable ai systems) Source : https://lnkd.in/gRdXR2Rz #Gisax #NVIDIA #AIInfrastructure #JensenHuang #FounderPerspective #AIStrategy #DeepTech #CustomSoftware

7

Building Production-Ready RAG Applications: Jerry Liu Large Language Models (LLM's) are starting to revolutionize how users can search for, interact with, and generate new content. Some recent stacks and toolkits around Retrieval Augmented Generation (RAG) have emerged where users are building applications such as chatbots using LLMs on their own private da... https://lnkd.in/gF8GB2ET

22

GPT-5.4 can now control your computer. Not APIs — actual mouse and keyboard. The implication nobody's grasping: we're moving from "AI helps me" to "AI acts for me." That changes the human role entirely. From orchestrator to conductor. #AI #AIAgents #FutureOfWork https://lnkd.in/gwYveVN5

5