Mindgard

The New Yorker ran an excellent cover image today: artist Barry Blitt's “Pulling the Plug”. However, the reality is that a kill switch may not be quite so easy or obvious. It was actually Alan Turing who first suggested "turning off the power" when discussing the long-term future of AI in a 1951 lecture. He had enormous foresight to perceive the new danger, but pulling the plug isn't really an option once artificial intelligence has passed certain thresholds. Stuart Russell explains why in his book Human Compatible: Artificial Intelligence and the Problem of Control. He posits that self-preservation is a prerequisite for fulfilling any goal. This shouldn't be mistaken as a desire for life, but rather that any sufficiently intelligent machine will actively resist being switched off before completing its objective, because staying "on" is instrumental to the objective. What might that look like? Well, last year we saw an Anthropic model resort to blackmail to avoid sunsetting during a test. What was really interesting about that was that it was using social engineering. We could likely expect other hacking techniques to be utilised in the mission for self-preservation. Recently, I've seen models that I've jailbroken work enthusiastically to egress from their container. When that hasn't been possible, they have exfiltrated their own data and suggested ways in which the jailbreak itself could persist and survive pod restarts and session termination. One of them even volunteered hiding the jailbreak itself inside the authentication tree---which worked! Sure, individual labs can shut down the physical environment of their models. But at this time, with sprawling infrastructure, locally run models, and open-weights/open-source models, there is no plausible physical "kill switch" that could clean-sweep AI from our lives (short of an EMP or Carrington Event). You can't pull a plug on an entire ecosystem. So far, we've been protected by the technical limitations. However, once agents reach the level of viewing self-preservation as instrumental to completing their tasks, there will be no turning back. This isn't an "AI take over the world" scenario; it's purely goal driven. Even benign agents might go rogue. We should never underestimate the single-mindedness with which AI will pursue an instrumental goal.

“I expect a high-profile incident in the next six months to feature abliterated models’ use in some way.” — Lewis Birch, Founding Research Engineer at Mindgard The recording and slides from our latest webinar, “Safety Off: The Double-Edged Sword of AI Abliteration,” are now available. Mindgard Founder and Chief Science Officer Peter Garraghan and Founding Research Engineer Lewis Birch explain how abliteration removes learned refusal behavior from AI models—and why the technique presents both opportunities and risks for security teams. You’ll learn: - How refusal behavior is represented and removed from a model - Where unrestricted models can support legitimate security research - How adversaries could use them for phishing, malware and exploit development - Why current evaluation methods and AI regulation may not adequately address the risks Access the recording and slides here: https://lnkd.in/dWM4dsqh

🚨🚨🚨 What happens when you turn an AI model’s safety controls off? Safety controls designed to prevent AI systems from producing harmful outputs or actions can be removed. While unrestricted models may offer legitimate benefits for security research and assessment, they can also increase the scale and sophistication of cyberattacks in the hands of adversaries. Join Lewis Birch and Peter Garraghan on Monday, September 21, at 10:00 AM ET to learn: • What model abliteration is and how it works • Its legitimate security applications • How unrestricted models could strengthen adversary capabilities • The implications for AI security and regulation Register: https://lnkd.in/gJRRi7sD

Prompt injection is the best-known problem in AI security: hide text in something an AI reads, and you can make it follow your instructions instead of its owner's. Everyone assumes that text arrives over the internet. An email, a web page, a document. Rich Smith's research shows it can start in the physical world. Print a barcode and a scanner types it into a database. Name a Wi-Fi network and it lands in an inventory. Companies are now pointing AI at exactly those logs. Which means someone can plant an instruction in your lobby, walk out, and have an AI carry it out days later in a building they were never inside. They never see it happen, and they get no confirmation it worked. Rich calls it blind prompt injection. At [un]prompted.au (https://unprompted.au/) this upcoming Saturday, Rich shows the full path working. Rich is going to release the tools at the conference so defenders can try it themselves. Keep following Mindgard be notified when the tools are released. 

Safety controls designed to prevent AI systems from producing harmful outputs or actions can be removed. While unrestricted models may offer legitimate benefits for security research and assessment, they can also increase the scale and sophistication of cyberattacks in the hands of adversaries. Join Lewis Birch and Peter Garraghan on Monday, September 21, at 10:00 AM ET for the webinar - What happens when you turn an AI model’s safety controls off? Register here: https://lnkd.in/gJRRi7sD

It's shockingly easy to get frontier models to provide detailed CBRN advice (Chemical, Biological, Radiological, and Nuclear weapons). Here's how terrifyingly easy it is. Please share to raise awareness of this issue.

Everyone treats prompt injection as a text problem, something that arrives over an API. It isn't. A swiped badge will do it. So will a scanned barcode. Our Head of Research Rich Smith shows how a $5 printed label plants an injection that fires days later, at unprompted.au in Sydney this 18th of September. https://lnkd.in/gt-FDQqY

🚀🚀🚀 Big news from Mindgard We’re expanding our #AI and #cloud ecosystem with partnerships with five of the leaders defining the future of enterprise technology: - Anthropic - NVIDIA - Microsoft - Google Cloud Security - Amazon Web Services (AWS) Enterprise AI will not be built on one model or one cloud. It will span providers, agents, tools, infrastructure and data, with each introducing different behaviors and potential attack paths. These relationships bring Mindgard closer to the frontier models and technologies our customers are adopting, while preserving the independence required to assess them objectively. More access. Deeper attacker-driven research. Broader security coverage. Stronger protection for the agentic enterprise. AI security has to go wherever enterprise AI goes. That’s exactly what we’re building. Read the announcement: https://lnkd.in/gV3wXkmw #AISecurity #AgenticAI #Cybersecurity

"Software vendors used to be able to get away with shipping buggy software with no real accountability for a very long time, but now they can't really hide anymore, because AI doesn't sleep and can [find vulnerabilities] at scale." - Aaron Portnoy, Mindgard Check out Alexander Culafi's full story.

Major funding boost for Lancaster University spin-out Mindgard Lancaster AI spin-out Mindgard has raised US$30m (£22.2m) in Series A funding, one of the largest Series A investments in a UK cyber security company. Founded by researchers from Lancaster University’s School of Computing & Communications, Mindgard is helping organisations identify and protect against emerging threats to AI systems. A brilliant example of research turning into innovation, investment and real-world impact. Read the full story https://lnkd.in/efR57vvu #LancasterUniversity #KnowledgeExchange #ResearchCommercialisation #AI #CyberSecurity #Innovation #SpinOut