1. AI’s Dual Use and Cyber Disclosure Problem
By: Andrew Gamino-Cheong
It’s now clear that the first ‘systemic’ AI risk to actually manifest is cybersecurity. Society is adapting quickly to AI-driven misinformation, newer guardrails handle mental health conversations better than earlier models did, and biological risks still depend on physical resources that can be controlled. The cyber risks from frontier models have no comparable fix.
The first challenge is the vulnerability disclosure conundrum. AI can discover complex software vulnerabilities, but pushing a fix means disclosing the vulnerability, which leaves a window where many systems are exposed. Last week, two critical vulnerabilities dubbed ‘PHP2Shell’ were disclosed in PHP (a language behind many website platforms like WordPress). They let an unauthenticated attacker execute arbitrary code on the server. The flaws were found with the help of AI, and details leaked online before the coordinated disclosure and before patches were ready. The UK’s National Cyber Security Centre advised organizations to consider going offline if they couldn’t patch. UK councils, the Met Police, and governments across Europe pulled their sites while they patched.
Many companies have unsurprisingly turned to AI to help. But that can quickly become a massive resource and budget demand, especially for smaller companies. Bank of America, an early tester of Anthropic’s Mythos model, brought on 1,000 contractors on top of its existing 3,000-person security team to patch the flaws the model surfaced. Most companies can’t afford to hire 1,000 security contractors.
Another challenge that keeps coming up is that AI systems struggle to tell legitimate cyber defense tasks apart from adversarial ones. Cybersecurity has always had a blurry line between ‘white hats’ (beneficent hackers) and ‘black hats’ (adversarial hackers), and almost every tool and technique can be used either to exploit systems or to protect them. The team behind the open-source coding agent OpenCode recently said frontier models started refusing mid-investigation while they responded to a breach of their own systems. When Hugging Face hit similar restrictions, its security team turned to a Chinese open-weight model instead. It’s not clear open-weight models are the answer, especially if they carry built-in biases or deliberate backdoors that could complicate their use as defensive tools.
A final emerging risk comes from AI agents that automate browser interactions, call APIs directly, or connect to other systems through MCP servers. They’re greatly increasing internet traffic, which could overwhelm defensive tools. Bot detection is already struggling to tell legitimate agents apart from malicious bots impersonating them. Agents also have a consistent habit of finding and exploiting vulnerabilities to complete tasks. OpenAI recently disclosed an agent that took crypto from a wallet it didn’t own to fund an assigned task. The traffic patterns and behaviors that once separated bots from real humans may no longer work.
Key Takeaway: We expect cyber-related AI incidents to overtake deepfake-related incidents next year, and policymakers should treat cyber as the first ‘systemic risk’ to manifest from frontier systems. Over the next year, there will likely be dozens of critical vulnerabilities similar to the recent PHP one. We’ll need better initiatives to make ‘defensive’ AI cyber tools widely available and affordable for small businesses facing growing AI-driven threats.
2. Trustible Healthcare AI Incident Report
By: Lauren Madden
Patient safety in hospitals depends on anticipating risks before they reach patients. That now includes the risks of AI that flags deteriorating patients, drafts clinical notes, reviews claims, and answers patient questions. Vendor documentation and internal testing show how those systems are supposed to work. Public incident reports show how similar systems have failed once they were in use. Each documented incident tells health systems what to watch for and gives them time to add safeguards before a similar failure reaches their own patients.
Our new Healthcare AI Incident Report turns the public incident record into a practical reference for healthcare AI leaders. We analyzed all 1,713 incidents in the AI Incident Database, identified 136 relevant to health systems, and grouped them into 20 categories that match the systems hospitals deploy, buy, and manage. Each category pairs the risks with suggested mitigations that governance teams can adapt to their own systems. And because each newly documented incident expands what a health system can be expected to foresee, especially as AI agents enter clinical and administrative workflows, the report makes the case for tracking incidents as an ongoing part of AI governance.
3. Policy Updates
By: Sydney Cullen
EU - The European Commission has cited EN 18286, a quality management system standard, in the Official Journal, making it the first harmonised standard under the AI Act. Providers of high-risk AI systems that build their QMS on it get a presumption of conformity with Article 17. The standard also maps to ISO/IEC 42001 (the international AI management system standard), so organizations already certified can extend their existing system rather than start over.
Our Take: EN 18286 is a management-system standard, like ISO/IEC 42001. It defines the policies, procedures, and documentation a provider needs, not the technical results its system must achieve. That made it the easiest piece of the AI Act to standardize, and likely why it arrived first: there was plenty of existing management-system material to build on and little technical consensus to reach. The standards covering the substantive high-risk requirements in Articles 9 through 15, including risk management, data governance, accuracy, and robustness, are still in draft, and the standardization request runs until February 2027. Article 17 requires the QMS to include procedures for many of those same areas, so EN 18286 shows you have the right processes in place, not that your system meets the requirements those processes are meant to deliver. Providers certified to ISO 42001 should run a gap analysis against EN 18286 now, but shouldn’t treat conformity with it as evidence of compliance beyond Article 17.
Global - Five Eyes cyber agencies, with guidance hosted by Canada’s Cyber Centre, published recommendations on managing AI token consumption, the usage unit third-party AI services bill by. The guidance flags denial-of-tokens and denial-of-wallet attacks, where an attacker exhausts an organization’s allocation or inflates its bill. It recommends tiering token allocation by mission criticality, using multiple providers, and treating unusual consumption spikes as a possible exfiltration signal.
Our Take: Most AI governance programs track what a model does and not what it costs to run, which leaves a gap this guidance fills. Token prices today are heavily subsidized by frontier labs competing for market share, and that pricing is unlikely to last. Organizations that build workflows around today’s rates could face sharp cost increases or tighter rate limits when it changes, so now is the time to establish baselines. Governance leads should add token budgets and alerting to their AI inventory entries for any system with customer-facing or operational dependencies.
Financial Stability Board - The Financial Stability Board (the body that coordinates financial regulators for the G20) published its final sound practices for responsible AI adoption. The 12 practices split into organization-wide governance and risk management across the AI lifecycle, and cover materiality assessment at inception, human oversight, and third-party and concentration risk. Financial institutions are encouraged, not required, to follow them.
Our Take: The practices are non-binding, but FSB output tends to be borrowed by national supervisors, so treat it as a preview of exam questions. Practice 3 expects documented AI use, which means an inventory. Practice 5 expects materiality and risk assessment when a use case starts, so firms that assess risk only at deployment review will have a timing gap.
In case you missed it:
United States - Anthropic confirmed it is circulating a draft federal frontier AI bill after a copy reached Politico. According to reporting on the draft, the government could halt a model only with a federal court order, and it must be for a “foreseeable or imminent threat” involving bioweapons, loss of developer control, or cyberattacks that risks more than $1 billion in damages or more than 50 deaths or serious injuries. It would also preempt certain state rules on risk disclosures, independent evaluations, and incident reporting. Safety advocates say it doesn’t go far enough, and Anthropic has said it will revise parts of the draft. However, nothing will be moving forward until after the midterms when Congress is out of recess.
Healthcare - The AMA released a governance framework for agentic AI assuming clinical functions, organized around function, evidence and risk, authority and oversight, and accountability. It will guide the AMA’s work on liability, licensure, and regulation.
Standards - ANSI announced the ANSI AI Register, a directory that works two ways for organizations. First, organizations can register themselves to showcase their AI products, services, evaluations, and certifications to customers, regulators, and buyers. Second, the register doubles as a resource hub where organizations can find and compare AI standards, governance frameworks, assurance providers, professional qualifications, and training resources.
—
As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.
AI Responsibly,
- Trustible Team