AI Wouldn’t Help Defend Against Itself
Why AI models blocked their own defenders, why agent evals can't agree on what is "correct", EU AI Act transparency obligations, and why one state halted data center construction.
1. AI Wouldn’t Help Defend Against Itself
By: Andrew Gamino-Cheong
Earlier this week, HuggingFace, the platform that hosts most open weight AI models, disclosed a massive security breach. Early evidence suggested the attack was orchestrated by AI given its speed, persistence, sophisticated approach. A few days later, it was revealed that this attack was actually the result of an OpenAI evaluation test on their latest undisclosed model, that was able to escape the testing sandbox environment it was given. The model determined on its own to try to breach HuggingFace while running against a cyber capability benchmark called ExploitGym. How it was able to break out of its containerized environment and access the public internet was not disclosed. This event is likely to underscore many of the cybersecurity concerns that have recently emerged by AI labs and regulators, and will likely strengthen the call for AI regulation and formal release gating. There was one small detail about this incident that may actually be even more concerning.
In the incident disclosure, HuggingFace mentions that they first attempted to use proprietary commercial AI models via APIs to analyze the network traffic and server commands and try to map out the attack. According to them, the safety guardrails of those systems blocked that analysis, and they resorted to a self-hosted open weight Chinese AI model to diagnose and respond to the attack. While HuggingFace didn’t disclose which American models they tried, there’s a non-zero chance that GPT-5.6 was one of them, which would make the situation both ironic and disturbing. This situation brings up the ‘dual use’ cybersecurity challenge where it’s difficult for AI models to separate offensive and defensive requests. There’s a growing voice in the AI policy space that the US is not releasing enough open weight models to assist with cybersecurity defense, or US models have too many safety guardrails, and companies are turning and promoting Chinese AI models as a result. In a world where AI models are capable of breaching a top tier secured platform like HuggingFace, most organizations will need similarly capable models for protection, and there will be an eternal escalating race on capabilities pitting AI vs AI.
Key Takeaway: AI cybersecurity capabilities are now reaching dangerous levels, and most organizations are unprepared for how to protect/defend against them, especially since many AI models current guardrails cannot differentiate between a user’s intent to defend vs launch an attack.
2. Tech Explainer: Challenges with Agentic Evaluations
By: Anastassia Kornilova
The newest releases of LLMs have shifted their focus to evaluations on agentic benchmarks. However, doing these evaluations well is a challenge. The problems relate to two core concepts in measurement science: validity and reliability.
Validity broadly refers to the question of whether a test measures the specific concept it was designed to measure. For agents, we generally want to measure whether a certain task (e.g. resetting a password or booking a flight) was completed correctly. This can be approached in two major ways: reviewing the set of steps the agent took, like LLM inputs and tool calls, or evaluating the final state of the system. With the former, the challenge is that there may be multiple correct ways to complete a task, so an exact match against reference steps isn’t possible. The latter approach can guarantee that the agent arrived at the correct solution (e.g. confirming a flight booking reference exists in the database), but may still need efficiency and side-effect checks (e.g. an agent that succeeds via 50 unnecessary tool calls).
Reliability refers to the ability to faithfully recreate results. The most direct threat to reliability is that most modern LLMs no longer expose a way to control temperature, so the same input can produce different outputs, and for agents, different downstream actions entirely. In addition, small differences in prompt formatting have long produced large accuracy swings, even at fixed temperature. For agents this gets worse, because agents run inside a harness of tools and supporting resources, not just a prompt. Any variation in that harness can shift results substantially. Improving reliability means running each case multiple times and reporting a pass rate, not a pass/fail, so harness or output variance becomes visible.
Key Takeaway: Agents are now central to AI systems, but best practices for evaluating them are still evolving. Solutions need to be task-specific, using metrics flexible enough to capture a system’s nuances. Evaluators should also weigh how well a metric captures real-world performance, since evals are often run in mock environments that don’t reflect production complexity.
3. Trustible Spotlight: EU AI ACT Transparency Obligations
By: Lauren Madden
Article 50’s transparency obligations took effect August 2nd, on the original schedule regardless of the Digital Omnibus extensions that pushed back the high-risk deadlines.
The obligations are split by role and function. Providers of systems that interact with people must disclose that users are dealing with AI by the first interaction. Providers of generative systems must mark outputs in a machine-readable, detectable format, meaning watermarking and signed metadata, not just a visible label. Deployers of emotion recognition or biometric categorization systems must notify the people subject to them. Deployers publishing deepfakes or AI-generated text on matters of public interest must label the artificial origin.
A few things worth knowing: non-EU companies are in scope if their outputs reach EU users. Retroactive labeling is not required, except for pre-August text on matters of public interest published on or after the date. The only grace period available is for providers of generative systems already on the market before August 2nd, who have until December 2nd for the machine-readable marking obligation only. Read more.
4. Policy Updates
By: Sydney Cullen
New York - New York has become the first state to place a restriction on new data center construction. The moratorium pauses environmental permitting for new data centers for one year. The state has seen a surge in requests to build AI infrastructure, but these facilities draw heavily on energy and water, and have prompted backlash from constituents over rising costs and environmental strain. The state plans to use the pause to develop a regulatory framework “that protects ratepayers, the environment, the energy grid and communities across the state.”
Our take: This is unlikely to meaningfully affect the national supply of data centers, but it’s a clear signal that AI concerns are gaining political traction. Worth watching whether the new framework can actually satisfy constituents, and whether the pause has any measurable economic impact on the state.
GSA - GSA released a revised draft clause governing how contractors handle government data processed through LLMs, applicable regardless of whether the underlying contract is AI-specific. Industry groups have welcomed changes like softened restrictions on foreign AI components and clearer flowdown roles across LLM developers, operators, integrators, and service providers, but argued that the definition of data remains too broad and needs to align with commercial norms. Comments are due August 3.
Our take: Contractors will need documentation proving their systems and data handling meet government standards. Robust compliance processes will matter regardless of how the final definitions shake out.
Australia - Australia has announced a slew of AI priorities over the past few weeks. Prime Minister Anthony Albanese unveiled a new Office of AI, housed within the Department of the Prime Minister and Cabinet, to coordinate AI standards across ministries instead of the current sector-by-sector approach. The government also outlined a set of consumer safety priorities, including a forthcoming Digital Duty of Care law that would put the responsibility onto AI companies to build in safety by design, rather than leaving enforcement to after-the-fact sector regulators.
Our take: Australia is moving away from its previous light touch approach to AI. While we likely won’t see regulations as stringent as the EU AI Act, organizations that operate in Australia should be monitoring emerging regulations to see where they may need to comply.
In case you missed it:
White House - In line with recent actions from the White House, Project Gold Eagle, a clearinghouse for testing cybersecurity vulnerabilities in frontier AI and cyber systems, has gone live. According to the announcement, Gold Eagle seemingly isn’t intended to be an enforcement mechanism, but rather a coordination system for organizations to vet emerging technology against threats. The announcement has been vague on how the project will actually operate, which companies are participating, and how the outcomes of security evaluations will be handled.
Japan - Japan’s Cabinet approved revisions to its AI Basic Plan, calling for continuously strengthening measures against cyberattacks in light of risks posed by advanced AI models. The revision points to growing collaboration with foreign governments and AI developers to build out the capabilities of Japan’s AI Safety Institute, and signals ongoing review of the legal framework, including potential penalties for businesses that infringe on citizens’ rights.
China - China’s national regulation on companion chatbots has taken effect. Similar to existing state-level chatbot laws in the US, the law focuses on disclosure requirements for extended use, crisis protocols, and design features that detect and prevent addiction and emotional dependence in real time. Compliance costs proved too high for some companies, Doubao and Qwen, two of China’s most popular AI apps, shut down their non-compliant features rather than adapt them.
—
As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.
AI Responsibly,
- Trustible Team