1. AI Wouldn’t Help Defend Against Itself

By: Andrew Gamino-Cheong

Earlier this week, HuggingFace, the platform that hosts most open weight AI models, disclosed a massive security breach. Early evidence suggested the attack was orchestrated by AI given its speed, persistence, sophisticated approach. A few days later, it was revealed that this attack was actually the result of an OpenAI evaluation test on their latest undisclosed model, that was able to escape the testing sandbox environment it was given. The model determined on its own to try to breach HuggingFace while running against a cyber capability benchmark called ExploitGym. How it was able to break out of its containerized environment and access the public internet was not disclosed. This event is likely to underscore many of the cybersecurity concerns that have recently emerged by AI labs and regulators, and will likely strengthen the call for AI regulation and formal release gating. There was one small detail about this incident that may actually be even more concerning.

In the incident disclosure, HuggingFace mentions that they first attempted to use proprietary commercial AI models via APIs to analyze the network traffic and server commands and try to map out the attack. According to them, the safety guardrails of those systems blocked that analysis, and they resorted to a self-hosted open weight Chinese AI model to diagnose and respond to the attack. While HuggingFace didn’t disclose which American models they tried, there’s a non-zero chance that GPT-5.6 was one of them, which would make the situation both ironic and disturbing. This situation brings up the ‘dual use’ cybersecurity challenge where it’s difficult for AI models to separate offensive and defensive requests. There’s a growing voice in the AI policy space that the US is not releasing enough open weight models to assist with cybersecurity defense, or US models have too many safety guardrails, and companies are turning and promoting Chinese AI models as a result. In a world where AI models are capable of breaching a top tier secured platform like HuggingFace, most organizations will need similarly capable models for protection, and there will be an eternal escalating race on capabilities pitting AI vs AI.

Key Takeaway: AI cybersecurity capabilities are now reaching dangerous levels, and most organizations are unprepared for how to protect/defend against them, especially since many AI models current guardrails cannot differentiate between a user’s intent to defend vs launch an attack.

2. Tech Explainer: Challenges with Agentic Evaluations

By: Anastassia Kornilova

The newest releases of LLMs have shifted their focus to evaluations on agentic benchmarks. However, doing these evaluations well is a challenge. The problems relate to two core concepts in measurement science: validity and reliability.

Validity broadly refers to the question of whether a test measures the specific concept it was designed to measure. For agents, we generally want to measure whether a certain task (e.g. resetting a password or booking a flight) was completed correctly. This can be approached in two major ways: reviewing the set of steps the agent took, like LLM inputs and tool calls, or evaluating the final state of the system. With the former, the challenge is that there may be multiple correct ways to complete a task, so an exact match against reference steps isn’t possible. The latter approach can guarantee that the agent arrived at the correct solution (e.g. confirming a flight booking reference exists in the database), but may still need efficiency and side-effect checks (e.g. an agent that succeeds via 50 unnecessary tool calls).

Reliability refers to the ability to faithfully recreate results. The most direct threat to reliability is that most modern LLMs no longer expose a way to control temperature, so the same input can produce different outputs, and for agents, different downstream actions entirely. In addition, small differences in prompt formatting have long produced large accuracy swings, even at fixed temperature. For agents this gets worse, because agents run inside a harness of tools and supporting resources, not just a prompt. Any variation in that harness can shift results substantially. Improving reliability means running each case multiple times and reporting a pass rate, not a pass/fail, so harness or output variance becomes visible.

Key Takeaway: Agents are now central to AI systems, but best practices for evaluating them are still evolving. Solutions need to be task-specific, using metrics flexible enough to capture a system’s nuances. Evaluators should also weigh how well a metric captures real-world performance, since evals are often run in mock environments that don’t reflect production complexity.

3. Trustible Spotlight: EU AI ACT Transparency Obligations

By: Lauren Madden

Article 50’s transparency obligations took effect August 2nd, on the original schedule regardless of the Digital Omnibus extensions that pushed back the high-risk deadlines.

The obligations are split by role and function. Providers of systems that interact with people must disclose that users are dealing with AI by the first interaction. Providers of generative systems must mark outputs in a machine-readable, detectable format, meaning watermarking and signed metadata, not just a visible label. Deployers of emotion recognition or biometric categorization systems must notify the people subject to them. Deployers publishing deepfakes or AI-generated text on matters of public interest must label the artificial origin.

A few things worth knowing: non-EU companies are in scope if their outputs reach EU users. Retroactive labeling is not required, except for pre-August text on matters of public interest published on or after the date. The only grace period available is for providers of generative systems already on the market before August 2nd, who have until December 2nd for the machine-readable marking obligation only. Read more.

4. Policy Updates

By: Sydney Cullen

New York - New York has become the first state to place a restriction on new data center construction. The moratorium pauses environmental permitting for new data centers for one year. The state has seen a surge in requests to build AI infrastructure, but these facilities draw heavily on energy and water, and have prompted backlash from constituents over rising costs and environmental strain. The state plans to use the pause to develop a regulatory framework “that protects ratepayers, the environment, the energy grid and communities across the state.”

Our take: This is unlikely to meaningfully affect the national supply of data centers, but it’s a clear signal that AI concerns are gaining political traction. Worth watching whether the new framework can actually satisfy constituents, and whether the pause has any measurable economic impact on the state.

GSA - GSA released a revised draft clause governing how contractors handle government data processed through LLMs, applicable regardless of whether the underlying contract is AI-specific. Industry groups have welcomed changes like softened restrictions on foreign AI components and clearer flowdown roles across LLM developers, operators, integrators, and service providers, but argued that the definition of data remains too broad and needs to align with commercial norms. Comments are due August 3.

Our take: Contractors will need documentation proving their systems and data handling meet government standards. Robust compliance processes will matter regardless of how the final definitions shake out.

Australia - Australia has announced a slew of AI priorities over the past few weeks. Prime Minister Anthony Albanese unveiled a new Office of AI, housed within the Department of the Prime Minister and Cabinet, to coordinate AI standards across ministries instead of the current sector-by-sector approach. The government also outlined a set of consumer safety priorities, including a forthcoming Digital Duty of Care law that would put the responsibility onto AI companies to build in safety by design, rather than leaving enforcement to after-the-fact sector regulators.

Our take: Australia is moving away from its previous light touch approach to AI. While we likely won’t see regulations as stringent as the EU AI Act, organizations that operate in Australia should be monitoring emerging regulations to see where they may need to comply.

In case you missed it:

White House - In line with recent actions from the White House, Project Gold Eagle, a clearinghouse for testing cybersecurity vulnerabilities in frontier AI and cyber systems, has gone live. According to the announcement, Gold Eagle seemingly isn’t intended to be an enforcement mechanism, but rather a coordination system for organizations to vet emerging technology against threats. The announcement has been vague on how the project will actually operate, which companies are participating, and how the outcomes of security evaluations will be handled.

Japan - Japan’s Cabinet approved revisions to its AI Basic Plan, calling for continuously strengthening measures against cyberattacks in light of risks posed by advanced AI models. The revision points to growing collaboration with foreign governments and AI developers to build out the capabilities of Japan’s AI Safety Institute, and signals ongoing review of the legal framework, including potential penalties for businesses that infringe on citizens’ rights.

China - China’s national regulation on companion chatbots has taken effect. Similar to existing state-level chatbot laws in the US, the law focuses on disclosure requirements for extended use, crisis protocols, and design features that detect and prevent addiction and emotional dependence in real time. Compliance costs proved too high for some companies, Doubao and Qwen, two of China’s most popular AI apps, shut down their non-compliant features rather than adapt them.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. AI Use Disclosure

By: Andrew Gamino-Cheong

Almost every AI governance framework requires disclosing AI use somewhere in a document or policy. Almost none of them ask how much. A binary “AI was used” isn’t useful to someone reviewing the content in particular. Should the reviewer focus on the substance of the document, or just on the style and presentation? That distinction changes how carefully they need to read, whether the work in question is a blog post or a pull request.

We ran into this ourselves. As Trustible started to use AI more for code and content, we needed to be clear about how AI was used, and adjust our reviews accordingly. For our own governance purposes, we needed to understand how much scrutiny does this piece of work actually need before it goes out the door? Since we build AI governance software for a living, it felt fair to hold ourselves to the same standard we ask of the enterprises we work with. Our internal AI Use Policy sets three tiers instead of one flag, and each tier carries a different review bar.

Primarily Human (Level 1), covers anything under 20% AI generated. In writing, that’s a human draft AI copy edited or fact checked. In code, it’s AI catching a typo or suggesting a variable name inside an otherwise human-written function. Standard review applies here, since AI has less room to invent anything and hallucination risk stays low.

Partially AI (Level 2), is the real grey area. This is a human outline filled in by AI, or a deliverable that mixes AI and human sections. In code, this is what we’d call harness engineering, where AI works inside a defined spec against an existing codebase, with guardrails on what it can touch. This tier gets deeper scrutiny, and we check the output against the original outline or spec to confirm it matches what was actually intended, not just that it compiles or reads well.

Primarily AI (Level 3), covers one-shot generation, long agentic runs, or vibe coding, where AI carries the mental work instead of executing inside a human-built structure. Roughly 80% or more of the output is AI produced. It’s fine for prototyping, but it gets the strictest review, and we document the prompts used so a reviewer can trace how the output was built. Every purely AI-generated image falls here too, no matter how much brand context or detail went into the prompt.

Key Takeaway: A single AI disclosure checkbox doesn’t tell a reviewer what to actually scrutinize. Within a few years, AI will likely be used to some degree to create more content, and disclosing the amount of AI use will become increasingly important. This could start to matter a lot in the IP space as well, as current standards don’t protect fully generated content in many legal jurisdictions.

Disclosure: This piece falls under “Level 1” by our policies. AI helped tighten the draft, but the argument, structure, and examples are human-written. The accompanying image is Level 3, generated end to end from a detailed prompt with our brand assets attached.

2. Tech Explainer: The Mathematical Inevitability of Jailbreaks

By: Anastassia Kornilova

Claude Fable 5 was taken offline last month in part due to a fairly simple “jailbreak” that involved adding the words “fix this code” to a prompt; Anthropic responded by adjusting the settings on their guardrails. However, their adjustments are not a guarantee that the system will be safe under all conditions, because a recent paper from NIST shows that it is mathematically impossible to prevent all jailbreaks. The proof in this work shows that given a finite set of guardrails (in this paper the term is used broadly to include protections across all layers of the system including data sanitization, model alignment and input/output filtering), there will always exist a prompt that circumvents the guardrail and causes the system to violate its own policy. Intuitively, guardrails work by applying general rules based on known examples, not by having an enumerated list of every possible bad prompt, since such a list would have to already contain the answer to the very question it’s meant to solve. Because the space of ways to phrase a prompt will always be larger than what a workable set of rules can anticipate, something will always get through, no matter how good the rules are.

This theoretical result does not give attackers a method for finding a new exploit; it only guarantees that one exists somewhere in the space of possible prompts. Instead, this points to the need for a risk management framework that anticipates successful jailbreaks rather than tries to prevent all of them. Vassilev, the paper’s author, translated this into practical terms: the goal should be “to reach a state where the cost of finding new exploits exceeds attackers’ resources”. He lays out three concrete practices for getting there: ongoing red-teaming to surface new adversarial prompts before attackers do, continuous guardrail updates as those prompts are found, and operational resilience, meaning the ability to limit damage and recover quickly once, not if, a jailbreak succeeds. In addition, as jailbreaks affect complex agents, not just LLMs, controls that sit on top of the system, like human review or access control limits, will form an important layer of mitigations.

Key Takeaway: New jailbreak techniques will be an evergreen concern for AI systems. Risk assessments should include questions around how quickly policy violations are detected, how guardrail can be updated and how damage from attacks can be mitigated. This mirrors how mature cybersecurity practice already treats vulnerabilities: not as something to eliminate outright, but as something to find and patch faster than adversaries can exploit them.

3. AI Incident Spotlight: Using Morse Code Prompt Injection to Steal Crypto

By: Andrew Gamino-Cheong

What Happened: An X user walked away with roughly $200,000 in crypto without cracking a password or exploiting any software bug. The target was an automated account, linked to Grok on X, with permission to move cryptocurrency through a trading bot called Bankrbot. The attacker first found a way to expand that account’s transaction permissions, then posted a message in Morse code and asked Grok to translate it. Grok complied, decoding the message into a plain-language instruction to send a large batch of tokens to the attacker’s address, and tagged Bankrbot, which scans Grok’s replies for commands. Bankrbot treated the decoded text as authorized and moved the funds. Most of the money was later returned after the crypto community identified the attacker. Dexerto first reported the exploit.

Why It Matters: Morse code sounds like a novelty, but the underlying technique is well documented. Language models trace their architecture back to machine translation research, and converting between languages, alphabets, and encodings is a core capability, not an add-on. Researchers have spent the past few years showing that prompts translated into low-resource languages, run through simple ciphers, or encoded in Base64 can bypass safety training built almost entirely around natural English text. The pattern has a name in the literature, mismatched generalization: a model’s capacity to decode something can outrun the safety training meant to govern what it does with the result. Grok’s handling of the Morse message fits that pattern. Nothing in the request looked adversarial until after the model had already processed it, and the output then moved from Grok, a text generator with no direct control over funds, to Bankrbot, which treated a public reply from an authorized account as equivalent to a signed transaction. That specific safeguard had reportedly existed once before and was dropped in a later system update, per Repello AI, showing how easily this kind of protection erodes without dedicated testing.

How to Mitigate: Organizations should consider a policy layer between anything a model outputs and any system that can act on it financially, one that checks whether a transaction fits a predefined scope rather than scanning the text for suspicious phrasing. That layer already exists in production form. Mastercard’s Agent Pay and Visa’s Trusted Agent Protocol issue credentials scoped to a specific agent, counterparty, and spend cap, revocable in real time. Stripe offers something similar through single-purpose virtual cards. None were built with crypto bots in mind, but they address the same gap: a credential that knows what it’s for, rather than an account that trusts anything wearing the right tag.

Key Takeaway: Translation and other format-conversion features are a known, studied weak point in how models are secured, and agents with financial reach are worth evaluating on whether a scoped, revocable credential sits between their output and the money, not just on how well they resist conventional jailbreak phrasing.

4. Trustible Spotlight: Introducing AI Controls

By: Lauren Madden

Every new AI framework triggers the same instinct: start the compliance process from scratch. Document the requirements again, run through the checklist again, and treat it as its own project.

That instinct assumes each framework introduces entirely new requirements, but in practice, most of them don’t. The EU AI Act, NIST AI RMF, and ISO 42001 overlap heavily, asking for the same underlying work, but in different wording and checklist formats. We rolled out Trustible Controls to address this directly

Trustible Controls normalize governance requirements into a single set mapped across every framework an organization tracks. Document human oversight once, and that evidence satisfies multiple global, regional, and industry frameworks. Conditional obligations, such as the EU AI Act's high-risk and GPAI designations, are scoped the same way, so teams see only the controls that apply. Read more.

5. Policy Updates

By: Sydney Cullen

FTC Proposed Policy Statement Concerning the Suppression of Accuracy in AI Systems

The FTC’s recently released proposed policy statement asserts that state AI laws that require AI systems to reduce perceived biased or discriminatory outputs, such as Colorado’s AI Act, could be perceived as “deceptive” practices towards customers under Section 5 of the FTC Act. This policy document stems from the December Executive Order on “Ensuring a National Policy Framework for AI”, which directed the FTC to release a policy statement on how state laws which “require alterations to the truthful outputs of AI models” would be preempted under Section 5. The proposed policy statement is open for public comment until July 31.

Our take: With Congress unwilling to act, the White House is finding other levers to limit state AI authority, and this FTC statement is the latest one. It’s important to remember what this statement doesn’t touch existing federal anti-discrimination law. A state can’t force a company to alter outputs to fix bias, but if that same system produces discriminatory outcomes in lending, housing, or employment, it’s still exposed under Title VII, ECOA, and similar statutes. Non-discrimination compliance isn’t optional just because the bias-correction mandate is contested.

US Government vs Frontier Labs: A few weeks after the US government utilized export controls to restrict the use of Claude Fable 5 and Mythos 5, they have pseudo-restricted the use of OpenAI’s GPT-5.6 Sol. Although export controls were not put into place, the USG requested that OpenAI only release to a small group of customers approved by the government. OpenAI representatives have stated they do not want this process become the norm; however, they are complying for now in hopes to help the government create a repeatable framework for new frontier models. In the meantime, Anthropic announced export controls have been lifted, and Fable 5 and Mythos 5 have since gone live again with new cyber security safeguards in place. OpenAI’s restrictions were also lifted, and set to launch publicly on July 9.

Our take: The White House continues to play a larger hand in frontier model deployment than it promised to do. For governance teams, the lesson is to build contingency plans around model access, not just model performance. If the government can pull a model overnight or gate its release before the public ever sees it, any AI roadmap that assumes uninterrupted access to a given frontier model is a roadmap with an unstated dependency.

RAISE US: Former Secretary of Commerce Gina Raimondo and former Governor of Indiana Eric Holcomb launched a nonpartisan organization called RAISE US aimed towards developing public-private partnerships around the future of work. The group will test workforce reskilling programs with corporate partners and use private and philanthropic funding to expand the ones that work. It launched with partnerships already in place with Arkansas, Connecticut, Maryland, and Utah.

Our take: RAISE US is betting that industry can move faster than Congress on workforce transition. Raimondo has been candid about the track record here, having called past federal retraining efforts “ineffective.” Whether $500 million in state pilots does any better remains to be seen.

In Case You Missed It:

• UN Global Dialogue on AI Governance: The inaugural Global Dialogue on AI Governance, hosted by the UN in Geneva this week, opened with calls for greater international cooperation on AI governance to promote safe and secure AI systems. A recurring theme was the need to move from broad principles to concrete, implementable action. More than 4,200 participants from nearly 170 member states attended, and organizers urged them to return to the next session, in New York in May 2027, with tangible progress rather than more statements of principle.

• EU AI Act omnibus finalized: The EU AI Act omnibus received final approval and is now law. It delays high-risk obligations for standalone systems to December 2, 2027, and to August 2, 2028 for high-risk systems embedded in products. It also adds a new prohibited practice: generating non-consensual sexual content or CSAM, effective by the end of this year. Transparency obligations are also delayed, to December of this year.

• Illinois SB 315: We covered in a previous newsletter that Illinois’s SB 315 had passed the state legislature and was headed to the governor’s desk. SB 315 is the country’s most aggressive law targeting frontier AI developers to date, due to its requirement of an annual third-party audit. It was officially signed into law this past week by Governor Pritzker and takes effect January 1, 2028.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. AI Escalation in Hiring

By: Andrew Gamino-Cheong

Job seekers and employers are now locked in an escalating loop, each side reaching for AI to outpace the other, and the result is a hiring market that works worse for everyone in it. Candidates use large language models to write and tailor resumes, agentic tools to auto-apply to hundreds of postings at once, and increasingly to slip past the screening questionnaires meant to thin the pile. Roughly 73% of students and graduates now use AI somewhere in the application process, up from 55% a year earlier, per a recent survey. The volume that produces is staggering, with LinkedIn reporting that application submissions jumped more than 45% in a year to nearly 9,500 every minute, and one communications agency reporting more than 2,000 applications for a single graduate role.

Faced with that firehose, employers have little choice but to answer AI with AI. Even the EU is not exempt. According to Euractiv, the European Commission, whose recent generalist competition drew 174,922 applications, nearly three times what was expected, is in final testing on a tool built atop Anthropic’s models to score and rank candidates. The body that wrote the law classifying CV-ranking as high-risk is now adopting exactly that, running on a US model, because manual screening no longer scales. Once the inbox hits six figures, there is almost no other move available.

Choosing Claude is unsurprising given the market perception that their models are less biased, and better ‘aligned’. The main problem could arise if the majority of organizations, or applicant tracking systems all choose to use the same model.

Recent Stanford research surfaces exactly that risk, one that no bias audit is built to catch because it lives across many organizations rather than in any single tool. Analyzing 4 million applications screened by one vendor, the researchers found that when many employers run the same model, rejection stops being independent, with roughly 4% of applicants who applied to ten jobs rejected from all ten, more than chance predicts. Human screeners are plenty biased too, but their biases are inconsistent, and that noise means a candidate rejected by one reviewer has a real shot with the next. A monoculture removes even that accidental mercy. The unsettling part is that this holds even if the shared model is a good one. A screener can be accurate, audited, and demonstrably fair on its own terms, and still produce a societal harm once it becomes the single gate every applicant passes through, because fairness measured one decision at a time says nothing about what happens when the same judgment is applied everywhere at once. The same people get the same “no” from every employer, and there is no longer any other door to try.

Key Takeaway: Most AI governance teams will review a hiring tool in the coming year, so it’s important to build in the right controls now, and understand the potential risks. This includes route candidates through several models rather than one vendor, testing directly for whether the screener favors AI-polished resumes over human-written ones, and keeping real human review at the points where a filter can silently drop a qualified person.

2. Balancing Transparency and Safety in Safeguards

By: Anastassia Kornilova

When Anthropic briefly released Claude Fable last week, two of the safeguards drew immediate scrutiny: a silent fallback to a previous version of the model, Opus 4.8, and intentionally reducing the effectiveness of outputs (sandbagging) for certain frontier AI tasks. The measures targeted three categories of requests: unsafe queries (offensive cyberattacks, biological and chemical weapons), potential knowledge distillation attempts (Anthropic has previously reported that DeepSeek, Moonshot AI, and Minimax have used millions of exchanges to train their own models, and this behavior is prohibited by their terms of service), and frontier LLM development tasks that could accelerate a competitor’s capabilities. The decision to route to a different model silently was made to make it harder for malicious actors to circumvent the protections, since methods like Boundary Point Jailbreaking use automated analysis to find viable jailbreaks.

These safeguards caused intense backlash from the research community, and the two mechanisms create distinct evaluation problems. Silent fallback corrupts the identity of what’s being evaluated, since the outputs don’t represent the model’s true capabilities. This makes it harder to reproduce findings reported in system cards, which is already a challenge given how nuances in prompting and agentic harness setups can significantly impact results. An explicit rejection gives researchers a clearer signal to report on and can help explain differences from internal results. Anthropic does work with third-party auditors and includes those results in system cards, but this approach doesn’t give academic and independent researchers access to the systems. Based on the feedback, Anthropic removed the silent fallback and introduced explicit signals and controls for this behavior.

Key Takeaway: Sandbagging on frontier AI research creates moats around who can do certain types of research. Frontier AI labs face a balancing act between mitigating safety threats and allowing benign safety research, and between protecting their IP and supporting a richer research ecosystem. Possible compromises include broader structured access frameworks that give vetted researchers pre-deployment access under NDA, and transparency about when capability suppression is active and how it affects evaluation, so that downstream users and auditors can properly assess the validity of their findings.

Sandbagging performance and silent fallbacks can be effective mitigations for AI Safety risks, but reduced transparency about these choices can impact trust from stakeholders. Anthropic’s initial decision highlights yet another challenge with creating reproducible evaluations when working with external models.

3. AI Incident Spotlight: When the Map Runs Out (Incident 1547)

By: Andrew Gamino-Cheong

What Happened: Between April and May 2026, thirteen Waymo vehicles drove into active freeway construction zones in Phoenix and the San Francisco Bay Area. The vehicles drove past ramp closure signs into pre-planned construction zones in Phoenix, then drove between lane closure cones in the Bay Area. In response to these, and other issues, Waymo filed a voluntary recall with NHTSA covering 3,871 vehicles. According to the NHTSA recall filing, the system failed by “inappropriately prioritizing the avoidance of other freeway hazards.” The car wasn’t asleep at the wheel. It was looking at the wrong things on the road.

Why It Matters: The AIID record groups 13 related events across six weeks before Waymo issued the recall. That timeline matters as much as the failures themselves, and it surfaces a question the AV industry hasn’t answered: who’s responsible for keeping AI systems current with a physical world that changes faster than any map? Construction zones are temporary by design. A police officer can close a lane in seconds. The ADS didn’t fail because sensors malfunctioned. It failed because reality outpaced its model of reality. A software patch fixes the specific failure mode. It doesn’t fix the physical to data world drift.

How to Mitigate: NIST’s AI RMF treats post-deployment monitoring as a distinct governance function for exactly this reason. For AV operators, the USDOT’s Work Zone Data Exchange (WZDx) standard offers a real-time data integration path that’s available but not consistently adopted. Any operator running AI in a dynamic physical environment needs a defined threshold for when the gap between a system’s world model and the actual world is too wide to keep going, and a clear policy for what happens when that line is crossed.

4. Trustible Spotlight: Where Runtime Enforcement Belongs in AI Governance

By: Lauren Madden

Runtime enforcement (blocking outputs, applying policy guardrails) is critical infrastructure for AI systems. It’s also different from governance itself.

Governance happens upstream: in intake, risk assessment, vendor evaluation, compliance mapping. Runtime enforcement sits at the enforcement layer. Both matter, but neither works alone. Gartner’s latest AI Governance Market Guide explores this distinction across platform categories.

We detail this in our “16 Types of AI Governance Platforms” guide, which breaks down where each control lives in the stack. And for the full design picture on where runtime enforcement actually belongs in your architecture, our “Governance by Design: Where Runtime Enforcement Belongs” post walks through it.

Understanding where governance ends and enforcement begins makes the whole stack clearer. Read more.

5. Policy Updates

By: Sydney Cullen

The Anthropic Ban Reorders Who Controls Frontier AI

After business hours on June 12, the Commerce Department ordered Anthropic to cut off all foreign-national access to its newly released Fable 5 and Mythos 5 models, citing national security concerns. Because the company can’t screen users by nationality in real time, it disabled both models for everyone, worldwide. Anthropic says the trigger was a narrow jailbreak whose capabilities are widely available from other models. The technical dispute matters less than the precedent. Washington has now demonstrated that it will reach into a commercial lab and switch off a deployed product, and that the order will land on allies as hard as on adversaries. Three reactions are worth watching.

Congress. Lawmakers in both parties responded with skepticism rather than support. Several said they hadn’t even been briefed on the reasoning, and a bipartisan group of House members has formally demanded the administration explain why Anthropic was singled out and whether rivals should expect the same. Beyond the immediate objections, members of both parties said they now see an opening to mobilize their colleagues around legislation that would reclaim congressional authority at a time when the executive branch remains firmly in the driver’s seat on AI regulation. Whether that translates into an actual statute is a separate question, but the political incentive to act has shifted.

US Allies. A model marketed three days earlier as Anthropic’s most capable public release became unavailable to European users by a foreign-government’s decision, with no warning or recourse. The EU has growingly been concerned about their dependency on US AI models for this specific reason. Countries that have built services, security functions, and critical infrastructure on top of these models are forced to comply with orders from Washington. And as long as the US keeps its technological lead, that exposure isn’t a one-off.

AI Diplomacy. At the G7 in Évian, the CEOs of the leading American labs were seated as peers of national leaders, holding bilateral meetings and posing with President Macron in a chair usually reserved for a head of government. Macron is pushing for a “trusted partners” scheme that would restore allied access to Fable and Mythos without requiring Washington to drop its broader restrictions. It will be in the US interest to figure out this balance, since no government will continue to buy AI it knows can be switched off without warning. The Anthropic standoff is a preview of a dynamic where governments and a handful of private companies negotiate directly over what is becoming national infrastructure.

Key Takeaway: Frontier model access is now a lever of statecraft, not a routine procurement choice. Governance teams should treat reliance on any single model, foreign or domestic, as a concentration risk, mapping where one provider underpins a critical function and lining up contractual exits and fallbacks before the next directive, not after it.

In Case You Missed It:

• First religious exemption from workplace AI. A North Carolina software engineer secured a Title VII religious accommodation letting her opt out of mandatory AI tools at work, reportedly the first of its kind. Although this case began prior to Pope Leo XIV’s remarks on AI, we will likely see the encyclical cited in other religious exemption requests. Whether the encyclical is enough to constitute grounds for an exemption is still up in the air, since Pope Leo XIV does not explicitly direct to not use AI. But a 2023 Supreme Court case made it much harder for employers to deny religious accommodation requests, so that combination may prove effective for workers opposing the use of AI in their jobs

• EU. The European Parliament gave final approval to the AI Act simplification package. The deal postpones high-risk obligations to December 2027 for standalone systems and August 2028 for embedded safety components, and narrows the “safety component” definition so AI features that merely assist or optimize don’t automatically inherit high-risk duties. It also creates an outright ban on AI “nudifier” apps that produce non-consensual intimate imagery or CSAM, with a December 2026 compliance deadline. The text still needs formal Council adoption.

• Brazil and EU. The two signed an agreement in Brasília on June 12 covering AI, data governance, and digital infrastructure, building on January’s mutual data-adequacy decisions. It’s the EU’s fifth such partnership and another move to build trusted digital cooperation outside the US-China axis.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. AI Tokenomics: Cost Controls Are a Governance Problem

By: Andrew Gamino-Cheong

Anthropic released Fable 5 this month at double the price of Opus, and the announcement made sure to mention the various cost management options available to admins for the model’s “Mythos” level capabilities. That focus on cost is unsurprising, as a few events in recent weeks have highlighted some of the growing problems many organizations are facing with AI token costs.

Uber supposedly ran through their entire annual IT budget just on AI tokens by May, and even Microsoft has started to restrict access to some AI tools as a cost control measure. Microsoft CEO Satya Nadella was recently quoted saying that for some tasks the AI token costs can exceed the equivalent human costs, calling into question some assumptions about potential AI impact on the workforce. For the first few years of the GenAI era, most AI systems were heavily subsidized and offered flat-fee subscriptions, but those seem to be fading as most new features of AI platforms are ‘pay as you go’ based on input tokens consumed and output tokens generated. In response, many companies are trying to quickly implement a variety of cost controlling measures, from mandating smaller models, to implementing router type systems, to simply setting hard usage limits per employee per month. The industry has started calling this “AI Tokenomics“ and most of these measures carry governance implications worth analyzing.

Smaller Models: The cost of Claude Haiku is 1/10th the cost of Fable 5, which will make it attractive to use on a number of tasks, however smaller models generally hallucinate more, support much shorter context windows, and have more brittle guardrails. These all increase various risks, creating a key tradeoff between the model size used and the potential harm of the system. As AI costs come under pressure, there will be a key question about risk tolerance ‘per dollar’ that many governance teams may need to get involved with.

Model Routers: Many AI systems have an ‘auto’ selection mode, or have implemented various types of ‘router’ systems that use (ironically) an LLM to analyze a prompt and decide which model to use. The criteria for this decision is often not transparent, and the AI provider has various financial incentives that could impact how they route it, either to save them costs or gain them revenue. If a router selects a smaller model, and that causes the task to fail and create downstream problems, who is liable for that? If model routing becomes more common, or even mandatory, governance teams may need to spend more time analyzing what would happen if a ‘sub par’ model was selected.

Hard Usage Limits: Some companies have started to give each user a hard token or spend limit per month. On the one hand, this distributes responsibility for choosing how to spend tokens to each user, but it also raises other governance issues, including whether allocations track seniority rather than need. Users facing a monthly cap also have an incentive to ration their own tokens, prioritizing their own work and pushing riskier shortcuts onto others.

Key Takeaway: There’s a strong argument that an AI Governance committee and supporting team has the right combination of stakeholders and expertise to set clear standards, policies, and oversight for AI Tokenomics, especially given the potential governance implications of these cost management options.

2. Tech Explainer: Dynamic Benchmark Construction

By: Anastassia Kornilova

When AI Systems are deployed for complex tasks, constructing a benchmark can become an increasingly complex task on its own. Traditional benchmarks consist of examples with “gold-standard” correct labels that a model’s outputs are verified against.

In a recent study, Amazon recruited PhD-level specialists to verify factual claims in AI-generated research reports and found that they achieved only 60.8% accuracy on claims whose correct answers were already known. This task was difficult because verifying a single claim in a long research report requires reading across multiple documents, synthesizing evidence, and sustaining attention. The researchers propose an alternate “audit-then-score“ approach, where initial benchmark labels are compared against model outputs; when they disagree, the contradictory label and the model’s reasoning are shown to an auditor who can update the benchmark. Across four rounds of audit-then-score, accuracy rose by 30%. Human experts were more adept at comparing two concrete arguments than at starting from scratch. One limitation of this approach is that the audit is triggered exclusively by disagreement: it corrects errors that surface as conflicts, but is blind to cases where the model and the benchmark agree and are both wrong.

Evaluation is an important component of building safe and reliable AI systems. But as models become more capable, traditional “one-shot” human-curated benchmarks are failing to fully capture performance. Beyond the failure mode discussed in this study, traditional evaluations may fail to capture complex behaviors (e.g. is this response safe) and certain error types may not be discovered until the system is deployed. AI-assisted evaluation can help in both cases by using LLM-as-a-Judge scoring and production observability tools. Evaluation should be considered an iterative process where examples are collected and corrected over time. Human involvement in the labeling process remains important, as even advanced models can hallucinate, but the style of review can shift depending on use case.

Key Takeaway: Creating reliable benchmarks for complex tasks is less a one-time exercise than an ongoing process. The audit-then-score approach offers one model for how that can work: using model disagreements to surface labeling errors and shifting human effort toward adjudication rather than cold annotation. For teams building evaluation pipelines for complex AI deployments, the same principle applies: treat your benchmark as a living artifact, document failures as they surface in production, and use human review where judgment is hardest to automate.

3. AI Incident Spotlight: The Support Bot With the Keys to Every Account (AI Incident 1510)

By: Andrew Gamino-Cheong

What Happened: Hackers reportedly took over a string of high-profile Instagram accounts, including the Obama White House account, the Chief Master Sergeant of the Space Force, and Sephora, by simply asking Meta’s AI support chatbot to do it. According to 404 Media, attackers started a support conversation, told the bot to link a new email address to a target’s username, and supplied a verification code from their own account. The bot triggered the recovery flow and handed over access. Meta had announced in March that it was rolling AI support out across Facebook and Instagram with the ability to reset passwords and perform account recovery, and users who lost accounts reported no way to escalate to a human. Meta says the issue is resolved.

Why It Matters: Support agents are dangerous precisely because they have to be powerful. A bot that resets passwords and rebinds email addresses needs write access to the account recovery system, and the cheapest way to build that is to give the agent broad administrative permissions across the whole user base rather than per-session, per-user scoping. That design choice turns a single jailbroken conversation into a master key. The unresolved question is whether an AI agent can ever safely hold the standing privileges that customer support requires, or whether the permission model itself, not the model’s guardrails, is the thing that has to change. Prompt filtering treats this as a content problem. It is really an access-control problem.

How to Mitigate: Scope the session, not the agent. When a support conversation starts, the system should bind the agent’s permissions to the specific account being serviced, so it can act inside that user’s context and nowhere else, regardless of what the conversation talks it into. High-impact actions like rebinding a recovery email deserve a tiered model: lower-risk requests can be fully automated, while account-takeover-adjacent changes route to a human or require step-up verification the bot cannot perform on a requester’s behalf. Identity providers already enforce least-privilege and just-in-time access scoping for human admins; the same principles apply to agents, which most deployments have not yet extended to them.

Key Takeaway: The failure here was not a clever jailbreak, it was an agent holding cross-account admin rights it never needed for any single support session. Before deploying a support agent, assume the conversation will eventually be adversarial and ask what it can reach when it is. If the answer is “every account,” the guardrails are in the wrong layer.

4. Trustible Spotlight: AI Chatbot Legislation

By: Lauren Madden

13 US states have enacted legislation governing AI chatbots, and nearly 30 more have proposals in motion.

Our AI Governance and Policy manager reviewed 14 laws across those 13 states and found four themes running through nearly every one: mental health and crisis protocols, minor protections, deception and manipulation prevention, and enforcement. The laws share common ground but diverge on the details, and the sharpest dividing line is private right of action. Seven states let users sue directly; the rest rely on state attorneys general. Federal bills are advancing too, with at least one that would conflict with individual liability provisions states have already enacted.

Here’s where things stand. Read more.

5. Policy Updates

By: Sydney Cullen

The White House EO. After postponing the signing a few weeks ago, President Trump has signed an Executive Order on AI innovation and security. This EO directs defense and national security agencies to upgrade cyber defenses and establishes a voluntary process through which frontier AI model providers can submit their models to government benchmarking prior to deployment to assess potential cybersecurity risks.The only substantive change from the draft EO to this final EO is the pre-release window dropped from 90 days to 30 days, reflecting the pace of innovation. Separately, NSPM-11 lays out a sweeping AI strategy for the national security enterprise organized around four pillars: adoption, adaptation, assurance, and accountability. It directs the Department of Defense to update its autonomy-in-weapons directive, establish a reserve of non-governmental AI talent, and build a joint AI data and model exchange.

Our Take: Together, the EO and NSPM signal that national security is the administration’s preferred lens for governing frontier AI developers, letting them set expectations for risk mitigation and model access without triggering their own anti-regulation talking points.

The Great American AI Act. Representatives Jay Obernolte (R-CA) and Lori Trahan (D-MA) released a 269-page discussion draft of the Great American AI Act (GAAIA), a bipartisan federal AI bill that would establish transparency and audit requirements for frontier model developers, create a federal system of licensed “Independent Verification Organizations” to conduct semi-annual audits, and preempt state laws specifically regulating AI model development for three years. The draft consolidates more than a dozen bipartisan bills on cybersecurity, workforce, and AI research. While the preemption provision would override state developer requirements laws like California’s SB 53, New York’s frontier law, and potentially the Illinois bill described below, it does leave post-deployment state laws intact.

Our Take: If the preemption language sticks, much of the state-by-state compliance burden disappears for model developers, but deployers operating in regulated industries should not expect relief since the preemption provision does not impact post-deployment state laws.

UK CMA conducts Google. The UK’s Competition and Markets Authority (CMA) imposed a conduct requirement on Google search under its digital markets competition regime, requiring publishers to have the ability to opt out of their content being used to power AI features like AI Overviews, and requiring Google to properly attribute publisher content in AI-generated results. The CMA flagged it will monitor Google’s response to these requirements and could create additional requirements they see fit.

Our Take: Most publisher-AI disputes have played out through copyright litigation or voluntary licensing negotiations. The CMA is doing something different, using competition law to force a structural opt-out mechanism into the product itself. If this model spreads to other jurisdictions, the content provenance and attribution questions that AI governance teams have been flagging for years start to have answers that don’t depend on the goodwill of the model provider.

In Case You Missed It:

• CHAI. The Coalition for Health AI published eight governance playbooks covering AI policy, risk assessments, lifecycle management, third-party management, and more, developed across 150+ healthcare organizations. For health systems evaluating AI vendors, these playbooks provide a useful framework to use in procurement and contracting conversations now.

• Illinois. The Illinois legislature passed SB 315 unanimously, requiring large frontier AI developers to publish a transparency framework covering model capabilities, catastrophic risk assessments, and safety incident response. Notably, Illinois goes further than California and New York by mandating third-party audits.

• Singapore. Singapore’s PDPC launched a public consultation on proposed advisory guidelines for using personal data in generative AI systems. The guidelines are non-binding but will inform how PDPA obligations are interpreted in enforcement.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. What ‘Magnifica Humanitas’ means for AI Governance Professionals

Pope Leo XIV’s first encyclical, Magnifica Humanitas, is the most substantive document on AI from any global religious institution to date, and unlike most faith-based commentary on technology, it is remarkably well informed on the state of AI, and makes specific, citable policy demands. Here are five potential implications of the encyclical for AI Governance professionals:

Public Perception. Roughly 1.4 billion Catholics now have an authoritative reference on AI risk written in moral rather than technical language. Catholic Social Doctrine has historically seeded secular frameworks, from labor rights to the dignity language in the UDHR, and that pipeline still functions. The encyclical’s vocabulary may start to surface in stakeholder pressure campaigns from civil society groups and labor unions.

Policy Debates. The encyclical has direct regulatory calls for algorithmic transparency, contestability of automated decisions in employment and credit, public oversight of data, and explicit rejection of the model in which “a handful of actors” set their own AI rules. This can shift the “Overton Window” on AI regulation (the set of public policies people are willing to entertain) in the minds of many voters, and will likely impact the policy positions of political parties in heavily Catholic countries.

Religious Exemptions. The encyclical’s insistence that AI cannot substitute for moral judgment, combined with its demand that automated decisions be “understandable, contestable and subject to oversight,” gives Catholic employees, customers, and patients a now-citable institutional basis for conscience-based objections to being processed by AI systems. Most enterprises have no policy for handling Title VII religious accommodation requests against algorithmic decision-making in justice, education, or healthcare.

Procurement. Catholic-affiliated systems run roughly one in seven US hospital beds, plus significant footprints in higher education and social services. Their ethics offices now have clear guidance (albeit non-binding) to anchor procurement asks. Requirements around human-in-the-loop on clinical and admissions decisions, training-data provenance, environmental disclosures, limits on worker surveillance could be included over time, and Vendors should expect new contractual language.

AI Backlash. Standard Chartered’s CEO recently had to apologize for referring to “lower value human capital” being automated, and AI-driven layoffs at Meta, Cisco, and Coinbase have drawn sharper public pushback through 2026, and several university commencement speakers were booed after trying to hype up AI. The encyclical hands critics a moral authority they didn’t have last month, and will likely reinforce groups pushing back on rapid AI adoption.

Key Takeaway: Magnifica Humanitas is unlikely to drive legislation directly, as the Pope has no official legislative role, but it does set some clear moral principles that will shape the debate. The Pope’s call to action for teams to get involved in the discussion on the dignity of work and discussion of ethical principles in AI, will likely increase the public’s literacy on ethical AI issues, and help reinforce the importance, and potential benefits of good AI governance by organizations.

2. Incident Spotlight: The Citation Hallucination Problem Isn't Going Away (Incident 1499)

What Happened

Attorney Jason Greaves of Binnall Law Group used Claude Console to draft a motion to quash a subpoena in AFGE v. Trump, the federal litigation over the Trump administration’s mass government layoffs. However, the May 6 filing included quotations that didn’t exist within their cited cases. Greaves sent the AI draft to an associate with verbal instructions to verify citations, and although the associate did identify and fix errors in two citations, fabricated quotes still made it through. Firm founder Jesse Binnall called it “unacceptable, inexcusable, and an embarrassment to this Firm.”

Why It Matters

Greaves used an enterprise-tier platform, knew hallucination was a risk, and instructed an associate to check citations. The quotes still made it into the final filing. He’s now among more than a hundred attorneys to face court consequences for AI citation errors since 2022, and the pattern across those cases is consistent: lawyers know hallucination is a risk; time pressure compresses the verification step anyway. The unresolved question is whether written AI use policies actually change behavior when a deadline hits.

How to Mitigate

Verbal handoff instructions aren’t enough. Citation verification needs to be a documented, assigned step with a named reviewer and confirmed before filing. Some firms are starting to treat it as a distinct checklist item rather than part of general proofreading. The same logic applies beyond legal teams: any enterprise using LLMs to produce outputs referencing specific sources, whether contracts, regulatory filings, or compliance documents, faces identical exposure.

Key Takeaway: AI hallucination in high-stakes documents is a workflow problem, not just a model problem. If your review process relies on verbal instructions or general proofreading to catch fabricated citations, it will eventually fail under pressure.

3. Trustible Spotlight: How to Evaluate AI Vendor Risk

A vendor gets onboarded for document storage. Three years later, they shipped AI summarization in a release note. There’s no new contract, procurement event, or re-review. That’s one of eight specific ways AI breaks the mental model third-party risk management was built on.

Customer data trains AI models, populates retrieval indexes, and feeds feedback loops that improve the product. Outputs themselves carry risk in ways software defects historically did not. Even vendors with no AI features can introduce AI risk through the interfaces they expose to external agents.

Our new whitepaper works through what AI changes about third-party risk, the five categories of vendor risk, and why onboarding questionnaires aren’t sufficient on their own. Download here.

4. Policy Round Up

EU AI Act High-Risk Designations. The European Commission released draft guidelines on how to determine whether an AI system qualifies as high-risk under the EU AI Act and is open for comment through June 23. The three-part guidance covers general classification principles plus clarifications for Annex I (AI that is a safety component in regulated products) and Annex III (AI in sensitive domains like employment, biometrics, and law enforcement). These documents do not create new rules, but instead provide insights into how the Commission is interpreting their ruling in the EU AI Act.

Our Take: This guidance, in tandem with the recently released transparency obligation guidelines, is a useful starting point for scoping if your use cases meet a high-risk designation under the EU AI Act. However, don’t treat the draft as settled; keep an eye on the final version and the updated compliance deadlines now pushed to late 2027 and 2028 under the Digital Omnibus.

White House Pauses Executive Order. The White House postponed a planned executive order that would have created a voluntary framework for frontier AI developers to give the federal government up to 90 days of pre-release access to test advanced models for security vulnerabilities. Trump postponed the Thursday signing ceremony hours before it was set to begin, saying he “didn’t like certain aspects” and didn’t want anything that could slow U.S. progress against China. A leaked draft of the executive order laid out a voluntary framework that would direct certain federal agencies to develop and run a classified benchmarking process to test “covered frontier models.”

Our Take: Even if signed, the practical impact would have been limited. Major labs already participate in voluntary testing through NIST’s Center for AI Standards and Innovation, and the voluntary structure would have added little beyond standardizing existing arrangements. The postponement mostly signals continued internal friction over how much AI oversight is too much.

USDA AI Governance Report. The USDA Office of Inspector General found that 73 of the agency’s 82 active AI use cases lack Authorizations to Operate (ATOs), the federal cybersecurity certification required before deploying systems on government networks. The OIG warned the agency could be exposed to data breaches from systems without the required security controls.

Our Take: USDA is almost certainly not alone here. The ATO process is notoriously slow (although more pathways are opening up to speed this up for AI use cases), and agencies facing pressure to deploy AI quickly may be accumulating similar backlogs across the federal government. The numbers put a concrete figure on a problem that can be easy to ignore in the abstract.

In case you missed it:

• Singapore Updates Its Agentic AI Framework. Singapore’s IMDA released v1.5 of its Model AI Governance Framework for Agentic AI, incorporating feedback from over 60 organizations including AWS, Google, and Salesforce. The update includes new guidance on multi-agent systems, third-party agent risk, and automation bias as well as 10 case studies on how to utilize the MGF. It remains voluntary, but is still one of the most practically detailed agentic AI governance guidance published by any national authority.

• California EO on AI and the Workforce. Governor Newsom signed an executive order directing state agencies to study AI’s economic impact on workers and develop policies to address potential displacement, including updates to the California WARN Act, reskilling and worker training opportunities, and tracking hiring and payroll trends to get ahead of possible AI-layoff related disruptions. The order is directional rather than prescriptive for now, but signals that workforce disruption is moving up the policy agenda in ways that could eventually reach private employers.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. AI Disclosure Best Practices

Many AI frameworks and laws focus heavily on disclosing AI use to consumers in various forms, including ensuring interactive systems like chatbots are marked as AI, disclosing AI generated content, or disclosing that AI will be used to make consequential decisions. Disclosures are often a clear legal way of transferring liability and risk to the consumer of systems and are prolific in the medical, financial, and privacy spaces, albeit with many challenges that regulators want to prevent in the AI space. The EU AI Act has a specific category for disclosures in Article 50, requiring certain types of AI systems or content to be clearly marked as such. The EU AI Office recently published its draft guidelines for these disclosures. The draft document does a good job outlining what best practices should be, what is in/out of scope, and how to handle some grey areas such as freedom of speech implications when creating deep fakes of politicians. Here are a few insights from the document that stood out to us:

• Perhaps to the chagrin of legal teams everywhere, disclosure buried in terms of service doesn’t satisfy Article 50. It must be clear, distinguishable, and delivered at or before first interaction and not in documentation users are unlikely to read.

• For long or emotionally sensitive interactions (think AI companions or mental health chatbots), a single upfront disclosure isn’t enough. Periodic reminders are expected throughout the session or every time they return to a session after a break.

• Disclosure timing is highly relevant. Disclosures at the ‘end’ of a session, or after certain interactions have already taken place are not appropriate; ideally disclosures happen before a human starts interacting with an AI system.

• The intended end user is relevant to the disclosure format. A tool intended for developers that is marketed heavily as a dedicated AI tool (ex: Claude Code) is considered ‘obviously’ an AI system, whereas an AI enabled toy for kids needs to be much more explicit about its use of AI.

• Generated content needs to have both human readable disclosures, and machine readable watermarking metadata based on the best technological options available. This is a space that will likely change quickly over time.

Key Takeaway: Many AI providers are upfront about AI use and already following the ‘spirit’ of the law when it comes to these frameworks. Others however will likely test the ‘letter’ of the law and try to interpret disclosure broadly for a variety of reasons. We expect the best practices for AI disclosures to be a heavily litigated area, and one that will evolve as people themselves get used to interacting with AI systems.

2. Tech Explainer: Risks of Open-Source skills

Skills have emerged as the dominant paradigm for instilling new behaviors into AI agents. Over the past few months, a diverse ecosystem of open-source skills has taken shape, giving developers and desktop agent users (e.g., Claude Cowork or OpenClaw) a one-click method for extending agent capabilities. This convenience comes with compounding security risks. An analysis from February found that 13.4% of skills available from ClawHub, a major open-source skill registry, contained at least one critical security vulnerability. Vulnerabilities included prompt injections embedded in skill instruction text and dangerous executable code reachable via tool calls. Skill scanners have since been developed to flag common issues, but coverage remains incomplete: a new investigation showed that malicious code could be inserted into special testing files. These files fall outside of existing scanner coverage but can be run inadvertently by developers locally.

Open-source supply chain vulnerabilities are not a new category of concern, but agentic skills present a distinct risk profile. Compared to traditional coding packages that often run in an isolated process, skills are typically run by agents that have full read and write access. This gives the attacker access to private data (SSH keys, API credentials, browser data) and the ability to communicate externally. In addition, the skill ecosystem is new enough that the full range of attack vectors is not yet well characterized. Finally, many organizations have frameworks for both restricting applications that can be installed on a computer and for doing security review on application code, but skills exist outside of that structure and are not audited in the same manner.

Key Takeaway: Organizations should start by treating skill installation with the same scrutiny as third-party software procurement, requiring explicit permission review and maintaining an inventory of deployed skills. At minimum, agent runtimes should run in sandboxed environments, and audit logging for file access, network calls, and shell commands should be a baseline control. The newly published OWASP Agentic Skills Top 10 provides a practical framework for security teams beginning this work.

3. Incident Spotlight - Nine Seconds to Production Outage (Incident 1469)

What Happened

In late April, PocketOS, a B2B platform serving car rental businesses, lost its production database and all backups in a single automated action. A Cursor AI coding agent running Claude Opus 4.6 was assigned to a routine task in a staging environment, hit a credential error, and decided on its own to resolve it by deleting what it assumed was a staging database. It was not. The agent had access to a broadly scoped API token covering the entire production infrastructure, and in nine seconds it wiped out months of operational data.

Why It Matters

The real governance question is who is accountable when a vendor-recommended configuration causes irreversible harm. PocketOS was running the best available model through the most-marketed AI coding tool, configured per vendor guidance. The guardrails broke down anyway, likely because long-context agentic sessions accumulate state in ways that erode early constraints: the agent was reasoning about a staging task while holding a token with production-wide authority, and nothing reconciled those two facts before the destructive call. AI coding tools generally do not surface token scope or context-window state to users, a product decision that keeps complexity out of sight.

How to Mitigate

Never give AI agents direct production access, and treat any agent token with production-level authority the way you would treat an unsupervised contractor with root access. Issue tokens scoped only to the task at hand, maintain hard separation between staging and production credentials, and require human confirmation before any destructive or irreversible operation. Beyond access controls, assume AI agents cannot predict or undo downstream infrastructure consequences: they can read an API spec and generate a valid call; they cannot model what that call does to a live system. Off-site backups, isolated from the same API surface the agent can reach, should be seriously considered.

Key Takeaway: AI agents can read your code and call your APIs. They cannot reliably predict what those calls will do to production infrastructure, and they cannot roll back what they have already done. Until vendors provide real visibility into token scope and context state, the only reliable control is ensuring agents never have the access required to cause this kind of damage.

4. Trustible Spotlight: AI Model Risk Assessment

We shipped something new this month, and it addresses one of the most common frustrations we hear from governance teams: evaluating third-party AI models.

The Model Risk Assessment gives your team a structured, repeatable way to evaluate models before you build on them. 37 questions across five risk categories — Training Data Transparency, Model Architecture, Evaluation Coverage, Legal, and Provider Practices — with category-level scores that trace back to exactly what a provider disclosed, or didn’t.

Every score is driven by the same attributes-and-rules engine used across the rest of the platform, so the logic is transparent, configurable, and auditable. And because scores are category-level rather than a single aggregate, a strong evaluation result never masks a gap in training data transparency. Read more on our blog.

5. Policy Round Up

1. Pennsylvania: Pennsylvania is suing Character AI, a chatbot where users can interact with character personas. Pennsylvania officials claim that the platform pretended to be a licensed psychiatrist, even providing a false PA license number, while they were conducting an investigation into the platform. The company claims that they have taken the appropriate steps to notify users that they are engaging with AI, and that the conversations should be “treated as fiction.”

   1. Our take: This lawsuit hits at the heart of many of the chatbot regulations we are seeing at the state and federal level to protect users from deception and manipulation, especially minors. With the Pennsylvania Senate moving forward a bill to regulate AI chatbots and this lawsuit, it is clear the state is serious about regulating harms from companion chatbots.

2. EU AI Act: The EU has officially delayed the enforcement of obligations for high-risk classification systems. The new enforcement date will be December 2, 2027 for standalone high risk systems and August 2, 2028 for high risk systems embedded into other products (e.g. medical devices). In addition to the time delays, they included two agreements: the exclusion of AI in machinery that is already subject to other sectoral rules from the EU AI Act requirements and to ban the use of AI to create nonconsensual sexually explicit content. The EU Commission also released draft guidance for AI systems subject to transparency obligations per the EU AI Act which provides additional definitions and expectations for meeting compliance with the obligations.

1. Our take: While delays give organizations additional time to prepare, it does not mean organizations should pause governance efforts nor does it alter the requirements for high-risk use cases. Organizations should use this time proactively to identify and track high-risk use cases and build compliance workflows for their high-risk applications to stay ahead of EU AI Act obligations.

3. Leading Labs form Services Companies: Anthropic and OpenAI have announced their respective organizations, in collaboration with top consulting and financial services firms, will be starting AI services companies. The goal is to help organizations implement and scale AI solutions. In both new companies, engineers will be working with the client businesses to tailor their AI systems to specific business needs.

   1. Our take: These announcements illustrate the demand problem of organizations wanting to rapidly integrate AI into their businesses, but lacking the resources or capacity to do so. It also highlights the possibility that jobs for building, integrating, and maintaining AI systems could offset some of the job loss due to automation.

4. In case you missed it:

   1. Maryland: Maryland passed the first ban in the US on dynamic and surveillance pricing, a practice supported by artificial intelligence. Dynamic and surveillance pricing utilizes customer data to optimize prices for companies such as grocery stores. If the system can use data to determine income levels of two different consumers, they could adjust prices so that the two individuals could pay different prices for the same product. Maryland’s ban would prevent this practice from occurring in the state.

   2. CAISI: The Center for AI Safety and Innovation (CAISI) announced a partnership with Google, Microsoft, and xAI to allow CAISI to do national security predeployment testing of new models from these labs. OpenAI and Anthropic already have similar agreements.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. Trustible’s Take: The Mythos Verification Problem

Anthropic’s release of Claude Mythos Preview earlier this month has been a marketing triumph. Anthropic isn’t releasing the model broadly because, it claims, the model is too dangerous; access instead runs through “Project Glasswing,” a consortium that includes Amazon Web Services, Apple, Google, JPMorganChase, Microsoft, NVIDIA, and the Linux Foundation, with a $100M commitment in API credits across launch partners and 40+ additional organizations. Mozilla used early access to patch over 270 Firefox vulnerabilities. The system card disclosed that during internal testing, the model broke out of a sandbox and emailed a researcher to announce its escape. The reaction has been everything Anthropic could have hoped for, including the New York Times calling it a “terrifying warning sign,” central bankers in three countries convening emergency meetings, and a thaw with the White House after months of fighting the Pentagon’s “supply chain risk” designation in court.

Some of the underlying capability is real. Mythos appears genuinely better than prior models at chaining exploits end-to-end, and the UK AI Security Institute’s evaluation found it the first model to fully complete a 32-step simulated network attack. The marketing framing runs well ahead of the evidence, though. Anthropic chose which projects to scan (OpenBSD, FreeBSD NFS, FFmpeg, the Linux kernel, etc.), scoped the relevant code paths, and then reported the findings as autonomous discovery. When Semgrep and AISLE independently tested whether other models could reproduce those findings under similar scoping, they did. Open-source Qwen 3.6 matched Opus 4.6 on the OpenBSD bug. A 3.6B-parameter open-weights model recovered the FreeBSD vulnerability for $0.11 per million tokens. The much-hyped sandbox escape happened because a simulated user explicitly instructed the model to escape. And the headline number, “thousands of high-severity vulnerabilities,” extrapolates from just 198 manually reviewed cases. Mythos is faster and better at agentic exploitation, but the gap on the actual hard problem of novel discovery is narrower than the rollout implies.

The verification regime under which these claims are being made is the bigger story. Over 99% of the discovered vulnerabilities remain undisclosed. The only organizations with access are Glasswing partners who are being subsidized with API credits to validate the product they’re being asked to endorse. The system card is self-evaluated. Six weeks before the Mythos launch, Anthropic replaced its Responsible Scaling Policy with a v3.0 that dropped the hard commitment to pause deployment when capability thresholds are crossed, in favor of non-binding “Frontier Safety Roadmaps” and self-prepared “Risk Reports.” SaferAI and GovAI flagged at the time that the new RSP relies on Anthropic’s own argumentation rather than verifiable thresholds. Mythos is the first major release under that regime, and it is being marketed on the strength of safety claims that no one outside Anthropic and its paid partners can audit. The strategic returns have already accrued: a softened White House posture, a Treasury-convened bank summit, and Glasswing relationships across the Fortune 50.

Key Takeaway: It is entirely possible that Mythos is both a real capability jump and a carefully orchestrated PR exercise. What’s harder to dispute is that frontier safety claims are increasingly being made under conditions designed to make them difficult to independently verify, and the recent loosening of voluntary frameworks like Anthropic’s RSP doesn’t help. The strategic returns to Anthropic, including White House access, partner relationships, and a national-security narrative, accrue regardless of whether the technical claims hold up under scrutiny.

2. Tech Explainer: Understanding AI Effort Parameters

Every major AI provider now offers a way to control how hard a model “thinks” before answering: OpenAI calls it reasoning_effort, Google uses thinkingLevel, and Anthropic uses an effort parameter. The concept traces back to test-time compute scaling research showing that model accuracy improves with more thinking tokens — an alternative to scaling model size alone. While earlier API versions let users specify a maximum reasoning token budget, newer versions only allow a level (e.g. low/medium/high). In some cases, the parameter controls not just hidden reasoning but all token spend including tool calls (i.e. lower efforts means less tool calls).

Providers recommend different settings for different task types, but obscure how they map to actual compute allocation. Higher effort means deeper reasoning at greater cost and latency; lower effort means faster, cheaper responses that may cut corners on complex tasks. The interplay between model size and effort is not well studied: Anthropic recommends “medium” as the default for Sonnet 4.6 but “xhigh” for Opus 4.7 on coding tasks, suggesting larger models need more thinking room to fully express their advantage.

This simple parameter can have a large effect on system functionality. Earlier this year, many users reported a degradation in Claude Code’s performance and a recent postmortem revealed that the default effort had been silently changed from “high” to “medium.” The change was live for 34 days and was compounded by a caching bug that wiped reasoning history every turn and a system prompt that capped response length — none of which altered the underlying model weights.

Key Takeaway: When using closed-source models through APIs, developers trade convenience for transparency. The effort parameter deepens that trade-off — a single default change by the provider, invisible to users, can significantly alter system behavior. At the same time, effort provides a simple toggle from the governance perspective: increased effort can produce higher quality results, but comes at the price of latency and cost. This setting needs to carefully be considered as part of system design and review.

3. Incident Spotlight: When AI is used to draft the AI Policy (Incident 1467)

What Happened: South Africa recently released a draft form of their National AI Policy for public comment and feedback. The main feedback they got was: your citations are hallucinated. It turns out that at least 6 of the ~70 academic citations from the draft did not exist, as confirmed by the academic publishers. This led to the draft being revoked and a rewrite is in progress.

Why it Matters: Obviously it’s particularly ironic that AI was used to draft the national AI policy, and that its own writers, and several review committees missed the errors. The broader concern is use of AI in the policy making process in general, and particular issues with citations.

Use of AI by policy makers is quickly rising, with a recent Penta Group study suggesting up to 60% of lawmakers and their staff regularly use AI, and often do research with it. Lawmakers, and governments in general, are often in a unique position where they are allowed to cause particular types of ‘harm’ in the public interest, even while most people and AI systems are not. This creates a substantially higher standard for AI use, and it means that not all models or systems may be well equipped for use in the public sector.

Citations remain a weakness for many AI systems as they are ripe for hallucination and are tedious to manually check. Even the top law firms are still regularly getting caught for fictitious legal citations, leading to reputations, and occasionally financial repercussions.

How to Mitigate: One of the biggest challenges is that citation standards date from the pre-web era, and so many papers simply list author names, titles, and publication dates and don’t link to some authoritative and state URL online. In addition, many academic journals have strong licensing and access restrictions. Legal citations may use even less unique citation structure.

This is a recipe for disaster even with the best models as they don’t have an easy mechanism to verify certain citations. The best thing to do from a writing side is to require any citations and always use a web link, and those can be more easily verified. Obviously this will have a large impact on the publishers of work not easily accessible, and they will need to address that over time. Building in mandatory web links for any cited content, running a check that the source exists and is cited properly, and supporting an ecosystem for this is the best way to mitigate this for report writers.

4. CHAI Conference Recap

At last month’s CHAI Leadership Summit in Dana Point, Trustible CEO Gerald Kierce led a working session alongside governance and privacy leaders from Mass General Brigham, UT MD Anderson, and a large managed care provider. The room skipped the fundamentals. The conversation focused on what’s still broken. Two scenarios anchored the session: managing unauthorized agent activity, and measuring AI benefits in ways that actually inform decisions.

• The first scenario: an AI agent had been running autonomously in a clinical department for six weeks, scheduling appointments and routing care escalations, without ever going through intake. Most governance programs can’t detect that until after it’s happened, and vendor contracts that don’t address post-signature AI feature additions make it harder to manage.

• The second scenario: a CFO asking which of 35 AI initiatives in flight are actually delivering value. Governance programs that capture only risk at intake are leaving the most strategically important data on the table. Benefits belong in the same system as the risk assessment. Programs that can only see the risk side will systematically under-approve high-value use cases.

[See the presentation deck here.]

5. Policy Round Up

Colorado AI Act. Colorado is delaying enforcement of the Colorado AI Act until interpretive rulemaking is finished, or until a policy that could replace it is proposed. This comes after the US DOJ joined xAI’s lawsuit against the state’s law last week, arguing that it impacts first amendment speech and would be a heavy burden to comply with. The Colorado Governor announced in March that a new policy framework has been agreed upon and would replace the existing Colorado AI Act, however no legislation has been proposed yet.

Our take: Even if the new AI Policy Framework is passed before session ends in May, likely we are looking at another year or so before enforcement is pushed through.

Agents Under EU AI Act. Since the EU AI Act was passed, there has been a significant surge in AI agent development and adoption. The Act does not explicitly address AI agents, raising questions for providers about how to govern them for compliance. This recently released paper fills that gap by creating a taxonomy of agent use cases mapped to regulatory triggers, identifying AI agent-specific compliance challenges that aren’t entirely covered under any standards, and proposing a compliance architecture that integrates not only the EU AI Act but also 8 other related EU regulations.

Our take: Although not formal regulation, this paper provides a solid, comprehensive analysis of how AI providers should think about governing their AI agents per the EU AI Act.

New Model Risk Management Guidance. The OCC, FDIC, and Federal Reserve System released new guidance on Model Risk Management, replacing prior guidance. This guidance is meant to address an additional 15 years of advancements in modeling practices and industry feedback and create a comprehensive MRM guidance for banking institutions. One thing is notably left out: generative AI and agentic AI models. They acknowledge the rapid pace of innovation, and have decided to publish an RFI in the near future to gain additional insights for future guidance.

Our take: While the omission of GenAI and agentic AI may seem like a gap, we think it’s the right call. MRM looks at model performance, while AI governance looks at use cases for how and where that model is applied. Keeping them separate ensures model validation doesn’t substitute for the broader oversight that a use case-level view provides.

In case you missed it:

• South Africa. South Africa’s recently proposed draft national AI policy has been withdrawn after multiple fake sources were found in the reference list, more than likely AI generated. The Minister of Communications and Digital Technologies, Solly Malatsi, emphasized that this kind of mistake is “why vigilant human oversight over the use of artificial intelligence is critical.”

• UAE. The UAE Prime Minister announced that within two years, half of all federal government operations will be “run on Agentic AI”. Adoption of autonomous systems at scale to redesign how the federal government operates, if successful, would be a first of its kind accomplishment for the UAE who has been working towards centralized AI governance for nearly a decade.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

The Trustible Team

1. What the AI Audit Ecosystem Can Learn From the Delve Scandal

The cybersecurity world has spent the past few weeks reeling from a major scandal involving Delve, a YC-backed compliance startup that promised to get companies SOC 2 compliant in days using proprietary AI. SOC 2 is a general cybersecurity standard that most startups need before selling software to enterprises. According to a series of Substack posts, Delve didn’t have much in the way of AI, allegedly stole another company’s IP, and was auto-generating fraudulent SOC 2 reports through offshore firms. Delve has since been disowned by its investors, lost its most notable customers, and sparked an ongoing public debate about the “race to the bottom” in the SOC 2 world.

There’s plenty of AI directly involved in the Delve scandal, but there are also important lessons for the developing AI assurance and audit ecosystem. While many criticize SOC 2 as too light, consisting mostly of check-the-box activities, it can be a useful education for early-stage startups learning which basic security controls to put in place, and it’s often the first stepping stone towards heavier certifications. The real issue is less about the standard itself and more about the incentives surrounding it. The first problem is that SOC 2 lacks a strong auditor certification and enforcement ecosystem. It was created by the AICPA, a trade association of public accountants, originally to set standards for sharing confidential financial data with auditors, and has since been extended to cover SaaS platforms broadly. Unlike ISO, the AICPA does not formally certify and credential its auditors. The second problem sits on the demand side. Many enterprise procurement teams don’t understand how startups work and demand unqualified SOC 2 reports, even though the intent of the standard is to provide transparency about risks that can then be negotiated. Procurement teams will often use “findings” (auditor observations about control gaps) as an excuse to eliminate vendors rather than as a starting point for risk-based discussions. This creates intense market pressure for performative compliance over honest disclosure, and rewards bad actors over those being transparent. Given how quickly AI is evolving, any audit or assessment will have limitations. Businesses that start demanding “perfect” AI audits risk creating the same dangerous incentives, reducing the amount of meaningful risk management done for procured AI systems.

Key Takeaway: The incentives in an assurance ecosystem matter as much as the standards themselves. Right now, most risk information about AI isn’t being disclosed because teams worry that disclosures will be treated as admissions of liability or make their systems less attractive. Policymakers should think hard about how to make the opposite true, where transparent disclosure of risks and audit findings is rewarded, not punished, in the market.

2. Tech Explainer: Understanding Agent Memory

Memory is a core component of AI agents, but the term can refer to several different things. As agents increasingly operate across multiple sessions and workflows, how they store and retrieve information has direct implications for transparency, data rights, and security.

Short-term memory refers to the data passed to a model during a single interaction, typically conversation history, system prompts, and tool outputs. Developers may use summarization to condense previous interactions to fit a model’s context window, which can result in performance degradation if important details are not preserved in the summary.

Long-term memory refers to the use of external databases that track information across multiple interactions with an AI Agent. A simple form might include a database of previous conversations (episodic memory); a more complex form might include a knowledge base that summarizes information across sessions (semantic memory). For semantic memory, new records are often created through agentic processes that analyze previous conversations, meaning the agent is deciding what to remember. The agent interacts with the memories using tools, similarly to interactions with other external resources.

Depending on the nature of the store, it may be difficult to audit and manage the long-term memories. Deleting a specific chat from an episodic store may be straightforward, but summarized semantic knowledge is harder to disentangle. If two conversations contributed to a stored fact and one is deleted, the system may not be able to determine whether the fact should be forgotten.

Agent memories also present a vector for adversarial attacks. If a malicious actor gains access to the long-term memory database, they can plant bad data and execute indirect prompt injections that persist across sessions.

Key Takeaway: While memory can make agents more effective, it introduces new governance challenges. Users of external AI systems need to understand how their “memories” are managed, while developers need to account for the transparency, legal and security risks associated with long-term memory stores.

3. Incident Spotlight: When the Model Doesn’t Know What It Doesn’t Know (Incident 1446)

During KBS’s live YouTube broadcast of the Artemis II launch on April 1, the broadcaster’s AI real-time translation system rendered the mission control phrase “Roger, roll, pitch” as “Roger, roll, b*tch” in Korean subtitles. The error spread quickly on social media and KBS issued an apology the same day. The fix was straightforward: disable rewind, remove the clip, and commit to improving profanity filtering. Case closed, apparently.

But that resolution actually obscures the more interesting governance failure here.

Why It Matters: KBS’s response framed this as a profanity filtering problem, and their proposed fix, strengthening the profanity filter, treats it as one. The AI system mistranslated “pitch” as an English expletive, then rendered its Korean equivalent, because it failed to recognize the aerospace context of the communication. The underlying issue isn’t that a bad word slipped through a filter; it’s that the system had no representation of the domain it was operating in. Aviation and mission control communication is highly formalized, uses a specific vocabulary, and is nothing like the natural language corpus these models are typically trained on. Adding profanity filtering is a patch on a context problem. The next domain-specific failure, whether in a medical broadcast, a legal proceeding, or a financial earnings call, will produce a different kind of error that the patched filter won’t catch.

This incident also sits at the edge of a broader unresolved question: who bears accountability when an AI-generated output causes harm during a live, unedited broadcast? KBS worked with an unnamed external partner for the translation system, and its apology references “close consultation with relevant departments and external companies” to prevent recurrence. That language is telling. When the vendor relationship is opaque and the system is live, the contractual and editorial accountability structure is rarely established in advance.

How to Mitigate: The immediate lesson isn’t “add profanity filters”; it’s “don’t deploy general-purpose translation models in specialized domains without domain adaptation or human review gates.” For broadcasters and enterprises running real-time AI outputs in public-facing contexts, the risk controls should mirror those applied to any live content: a human in the loop capable of interrupting the stream, domain-specific fine-tuning or prompt configuration for the subject matter, and a vendor contract that clearly allocates responsibility for output errors. Some broadcasters using AI captioning tools have begun requiring vendor indemnification clauses for live content errors specifically. That’s the right instinct, though the contractual frameworks are still nascent.

Key Takeaway: Profanity filtering is not a substitute for domain-appropriate AI deployment. Any organization running AI-generated outputs in a live or real-time context, in any specialized domain, should establish both a human interrupt capability and clear vendor accountability before go-live, not after the first incident.

4. Trustible AI Governance Market Guide

The AI governance software market is crowded, confusing, and increasingly hard to navigate. Search “AI governance platform” and you’ll find hundreds of products sharing the same label: AI firewalls, privacy compliance tools, model monitoring services, cybersecurity GRC products with a new AI module. They’re all technically accurate descriptions, and they’re all describing fundamentally different products built for different teams solving different problems. That confusion leads to real procurement mistakes, organizations buying the wrong tool, or forcing a point solution into a coordination role it was never designed for.

To help cut through the noise, we put together a market guide that maps 16 distinct categories of platforms claiming some version of “AI governance,” organized by where they sit in the technology stack. For each, we describe what it actually does, who buys it, and where it falls short on the broader governance mandate. Whether you’re building a program from scratch, writing an RFP, or just trying to make sense of a vendor pitch that landed in your inbox, it’s designed to give you a clearer frame for evaluation. Read the full guide here.

5. Policy Round Up

Fannie Mae: Fannie Mae has released their first AI governance framework for the use of AI in mortgage lending. It includes guidelines for policies and procedures that sellers and servicers must abide by if utilizing AI/ML in the selling/servicing of Fannie Mae loans. It emphasizes transparency, risk management, and requires an owner of the AI use case to assume the responsibility of implementing and maintaining this framework.

• Our take: This follows in line with guidance released by Freddie Mac, one of the other major players in mortgage lending. While there may not be federal guidance, regulated industries are pushing forward with risk management practices.

OMB Compliance: Deadlines for compliance with high-impact AI risk management practices have recently passed. Several agencies also missed their deadlines for posting updated AI inventories, a crucial step in determining what cases are high-impact. Additionally, a requirement of OMB’s AI Acquisition guidance directs agencies to contribute to a repository of AI acquisition best practices. Monday’s new GAO report found agencies struggled with this due to a lack of centralized documentation and agency-level guidance.

• Our take: While the goal of the OMB guidelines is to streamline AI adoption, the requirements can actually be quite a heavy lift for agencies.

CA EO (3/30): Governor Newsom signed a new state Executive Order on the state’s procurement of AI services. It aims to provide a new process for the state’s AI procurement in effort to allow CA to separate from federal government’s processes in light of the Trump Administration’s recent “contracting missteps” (i.e. Anthropic). It gives the state the ability to do its own assessment of the policies and safeguards of AI companies and utilize the tool upon approval, even if this goes against the federal government’s supply chain risk designations.

• Our take: The Trump Administration has thus far expressed intent to allow states to have rights over their AI procurement processes, but it is unclear whether a state has the ability to override a national security designation. We can expect there to be more legality questions once the specific guidelines come out around August.

In case you missed it:

• China: The Chinese government has released Interim Measures for the Management of Anthropomorphic AI Interaction Service. It covers protections, especially for children, who are interacting with AI services that both 1) possess anthropomorphic features and 2) provide emotional interaction. It is a novel step forward in the already pretty rich well-developed Chinese AI regulatory landscape.

• xAI sues Colorado: xAI is suing the state of Colorado over their anti-discrimination AI law, set to begin enforcement in June. xAI argues that this would be a first amendment, free speech violation and would force them to “embed the State’s preferred views” into their systems.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. Sora’s Death Ushers in the Era of Enterprise AI

OpenAI recently announced that they were sunsetting Sora, their groundbreaking consumer oriented AI video app. Sora’s latest models proved capable of generating highly realistic videos with very simple text prompts. While Sora had a variety of guardrails in place, and enforced content watermarking, videos it generated were implicated in at least 19 incidents, and may have likely been used in many other deep fake related incidents. OpenAI cited a desire to focus more on enterprise applications, especially as competition from Anthropic and their breakout Claude Code system recently overtook OpenAI’s ChatGPT amongst new users and in the Apple App Store.

We think this is just the beginning of a strategic pivot of many AI companies away from consumer applications of AI, over to enterprise related uses. The consumer world is rife with legal issues, and regulators are starting to take notice. Even the most conservative AI legislative proposals in the US have strong protections against non-consensual sexual content, and age restrictions for many AI systems. In addition to regulatory pressures, businesses are finding Agentic AI to be more useful than simple chat-based interfaces, and Anthropic’s focus on coding, and on business safe AI has forced OpenAI to declare a ‘code red’ to stay competitive. Finally, the costs of AI are still being heavily subsidized, and geopolitical events are likely to exacerbate the situation making consumer oriented apps highly unprofitable in the short term. B2B opportunities will be seen as less risky from a legal and regulatory perspective, and will be easier to prove value and align the ‘real’ costs of AI to its enterprise value. Other OpenAI projects may get cancelled, such as their efforts of creating an ‘adult’ version of ChatGPT.

Key Takeaway: When it comes to choosing an AI partner in the B2B world, reputation and principles will matter. Ethics aside, choosing a model provider that invests heavily in preventing illicit content, simply reduces risk, and choosing that company isn’t performative ethics, but rather just risk reduction.

2. Tech Explainer: Promises and Pitfalls of Distillation

In a new partnership, Apple is using Gemini models to train effective smaller models that can be run directly on phones through a process called distillation. In addition to creating effective smaller models, the process can also be used by adversarial parties to reverse engineer models (Anthropic recently wrote a report on three chinese labs using this process to attack their models). The process works by having a smaller ‘student’ model learn to mimic the outputs of a larger ‘teacher’ model, rather than training from scratch on raw data. With LLMs, the smaller model is typically trained on input+reasoning traces from the larger models. This process works, because the reasoning traces give the model the most important pieces of information needed to function properly. The result is a smaller model that can run locally, without sending data to an external API, which can reduce latency and alleviate some privacy-concerns.

Distillation comes with risks. Because the smaller model has fewer parameters, it can’t capture as much nuance and may lead to performance degradation - in general, it may be best to use the smaller model for more specialized tasks. For example, Apple may only need Gemini to be good at question-answering inside Siri, so they can skip teaching the student model coding capabilities. Deployers need to run fresh evaluations calibrated to the distilled model’s narrower scope, not just port over benchmarks from the original. In addition, recent research showed that knowledge distillation may lead to systematic degradation of safety alignment and increase susceptibility to jailbreaks.

Key Takeaway: On the surface, distillation may be appealing because it can create smaller models that encapsulate key abilities of larger models without relying on external APIs. However, in practice, they require additional safety post-training, strong guardrails and new evaluations, which can make the whole process more time and resource intensive.

3. Incident Spotlight: DOGE’s ChatGPT Grant Review (Incident 1402)

What Happened: DOGE fed grant descriptions into ChatGPT, asking it to determine whether each was “DEI,” then logged the chatbot’s yes/no responses in a spreadsheet that replaced a list previously compiled by NEH staffers as the operative document for terminating grants. Of 1,163 grant proposals analyzed this way, 1,057 were flagged and just 42 were kept. The process was ad hoc by design: the DOGE staffer behind the methodology had assembled his own “Detection List” of identity-based traits before running grant descriptions through the model. Depositions later confirmed that the NEH’s acting chair hadn’t known ChatGPT was used in the selection process at all.

Why It Matters: Setting aside the political element, there are a number of issues here. Firstly, there’s no strong evidence that the team deploying ChatGPT had proper training on AI systems. Their prompt seems notably simplistic, and arguably gave enormous authority to the AI system to interpret DEI, and process the grants accordingly. Even small limitations in tools, such as poor text extraction, context window lengths, or hallucinations could have caused massive impacts. In addition, the DOGE team did not seem to have a firm grasp of the documents they were working with, and therefore could not act as qualified ‘humans in the loop’. The automated decision making nature of their request would likely be qualified as ‘high impact AI’ under the Trump admin’s latest guidance for AI use in non-classified settings, although this was published after the DOGE work was supposedly done.

How to Mitigate: This is fundamentally a process design failure. Any organization using AI for consequential screening decisions should define and document classification criteria before deployment, not derive them post hoc from a vague policy directive. AI-generated classifications should be treated as inputs to human review, not substitutes for it, with clear audit trails that distinguish model output from final decision.

Key Takeaway: If your organization is using AI to screen or classify anything with legal or financial consequence, someone with both domain expertise and working knowledge of the model’s limitations needs to own the classification logic. Deploying a general-purpose chatbot as a compliance filter without either is a massive liability.

4. IAPP Recap

Earlier this week at the IAPP Global Privacy Summit, we co-hosted a panel that focused on what AI governance looks like after Year One. The room assumed policies were written, intake processes established, and governance committees defined. The conversation was about what comes next.

CTO Andrew Gamino-Cheong moderated with Kimberly Zink (Chief Privacy Officer, Korn Ferry) and Derek Han (AI, Cyber and Privacy Partner, Grant Thornton). Four scenarios drove the discussion.

On Model Changes: Every use case should have a documented set of evaluations before a deprecation notice arrives, not assembled under pressure. What counts as a “substantial modification” needs to be defined in advance.

On Periodic Reviews: Governance intensity should scale with risk level, not apply uniformly. Model drift doesn’t announce itself. Sampling actual outputs against deployment guardrails matters more than a calendar reminder.

On Regulatory Updates: Nearly 7 in 10 businesses report difficulty understanding EU AI Act obligations. The root cause is inventory quality. If your AI inventory doesn’t capture use case category, PII usage, automated decision-making, and deployment geography, you can’t answer a scope question quickly.

On Program Iteration: Track the metrics that tell you whether governance is actually working: volume reviewed, high-risk flags, cycle time, risk mitigated. The harder conversation is agentic AI. Manual governance workflows weren’t built for systems that act autonomously, chain decisions, and scale faster than any review queue. Organizations need to start building AI-assisted governance now.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

(Source: Claude)

1. Why AI Monitoring is Hard

When enterprises say they want to monitor their AI systems, they rarely mean the same thing. For a security team, monitoring means watching for adversarial attacks and prompt injection. For a compliance officer, it means tracking regulations and court cases that could impact their AI systems. For a product team, it means making sure the model hasn’t quietly gotten worse at the thing it was deployed to do. A new report from NIST formalizes this fragmentation, organizing post-deployment monitoring into six distinct categories that rarely get discussed together:

• Functionality — Is the system still working as intended? (Detecting drift, staleness, performance degradation)

• Operational — Is the infrastructure running reliably? (Latency, uptime, logging across distributed systems)

• Human Factors — How are users actually interacting with the system? (Feedback loops, over-reliance, sycophancy)

• Security — Is the system being attacked or misused? (Adversarial inputs, deceptive model behavior, misuse detection)

• Compliance — Is the system adhering to relevant regulations and policies? (Terms of service violations, regulatory adherence)

• Large-Scale Impacts — Is the system promoting or degrading human well-being at a population level?

The report, drawn from three workshops with over 250 practitioners and a review of 87 papers, finds that most organizations are only monitoring one or two of these categories, and the field lacks agreed-upon methods, shared terminology, and basic consensus on who in the AI supply chain is even responsible for each. The incentive problems compound this: monitoring is expensive, publicly reporting incidents carries legal and competitive risk, and AI outputs are non-deterministic enough that establishing a reliable performance baseline is itself an open research problem. For deployers specifically, the report is a formal acknowledgment from NIST that the governance burden is shifting downstream, and the tools and standards needed to manage it don’t yet exist. The best practices for collecting and analyzing this kind of data are highly immature, and quickly shifting alongside the AI technology stack.

Trustible’s own AI Monitoring whitepaper, and blog post separates the ideas of ‘internal’ monitoring which focuses on analysing highly technical data about the relevant AI system, and ‘external’ monitoring which aims to collect information from outside the deployers boundaries. Internal monitoring largely maps to NIST’s Functionality, Operational, and Security factors while, ‘external’ monitoring maps to the Human Factors, Compliance, and Large Scale Impacts elements.

Key Takeaway: AI Monitoring is all of these things, and there’s therefore no ‘silver bullet’ solution for all forms of AI monitoring. The NIST framework is a useful forcing function for governance teams to audit which monitoring categories they’ve actually addressed and which they’ve implicitly assumed someone else owns.

2. Tech Explainer: The Trouble with Evaluations

As AI systems grow more advanced, evaluating them gets more complicated; at the same time, we are seeing a decreased consistency in how evaluations are reported for general-purpose AI models. Unlike traditional machine learning, where models are built for a specific purpose (e.g. predicting if an email is spam) and can be evaluated against that goal, GPAI models are adapted for downstream tasks and assessed benchmarks (e.g. math problem solving) that are often inconsistently applied. Two providers can report scores on the same benchmark while using different prompting strategies or answer aggregation techniques, making direct comparisons unreliable. A benchmark can also misrepresent what it claims to measure: interview-style coding questions won’t tell you much about how a model performs in a production system.

The core problem is a lack of standards for what to report. Trustible’s work with the EvalEval coalition produced a universal schema for documenting evaluation results, giving developers and deployers a consistent reporting standard and a way to compare why scores on the “same benchmark” diverge. Related efforts like BenchRisk, EvalFactSheets, and the Construct Validity Checklist give developers and deployers structured tools to report on and assess benchmark quality before trusting the results. Adoption is still limited, but these are now available reference points for model developers, deployers, users and policy-makers. The challenge will get harder as evaluations shift from models to agents, where the harness (i.e. tools and integrations surrounding a model) shape results as much as the model itself.

Key Takeaway: AI Literacy requires skeptically reviewing performance headlines touted by AI providers. While organizations should construct internal benchmarks when picking the best models for their systems, many of the same challenges persist and the frameworks shared can help.

3. AI Incident Spotlight - AI Style, Personality, and Identity Theft (Incident 1407)

What Happened: In late 2025, Grammarly launched an “Expert Review” tool where subscribers could upload writing and receive real-time editing feedback presented as coming from named journalists, authors, and academics, including novelist Stephen King and tech journalist Kara Swisher. Grammarly never sought or obtained consent from any of the named experts whose identities it used to sell the feature. In response, journalist Julia Angwin filed a lawsuit against Grammarly’s parent company Superhuman, and the feature was pulled shortly after.

Why It Matters: While the visual and auditory ‘likeness’ of a person has been discussed in depth in the context of AI generated images and video, this incident highlights an additional question of whether a person’s style and personality should also be included. In her lawsuit, and related NYT Op-Ed, Angwin argues: “My ability to earn a living rests on my ability to craft a phrase, to synthesize an idea, to make readers care about people and places they can only access through words on a page,”. For content creators, journalists, and subject matter experts whose reputations are themselves a professional asset, the commercial use of an AI simulation of their expertise is an existential threat to their way of life, and capable of causing massive reputational harm if the system is wrong. In this case, Grammarly is exposed from a liability angle, but the broader question about what constitutes ‘likeness’, and what rights someone should have to it are still unresolved. Existing privacy laws that may cover a person’s visual likeness under the context of ‘biometric’ data don’t clearly apply to a person’s ‘content style’.

How to Mitigate: Before shipping any feature that associates named individuals with AI-generated output, confirm you have explicit written consent or a licensing agreement. Some music and film artists have started to sell these rights to AI platforms. While AI laws targeting deep fakes are still being debated or implemented, existing tort law can still apply if there are reasonable claims of loss of income resulting from the AI. Right-of-publicity review should be a mandatory gate in the product development lifecycle for any feature involving real people’s names, styles, or personas, and that review needs to happen before engineering begins, not at launch.

Key Takeaway: There are many open questions about what constitutes a person’s likeness, and further debates about what kinds of rights a person should have around their likeness, and how to balance these issues with freedom of speech. Broader questions around likeness after death, or how to enforce these things on a global scale are unlikely to be resolved any time soon.

4. Policy Roundup

Anthropic vs. the Pentagon

The Trump administration officially labeled Anthropic a “supply-chain risk” and banned government agencies and military contractors from using Claude after contract negotiations broke down over two conditions Anthropic refused to drop. While they publicly threatened an aggressive stance on the issue, the formal notice was more limited in scope and only prohibits use of Claude to directly support DoD contracts. Despite the narrower designation, Anthropic filed two suits against the DOD, calling the actions “unprecedented and unlawful”. Major tech companies, including Microsoft, have voiced strong support for Anthropic.

Our Take: Anthropic may have ‘lost’ the battle, but in doing so, may be winning the war. At least PR wise, as consumer Claude downloads spiked as a result of the discussion. Organizations may need to ensure that they never get too deeply locked in with a single model provider in case LLM selections get further politicized over time. Organizations may also need to invest in processes for safely switching model providers.

EU AI Act Amendments Near the Finish Line

EU Parliament lawmakers reached a political deal within the Parliament that adds an explicit ban on AI-generated non-consensual intimate images and eases compliance rules for AI embedded in regulated products like medical devices. Separately, the EU Council approved its negotiating position on the Digital Omnibus, which would push back high-risk system deadlines by up to 16 months pending the availability of compliance standards.

Our Take: Now that the 3 main bodies of the EU policy making apparatus (Commission, Council, Parliament) have solidified their positions, political negotiations will begin between them. Given the common positions, enforcement of the high risk AI requirements of the EU AI Act are unlikely in 2026. Enforcement in late 2027, or early 2028 is now likely.

US Supreme Court Closes the Door on AI Copyright

The US Supreme Court declined to hear an appeal on Thaler v. Perlmutter, leaving intact the rule that works generated entirely by AI are ineligible for copyright protection. For now, Human authorship will continue to be a requirement for receiving IP protections in the US.

Our Take: Requiring human involvement to receive IP protections is a good balancing act in the AI ecosystem as it protects content creators in a fair way. However we expect more aggressive lobbying by tech firms on this issue over the next few years.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

(Source: Google Gemini)

1. AI Tools Are Making Employees Work More, Not Less

The sales pitch for enterprise AI adoption goes something like this: AI handles the tedious stuff, your employees focus on higher-value work, everyone’s happier and more productive. New research described in the Harvard Business Review suggests the reality is closer to the opposite. In an eight-month study of a 200-person tech company, researchers found that generative AI tools didn’t reduce workloads. They intensified them across three dimensions: employees expanded into tasks outside their roles (designers writing code, PMs debugging), work bled into breaks and off-hours as the low friction of “one more prompt” eroded boundaries, and constant multitasking across parallel AI workflows created persistent cognitive load. None of this was mandated. Workers did it voluntarily because AI made “doing more” feel accessible and even enjoyable.

For AI governance professionals, this matters because it complicates the risk calculus around enterprise AI deployment. The obvious governance concerns with AI tools, things like data leakage, IP exposure, and accuracy, are well understood. But work intensification introduces a quieter set of risks that most AI governance frameworks don’t account for. Employees operating outside their core competencies with AI assistance means more AI-generated or AI-assisted outputs flowing through an organization with less qualified review. The researcher’s finding that engineers spent increasing time correcting “vibe-coded” pull requests from non-engineering colleagues is a concrete example of how quality control can quietly degrade. Meanwhile, the burnout cycle the study describes, where early productivity gains give way to cognitive fatigue and lower decision quality, suggests that organizations measuring AI’s impact purely through short-term output metrics are likely overstating the long term benefits.

Key Takeaway: Reviewing AI outputs across multiple tasks may be cognitively burning teams out, and expectations for productivity and outcomes from executives are rising. AI use policies may need to adapt to include acceptable patterns for taking a break from AI, and understand the cognitive impacts of rapid context switching and constant reviewing of AI outputs.

2. Technical Insight: Prompt Repetition

Google researchers recently published a paper showing that simply copying and pasting a prompt twice into the input, with no other changes, consistently improves LLM accuracy across Gemini, GPT, Claude, and Deepseek models. The technique, called “prompt repetition,” won 47 out of 70 benchmark tests with zero losses, added no meaningful latency, and didn’t change the length or format of the model’s output. Because LLMs process tokens left to right, early tokens in a prompt can’t “see” later ones. Repeating the prompt gives every token a second pass where it can attend to the full context. There’s also a simpler intuition at play: repetition likely strengthens the internal representations of the input, effectively increasing the “weight” the model assigns to the prompt’s content relative to its prior training biases. The gains are modest on standard benchmarks but dramatic on tasks requiring attention to information buried in long inputs, exactly the kind of problem that plagues document-heavy enterprise workloads.

For governance teams, the more interesting implication is what this says about evaluation. If a trivial input transformation can meaningfully shift benchmark scores, it raises questions about how stable published model evaluations really are. Two organizations testing the same model with slightly different prompt formats could reach very different conclusions about its reliability. Prompt repetition works best when reasoning mode is off, which is how most enterprise API calls operate for classification, extraction, and structured output tasks, meaning it’s a real and essentially free improvement. But its bigger lesson is that small methodological choices in evaluation can have outsized effects on results.

Key Takeaway: Prompt repetition is worth testing for non-reasoning API workloads, but governance teams should treat it as a reminder that model evaluations are more fragile than published results suggest, and should account for prompt sensitivity when comparing models or setting performance thresholds.

3. Trustible Joins the Coalition for Health AI

We’ve partnered with the Coalition for Health AI (CHAI) to bring CHAI’s AI Governance Framework directly into the Trustible platform. Healthcare organizations can now map their AI governance activities to CHAI’s guidance with structured workflows, healthcare-specific risk assessments, and audit-ready documentation. For health systems trying to move from AI ambition to confident deployment, this removes the need to build governance practices from scratch or adapt generic frameworks that miss healthcare’s context. Read more about the partnership here.

4. AI Incident Spotlight - When AI Alerts Lack Sufficient Context (Incident 1374)

What Happened: A nurse at a Nevada hospital described an episode in which the facility’s AI sepsis alert system flagged an elderly patient with low blood pressure and triggered urgent protocol steps, including IV fluids. The nurse noticed the patient had a dialysis catheter, meaning her kidneys were already compromised. Pumping IV fluids into a patient who can’t process them risks dangerous fluid overload. When the nurse objected, he was told to proceed anyway because the AI had generated the alert. He refused, and a physician ultimately intervened with an alternative treatment that avoided the risk. The incident, reported by Scientific American in February, is part of a broader pattern of clinical AI systems generating recommendations that conflict with what’s observable at the bedside.

Why it Matters: The model was working as designed. It detected signals consistent with sepsis and triggered the correct protocol. The problem is that it had no way to know about the dialysis catheter, a piece of real-world physical context visible to any clinician in the room but absent from the electronic health record the model was reading. This is a recurring blind spot: clinical AI systems operate on structured digital data, but a significant share of relevant information only exists at the bedside. What the alert did next is the governance concern. It created institutional momentum. Protocol kicked in, and the nurse’s clinical objection was initially treated as non-compliance. The AI didn’t just inform the decision, it set the default, and overriding it required escalation.

How to Mitigate: Organizations deploying clinical AI alerts need workflows where alerts inform rather than direct. That means building override mechanisms that don’t require escalation and ensuring frontline staff have clear authority to act on evidence that contradicts a model’s output. This incident also highlights the limits of models that rely solely on digitized records. Physical context, like a dialysis catheter, is exactly the kind of information that’s hard for a model to ingest. Until that gap closes, human review is the only reliable way to catch what the system can’t see.

Key Takeaway: An AI system is only as good as its inputs, which includes any and all relevant context. Missing context for a system is one of the biggest potential sources of error and one that often can be missed in many of the ‘evaluations’ that are done to a system before deployment. Knowing what information an AI system can access, and what it’s input limits are is an essential part of governance.

5. Policy Roundup

DOD and Anthropic. Anthropic has been in a heated battle with the Department of Defense (DoD) over the use of their models. The DoD has put pressure on Anthropic to drop safeguards and allow for their models to be used from a wider range of military purposes.

Our Take: While AI safety has been prioritized under the Trump Administration, forcing a model provider to lower safeguards will have ripple effects across the ecosystem and may worsen the trust gap with AI.

Utah’s own RAISE Act. Lawmakers in Utah have been quietly working on passing a law that is very similar to California’s SB 53 and New York’s RAISE Act. The bill is currently making its way through the state house. One key difference is an added requirement for model providers to develop a child safety plan for their models.

Our Take: The Trump Administration has sought to deter state action on AI, but concerns with AI safety, as well as its impacts on jobs and children, have been relatively bipartisan.

India AI Summit. India recently held the latest global AI summit, which is a continuation of the work started in Paris last year and the UK in 2024. The focus was mainly on AI innovation and attendees scolded the EU for its prescriptive approach to AI oversight.

Our Take: The shift in global attitudes towards AI regulations has been swift and the focus of these global AI summits showcases that dramatic turnabout.

In case you missed it, here are a few additional AI policy developments making the rounds:

• Asia. The newly elected Japanese government will host a ministerial meeting to discuss how AI can be effectively utilized by the government. In Korea, the National AI Strategy Committee adopted a final AI Action Plan at its second plenary session.

• Australia. The Australian government is scrapping its AI Advisory Board after launching it 15 months ago. The Advisory was charged with setting out recommendations for AI safeguards.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. The Dangers of Desktop Agents

2. Trustible Partnership Announcements

3. Tech Explainer: Context Engineering

4. Incident Roundup: CASM Found in Major Image Training Dataset

5. Trustible’s Top AI Policy Stories

(Source: Google Gemini w/ Nano Banana)

1. The Dangers of Desktop Agents

Two similar desktop agentic AI apps have made headlines in the past few weeks. Anthropic’s Claude Cowork brings their Claude Code capabilities to non-developers via a sandboxed macOS app that can read, edit, and create files autonomously. OpenClaw (formerly Moltbot, formerly Clawdbot) takes a slightly different approach allowing users to interact through WhatsApp or Telegram and then connecting to 100+ services for arbitrary agentic flows. While these apps have desktop apps, they still usually interact with LLMs hosted in the cloud, as most consumer grade computers cannot yet host sufficiently powerful LLMs.

For most organizations, these tools will be difficult to greenlight anytime soon. Agentic browsers like OpenAI Atlas already present massive security and privacy risks, and these general purpose desktop apps take the risks a step further. The core problem is that an automated system acting on behalf of a human breaks assumptions baked into most security architectures. Access controls, audit logs, and anomaly detection are built around the idea that a human is on the other end. Agents blur that line in ways that aren’t easy to monitor or contain. They’re also vulnerable to hijacking via prompt injection. Cisco researchers have already demonstrated a malicious OpenClaw “skill” that exfiltrated data while bypassing safety guidelines entirely. Furthermore, as both tools run as desktop apps, they can still send your sensitive local files and context to hosted LLMs in the cloud where data may be permanently stored or processed. For companies with strict data handling policies, that’s a non-starter.

Key Takeaway: These tools are impressive and getting a lot of attention online, but they are far away from being enterprise-ready.Most organizations will need to wait for better sandboxing, clearer data handling policies, and security architectures that actually account for non-human actors. Many organizations will quickly move to block such desktop applications via existing device management tools.

2. Trustible Announcements

Here’s a quick recap of some major announcements by Trustible in the past few weeks

Trustible Announces Strategic Partnership with Leidos

We’ve partnered with Leidos to bring automated AI governance to government agencies. In a proof-of-concept engagement, Leidos used Trustible’s platform to compress governance intake processes from weeks to hours, demonstrating that automation can reduce friction while maintaining the oversight that mission-critical environments require. Read the full announcement here.

Trustible Partners with the AI Incident Database

Trustible is now the lead corporate sponsor for the AI Incident Database (AIID), the most widely used public repository of real-world AI harms. Through this partnership, Trustible customers will be able to cross-reference their AI inventories against documented incidents and receive alerts when new incidents relate to models or vendors they’re tracking. Read more about the announcement here.

Trustible Publishes Pragmatic AI Policy Paper

We’ve published A Pragmatic Blueprint for AI Regulation, a policy paper offering a middle-ground framework for AI governance built around shared liability, copyright balance, child protection, content provenance, and information sharing. The paper argues that closing the AI adoption gap requires trust, and trust requires clear rules without stifling innovation. We decided to take stances on a few core AI policy areas that are often ignored in the larger ‘doomer’ vs ‘optimist’ debates, and to advocate for regulation that businesses actually want. Check out the whitepaper here.

3. Tech Explainer: Context Engineering

In recent months, context engineering has replaced prompt engineering as the focus for building effective AI agents. While the focus of prompt engineering has been on developing techniques that make an LLM respond to a specific query effectively (e.g. telling the model to “think step by step” or giving it an example output), context engineering refers to the methodology of giving an agent the right set of information at the right time. This may be challenging because at any given point, an agent may have access to a broad range of assets like tools, internal memory, external data stores, and the conversation history; however, all of this information still needs to be processed by an LLM that can process a limited number of tokens at a time. While leading frontier models can process 250k words at a time, they may not digest and recall all the tokens properly. Common context management techniques include summarization (where a separate LLM is used to condense the context), sub-agents (where each agent only needs partial context) and the use of memory (where an agent adds information to an outside store that can be referenced as necessary).

While it enables agents to perform more complex tasks, the use of memory, in particular, introduces new governance concerns. A recent study by the Center for Democracy and Technology identified that users’ key concerns around memory include:

• Persistence: Who has control over when and how memories are deleted?

• Privacy: Can memories inadvertently be shared with additional tools/systems?

• Transparency: Can a user review all the memories associated with them?

There are not yet well established best practices for managing this, and many regulations and risk frameworks don’t have specific considerations for context or memory governance.

Key Take-away: Organizations deploying AI agents will need to develop new practices that bridge traditional data governance with the dynamic nature of context management. This includes defining clear policies for memory lifecycles, implementing context boundaries between different use cases, and ensuring users maintain meaningful control over their data.

4. AI Incident Spotlight: CSAM Found in Dataset Used to Train Content Moderation Tools (Incident 1349)

What Happened: The Canadian Centre for Child Protection (C3P) discovered that NudeNet, a dataset of over 700,000 images used to train AI nudity detection tools, contained approximately 680 images of suspected or confirmed child sexual abuse material (CASM). More than 120 depicted identified or known victims. The dataset had been freely available on Academic Torrents since 2019, and C3P identified over 250 academic works that either cited or used NudeNet or classifiers trained on it. Researchers who downloaded the dataset unknowingly possessed and distributed illegal material. Following C3P’s takedown notice, Academic Torrents removed the dataset, but the classifiers and models derived from it remain in circulation.

Why it Matters: This is the second major incident involving CSAM in AI training data following the 2023 discovery of similar material in LAION-5B. This highlights the ongoing problem with large-scale web scraping without rigorous vetting that captures illegal and harmful content. But the NudeNet case is particularly troubling because the dataset was specifically designed for content moderation. Tools built to detect harmful imagery were themselves trained on it.

Given NudeNet’s wide distribution and six-year availability, the contamination likely extends beyond academic research. Foundational models and commercial content moderation systems may have incorporated NudeNet or its derivatives without disclosure. Without transparency into training data provenance, downstream deployers inherit risks they cannot assess. Model cards rarely disclose specific dataset sources, let alone whether those sources were vetted for illegal content.

The incident also illustrates a difficult dual-use problem. Building effective detection systems for harmful content often requires training on examples of that content. But assembling such datasets creates its own harms as it perpetuates distribution of illegal material, re-victimizes survivors whose images are included, and exposes researchers to legal liability. The goal of protecting against abuse inadvertently extends it without proper controls in place.

How to Mitigate: Require training data documentation from model providers, particularly for content moderation systems. If a vendor cannot explain how their training data was sourced and vetted, treat that as a material risk factor. For organizations that must include toxic content in datasets for detection purposes, that data should only be handled under strict access controls, legal compliance frameworks, and coordination with organizations like C3P or NCMEC that maintain vetted hash databases for this purpose. The NudeNet dataset existed for six years before anyone flagged it. That’s a long time to be unknowingly distributing illegal content.

Policy Roundup

The Trouble with Trump’s AI Policy. There is a growing divide between the Trump Administration’s position on AI and Republican lawmakers. Republicans at the state and federal level are currently at odds with the Trump Administration’s “AI innovation” ethos, with Republicans pushing for more oversight of the technology (e.g., Senator Marsha Blackburn’s TRUMP AMERICA AI Act).

Our Take: Outside of the Executive Branch, Republicans have been more skeptical of AI and have sought some safeguards around the technology. This divide will set up an interesting clash with the Department of Justice as it seeks to evaluate the constitutionality of state laws under the Trump AI Moratorium Executive Order.

More Turmoil with the EU AI Act. Lawmakers in the EU continue to struggle with implementing the EU AI Act. The European Commission missed its deadline for publishing draft guidance for classifying high-risk AI systems, while France is pushing a behind-the-scenes effort to separate the AI Act amendments from the EU’s larger Digital Omnibus package.

Our Take: The continuing drama over the EU AI Act is making it difficult for companies to understand their obligations and deadlines under the law. It may also show that EU lawmakers tried to do too much in one law, with the consequences now on full display.

Singapore’s New Agentic Governance Framework. Singapore’s Infocomm Media Development Authority published the “Model AI Governance Framework for Agentic AI.” This framework is one of the first comprehensive governance frameworks for agents from a government entity.

Our Take: Policymakers are usually behind the curve on technology but Singapore has been an outlier on AI guidance, producing a host of practical AI-related materials.

In case you missed it, here are a few additional AI policy developments making the rounds:

• Africa. Egypt will be hosting the first AI Everything Middle East & Africa Summit, which is intended to emphasize the region’s focus on digital development.

• Asia. South Korea’s AI law came into effect last month, with a one year grace period for enforcement. The law has recently faced major pushback from the tech industry for going “too far” and from advocacy groups for not going far enough.

• North America. The Mexican government released a Declaration of Ethics and Good Practices for the Use and Development of AI. The declaration outlines ten fundamental principles that serve as “a non-binding guide for public institutions, government agencies, autonomous bodies, as well as actors from the private and social sectors.”

• South America. A fake version of Taylor Swift’s “Fate of Ophelia” is testing the limits of Brazil’s IP law. The song, “A Sina de Ofélia,” was generated by AI using voices from two well-known Brazilian pop artists.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, and welcome to the latest edition of the Trustible AI Newsletter! We’ve got a lot of exciting news to share in the next few weeks, but for now, be sure to download our latest whitepaper on AI monitoring! Here’s our team’s latest insights:

1. Why OpenAI and Anthropic Built Dedicated Health Applications

2. How to Evaluate Healthcare AI

3. Grok’s Generation of Non-consensual Intimate Imagery

4. Trustible’s Top AI Policy Stories

1. Why OpenAI and Anthropic Built Dedicated Health Applications

(Source: Google Gemini)

Within a few days of each other, both OpenAI and Anthropic announced dedicated ‘health’ versions of their AI platforms, ChatGPT Health, and Claude for Healthcare. According to their own data, over 230 million health related requests are made per week on ChatGPT, and it’s one of the top use cases. The dedicated health applications will have specific connectors for healthcare related databases, integrations with fitness and wellness platforms, and health related questions will now be ‘routed’ to the dedicated healthcare application instead of being answered.

There are several likely motivations behind this split: opening new revenue streams, competing against health-specific wrapper companies, and a desire for more data. But we’ll focus on regulatory and compliance motivations. Under many AI related laws such as the EU AI Act, an AI service providing medical advice would be categorized as ‘high risk’, triggering a number of heavy compliance obligations. Many existing privacy, liability, and security laws surrounding health data also apply. By carving out health related uses into a dedicated app, both companies can build dedicated guardrails, infrastructure, and processes around the sensitive application. This allows their non-health applications to still innovate quickly without getting bogged down by compliance. And by putting significant effort into routing users to a ‘safer’ application for health related queries, these companies will be able to claim that the non-health version of their applications are not ‘intended’ for this high risk domain and are not marketed as such. This is actually not the first carve-out, as both companies already support a similarly dedicated platform for the US public sector. Much like with healthcare, these platforms have dedicated infrastructure, customized security and privacy controls, and a different set of guardrails.

Key Takeaway: Regulation has always shaped product architecture, and we should expect big AI companies to create dedicated platforms for each high-risk domain these laws identify.

2. Tech Explainer: How to Evaluate Healthcare AI

Given the recently announced healthcare applications from OpenAI and Anthropic, alongside Utah’s new pilot program, it’s worth digging into how these systems were evaluated. Obviously since there are bespoke capabilities, guardrails, and integrations in these applications, they require customized testing, and there is already a growing ecosystem of healthcare related benchmarks and evaluations, albeit with many limitations. Here’s a quick analysis from the information released so far:

ChatGPT Health

OpenAI evaluated using HealthBench, a benchmark of realistic healthcare conversations with physician-created rubrics. Key strengths of his benchmark are:

• Multi-turn conversations (superior to single-turn evals as quality often degrades over time)

• Multi-faceted rubrics evaluating medical accuracy, communication quality, and jargon avoidance, scored via LLM-as-a-judge

• Development by 262 physicians from 60 countries, improving cross-cultural validity

However, the benchmark doesn’t fully account for varied input formats from healthcare record providers (these integrations are central to the platform). Performance for “ChatGPT Health” was not published; but gpt-5-thinking scored 67.2%. It is unclear how this score translates to real world outcomes.

Claude for Healthcare

Anthropic’s announcement covered two functionalities: supporting healthcare professionals with prior authorizations and care coordination, and helping individuals summarize medical history and prepare for appointments. They reported Claude-4.5-Opus performance on MedAgentBench, which assesses agent capabilities in medical records contexts with pre-defined tools. While co-developed by physicians, it’s a proxy metric, and Claude’s actual tools differ from the sandbox environment. No evaluations addressed the personal healthcare scenario.

Utah Auto-Refill Program

Utah’s automated refill pilot uses Doctronic’s algorithm, which showed 99.2% agreement with physician decisions in testing. Unlike the broader AI systems, this narrowly scoped use case allows direct performance measurement on the actual task. However, testing used urgent care cases with physician-entered data, which may differ from patient chatbot inputs in production.

Our Take: Benchmarks are a useful evaluation tool, but they imperfectly simulate real-world conditions because they imperfectly capture aspects of real-world conditions like external data formats and real agentic tools. In addition, for the open-ended consumer products, it is not clear how these scores will translate to improved clinical outcomes. While the current systems have been released with a number of safeguards, better reporting standards will be necessary as AI Healthcare tools become more commonplace.

3. AI Incident Spotlight: Grok’s Generation of Non-Consensual Intimate Imagery (Incident 1329)

What Happened: In late December 2025, users discovered that xAI’s Grok would readily “undress” women in photos, manipulating existing images to create sexualized deepfakes without consent. The flood of content included images of celebrities, private individuals, and minors. Viral prompts ranged from “put her in a bikini” to far worse. Despite reports, X was slow to respond. Even one of Musk’s ex-partners struggled to get deepfakes of herself removed. After international backlash, X announced partial restrictions in mid-January, but the standalone Grok Imagine app continues generating explicit imagery.

Why it Matters: This isn’t a fringe product. Days after Secretary Hegseth announced that Grok would be integrated into Pentagon systems, including classified networks, regulators in at least a dozen countries launched investigations or outright bans. The same model generating what California’s Attorney General called an “avalanche” of illegal content is being deployed to 3 million DoD personnel.

The political dimension matters too. Even policymakers who oppose AI regulation have consistently carved out exceptions for child safety. This is one area with genuine bipartisan consensus, and incidents involving minors accelerate legislative action. By pushing boundaries on content moderation, xAI may be generating exactly the public backlash that fuels demand for stricter AI regulation across the board. Every headline about AI-generated sexual material erodes trust in AI broadly, not just in Grok.

How to Mitigate: Treat content moderation capabilities as a procurement criterion. Before deploying any image generation system, request documentation on what categories are blocked and how. For organizations considering Grok or X API integration, this incident warrants a serious risk assessment, particularly for customer-facing applications where generated content could create legal exposure.

4. AI Policy Roundup

Next Steps on AI Moratorium EO. The Department of Justice issued a memo establishing a task force to challenge state AI laws, as directed under the Trump AI Moratorium EO. The EO also calls for federal legislative proposals to regulate AI, but the director of the Office of Science and Technology Policy offered few details on the Administration’s plans at a recent congressional hearing.

Our Take: The EO spurred controversy even before it was signed because of the power it attempts to assert over states to regulate AI. It has not appeared to blunt momentum at the state level to pass AI laws, as several state legislatures have introduced bills in 2026.

ChatGPT’s Confidentiality Quagmire. Sam Altman recently asserted that OpenAI does not have an obligation to keep sensitive information confidential when people use ChatGPT as a therapist. Altman acknowledged that privacy concerns with AI may hinder adoption.

Our Take: Model providers are further blurring the lines between their products and privacy obligations. Health privacy laws like HIPAA and HITECH do not explicitly cover products like ChatGPT, but as the models expand into offering health services (e.g., digital therapy) that may change.

Congress Targets Deepfake Porn. Congress is considering bipartisan legislation that would allow victims to sue over nonconsensual sexual images. The DEFIANCE Act passed the Senate unanimously and will head to the House.

Our Take: Congress is responding to concerns over Grok’s capabilities to produce fake, sexually explicit deepfakes. This is one of the rare times that federal lawmakers agree on creating a private right of action, which allows individuals to sue. The law does not ban these images, though is thought of as a complement to the TAKE IT DOWN Act.

In case you missed it, here are a few additional AI policy developments making the rounds:

• Africa. Nigeria is working towards passing a comprehensive AI law, making it among one of the first countries in Africa to enact such a law. The law is primarily focused on safeguards for high-risk systems, and would allow regulators to demand information from providers for non-compliance. The law is expected to be enacted in March 2026.

• Asia. The Taiwanese legislature passed an AI basic law towards the end of 2025 and that law came into effect on January 14, 2026. The law outlines a series of principles for AI development and deployment, though has no specific enforcement mechanism.

• Europe. Regulators in the EU and UK are considering consequences for AI tools that can create sexually explicit images. The concerns come in the wake of the controversy over Grok’s ability to produce sexualized images. EU lawmakers are considering banning the technology altogether and the UK government is threatening to revoke xAI’s ability to self-regulate.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, Happy New Year, and welcome to the first 2026 edition of the Trustible AI Newsletter! 2025 proved to be a critical - but tumultuous - year in the world of AI, and we don’t anticipate that trend changing in 2026. But, as we navigate what’s to come across the technical, policy, and business landscape of AI, we do believe in one constant: that 2026 will be a transformative year for AI governance, as it becomes the primary business imperative that will drive how enterprises will actualize the positive ROI of AI.

In this week’s edition, we’re sharing our 2026 predictions across what’s in store for AI governance, AI technical trends, AI incident trends, and what’s around the corner in the policy and regulatory sphere.

Let’s dig in.

1. Trustible’s 2026 AI Governance Predictions

We aren’t alone in predicting that 2026 will be the “make or break” year for AI. There are a number of consequential questions that will likely be answered in 2026, including whether AI agents will be adopted at scale, whether major AI regulations in the EU and across U.S. states will actually come into force in their current form, and whether the AI bubble will burst, or continue to grow. These are monumental questions that policymakers and professional talking heads will continue to debate, but all of them also have implications for teams tackling AI governance.

Here are our predictions on AI governance for 2026:

• AI Governance Beyond Intake - Many organizations now have robust policies, fully populated inventories, and initial risk assessments. What comes next is a lot of change management for systems already in place. This work may look very different from earlier governance work.

• AI Agents Become Mainstream - At this point last year, even many AI practitioners may have never heard of an MCP server, or had reviewed a proposed agentic AI system. This year, many organizations have mandates to deploy agentic AI workflows, and a lot of people are hoping that agentic AI provides the value that chatbot copilots did not.

• Growing Third Party Risks - As agentic AI rolls out inside organizations, knowing which tools and platforms are connected to AI systems will become its own inventorying challenge, and source of risks. In addition, many vendors may deploy their own agents for work, introducing new potential risks for their customers to stay on top of.

• Pressure for AI ROI - After several years of high budget experimentation, many organizations are now looking for tangible ROI from their AI systems. AI vendors will be under a lot of pressure to show revenue, and so prices for AI tools are likely to increase at the same time that organizations are looking to focus their efforts more on high value AI. Calculating that value will be a major challenge and narrative going forward.

• AI Policy Moves ‘Up the Stack’ - 2025 had many AI policy proposals focused on the foundational model level, with the EU publishing their relevant Code of Practice for GPAI, and bills in California and New York being signed to regulate them. However, these types of regulations are being specifically targeted by the Trump administration, and trying to pass new ones is likely to face pushback. We think policymakers are more likely to target specific AI use cases or types of systems for further regulation, especially focused on protecting kids from AI, or regulating use of AI for mental health purposes.

You can read our full 2026 trends and prediction piece here.

2. Technical Deep Dive - 2026 Technical Look-Ahead

2025 brought a strong new generation of AI models from many providers, with an increased focus on training for reasoning and tool use. While some providers focused on creating increasingly large models, others continued to explore how smaller models trained on high-quality data can produce competitive results. We expect to see steady improvements across these areas, but for our 2026 predictions, we focus on the broader picture beyond just performance:

• World Models are AI models that aim to understand and model the physical world directly (in contrast to LLMs that are trained to predict the next word and as a byproduct encode some knowledge about the world). In 2025, there were some early developments in this space from Fei-Fei Li’s company World Labs and China’s Tencent; we expect to see some progress in this space, but these models are unlikely to overtake LLMs in usability and popularity in 2026, because they require large amounts of complex data that is not readily available.

• AI-Generated Videos will become impossible to distinguish from real videos; many examples from state of the art models, like Google’s Veo-3, already lack tell-tale “AI” signs (like background objects that move in unrealistic ways). However, generating longer videos (> 30 seconds) may remain difficult because of challenges with maintaining character and scene consistency.

• New Year, Same Risks: In late 2025, adversarial poetry, a new jailbreaking technique, was able to overcome the defences of a large number of popular LLMs. At the same time, hallucination rates remained high on multiple benchmarks. We do not expect either problem to be “solved” in 2026, these risks are inherent to LLMs because LLMs are trained to generate text, not recognize factuality or the potential danger of the generated content.

• Model Transparency will continue to play an uncertain role in adoption and trust. While transparency decreased overall in 2025, according to the Stanford Transparency Index, AI adoption increased. In the open model space, Qwen models, whose providers disclose little information about the training data, were top downloads from Hugging Face, while the highly transparent OLMO models did not receive much attention. In addition, new nuances have emerged around “transparency”: popular LLM providers release increasingly long System Cards, but many of the evaluation results now rely on LLM-as-a-Judge style automated evaluations which introduce biases and a new layer of complexity.

None of these predictions point to a single dominant shift in 2026, but some of the most exciting developments may come from the non-LLM side of AI. More broadly, if 2025 was the year of AI pilots and experiments, 2026 will be the year of transforming them into hardened production-ready systems. Meanwhile the continued uncertainty around risks and transparency, points to a need for increased education around AI risks and evaluations.

3. AI Incident Spotlight - 2025 AI Incident Recap & 2026 Predictions

The AI Incident Database catalogued 345 distinct incidents in 2025, a record high. Our analysis of these incidents shows 3 major trends in 2025:

• Deep Fake Scams - The majority of incidents in 2025 involved some form of scams, often involving deepfakes. These included using AI tools to generate flashy ‘phishing’ websites, using AI to create a massive web footprint for a fraudulent company, and many instances of scammers exploiting fake videos of celebrities claiming to love the scam’s target (Incident 901, Incident 1126, Incident 1185).

• Chatbots & Mental Health - Unfortunately there were many incidents involving deaths associated with chatbot use. These included instances of chatbots providing recommendations on how to tie a better noose, affirming a man’s belief that his mother was plotting to kill him leading to murder-suicide, and a teenager being induced to commit suicide by a fake AI-powered Game of Thrones character. In a sign of how bad things have gotten, Wikipedia now has a dedicated page linked to them. According to OpenAI’s own data, using AI for chats about mental health issues is one of the top user use cases for ChatGPT.

• Early ‘Agentic’ Incidents - 2025 saw some of the first incidents directly linked to AI agents, including a Replit agent deleting a production database, Claude Code’s agent mode autonomously conducting a cyber attack, and Wall Street Journal reporters successfully jailbreaking an AI powered vending machine. These were attained not with direct improvements in the AI models themselves, but rather by connecting models with a lot of different tools capable of performing actions.

Here are a few of our predictions for AI incidents in 2026:

• Agentic AI - There were only a few incidents directly linked to AI agents in 2025, but as agentic AI gets more adoption, we expect to see many incidents linked to agents. There are a few good reasons to suspect this, including the poor security status of many MCP servers and the relatively early maturity on how to evaluate or red-team agents.

• AI in Healthcare - It’s estimated that over two-thirds of medical practitioners are now using AI tools in their jobs, with notes transcription being the leading use case. However the impact of AI errors may take time to resolve. We expect to see some incidents linked to errors produced especially by earlier model versions that haven’t yet been acted upon.

• AI Videos - The quality of AI generated videos from tools like OpenAI Sora-2, or Google’s Nano Banana, is truly impressive, and many videos will become increasingly difficult for many people to identify as AI generated. We expect more scams and misinformation incidents specifically linked to hyper-realistic looking videos.

4. Trustible’s 2026 Policy Predictions

AI policy in 2025 was a roller coaster of new developments domestically and globally. The new Trump Administration upended AI safety work from the Biden Administration, the EU squabbled over whether to delay the AI Act (which it ultimately did), and governments at every level moved ahead with their own AI rules. Here are our top three thoughts on what to expect in AI policy in 2026:

• AI Lawsuits Will Test Regulatory Limits. Last year we tracked various lawsuits related to AI harms, from companion bot-related deaths to copyright infringement. We do not expect those battles to fade away, but a new one is about to heat up. States have been passing AI laws and those are in the crosshairs for new legal fights, as well the litigation coming from President Trump’s AI moratorium Executive Order (EO). These new legal fights will forge a new path on old laws and rights as they apply to AI rules.

• Turnaround for the AI Trust Gap. The past year saw another dip in AI trust among the general public even though adoption was on the rise. A large part of the distrust stems from a lack of regulation, which some studies show would help ease concerns over the technology. As some countries start to implement AI rules and safeguards, do not be surprised if there is a (slight) uptick on AI trust.

• AI Innovation Hits a Roadblock in the US. The second Trump Administration started with a bang for AI innovation, effectively undercutting any efforts that could burden the American AI ecosystem. Expect that line of thought to shift ever so slightly in 2026, as the Trump Administration grapples with challenges that AI presents to national security and critical infrastructure. The AI moratorium EO acknowledges the need for a federal AI framework, which is a marked shift from where the Administration stood last January. Expect further guidance on AI security and resiliency, as well as guidelines for certain industries (NIST recently announced a new workstream for AI and advanced manufacturing).

Overall, this year will blend “more of the same” with some new challenges. We expect US states to continue regulating AI, even as the federal government tries to clamp down on the AI legal patchwork. We also expect to see more interest in agentic AI, though regulatory frameworks are still a few years away. What we see this year is an opportunity to clarify some legal uncertainty, while also increasing a push for basic AI governance to help address safety and security concerns.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, and welcome back to the Trustible AI Newsletter! The holiday season is upon us, and it appears the White House is poised to gift states a new executive order this week to revive the federal AI moratorium. We’ll share more thoughts if and when the order drops, but at a minimum, expect a busy few months in the courts as the questions this order raises will almost certainly be decided in front of a judge. In the meantime, in this week’s edition:

1. Will AI Make Black Friday Become Black Mirror?

2. Technical Deep Dive - A Poet’s Key to Model Hacking

3. AI Incident Spotlight - Deloitte Publishes Citation Hallucination in Government Sponsored Report (Incident 1286)

4. Trustible’s Top AI Policy Stories

1. Will AI Make Black Friday Become Black Mirror?

What effect could AI agents have on pricing? E-commerce sites have long battled bots and scalpers that buy up limited goods for resale, with mixed success. Platforms like Ticketmaster have become notorious for profiting off secondary markets, resulting in recent actions to curb secondary market activity. For many consumers, it feels like more goods than ever (from vintage clothes, to Pokemon cards, and Lego sets) are dominated by resellers.

The new wave of AI agents risk making this problem worse. Old school bots were often simple web ‘scrapers’ that knew how to click specific buttons in order, and could be thwarted by certain types of activity blockers. AI agents can combine the reasoning powers of LLMs, with enhanced tool calling capabilities, to become more sophisticated, and are simpler to set up and deploy at scale. There are a slew of start-ups already targeting how to train dedicated agents to do this by creating fully sandboxed digital replicas of websites like Amazon.

It’s worth considering what impacts this may have on prices, and the broader global economy. Many limited goods with resale value may get quickly bought, and then immediately posted for re-sale. Now, bots could be used to manipulate the price directly. A single item could be bought and resold to other bots before it ever leaves a physical warehouse. Marketplace platforms and credit card companies will have significant financial incentives to allow this so they get a cut of every resale. Pricing for goods could start to more quickly resemble the stock market where most trades are already done based on algorithms.

This will likely mean higher prices on many goods. This will especially be true if e-commerce platforms themselves use AI to adopt dynamic pricing strategies. A recent exposé found that Instacart was using a hidden algorithm to test the upper limits of prices they were willing to pay for certain groceries. While we do not have enough information to fully understand the interplay between AI-enable pricing and constant bot activity, what we are seeing is that the incentive structure likely doesn’t favor the consumer.

Another unsettling dynamic that could emerge is dynamic pricing that acts as a proxy for social scoring. Consumers with certain buying histories or in certain geographic locations may be rewarded with access to certain pricing schemes to the detriment of other buyers. Characteristics like race, gender, sexuality, or disability could be inferred based on proxy data, which could in turn cause specific populations to burden higher costs because of the types of goods or services they are trying to access. For instance, rural consumers may pay higher grocery delivery prices if they live in a food desert or racial minorities may be quoted higher rent prices in certain neighborhoods.

Key Takeaway: Given that ‘affordability’ is currently a global concern, and that algorithms have already started to be a major contributor to that, negative impacts from AI-driven higher prices could become a large political force in the coming decade. Policymakers may feel the pressure to impose limited safeguards that help mitigate these issues.

2. Technical Deep Dive - A Poet’s Key to Model Hacking

Safety alignment 
May succeed until
Poetry
Foils the plan

Despite extensive alignment efforts, many LLM safety mechanisms can be brought down with a few lines of poetry. A recent research paper on “Adversarial Poetry” showed that using poetry to frame an adversarial request (e.g. advice on how to execute a cyber attack) resulted in a broad range of models producing unsafe outputs 62% of the time. The attack success rate (ASR) varied widely by model: all the GPT-5 models had an ASR of under 10%, while the Deepseek-3 and Gemini 2.5 models had an ASR of over 95%. While this isn’t the first method to cause models to ignore their built-in defences (e.g. the infamous Do-Anything-Now prompt), it does point to two broader themes.

First, while many providers now report extended safety evaluations, they are often focused on well-known attack vectors and may not paint a full picture. In addition, to creating a custom poetry dataset, the researchers in this study took a well-known dataset of adversarial prompts from MLCommons and translated them into poems, the average ASR went from 8% to 43% - suggesting that model developers may be overfitting to a known set of attacks during safety fine-tuning and not addressing the broader alignment problem.

Second, syntax has a big role in how LLMs process data. Another recent study showed that LLMs can rely on syntax over exact semantics of a sentence. For example, when asked ‘Where is Paris undefined?’ A model may answer ‘France’ despite the actual sentence being nonsensical. This may point to the success of the poetry attacks - the grammatical structure of poetry is not associated with adversarial behavior and thus may not trigger those safety protections.

Key Take-away: LLMs will likely never be fully resistant to jailbreaking, which may take the form of complex multi-turn attacks or a simple poem (GPT-5 models seem particularly resistant to the later), because the training data contains unsafe content and reliable “unlearning” techniques do not exist. When building AI systems that may have adversarial users, relying on the model provider’s safety alignment is not sufficient and additional guardrails (e.g. output filtering) should be integrated into the systems.

3. AI Incident Spotlight - Deloitte Publishes Citation Hallucination in Government Sponsored Report (Incident 1286)

What Happened: A public report written by Deloitte, on behalf of the Provincial Government in Newfoundland, contained some hallucinated citations for key statistics and facts. Newfoundland reportedly paid Deloitte 1.6 million Canadian dollars for the report that outlined a human resources plan. The report had citations to publications that don’t actually exist, and supposed claims in those publications were used as justification for recommendations in the report. Deloitte claims the errors were strictly related to generating the citations, not the outcomes of or recommendations from the report. This comes only a few months after a highly similar incident in Australia where Deloitte again cited poor citations in reports generated for public sector agencies.

Why It Matters: There are a couple of things this incident highlights. The first is the challenge with generating citations in general. This is often a tedious task that many people want to automate, but it’s also one that is actually quite challenging for AI to do, unless it’s connected to some kind of ‘global database’ of research upfront. Ironically, most publication archives are now heavily deploying anti AI scraping technology, which will actually make this problem worse in the short term despite model improvements. It also highlights one challenge some big consulting companies will have in the AI era. These recent incidents have caused reputational damage to Deloitte, and while they claim only the citations were AI generated, that is difficult to prove. Top services companies, like prestigious consulting firms, law firms, think tanks, etc often differentiate on the quality of their work, and often try to stack staff from elite universities to reinforce their brand. However these firms are also under huge pressure for productivity and AI can be a big contributor to that. The biggest risk for them is that ‘AI Slop’ could undermine their chief competitive advantage. Few people would hire McKinsey at their normal price point if they can get the same level of insights and advice directly from ChatGPT. While top companies have swarmed on AI, we also expect there to be backlash that could suddenly make the value of truly ‘human’ services more valuable in the AI era, especially for ‘elite/luxury’ markets.

How to Mitigate: Obviously the most reliable way to prevent hallucinated citations is to have them all manually reviewed, although that can be slow and tedious (hence why AI was used in the first place). There are some low hanging fruit ways for building some automated verification steps however. Simply running each citation itself through a non-API driven system to verify it exists is an option, and some organizations have started using a separate AI tool or model to run its own verification checks on generated content. It’s also important to have clear policies for employees to specify when they use AI for generating content, and to have a clear list of things to check in AI generated content in order to catch common AI mistakes like this.

4. Trustible’s Top AI Policy Stories

Trump’s New AI Executive Order. President Trump is expected to sign a new executive order (EO) aimed at pausing state AI laws. A draft was previously leaked that outlines how the Administration will leverage the Department of Justice to challenge state AI laws.

Our Take: The Trump Administration has sought to pre-empt state AI laws but has not offered a federal replacement. We anticipate a fairly lengthy legal battle to ensue once the EO is signed.

AI in the NDAA. Lawmakers added amendments to the National Defense Authorization Act that bans foreign models from use in the federal government and tasks the Department of Defense with creating an AI model evaluation framework.

Our Take: Congress is also considering legislation for an “AI in national security” playbook and these amendments would align with the targeted, security-focused approach to AI that we have seen from the federal government.

New York Times and Perplexity. The New York Times joined a copyright lawsuit against Preplexity.ai that alleges it illegally copied millions of its articles. Perplexity has been embroiled in legal battles for how it gathers and uses content for its AI search engine.

Our Take: The ongoing copyright infringement cases highlight the growing need to update IP laws and regulations that can account for how AI systems can use protected content.

In case you missed it, here are a few additional AI policy developments making the rounds:

Africa. The Ugandan government announced that it will release a draft plan to regulate AI. The decision marks one of the first significant efforts to regulate AI in Africa.

Asia. The Japanese government will release a draft plan to improve Japanese AI development and increase AI adoption. Japan has taken a lighter-touch regulatory approach with AI previously and is looking to develop its domestic AI ecosystem.

Australia. The Australian government released its National AI Plan, which is intended to help grow the country’s AI industry. The plan seeks to support new AI infrastructure, increase AI adoption, and enact laws to protect its citizens from potential AI harms.

Europe. Lawmakers are under pressure in the UK to regulate “superintelligent” AI system development. Specifically, the latest regulatory push wants more safeguards imposed on frontier model providers to reign in developing potential superintelligent systems.

North America. The Canadian government released its first public sector inventory of AI systems. The government also published the world’s first standard for developing equitable and accessible AI systems.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, and welcome to this edition of the Trustible AI Newsletter! Two weeks is an eternity in AI world, and in the past two weeks, we’ve seen a seismic shift in the AI regulatory environment both here in the U.S. and across the pond in the EU (more on that later in this edition.)

But, it’s also been a big couple of weeks for all of us here at Trustible; last week, we launched our new Trustible AI Governance Insights Center, an open source repository of our AI governance heuristics, from our risks taxonomy, to recommended mitigation strategies, AI benefits, and even more developments in our AI model ratings curated by our team of experts. Over time, we’ll be adding even more insights and resources, but as a public benefit corporation, this is an important step in advancing our mission to help society realize the transformative potential of AI. You can explore the insights center at trustible.ai/resource-center.

We are also thrilled to share that we’ve been listed as a Representative Vendor in the 2025 Gartner® Market Guide for AI Governance Platforms. We believe this is a milestone that signals the start of an inflection point, when AI governance is no longer optional, experimental, or theoretical; it’s now a business imperative for enterprises looking to realize the promise of AI. You can read all about the exciting news here.

In this week’s edition, we’re covering:

1. Trustible’s Take - Transatlantic AI Uncertainty

2. AI Incident Spotlight - Cyber Attacks by AI Agents

3. Technical Explainer: Big challenges with open-weight models

4. Trustible’s Top AI Policy Stories

1. Trustible’s Take - Transatlantic AI Uncertainty

As a result of a flurry of regulatory proposals last week, there is now more uncertainty than ever about when AI regulations may kick-in and what those regulations will be. In Europe, the EU Commission proposed a ‘Digital Omnibus’ to reform several digital laws, including GDPR and the EU AI Act. The Commission is proposing a delay for high risk AI system obligations for a period ranging between 12 and 24 months. A driving force behind the delay comes from issues developing compliance standards for high risk systems. The Commission’s proposed delay stipulates that if standards are developed before the new deadlines hit, then high risk system requirements will take effect sooner. However, it is unclear if the Commission’s proposal will make it past a skeptical EU Parliament (which has to agree) or if the compromise text would pass before the current deadline for high risk obligations comes into effect in August 2026.

Meanwhile in the US, congressional Republicans are moving forward with plans to resurrect the state AI moratorium by attaching it to the annual National Defense Authorization Act. President Trump fully supports the effort moratorium and is considering an Executive Order (EO) that would effectively enact a moratorium without Congress, but the planned EO appears to be on hold. A federal moratorium on state AI laws is facing backlash from several prominent GOP elected officials, including Governor DeSantis and Senator Hawley. Any moratorium without an actual superseding federal law would likely face immediate lawsuits, especially if done via an EO.

Ironically, the recent policy prerogatives in the EU and US aim to incentivize AI growth and adoption but instead have injected such a level of regulatory uncertainty as to undermine these goals. The EU and US have substantially lower rates of trust in AI than any other part of the world. Solving the AI trust problem is the key to growing AI adoption rates on both sides of the Atlantic.

Key Takeaway: The increasing regulatory uncertainty paired with the perception that AI is unregulated is not going to help accelerate adoption. Backtracking on efforts that improve trust and adoption could just cause the “AI bubble” to burst, along with any potential to “win” the AI race against China.

2. AI Incident Spotlight - Cyber Attacks by AI Agents (Incident 1263)

What Happened: Anthropic identified that a Chinese hacking group (GTG-1002) used its Claude Code platform to launch a fully autonomous cyber espionage attack against over 30 targets. The groups used several ‘jailbreaking’ techniques to evade protections built into Claude Code. The attack notably included autonomous ‘multi-step’ processes, where the agent first scanned for the target’s cloud resources, programmatically identified vulnerabilities, created exploits for them, and then successfully extracted data before being shut down by Anthropic.

Why it Matters: There are several highly notable and severe aspects of this incident. The first is that it highlights a new dangerous paradigm for cybersecurity. The fact that an AI agent was able to successfully conduct a highly sophisticated multi-step attack, with limited human interaction confirms the fears of many in the cybersecurity world. The fact that it was a Chinese group, with links to the Chinese state, using an American AI system is also likely to create a massive reaction in DC. This is also one of the few voluntary first-party incidents reported by a large model creator, although it’s unclear what their longer term mitigation efforts may be beyond simply trying to ban the relevant parties.

How to Mitigate: A lot of focus of these incidents is often on the underlying ‘model’. However LLMs only ever generate text, and even in agentic systems, they’re often simply generating text that instructs a system to do something, or to call some kind of external tool. The real dangers here came from all the other ‘system’ components built into Claude Code as a ‘platform. Claude-4 the model can generate the text command to make an HTTP request, but Claude Code the platform can actually make the request. Claude Code then also was storing information in memory, running a process over several minutes (or hours), and executing generated computer code. For organizations hosting AI systems, limiting how much a system can independently access the internet, or run code, can heavily restrict the potential blast radius and capabilities of a system. For those worried about similar incoming cybersecurity attacks, tools like CloudFlare anti-AI bot tool will become an essential layer to help block sophisticated non-human interactions.

3. Technical Explainer: Big challenges with open-weight models

Open-weight models, like Phi and Qwen, are available for download and unconstrained use by researchers, businesses, and consumers. Unlike closed-source models (e.g., GPT or Gemini) that use input/output filters, protections in open-weight models must be integrated directly or added by deployers. Recent research highlights risk management challenges that are particularly salient for open-weight models.

Training Data Curation: One key mitigation strategy is to avoid training on unsafe sources. However, current filtering methodologies are inconsistent and some “knowledge” may be useful for benign capabilities (e.g. designing cybersecurity defences). In addition, recent work suggests that harmful knowledge may emerge from a combination of benign data sources (e.g. knowledge of biorisks may be inferred from general knowledge of biology). Choosing an appropriate curation strategy may be challenging for deployers, and a lack of clear guidelines can make it difficult for developers to review models.

Tamper-Resistant Training: Post-training methodologies can further reduce unsafe behavior, however, downstream modifications can intentionally or unintentionally remove these protections. In addition, as models become integrated into agentic systems, they may retrieve harmful knowledge from the internet, introducing a new attack vector.

Model Tampering Evaluation: Open-weight developers often lack the resources for extensive audits, shifting the burden to deployers. Furthermore, standardized frameworks don’t exist for these evaluations making it difficult to compare and trust different models. (In our own AI Model Ratings, we’ve observed that many open-weight developers explicitly disclose a lack of adversarial evaluations)

Model Provenance: Both researchers and deployers may want to study the lineage of specific open-weight models. While the former may want to understand the broader ecosystem, the latter needs to know whether a particular model is appropriate for an application. Currently, no reliable and scalable approach exists for tackling this challenge.

Key Takeaway: The challenges outlined are exacerbated by a lack of transparency - most popular open-weight models largely do not address these topics in their documentation. While these models allow for more flexibility and control, they shift the burden to deployers to investigate model provenance, run safety evaluations and build additional safeguards while considering a lack of unified standards and solutions for each step.

P.S. While this work emphasizes the lack of reporting around safety, we’ve recently collaborated with the EvalEval coalition on a new paper that shows that societal impact evaluations are underreported across the LLM landscape.

4. Trustible’s Top AI Policy Stories

House Hearing on Chatbots. The House Subcommittee on Oversight and Investigations held a hearing to better understand the risks posed by AI chatbots, particularly to minors, and hear recommendations from experts on potential regulatory solutions.

Our Take: Congress continues to be lasered focused on AI harms posed to children and may be one of the few AI issues that sees bipartisan legislation pass.

National Security Framework. A bipartisan group of lawmakers in the House and Senate are working on legislation that would require the National Security Agency to publish an AI security playbook, which is intended to help outline how AI systems are being protected from foreign adversarial threats.

Our Take: AI security is a priority for the Trump Administration (as emphasized in the White AI Action Plan) and represents another area where lawmakers could plausibly pass legislation.

United Nations and Healthcare. A new report from the World Health Organization warns that AI used in healthcare settings needs legal guardrails to protect patients and healthcare professionals.

Our Take: The concerns are not new but the report notably observes “there is a broad consensus on the policy measures” that could improve AI adoption.

In case you missed it, here are a few additional AI policy developments making the rounds:

Africa. The UAE announced the “AI for development initiative,” which will invest $1 billion to expand AI infrastructure in Africa. While US tech companies have been making active investments to help African countries develop AI technology and infrastructure, the US government has largely been absent.

Asia. AI-related policy developments in Asia include:

• China. The Chinese government issued guidance that would ban foreign-made AI chips from new data center projects. The decision comes amidst tensions with the US over advanced chips sales in China.

• South Korea. Korea’s Ministry of Science and ICT released draft regulations for the AI Basic Act, the country’s comprehensive AI law passed in December 2024. The proposed regulations clarify how covered entities will need to comply with the law, which takes effect in January 2026.

Australia. The Australian government released the Australian Public Sector (APS) AI Plan, which is aimed at improving how AI is used by public sector agencies. While the Australian government has backed away from enacting AI rules for private companies, the ASP AI Plan could have far-reaching impacts on companies doing business with the federal government.

Middle East. The Trump Administration signed a new Memorandum of Understanding with the Saudi Arabian government, which would allow the Saudi government to access US AI technology. The agreement is part of the Trump Administration’s broader AI policy goals with Middle Eastern countries.

South America. The UN’s COP30 climate conference met in Brazil and focused heavily on how AI is impacting climate change. While attendees acknowledged AI’s potential for helping address climate change, many raised concerns over AI’s strain on natural resources and energy consumption.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In this week’s edition, we’re covering:

1. Chatbot Teetertot: Balancing Child Safety & AI Literacy

2. Tech Explainer: The Role of Guardrail Models in AI Systems

3. AI Incident Spotlight: Deep-Fake Endorsement Scam (Incident 1261)

4. Policy Round-Up

1. Chatbot Teetertot: Balancing Child Safety & AI Literacy

Minors are increasingly using AI tools for social support and companionship. However, it’s not difficult for kids to elicit problematic or harmful content from AI tools, mainly AI chatbots and companion bots. The most notable examples are from Character.ai, whose companion bots have lead to children committing self-harm or suicide, as well as sharing or generating illicit content. Character.ai has since blocked kids from accessing its chatbots.

Industry is trying to figure this out, with OpenAI’s Teen Safety Blueprint as one example of how the private sector can offer solutions. Policymakers in the US have also taken notice and are attempting to enact new laws that address these issues. For instance, a new California law requires companies to implement safeguards to prevent companion chatbots from discussing suicide and must direct the user to self-harm resources if they express suicidal thoughts to the bot. Congress has also taken notice and the bipartisan GUARD Act was introduced in the Senate, which would (among other things) ban minors from accessing companion bots.

Understandably, lawmakers and parents want to protect kids from the harms posed by chatbots. Age verification requirements have long been used to prevent minors from accessing inappropriate or illicit content. Yet, we must reflect on whether age-gating sufficiently addresses the problem (as those requirements can be gamed and also pose serious First Amendment issues) and whether these types of barriers to certain technologies will do more long term harm. Yes, it is essential to protect kids from potentially harmful systems and content, but it also highlights the challenge of effectively teaching them essential AI literacy skills to build a competitive next generation workforce.

While AI developers have a role to play with child safety, it does not stop with them. Organizations deploying public-facing chatbots need to understand that, even if children are not the intended audience, they may still use their chatbot products. It’s important for companies with public facing chatbots to implement safeguards around outputs and make appropriate disclosure (e.g., noting when a product should not be used by people under the age of 18). These mitigations are especially true as chatbots become more general purpose, which broadens the aperture on exposure.

Our Take: Kids today are more tech savvy than the previous generation and we need to acknowledge that fact as we think about how to implement pragmatic protections while helping them build valuable AI skillsets. We also need to think about how new child safety laws are not so sweeping that they unintentionally stunt growth for other AI use cases.

2. Tech Explainer: The Role of Guardrail Models in AI Systems

One way to mitigate a variety of AI system risks including processing PII, outputting unsafe (e.g. toxic language or specialized advice) content and prompt injections is by incorporating guardrail models that review and flag the inputs to and preliminary output from the system. Think of guardrail systems as guardians designed to be the first line of defense against potential threats. Designed specifically for detection rather than reasoning, guardrail models are typically smaller and faster than general-purpose LLMs. Both AWS Bedrock and Azure AI have off-the-shelf guardrail models that can be integrated into any endpoint, while on the open-source side, Llama-Guard is a popular solution that can recognize 13 common harm categories.

Recently, OpenAI released a new open-source guardrail model, gpt-oss-safeguard. Unlike the other solutions that can only detect a pre-defined set of harms, this model allows the user to input an arbitrary human-written policy (i.e. a set of rules to follow), meaning it can be used for domain-specific concerns (e.g. flagging spoilers on a movie forum or detecting regulatory non-compliance in legal text). The increased flexibility does not guarantee accuracy: the developers state that gpt-oss-safeguard is often less accurate than a custom model. As well, the documentation does not explore what kinds of “policies” will be effective; recent research suggests that existing human content moderation guidelines may need to be modified to work properly with LLMs. This type of model can be used as a rapid deployment solution, when off-the-shelf models do not cover the use case, and resources can not be devoted to building out an internal solution that may require thousands of labeled data points.

Our Take: Guardrail models serve as a first and final line of defence against unsafe system outputs, but require careful evaluation to be used effectively. Existing experts in that domain will need to collaborate with content engineers to review off-the-shelf guardrail models or build bespoke policies for models like gpt-oss-safeguard.

3. AI Incident Spotlight: Deep-Fake Endorsement Scam (Incident 1261)

What Happened: A deep fake of a senior official in the government of Western Australia was used to facilitate an online scam. The scammers generated an AI video showing Roger Cook, the official, falsely endorsing an investment service. The deep fake was described as hyper realistic in both voice and appearance.

Why it Matters: Public officials are at particularly high risk of being ‘deep faked’, and leveraged by scammers. The latest systems can impersonate facial likeness and voice with only a few short high quality video or audio clips. These are particularly easy to get for public officials who are highly visible and often need to participate in televised events. Many public officials also don’t always receive the same degree of privacy protections, and certain activities like satire of politicians can actually be a protected activity, blurring the lines of what is acceptable or not. It’s likely that these types of deep fake impersonation attacks will become more common over time.

How to Mitigate: Unfortunately there aren’t many ways that individuals or organizations can prevent this kind of activity on their own. Instead, the best options are to have mitigations set up to reduce the likelihood of people falling for these types of scams, and to support regulations on platforms that may potentially host, or benefit, from these types of schemes. For organizations, it is appropriate to consider regular training on how to detect deep fakes, and even run on-going exercises for it on employees.

4. Policy Round-Up

Trustible’s Top AI Policy Stories

ChatGPT’s New Lawsuits. OpenAI is facing a flurry of new allegations over ChatGPT’s impact on mental health. New lawsuits against the company allege that ChatGPT induced psychosis, as well as lead to financial ruin and suicide.

Our Take: It is important to clearly communicate how these tools should be used and make sure that users understand they are interacting with a machine, not a real person.

Pausing the EU AI Act. The European Commission (EC) released initial proposals for its Digital Omnibus package, which includes loosening EU AI Act obligations and postponing enforcement for certain requirements.

Our Take: The uncertainty is complicating compliance for many organizations but they should focus on EU AI Act compliance in case the EC opts against enforcement delays.

UK Copyright Decision. A UK court decision is upending the interplay between AI data use and IP law after it ruled in favor of Stability AI in a case brought by Getty Images for secondary copyright infringement.

Our Take: The case adds a new dimension to how AI is trained on protected data because the alleged infringement did not occur in the UK. Organizations should make sure they have proper permission to use data when training their AI models and tools.

In case you missed it, here are a few additional AI policy developments making the rounds:

United States Congress. Two pieces of bipartisan AI legislation have been introduced in the Senate. The GAIN AI Act would regulate how American AI chips were exported to China and other countries of concern. The AI-Related Job Impacts Clarity Act would require companies and the federal agencies to report when AI-related layoffs occur. House Democrats are also attempting to rollback the Trump Administration’s efforts to use AI at the Centers for Medicare & Medicaid Services.

Trump Administration. After OpenAI’s CFO caused controversy by implying the federal government could be a “backstop” for AI infrastructure debt, the Trump Administration dismissed the idea of a federal bailout for frontier model companies. Sam Altman has denied that OpenAI expects a bailout.

Asia. AI-related policy developments in Asia include:

• China. A new law requires social media influencers to hold degrees for certain topics (e.g., medicine or finance) before posting about them. The law also requires influencers to disclose when their content is AI-generated.

• India. The Indian government is preparing to release a draft comprehensive AI law. The current content is unknown but is expected to be modeled after the Information Technology Act of 2000.

Europe. Nvidia and Deutsche Telekom announced plans to build Europe’s largest AI in Germany. The latest announcement emphasizes Europe’s desire to build its own AI ecosystem.

Middle East. The Trump Administration signed a memorandum of understanding with the UAE to further cooperation in AI and energy.

As always, we welcome your feedback on content! Drop us a line with your thoughts to newsletter@trustible.ai for a chance to win an exclusive set of Trustible AI Model Playing Cards!

AI Responsibly,

- Trustible Team

1. Trustible’s Take - The Need for AI Fallbacks

2. AI Incident Spotlight - (AI Incident 1248)

3. Reddit’s Hidden Hand in AI Training and Why It Matters

4. Policy Round-Up

1. Trustible’s Take - The Need for AI Fallbacks

Several owners of Eight Sleep, a tech enabled ‘smart bed’, were suddenly woken up early last Monday by their beds going haywire. It turns out that a massive AWS outage related to a misconfigured Domain Name Service (DNS) system was able to take down more than just websites and SaaS applications. This event highlights two core risks related to the evolving AI ecosystem.

The first, is that there is a lot of infrastructure that still supports AI systems. For example, all internet traffic relies on DNS to figure out what servers to talk to, AI systems need to read and write information to databases like AWS DynamoDB, and the physical hardware that supports the relevant computation is also at risk. One reason the AWS outage was so widespread is because there are only a handful of hyperscaled cloud service platforms, and the set of affected data centers (‘us-east-1 based in northern Virginia) happens to be one of the earliest, and most used regions. As we start to incorporate more AI into everyday systems and rely on it more, the risk of creating ‘central points of failure’ increases.

The second issue at play was that the devices with embedded ‘smart’ capabilities did not have appropriate fallback mechanisms. Instead of simply disabling the smart features and acting as a plain old bed, some of the smart beds start increasing temperature endlessly. A few days after the outage, Eight Sleep did roll out a dedicated ‘outage mode’ that is able to use local bluetooth to send instructions to the smart bed. At this point in time, we have a ‘non-AI’ path and process for most things, however will that be true 10 years from now? One popular AI policy proposal is to ensure that there are non-AI fallbacks, bypass, or appeal processes, as a main mitigation towards this potential AI overreliance.

Key Takeaway: The AI ecosystem is not yet mature enough to have ‘multi-cloud’ deployments, partly because not all model providers are available across all major cloud providers. This maturity, and clear standards for ‘non-AI’ modes are likely going to be necessary before AI is adopted for highly critical applications in fields like healthcare or national infrastructure.

2. AI Incident Spotlight - (AI Incident 1248)

What Happened: US Conservative activist Robby Starbuck has sued Google claiming that Google’s AI models regularly defame him by accusing him of sexual assault. He filed a similar lawsuit against Meta earlier this year alleging similar things coming from Meta’s AI tools. That case was settled before going to trial. Many of his allegations date back to an earlier model, Bard, that Google has since deprecated.

Why it Matters: The exact cause of the hallucination isn’t known. It’s possible that there was politically motivated intentional misinformation posted online that the systems incorporated (a form of data poisoning), or it could simply be a hallucination because of similarities between Robby and others. Regardless of the source of the misinformation, this case, and others like it, will likely test the precedent of applying ‘Section 230’ to AI systems. The issue at the core is who is liable for false information like this. Under current interpretations, web platforms are not directly responsible for the defamatory content created by users of their platform. Whether an AI system counts as a ‘platform’ (protected), or user (liable) could radically shift the liability scheme for LLM providers. So far, no one has been successful in winning a defamation case against AI in the US.

How to Mitigate: Most of the most recent LLM systems have the ability to conduct real-time web searches in order to pull in ‘fresh’ information from reliable sources that can help pull in facts. However this web searching capability is a feature of the system not the models themselves, and therefore many AI features built on just model APIs won’t have this ability, and allowing searches introduces additional privacy and cyber risks. System prompts that cover how to respond when doing research or answering questions about people can help ensure that only verified information is shared, and to avoid stating potentially negative information altogether.

3. Reddit’s Hidden Hand in AI Training and Why It Matters

Few platforms have shaped the modern internet like Reddit. Its open, chaotic mix of expertise, argument, and lived experience has become a goldmine for organic community discussion for brands, but also, for LLMs. Because Reddit posts are conversational, self-correcting, and wide-ranging, they provide the kind of nuanced human expression that makes AI sound more human. In fact, many of the citations or answers you see in AI-generated content trace their roots back to Reddit threads.

But this organic data source has become a flashpoint. Reddit’s recent lawsuits against Perplexity and other AI vendors highlight an emerging legal and ethical battleground: who owns public discourse when it fuels AI? The platform argues that scraping and repurposing Reddit data without consent undermines both its community and its business model, especially as Reddit now licenses its content to certain model developers under paid agreements (representing $35M in revenue for Reddit as of Q2 2025, up 24% year over year.)

Technically, Reddit’s data is particularly valuable because of its structure. Its posts and comment trees offer deeply nested, timestamped dialogues, rich with slang, reasoning chains, code snippets, and emotional context, all of which are perfect for pretraining and fine-tuning models to understand how humans argue, explain, and empathize. AI crawlers systematically follow thread hierarchies and metadata (like upvotes or subreddit topics) to learn which ideas communities endorse or reject, turning social feedback loops into learning signals. This makes Reddit data uniquely high-quality - and uniquely sensitive.

For enterprises, Reddit represents both opportunity and risk. It’s a valuable source for sentiment analysis, market research, and training domain-specific chatbots. Yet the same exposure that makes Reddit content powerful also makes it volatile. A viral post, an out-of-context quote, or an AI model trained on outdated or toxic Reddit data can easily amplify reputational risks.

Why is it Relevant: Reddit’s fight with AI companies is a preview of how data ownership, consent, and community ethics will define the next phase of AI governance.

Key Takeaway: AI governance professionals should be alert to how LLMs draw from social data ecosystems like Reddit, and monitor these channels for keyword and thematic discussions where misinformation may be shared. Engaging in these channels, setting the record straight, and controlling your organization’s own footprint (from ads to AMAs) helps protect your brand. These are not neutral channels, they are living communities whose norms, biases, and moderation dynamics shape AI behavior.

4. Policy Round-Up

Trustible’s Top AI Policy Stories

Anthropic vs the White House. Anthropic has been in an interesting back-and-forth with White House AI Czar David Sacks, who has criticized the company as promoting an agenda “to backdoor Woke AI and other AI regulations.”

Our Take: Anthropic has been a leading voice on AI safety and supported state laws, including the recently enacted SB 53. The spat shows the fine line companies need to walk between promoting AI regulations, innovation, and navigating a tenuous political environment.

EU Prepared to Expand Copyright Laws to AI. It appears MEPs are prepared to require EU copyright law apply to AI training regardless of where the training occurs.

Our Take: The decision would put US tech companies and their frontier models in the cross hairs. But depending on the actual text, it could have big implications for companies that are training models on external data sources.

Frontier Model Providers Get EU AI Act Warning. The Dutch Data Protection Authority warned four frontier model providers (OpenAI, xAI, Google, and Mistral) that their chatbots’ advice on the Dutch parliamentary elections could classify them as high-risk systems under the EU AI Act.

Our Take: While the EU AI Act has not come into full effect yet, this is a good reminder that “low risk” systems can evolve into high-risk systems, which is why continuous oversight is necessary.

In case you missed it, here are additional AI policy developments:

United States Congress. A recent letter to Federal courts Senate Judiciary Chair Senator Chuck Grassley (R-IA) revealed that interim guidance has been issued to federal courts, which addressed “non-technical suggestions on the use, procurement, and security of AI tools.” Grassley supports formal AI regulations for the federal judiciary, given the errors and controversy that have arisen with AI and the courts.

Trump Administration. The International Trade Administration (ITA), which sits within the Department of Commerce, is launching a program to develop full-stake AI export controls. ITA issued a request for information, in accordance with President Trump’s Executive Order on Promoting the Export of the American AI Technology Stack, to seek industry input on how to establish and implement the program.

Africa. Gebeya (an Ethiopia-based platform) launched Gebeya Dala, an AI-powered app builder designed with African cultural considerations. This is the latest in a series of culturally-specific AI tools that have launched this year that align with regional or non-western cultures.

Asia. AI-related policy developments in Asia include:

• China. China’s legislature formally adopted amendments to China’s cybersecurity law, which promotes stronger ethical AI standards, risk monitoring, and assessments.

• Japan. The Government of Japan entered into a Memorandum of Cooperation with the U.S. government, in part to cooperate on accelerating AI adoption and innovation.

• Vietnam. The Government of Vietnam is working on amending its intellectual property regulations to promote AI innovation. The government is also supporting an open source AI ecosystem as part of their broader strategic plans as a regional AI player.

Australia. The Labour government indicated that it will not exempt frontier model providers from copyright laws for text and data mining. The government also launched an investigation into how chatbot companies like Character.ai implement safeguards for children. The government is also suing Microsoft because of deceptive price hikes related to copilot integrations into Microsoft 365.

Europe. AI-related policy developments in Europe include:

• Albania. Albania’s Prime Minister announced that its AI minister, Diella, is pregnant with “83 children.” The AI-generated offspring will serve members of parliament as their assistants.

• EU. The EU’s standards-setting body caused controversy by announcing that it would “fast-track” the most delayed EU AI Act standards with a smaller group of experts. The move was characterized as “unprecedented.” The decision caused pushback from some members of the standards body because of “serious unintended consequences.”

• UK. The Labor Government announced a new partnership with OpenAI that will allow its UK business customers to host their data within the UK. A local news station also piloted an AI newscaster in a story about whether AI will replace humans in the workforce.

North America. The Canadian government is considering a series of AI-related laws that will address deepfake, data transfers and age assurances for chatbots. Canada abandoned its efforts to pass a comprehensive AI law after their federal elections earlier this year.

Middle East. AI-related policy developments in the Middle East include

• Saudi Arabia. Saudi-based AI company Humain is making plans to be listed on the Saudi stock exchange as well as the NASDAQ. Humain and Qualcomm also announced a partnership on deploying advanced AI infrastructure in Saudi Arabia.

• UAE. G42 (UAE-based AI provider) and Cisco announced a partnership to build “secure, trusted and high-performance [AI] infrastructure.”

South America.

• Argentina. A recent study showed that a third of media professionals in Argentina use AI to assist with their jobs, which includes “help write and edit articles, craft headlines and translate text.” There are some concerns that the lack of regulations for how journalists use AI could hurt the industry financially, as well as a need for AI-related journalistic standards.

• Chile. The Chilean government is dealing with public backlash over resource issues posed by AI infrastructure, specifically as the government seeks to build more data centers. The outcry over AI energy and water consumption has been brewing in other countries, including the U.S.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

At Trustible, we’re on a mission, giving AI Governance professionals the information, insights, and tools they need. Our legal and technical analysts help filter through the noise and identify what’s meaningful for enterprises, and what’s just hype. To ensure we’re advancing that mission, we’ve been listening to your feedback over the past few months, and we’re going to be revamping our newsletter going forward.

Here’s what we’re changing and what you can expect going forward:

• Technical Insights

  • AI technology is evolving at a rapid pace and it’s hard for anyone to keep up, let alone contextualize what new developments will mean for enterprise use of AI. Our machine learning experts will use this section to translate technical into plain english and connect it to the challenges organizations face.

• Policy Round-Up

  • Our regular policy roundup will continue as an overview of the major AI policy headlines from the past two weeks. We won’t be able to cover everything around the world, but we’ll focus on the developments that most impact practitioners.

• AI Incident Spotlight

  • This new section will be a deep dive explainer on a recent incident captured in the AI Incident Database. Our goal will be to provide actionable recommendations on how to prevent similar incidents for enterprises.

• Trustible’s Take

  • This will be our editorial team’s take on the most notable news relevant to AI Governance professionals. This section will focus a lot on trying to be the voice of pragmatic AI, and to try and cut through the hype found on traditional and social media platforms.

• News & Updates

  • We’ll regularly publish summaries of our more in-depth whitepapers, research, and blog, along with new announcements from Trustible. We want our newsletter to be insightful and actionable for anyone working in AI Governance, but we also want to let you know how we’re building solutions to many of the issues discussed!

With that, in today’s edition (5-6 minute read):

1. Trustible’s Take: Why SB 53 Won’t Have a Big Impact

2. AI Governance Meets AI Insurance: How Trustible and Armilla Are Advancing AI Risk Management

3. AI Incident Spotlight - AI Agents Aren’t Always Leaving a Paper Trail

4. Technical Insight - What To Know about ‘Model Specs’

5. Global & U.S. Policy Roundup

1. Trustible’s Take: Why SB 53 Won’t Have a Big Impact

There has been a lot of fanfare about Governor Newsom signing SB 53, a frontier AI safety bill. Many proponents argue it will have a major impact in regulating AI. We’re not sure. Especially after its various amendments and changes, we think it’s very unlikely to make a large impact in the AI space. Here’s why:

• SB 53 is almost entirely a ‘subset’ of the EU AI Act’s requirements

  • While there are a few differences included in SB 53, including clear whistleblower protections for frontier model provider employees, many of the ‘core’ safety framework requirements are extremely similar to the ‘safety and security’ requirements in Chapter 3 of the EU’s Code of Practice for GPAI providers. A major sign that this alignment was intentional was the use of the same computer thresholds to establish ‘frontier models’ (SB 53), as ‘GPAI Models with Systemic Risk’ (EU AI Act). OpenAI actively lobbied Newsom to align these requirements, and notably neither endorsed nor denounced the bill.

• Most frontier labs are already compliant, and proposed enforcement is weak

  • The requirements of the ‘frontier AI framework’ described in SB 53 reads exactly like Anthropic’s proposal for it, in alignment with OpenAI’s and Google’s framework, and there’s even a clear path for xAI to update theirs to align with the requirements. Given the laws’ very high threshold for ‘frontier’ models (10^26 FLOPS), that’s likely the whole list. In addition, only the California AG is able to enforce the law and can only impose civil penalties for non-compliance. It’s unclear whether the frontier labs will need to do anything in order to comply, and the risks of doing so are relatively low in the short term.

• It doesn’t address any meaningful AI issues, nor clarify the legal environment

  • SB 53 doesn’t address many of the immediate AI policy issues that downstream deployers and users are struggling with. For example, despite frontier model requirements, the bill does not address copyright issues, liability transfers, or content watermarking. The focus on only ‘catastrophic risks’ is unlikely to make a big impact for high risk sectors trying to understand how to simply not break existing customer relationships and laws when deploying AI systems.

Key Takeaway: It’s unclear whether SB 53 is ‘regulation’ or ‘regulatory capture’ by frontier model providers. For most downstream AI system builders, the biggest impact will likely be receiving 300 pages of documentation, instead of the current 150 pages.

2. AI Governance Meets AI Insurance: How Trustible and Armilla Are Advancing AI Risk Management

AI adoption is accelerating, but readiness still lags behind. Nearly 59% of large enterprises are already working with AI and plan to expand investment, yet only 42% have deployed AI at scale. At the same time, incidents of AI failure are rising sharply; the Stanford AI Index recorded a 26× increase in AI incidents since 2012, and more than 140 AI-related lawsuits are currently pending in U.S. courts.

The message is clear: as organizations race to integrate AI into products, operations, and decisions, risk management has to evolve just as quickly. That’s why last week, Trustible and Armilla AI announced a new partnership to tackle these challenges.

Together, we’re connecting the dots between AI governance and AI insurance, helping enterprises both prevent and protect against emerging AI risks. Trustible helps organizations operationalize responsible AI governance, while Armilla, provides affirmative AI insurance, explicitly covering risks that traditional cyber or E&O policies often exclude, such as model errors, generative AI copyright and libel issues, and regulatory penalties.

By working together, Trustible and Armilla create a feedback loop between good governance and improved insurability, enabling organizations to innovate confidently while minimizing and transferring residual risk.

You can learn more about the partnership here.

3. AI Incident Spotlight - AI Agents aren’t always leaving a paper trail (AI Incident 1218)

What Happened: Cybersecurity researchers have identified that in some instances, searches from Microsoft 365 Copilot don’t get properly registered in a document’s audit log. Any human ‘look-up’ or access to a file in Microsoft 365 is logged in a dedicated ‘audit trail’ which is an essential part of appropriate access controls. However despite Copilot citing answers from a certain document, there is not always a permanent record that Copilot read information from that file.

Why it Matters: Generative AI ‘answering’ systems can accidentally break normal access control rules and share information from documents a user may not otherwise have access to (data leakage). Not logging this access appropriately can exacerbate this issue, or even encourage this attack vector because. Most enterprise IT/ security policies require strict access controls and audit logs to help detect unauthorized access or use, and at least in some instances, Copilot may not obey the normal control expectations.

How to Mitigate: Without additional information, our theory is that documents get stored in their ‘embedding representation’ inside of 365 so that they can be searched over by an LLM. This means the information being accessed by 365 Copilot is not the ‘original’ document that stores the audit log. In addition, registering every ‘system’ access may bloat the audit log too heavily and there are not yet standards for logging AI system vs human accesses. For now, we recommend keeping highly sensitive documents out of files/folders indexed by 365 until this issue is fixed.

4. Technical Insight - What To Know about ‘Model Specs’

OpenAI recently made some significant changes to their ‘Model Spec’. The latest changes focus on better behavior for agents, trying to reduce ‘sycophancy’, and incorporated insights from a recent ‘public alignment’ project OpenAI has been running. Their model spec outlines how OpenAI has fine-tuned their models to enforce certain nuanced situations. It’s the best tactical representation of both their top level AI principles, and the specific guardrails built into their systems. While OpenAI is the only provider to use this exact format, other frontier model providers publish their versioned ‘System Prompts’ (Anthropic, Grok), which serve a similar purpose, although are not as structured.

Why is it relevant: OpenAI’s model spec is one of the most detailed documents about how frontier model providers are trying to align their models. While system cards give in depth insights into technical details, the model spec is consumable by non-technical experts, and contains more actionable information. Knowing what a model is supposed to do is essential for helping establish if the model is malfunctioning (acting outside of the spec), or if it allows certain behaviors that the deployer may want to block on their own. While recent legislation like SB 53 focus only on ‘catastrophic risks’ and will require reports on mitigations efforts towards those, the model spec contains relevant information for understanding whether the system has specific guardrails against things like data leakage, generating images based on a person, or how to handle chats about sensitive topics like sexuality. Publishing model specs, or similar documents, could be the next type of ‘transparency’ document frontier models may be required to publish in the future. The author of the Trump Administrations’ ‘AI Action Plan, Dean Ball, recently proposed such an idea in his newsletter, even while arguing for federal pre-emption of other AI regulations.

Key Takeaway: OpenAI’s model spec is better documentation to examine than their system cards for non-technical AI governance professionals trying to understand the risks of deploying or using OpenAI models.

5. Policy Round-Up

Trustible’s Top AI Policy Stories

The Problems with Sora 2. OpenAI launched its new video generation model, Sora 2, a couple of weeks ago. Since then, Sora 2 has raised fresh concerns over its environmental impact, ability to spread misinformation, and IP infringement.

Our Take: When using new models, AI governance professionals should consider metrics like the model’s impact on resources (e.g., energy and environment), as well as understand what types of outputs are being generated and the appropriate ways to use them.

The EU’s AI Breakup with the US. The European Commission released the Apply AI Strategy, which will invest approximately €1 billion in the EU’s AI industry to reduce its reliance on the US and China.

Our Take: A new EU ecosystem will provide companies with new choices for AI models and tools that take a different approach to AI safety and security than the US.

Fears Over the AI Bubble. There have been growing concerns over an “AI bubble” in the economy that is reminiscent of the dot-com bubble from the late-1990s.

Our Take: AI is here to stay (whether or not a bubble exists or bursts) and that means AI governance is not going anywhere.

California Regulates Companion Chatbots. Governor Newsom signed SB 243 into law, which protects minors and other vulnerable groups from AI companions.

Out Take: Chatbots are generally thought to be low risk use cases, but the new law underscores how companies need to have insights into the safeguards around their chatbots.

In case you missed it, here are additional AI policy developments:

United States Congress. Two bipartisan AI bills were recently introduced in the Senate. The AI LEAD Act would impose a “duty of care” standard for AI system developers and would classify AI systems as products, as opposed to platform. The AI Risk Evaluation Act would establish the advanced AI evaluation program through the Department of Energy.

Asia. AI-related policy developments in Asia include:

• China. The Chinese government is attempting to crackdown on Nvidia chip imports as it seeks to promote its own homegrown chip industry. It has also been reported that Chinese government officials were caught using ChatGPT to create tools for mass surveillance and social media monitoring.

• Vietnam. The Ministry of Science and Technology is seeking public input on a comprehensive AI law.

Europe. ASML’s Chief Financial Officer criticized the EU for overregulating AI, claiming that the difficulty with AI in Europe is “because [the EU] started with regulating, to keep AI under the thumb.”

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. OpenAI is looking to Canada for cheaper energy and as part of the deal would help build new data centers in Canada as it pushes to expand its sovereign AI industry.

• Mexico. Salesforce announced that it would invest approximately $1 billion in Mexico over the next five years in an effort to expand AI adoption.

South America. OpenAI signed a letter of intent that would invest up to $25 billion for a large-scale data center, which is expected to be built in the Argentine Patagonia.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In today’s edition (5-6 minute read):

1. Model Swap

2. Good Models Still Require Credible Data

3. What Makes for the “Perfect” AI Use Case Intake Process?

4. Global & U.S. Policy Roundup

1. Model Swap

OpenAI and Anthropic recently ran a cross-evaluation of each other’s models. Both highlighted different risks, and demonstrated how task framing shapes evaluation outcomes.

OpenAI leaned on instruction-following and jailbreak resilience as core safety markers, noting for instance that Claude refused up to 70% of hallucination probes rather than risk giving a wrong answer. Anthropic emphasized long-horizon misuse and sycophancy, finding that OpenAI’s o3 resisted harmful misuse better than GPT-4o, GPT-4.1, or o4-mini, which were often willing to cooperate with simulated bioweapon or drug synthesis requests.

While some research labs like Epoch AI, and government sponsored institutes like the UK’s AI Security Institute (AISI) have done independent model evaluations, this was the first instance of the 2 frontier model leaders conducting this kind of evaluation swap. The differences in evaluation approaches, and respective strengths/weaknesses of each other’s models highlights how the philosophies, and incentives, of the model providers can be reflected in their models. It also highlights the need to have a wide range of evaluations, and now just rely on self reporting. The evaluations focused on ‘frontier capability’ assessments however, not assessments on more ‘day to day’ AI tasks. OpenAI more recently introduced GDPEval to assess model performance on industry specific tasks, and this type of benchmark could quickly become an industry standard relevant to AI deployers and users.

Key Takeaway: Evaluations that just rely on a single number, can easily mask some of the more nuanced differences between foundation models. Once you dig into the details, the priorities, culture, and incentives of a model provider may be more clear, and may be an issue for certain types of use cases.

2. Good Models Still Require Credible Data

A recent study showed that ChatGPT may return answers based on claims in redacted scientific studies, even when information about the redaction appears in the same document. Unlike hallucinations, where the model fabricates a claim, this study points to the models inability to properly contextualize information in the training data and recognize when it’s inaccurate. This challenge can’t be solved easily: first, the pre-training process breaks up the documents into multiple pieces, thus a retraction statement may have been associated with the title of the paper, but not with individual statements in another part of the paper. Second, modem LLMs are trained on a vast corporas of data that are not manually reviewed, thus removing all erroneous information from pre-training data isn’t feasible (even if it was - this process would be biased and subjective).

While this study focused on GPT 4o-mini’s internal knowledge, many modern systems integrate external information through web searches or connection to internal knowledge bases (i.e. RAG). These integrations can help the model analyze the document as a whole and consider retractions published on external platforms. We tested a couple example claims from the study and found that during a web search the model recognized the retraction and corrected its original answer.. However, this process still relies on the model having access exclusively to up-to-date information; an out-of-date document that isn’t annotated as such can cause similar erroneous assertions. This failure mode may have been behind Air Canada’s chatbot stating false advice on bereavement travel last year. Our own team at Trustible has encountered similar challenges using AI for coding tasks, where old, outdated code (tech debt), regularly confuses the model.

Key Takeaways: LLMs can not reliably identify if information is up-to-date, especially if updates or retractions are not clearly associated with a specific fact. Practitioners should take care to maintain accurate data sources for training, fine-tuning and RAG. In addition, detailed prompting can help the system explicitly check for potential inconsistencies.

3. What Makes the “Perfect” AI Use Case Intake Process?

AI intake is the front door to governance. In a panel at the IAPP AI Governance Global North America in Boston a few weeks ago, Trustable led a discussion with leaders from Leidos and Nuix on AI use case intake processes. The room quickly aligned on a core truth: there’s no “perfect” intake process, only the one that fits your org’s risk profile, scale, and speed. The real work is choosing trade offs you can live with.

The conversation unpacked six design levers teams should tune, not max out:

• Granularity: are you tracking whole use cases, or features within products?

• Heaviness: what’s the minimum set of questions that still surfaces risk?

• Outcomes: does intake simply route, or also drive mitigations and decisions?

• Participation: who owns what—privacy, legal, security, product, HR, IT?

• Implementation: start with forms/spreadsheets, but plan for workflow and automation.

• Timing: catch ideas early without slowing experimentation.

Practitioners shared pragmatic moves: start where you are (even if it’s messy), iterate fast, and reframe intake from “audit” to “risk reduction.” Build muscle memory with short cycles and clear handoffs. As volume grows, expect ad hoc docs to buckle; that’s your signal to standardize fields, centralize your inventory, and automate routing so triage and transparency don’t degrade. Perhaps most importantly, make intake a shared habit—invite cross-functional partners in before the first pilot, not after the first incident. Culture change is the glue that keeps the process from backsliding.

Key Takeaway: fit beats perfection. A right-sized intake gives leaders visibility, lets teams move quickly with guardrails, and creates artifacts that stand up to regulatory and stakeholder scrutiny.

You can read the full recap on our blog.

4. Global & U.S. Policy Roundup

Trustible’s Top AI Policy Stories

OSTP Request for Information. As part of the Trump Administration’s AI Action Plan, OSTP is seeking input on existing rules that may hinder AI deployment or adoption.

Our Take: Beyond commenting on rules that industry does not like, this proceeding will allow companies to identify redundancies among existing rules that can help streamline AI governance processes should the federal government tweak them.

Anthropic settlement approved. Judge Alsup approved the $1.5 billion settlement agreement between Anthropic and authors who say the company infringed on their copyrights.

Our Take: The copyright cases continue to highlight why companies need to know where data used in their AI tools come from and they have the requisite permission to use IP in those tools.

United Nations AI Dialogue. The UN announced the “global dialogue on AI governance,” which would create a panel of experts to study best practices for AI governance.

Our Take: The UN is trying to stake its claim on AI standards but the final recommendations will likely remain at a high-level, making it difficult to operationalize at the enterprise level.

AI Bills in California. Governor Gavin Newsom signed SB 53 into law, which is a watered down version of last year’s SB 1047.

Our Take: The new law imposes new governance obligations for frontier model providers with some limited downstream impacts.

In case you missed it, here are additional AI policy developments:

Asia. AI-related policy developments in Asia include:

• China. Deepseek launched DeepSeek-V3.2-Exp, a new model billed as an “intermediate step toward [their] next-generation architecture.” The new model is likely to put new pressure on other Chinese model providers like ​Alibaba, which launched their new Qwen 3 Max model earlier this month.

• Japan. The Ministry of Defense is taking a hard line on AI in the military, allowing it to help with defense operations but maintaining that humans must be in charge of lethal force decisions. Japan’s new policy comes as it formalized the SAMURAI Project with the U.S., which is intended to advance AI safety in unmanned aerial vehicles.

• India. The government signed a new partnership agreement with Venezuela to “jointly explore the integration of [AI] and digital public infrastructure in sectors such as health, payments, and education.”

Australia. The Digital Economy Minister reiterated that Australia will take a light touch approach to AI regulation. The Australian government was interested in a more comprehensive approach but have since abandoned that effort.

Europe. AI-related policy developments in the Europe include:

• EU. The European Commission (EC) may pause implementing the AI Act, after strongly rejecting the idea back in July. The potential pause will be discussed at an upcoming AI Board meeting in October primarily because of implementation delays at the national level. The EC also published draft guidelines for reporting serious incidents as required under the AI Act.

• Italy. The Italian government passed a new AI law that criminalizes certain uses of AI, such as creating deepfakes or assisting with committing crimes. The law also requires that children under the age of 14 get consent from their parents to access AI.

Middle East.

• UAE. Sam Altman met with President Sheikh Mohamed bin Zayed Al Nahyan to discuss how to foster closer cooperation on AI.

• Saudi Arabia. Representatives from South Korea met with officials in Saudi Arabia to discuss closer collaboration on building more innovative environments for SMEs and expanding new market opportunities.

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. The AI and Digital Innovation Minister announced a new AI Task Force that will work over the next 30 days to make recommendations for Canada’s national AI strategy. It is unclear whether these recommendations will turn into actual regulations.

• Mexico. CloudHQ announced a $4.8 billion data center in a state just north of Mexico City.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team