1. What the AI Audit Ecosystem Can Learn From the Delve Scandal

The cybersecurity world has spent the past few weeks reeling from a major scandal involving Delve, a YC-backed compliance startup that promised to get companies SOC 2 compliant in days using proprietary AI. SOC 2 is a general cybersecurity standard that most startups need before selling software to enterprises. According to a series of Substack posts, Delve didn’t have much in the way of AI, allegedly stole another company’s IP, and was auto-generating fraudulent SOC 2 reports through offshore firms. Delve has since been disowned by its investors, lost its most notable customers, and sparked an ongoing public debate about the “race to the bottom” in the SOC 2 world.

There’s plenty of AI directly involved in the Delve scandal, but there are also important lessons for the developing AI assurance and audit ecosystem. While many criticize SOC 2 as too light, consisting mostly of check-the-box activities, it can be a useful education for early-stage startups learning which basic security controls to put in place, and it’s often the first stepping stone towards heavier certifications. The real issue is less about the standard itself and more about the incentives surrounding it. The first problem is that SOC 2 lacks a strong auditor certification and enforcement ecosystem. It was created by the AICPA, a trade association of public accountants, originally to set standards for sharing confidential financial data with auditors, and has since been extended to cover SaaS platforms broadly. Unlike ISO, the AICPA does not formally certify and credential its auditors. The second problem sits on the demand side. Many enterprise procurement teams don’t understand how startups work and demand unqualified SOC 2 reports, even though the intent of the standard is to provide transparency about risks that can then be negotiated. Procurement teams will often use “findings” (auditor observations about control gaps) as an excuse to eliminate vendors rather than as a starting point for risk-based discussions. This creates intense market pressure for performative compliance over honest disclosure, and rewards bad actors over those being transparent. Given how quickly AI is evolving, any audit or assessment will have limitations. Businesses that start demanding “perfect” AI audits risk creating the same dangerous incentives, reducing the amount of meaningful risk management done for procured AI systems.

Key Takeaway: The incentives in an assurance ecosystem matter as much as the standards themselves. Right now, most risk information about AI isn’t being disclosed because teams worry that disclosures will be treated as admissions of liability or make their systems less attractive. Policymakers should think hard about how to make the opposite true, where transparent disclosure of risks and audit findings is rewarded, not punished, in the market.

2. Tech Explainer: Understanding Agent Memory

Memory is a core component of AI agents, but the term can refer to several different things. As agents increasingly operate across multiple sessions and workflows, how they store and retrieve information has direct implications for transparency, data rights, and security.

Short-term memory refers to the data passed to a model during a single interaction, typically conversation history, system prompts, and tool outputs. Developers may use summarization to condense previous interactions to fit a model’s context window, which can result in performance degradation if important details are not preserved in the summary.

Long-term memory refers to the use of external databases that track information across multiple interactions with an AI Agent. A simple form might include a database of previous conversations (episodic memory); a more complex form might include a knowledge base that summarizes information across sessions (semantic memory). For semantic memory, new records are often created through agentic processes that analyze previous conversations, meaning the agent is deciding what to remember. The agent interacts with the memories using tools, similarly to interactions with other external resources.

Depending on the nature of the store, it may be difficult to audit and manage the long-term memories. Deleting a specific chat from an episodic store may be straightforward, but summarized semantic knowledge is harder to disentangle. If two conversations contributed to a stored fact and one is deleted, the system may not be able to determine whether the fact should be forgotten.

Agent memories also present a vector for adversarial attacks. If a malicious actor gains access to the long-term memory database, they can plant bad data and execute indirect prompt injections that persist across sessions.

Key Takeaway: While memory can make agents more effective, it introduces new governance challenges. Users of external AI systems need to understand how their “memories” are managed, while developers need to account for the transparency, legal and security risks associated with long-term memory stores.

3. Incident Spotlight: When the Model Doesn’t Know What It Doesn’t Know (Incident 1446)

During KBS’s live YouTube broadcast of the Artemis II launch on April 1, the broadcaster’s AI real-time translation system rendered the mission control phrase “Roger, roll, pitch” as “Roger, roll, b*tch” in Korean subtitles. The error spread quickly on social media and KBS issued an apology the same day. The fix was straightforward: disable rewind, remove the clip, and commit to improving profanity filtering. Case closed, apparently.

But that resolution actually obscures the more interesting governance failure here.

Why It Matters: KBS’s response framed this as a profanity filtering problem, and their proposed fix, strengthening the profanity filter, treats it as one. The AI system mistranslated “pitch” as an English expletive, then rendered its Korean equivalent, because it failed to recognize the aerospace context of the communication. The underlying issue isn’t that a bad word slipped through a filter; it’s that the system had no representation of the domain it was operating in. Aviation and mission control communication is highly formalized, uses a specific vocabulary, and is nothing like the natural language corpus these models are typically trained on. Adding profanity filtering is a patch on a context problem. The next domain-specific failure, whether in a medical broadcast, a legal proceeding, or a financial earnings call, will produce a different kind of error that the patched filter won’t catch.

This incident also sits at the edge of a broader unresolved question: who bears accountability when an AI-generated output causes harm during a live, unedited broadcast? KBS worked with an unnamed external partner for the translation system, and its apology references “close consultation with relevant departments and external companies” to prevent recurrence. That language is telling. When the vendor relationship is opaque and the system is live, the contractual and editorial accountability structure is rarely established in advance.

How to Mitigate: The immediate lesson isn’t “add profanity filters”; it’s “don’t deploy general-purpose translation models in specialized domains without domain adaptation or human review gates.” For broadcasters and enterprises running real-time AI outputs in public-facing contexts, the risk controls should mirror those applied to any live content: a human in the loop capable of interrupting the stream, domain-specific fine-tuning or prompt configuration for the subject matter, and a vendor contract that clearly allocates responsibility for output errors. Some broadcasters using AI captioning tools have begun requiring vendor indemnification clauses for live content errors specifically. That’s the right instinct, though the contractual frameworks are still nascent.

Key Takeaway: Profanity filtering is not a substitute for domain-appropriate AI deployment. Any organization running AI-generated outputs in a live or real-time context, in any specialized domain, should establish both a human interrupt capability and clear vendor accountability before go-live, not after the first incident.

4. Trustible AI Governance Market Guide

The AI governance software market is crowded, confusing, and increasingly hard to navigate. Search “AI governance platform” and you’ll find hundreds of products sharing the same label: AI firewalls, privacy compliance tools, model monitoring services, cybersecurity GRC products with a new AI module. They’re all technically accurate descriptions, and they’re all describing fundamentally different products built for different teams solving different problems. That confusion leads to real procurement mistakes, organizations buying the wrong tool, or forcing a point solution into a coordination role it was never designed for.

To help cut through the noise, we put together a market guide that maps 16 distinct categories of platforms claiming some version of “AI governance,” organized by where they sit in the technology stack. For each, we describe what it actually does, who buys it, and where it falls short on the broader governance mandate. Whether you’re building a program from scratch, writing an RFP, or just trying to make sense of a vendor pitch that landed in your inbox, it’s designed to give you a clearer frame for evaluation. Read the full guide here.

5. Policy Round Up

Fannie Mae: Fannie Mae has released their first AI governance framework for the use of AI in mortgage lending. It includes guidelines for policies and procedures that sellers and servicers must abide by if utilizing AI/ML in the selling/servicing of Fannie Mae loans. It emphasizes transparency, risk management, and requires an owner of the AI use case to assume the responsibility of implementing and maintaining this framework.

• Our take: This follows in line with guidance released by Freddie Mac, one of the other major players in mortgage lending. While there may not be federal guidance, regulated industries are pushing forward with risk management practices.

OMB Compliance: Deadlines for compliance with high-impact AI risk management practices have recently passed. Several agencies also missed their deadlines for posting updated AI inventories, a crucial step in determining what cases are high-impact. Additionally, a requirement of OMB’s AI Acquisition guidance directs agencies to contribute to a repository of AI acquisition best practices. Monday’s new GAO report found agencies struggled with this due to a lack of centralized documentation and agency-level guidance.

• Our take: While the goal of the OMB guidelines is to streamline AI adoption, the requirements can actually be quite a heavy lift for agencies.

CA EO (3/30): Governor Newsom signed a new state Executive Order on the state’s procurement of AI services. It aims to provide a new process for the state’s AI procurement in effort to allow CA to separate from federal government’s processes in light of the Trump Administration’s recent “contracting missteps” (i.e. Anthropic). It gives the state the ability to do its own assessment of the policies and safeguards of AI companies and utilize the tool upon approval, even if this goes against the federal government’s supply chain risk designations.

• Our take: The Trump Administration has thus far expressed intent to allow states to have rights over their AI procurement processes, but it is unclear whether a state has the ability to override a national security designation. We can expect there to be more legality questions once the specific guidelines come out around August.

In case you missed it:

• China: The Chinese government has released Interim Measures for the Management of Anthropomorphic AI Interaction Service. It covers protections, especially for children, who are interacting with AI services that both 1) possess anthropomorphic features and 2) provide emotional interaction. It is a novel step forward in the already pretty rich well-developed Chinese AI regulatory landscape.

• xAI sues Colorado: xAI is suing the state of Colorado over their anti-discrimination AI law, set to begin enforcement in June. xAI argues that this would be a first amendment, free speech violation and would force them to “embed the State’s preferred views” into their systems.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. Sora’s Death Ushers in the Era of Enterprise AI

OpenAI recently announced that they were sunsetting Sora, their groundbreaking consumer oriented AI video app. Sora’s latest models proved capable of generating highly realistic videos with very simple text prompts. While Sora had a variety of guardrails in place, and enforced content watermarking, videos it generated were implicated in at least 19 incidents, and may have likely been used in many other deep fake related incidents. OpenAI cited a desire to focus more on enterprise applications, especially as competition from Anthropic and their breakout Claude Code system recently overtook OpenAI’s ChatGPT amongst new users and in the Apple App Store.

We think this is just the beginning of a strategic pivot of many AI companies away from consumer applications of AI, over to enterprise related uses. The consumer world is rife with legal issues, and regulators are starting to take notice. Even the most conservative AI legislative proposals in the US have strong protections against non-consensual sexual content, and age restrictions for many AI systems. In addition to regulatory pressures, businesses are finding Agentic AI to be more useful than simple chat-based interfaces, and Anthropic’s focus on coding, and on business safe AI has forced OpenAI to declare a ‘code red’ to stay competitive. Finally, the costs of AI are still being heavily subsidized, and geopolitical events are likely to exacerbate the situation making consumer oriented apps highly unprofitable in the short term. B2B opportunities will be seen as less risky from a legal and regulatory perspective, and will be easier to prove value and align the ‘real’ costs of AI to its enterprise value. Other OpenAI projects may get cancelled, such as their efforts of creating an ‘adult’ version of ChatGPT.

Key Takeaway: When it comes to choosing an AI partner in the B2B world, reputation and principles will matter. Ethics aside, choosing a model provider that invests heavily in preventing illicit content, simply reduces risk, and choosing that company isn’t performative ethics, but rather just risk reduction.

2. Tech Explainer: Promises and Pitfalls of Distillation

In a new partnership, Apple is using Gemini models to train effective smaller models that can be run directly on phones through a process called distillation. In addition to creating effective smaller models, the process can also be used by adversarial parties to reverse engineer models (Anthropic recently wrote a report on three chinese labs using this process to attack their models). The process works by having a smaller ‘student’ model learn to mimic the outputs of a larger ‘teacher’ model, rather than training from scratch on raw data. With LLMs, the smaller model is typically trained on input+reasoning traces from the larger models. This process works, because the reasoning traces give the model the most important pieces of information needed to function properly. The result is a smaller model that can run locally, without sending data to an external API, which can reduce latency and alleviate some privacy-concerns.

Distillation comes with risks. Because the smaller model has fewer parameters, it can’t capture as much nuance and may lead to performance degradation - in general, it may be best to use the smaller model for more specialized tasks. For example, Apple may only need Gemini to be good at question-answering inside Siri, so they can skip teaching the student model coding capabilities. Deployers need to run fresh evaluations calibrated to the distilled model’s narrower scope, not just port over benchmarks from the original. In addition, recent research showed that knowledge distillation may lead to systematic degradation of safety alignment and increase susceptibility to jailbreaks.

Key Takeaway: On the surface, distillation may be appealing because it can create smaller models that encapsulate key abilities of larger models without relying on external APIs. However, in practice, they require additional safety post-training, strong guardrails and new evaluations, which can make the whole process more time and resource intensive.

3. Incident Spotlight: DOGE’s ChatGPT Grant Review (Incident 1402)

What Happened: DOGE fed grant descriptions into ChatGPT, asking it to determine whether each was “DEI,” then logged the chatbot’s yes/no responses in a spreadsheet that replaced a list previously compiled by NEH staffers as the operative document for terminating grants. Of 1,163 grant proposals analyzed this way, 1,057 were flagged and just 42 were kept. The process was ad hoc by design: the DOGE staffer behind the methodology had assembled his own “Detection List” of identity-based traits before running grant descriptions through the model. Depositions later confirmed that the NEH’s acting chair hadn’t known ChatGPT was used in the selection process at all.

Why It Matters: Setting aside the political element, there are a number of issues here. Firstly, there’s no strong evidence that the team deploying ChatGPT had proper training on AI systems. Their prompt seems notably simplistic, and arguably gave enormous authority to the AI system to interpret DEI, and process the grants accordingly. Even small limitations in tools, such as poor text extraction, context window lengths, or hallucinations could have caused massive impacts. In addition, the DOGE team did not seem to have a firm grasp of the documents they were working with, and therefore could not act as qualified ‘humans in the loop’. The automated decision making nature of their request would likely be qualified as ‘high impact AI’ under the Trump admin’s latest guidance for AI use in non-classified settings, although this was published after the DOGE work was supposedly done.

How to Mitigate: This is fundamentally a process design failure. Any organization using AI for consequential screening decisions should define and document classification criteria before deployment, not derive them post hoc from a vague policy directive. AI-generated classifications should be treated as inputs to human review, not substitutes for it, with clear audit trails that distinguish model output from final decision.

Key Takeaway: If your organization is using AI to screen or classify anything with legal or financial consequence, someone with both domain expertise and working knowledge of the model’s limitations needs to own the classification logic. Deploying a general-purpose chatbot as a compliance filter without either is a massive liability.

4. IAPP Recap

Earlier this week at the IAPP Global Privacy Summit, we co-hosted a panel that focused on what AI governance looks like after Year One. The room assumed policies were written, intake processes established, and governance committees defined. The conversation was about what comes next.

CTO Andrew Gamino-Cheong moderated with Kimberly Zink (Chief Privacy Officer, Korn Ferry) and Derek Han (AI, Cyber and Privacy Partner, Grant Thornton). Four scenarios drove the discussion.

On Model Changes: Every use case should have a documented set of evaluations before a deprecation notice arrives, not assembled under pressure. What counts as a “substantial modification” needs to be defined in advance.

On Periodic Reviews: Governance intensity should scale with risk level, not apply uniformly. Model drift doesn’t announce itself. Sampling actual outputs against deployment guardrails matters more than a calendar reminder.

On Regulatory Updates: Nearly 7 in 10 businesses report difficulty understanding EU AI Act obligations. The root cause is inventory quality. If your AI inventory doesn’t capture use case category, PII usage, automated decision-making, and deployment geography, you can’t answer a scope question quickly.

On Program Iteration: Track the metrics that tell you whether governance is actually working: volume reviewed, high-risk flags, cycle time, risk mitigated. The harder conversation is agentic AI. Manual governance workflows weren’t built for systems that act autonomously, chain decisions, and scale faster than any review queue. Organizations need to start building AI-assisted governance now.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

(Source: Claude)

1. Why AI Monitoring is Hard

When enterprises say they want to monitor their AI systems, they rarely mean the same thing. For a security team, monitoring means watching for adversarial attacks and prompt injection. For a compliance officer, it means tracking regulations and court cases that could impact their AI systems. For a product team, it means making sure the model hasn’t quietly gotten worse at the thing it was deployed to do. A new report from NIST formalizes this fragmentation, organizing post-deployment monitoring into six distinct categories that rarely get discussed together:

• Functionality — Is the system still working as intended? (Detecting drift, staleness, performance degradation)

• Operational — Is the infrastructure running reliably? (Latency, uptime, logging across distributed systems)

• Human Factors — How are users actually interacting with the system? (Feedback loops, over-reliance, sycophancy)

• Security — Is the system being attacked or misused? (Adversarial inputs, deceptive model behavior, misuse detection)

• Compliance — Is the system adhering to relevant regulations and policies? (Terms of service violations, regulatory adherence)

• Large-Scale Impacts — Is the system promoting or degrading human well-being at a population level?

The report, drawn from three workshops with over 250 practitioners and a review of 87 papers, finds that most organizations are only monitoring one or two of these categories, and the field lacks agreed-upon methods, shared terminology, and basic consensus on who in the AI supply chain is even responsible for each. The incentive problems compound this: monitoring is expensive, publicly reporting incidents carries legal and competitive risk, and AI outputs are non-deterministic enough that establishing a reliable performance baseline is itself an open research problem. For deployers specifically, the report is a formal acknowledgment from NIST that the governance burden is shifting downstream, and the tools and standards needed to manage it don’t yet exist. The best practices for collecting and analyzing this kind of data are highly immature, and quickly shifting alongside the AI technology stack.

Trustible’s own AI Monitoring whitepaper, and blog post separates the ideas of ‘internal’ monitoring which focuses on analysing highly technical data about the relevant AI system, and ‘external’ monitoring which aims to collect information from outside the deployers boundaries. Internal monitoring largely maps to NIST’s Functionality, Operational, and Security factors while, ‘external’ monitoring maps to the Human Factors, Compliance, and Large Scale Impacts elements.

Key Takeaway: AI Monitoring is all of these things, and there’s therefore no ‘silver bullet’ solution for all forms of AI monitoring. The NIST framework is a useful forcing function for governance teams to audit which monitoring categories they’ve actually addressed and which they’ve implicitly assumed someone else owns.

2. Tech Explainer: The Trouble with Evaluations

As AI systems grow more advanced, evaluating them gets more complicated; at the same time, we are seeing a decreased consistency in how evaluations are reported for general-purpose AI models. Unlike traditional machine learning, where models are built for a specific purpose (e.g. predicting if an email is spam) and can be evaluated against that goal, GPAI models are adapted for downstream tasks and assessed benchmarks (e.g. math problem solving) that are often inconsistently applied. Two providers can report scores on the same benchmark while using different prompting strategies or answer aggregation techniques, making direct comparisons unreliable. A benchmark can also misrepresent what it claims to measure: interview-style coding questions won’t tell you much about how a model performs in a production system.

The core problem is a lack of standards for what to report. Trustible’s work with the EvalEval coalition produced a universal schema for documenting evaluation results, giving developers and deployers a consistent reporting standard and a way to compare why scores on the “same benchmark” diverge. Related efforts like BenchRisk, EvalFactSheets, and the Construct Validity Checklist give developers and deployers structured tools to report on and assess benchmark quality before trusting the results. Adoption is still limited, but these are now available reference points for model developers, deployers, users and policy-makers. The challenge will get harder as evaluations shift from models to agents, where the harness (i.e. tools and integrations surrounding a model) shape results as much as the model itself.

Key Takeaway: AI Literacy requires skeptically reviewing performance headlines touted by AI providers. While organizations should construct internal benchmarks when picking the best models for their systems, many of the same challenges persist and the frameworks shared can help.

3. AI Incident Spotlight - AI Style, Personality, and Identity Theft (Incident 1407)

What Happened: In late 2025, Grammarly launched an “Expert Review” tool where subscribers could upload writing and receive real-time editing feedback presented as coming from named journalists, authors, and academics, including novelist Stephen King and tech journalist Kara Swisher. Grammarly never sought or obtained consent from any of the named experts whose identities it used to sell the feature. In response, journalist Julia Angwin filed a lawsuit against Grammarly’s parent company Superhuman, and the feature was pulled shortly after.

Why It Matters: While the visual and auditory ‘likeness’ of a person has been discussed in depth in the context of AI generated images and video, this incident highlights an additional question of whether a person’s style and personality should also be included. In her lawsuit, and related NYT Op-Ed, Angwin argues: “My ability to earn a living rests on my ability to craft a phrase, to synthesize an idea, to make readers care about people and places they can only access through words on a page,”. For content creators, journalists, and subject matter experts whose reputations are themselves a professional asset, the commercial use of an AI simulation of their expertise is an existential threat to their way of life, and capable of causing massive reputational harm if the system is wrong. In this case, Grammarly is exposed from a liability angle, but the broader question about what constitutes ‘likeness’, and what rights someone should have to it are still unresolved. Existing privacy laws that may cover a person’s visual likeness under the context of ‘biometric’ data don’t clearly apply to a person’s ‘content style’.

How to Mitigate: Before shipping any feature that associates named individuals with AI-generated output, confirm you have explicit written consent or a licensing agreement. Some music and film artists have started to sell these rights to AI platforms. While AI laws targeting deep fakes are still being debated or implemented, existing tort law can still apply if there are reasonable claims of loss of income resulting from the AI. Right-of-publicity review should be a mandatory gate in the product development lifecycle for any feature involving real people’s names, styles, or personas, and that review needs to happen before engineering begins, not at launch.

Key Takeaway: There are many open questions about what constitutes a person’s likeness, and further debates about what kinds of rights a person should have around their likeness, and how to balance these issues with freedom of speech. Broader questions around likeness after death, or how to enforce these things on a global scale are unlikely to be resolved any time soon.

4. Policy Roundup

Anthropic vs. the Pentagon

The Trump administration officially labeled Anthropic a “supply-chain risk” and banned government agencies and military contractors from using Claude after contract negotiations broke down over two conditions Anthropic refused to drop. While they publicly threatened an aggressive stance on the issue, the formal notice was more limited in scope and only prohibits use of Claude to directly support DoD contracts. Despite the narrower designation, Anthropic filed two suits against the DOD, calling the actions “unprecedented and unlawful”. Major tech companies, including Microsoft, have voiced strong support for Anthropic.

Our Take: Anthropic may have ‘lost’ the battle, but in doing so, may be winning the war. At least PR wise, as consumer Claude downloads spiked as a result of the discussion. Organizations may need to ensure that they never get too deeply locked in with a single model provider in case LLM selections get further politicized over time. Organizations may also need to invest in processes for safely switching model providers.

EU AI Act Amendments Near the Finish Line

EU Parliament lawmakers reached a political deal within the Parliament that adds an explicit ban on AI-generated non-consensual intimate images and eases compliance rules for AI embedded in regulated products like medical devices. Separately, the EU Council approved its negotiating position on the Digital Omnibus, which would push back high-risk system deadlines by up to 16 months pending the availability of compliance standards.

Our Take: Now that the 3 main bodies of the EU policy making apparatus (Commission, Council, Parliament) have solidified their positions, political negotiations will begin between them. Given the common positions, enforcement of the high risk AI requirements of the EU AI Act are unlikely in 2026. Enforcement in late 2027, or early 2028 is now likely.

US Supreme Court Closes the Door on AI Copyright

The US Supreme Court declined to hear an appeal on Thaler v. Perlmutter, leaving intact the rule that works generated entirely by AI are ineligible for copyright protection. For now, Human authorship will continue to be a requirement for receiving IP protections in the US.

Our Take: Requiring human involvement to receive IP protections is a good balancing act in the AI ecosystem as it protects content creators in a fair way. However we expect more aggressive lobbying by tech firms on this issue over the next few years.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

(Source: Google Gemini)

1. AI Tools Are Making Employees Work More, Not Less

The sales pitch for enterprise AI adoption goes something like this: AI handles the tedious stuff, your employees focus on higher-value work, everyone’s happier and more productive. New research described in the Harvard Business Review suggests the reality is closer to the opposite. In an eight-month study of a 200-person tech company, researchers found that generative AI tools didn’t reduce workloads. They intensified them across three dimensions: employees expanded into tasks outside their roles (designers writing code, PMs debugging), work bled into breaks and off-hours as the low friction of “one more prompt” eroded boundaries, and constant multitasking across parallel AI workflows created persistent cognitive load. None of this was mandated. Workers did it voluntarily because AI made “doing more” feel accessible and even enjoyable.

For AI governance professionals, this matters because it complicates the risk calculus around enterprise AI deployment. The obvious governance concerns with AI tools, things like data leakage, IP exposure, and accuracy, are well understood. But work intensification introduces a quieter set of risks that most AI governance frameworks don’t account for. Employees operating outside their core competencies with AI assistance means more AI-generated or AI-assisted outputs flowing through an organization with less qualified review. The researcher’s finding that engineers spent increasing time correcting “vibe-coded” pull requests from non-engineering colleagues is a concrete example of how quality control can quietly degrade. Meanwhile, the burnout cycle the study describes, where early productivity gains give way to cognitive fatigue and lower decision quality, suggests that organizations measuring AI’s impact purely through short-term output metrics are likely overstating the long term benefits.

Key Takeaway: Reviewing AI outputs across multiple tasks may be cognitively burning teams out, and expectations for productivity and outcomes from executives are rising. AI use policies may need to adapt to include acceptable patterns for taking a break from AI, and understand the cognitive impacts of rapid context switching and constant reviewing of AI outputs.

2. Technical Insight: Prompt Repetition

Google researchers recently published a paper showing that simply copying and pasting a prompt twice into the input, with no other changes, consistently improves LLM accuracy across Gemini, GPT, Claude, and Deepseek models. The technique, called “prompt repetition,” won 47 out of 70 benchmark tests with zero losses, added no meaningful latency, and didn’t change the length or format of the model’s output. Because LLMs process tokens left to right, early tokens in a prompt can’t “see” later ones. Repeating the prompt gives every token a second pass where it can attend to the full context. There’s also a simpler intuition at play: repetition likely strengthens the internal representations of the input, effectively increasing the “weight” the model assigns to the prompt’s content relative to its prior training biases. The gains are modest on standard benchmarks but dramatic on tasks requiring attention to information buried in long inputs, exactly the kind of problem that plagues document-heavy enterprise workloads.

For governance teams, the more interesting implication is what this says about evaluation. If a trivial input transformation can meaningfully shift benchmark scores, it raises questions about how stable published model evaluations really are. Two organizations testing the same model with slightly different prompt formats could reach very different conclusions about its reliability. Prompt repetition works best when reasoning mode is off, which is how most enterprise API calls operate for classification, extraction, and structured output tasks, meaning it’s a real and essentially free improvement. But its bigger lesson is that small methodological choices in evaluation can have outsized effects on results.

Key Takeaway: Prompt repetition is worth testing for non-reasoning API workloads, but governance teams should treat it as a reminder that model evaluations are more fragile than published results suggest, and should account for prompt sensitivity when comparing models or setting performance thresholds.

3. Trustible Joins the Coalition for Health AI

We’ve partnered with the Coalition for Health AI (CHAI) to bring CHAI’s AI Governance Framework directly into the Trustible platform. Healthcare organizations can now map their AI governance activities to CHAI’s guidance with structured workflows, healthcare-specific risk assessments, and audit-ready documentation. For health systems trying to move from AI ambition to confident deployment, this removes the need to build governance practices from scratch or adapt generic frameworks that miss healthcare’s context. Read more about the partnership here.

4. AI Incident Spotlight - When AI Alerts Lack Sufficient Context (Incident 1374)

What Happened: A nurse at a Nevada hospital described an episode in which the facility’s AI sepsis alert system flagged an elderly patient with low blood pressure and triggered urgent protocol steps, including IV fluids. The nurse noticed the patient had a dialysis catheter, meaning her kidneys were already compromised. Pumping IV fluids into a patient who can’t process them risks dangerous fluid overload. When the nurse objected, he was told to proceed anyway because the AI had generated the alert. He refused, and a physician ultimately intervened with an alternative treatment that avoided the risk. The incident, reported by Scientific American in February, is part of a broader pattern of clinical AI systems generating recommendations that conflict with what’s observable at the bedside.

Why it Matters: The model was working as designed. It detected signals consistent with sepsis and triggered the correct protocol. The problem is that it had no way to know about the dialysis catheter, a piece of real-world physical context visible to any clinician in the room but absent from the electronic health record the model was reading. This is a recurring blind spot: clinical AI systems operate on structured digital data, but a significant share of relevant information only exists at the bedside. What the alert did next is the governance concern. It created institutional momentum. Protocol kicked in, and the nurse’s clinical objection was initially treated as non-compliance. The AI didn’t just inform the decision, it set the default, and overriding it required escalation.

How to Mitigate: Organizations deploying clinical AI alerts need workflows where alerts inform rather than direct. That means building override mechanisms that don’t require escalation and ensuring frontline staff have clear authority to act on evidence that contradicts a model’s output. This incident also highlights the limits of models that rely solely on digitized records. Physical context, like a dialysis catheter, is exactly the kind of information that’s hard for a model to ingest. Until that gap closes, human review is the only reliable way to catch what the system can’t see.

Key Takeaway: An AI system is only as good as its inputs, which includes any and all relevant context. Missing context for a system is one of the biggest potential sources of error and one that often can be missed in many of the ‘evaluations’ that are done to a system before deployment. Knowing what information an AI system can access, and what it’s input limits are is an essential part of governance.

5. Policy Roundup

DOD and Anthropic. Anthropic has been in a heated battle with the Department of Defense (DoD) over the use of their models. The DoD has put pressure on Anthropic to drop safeguards and allow for their models to be used from a wider range of military purposes.

Our Take: While AI safety has been prioritized under the Trump Administration, forcing a model provider to lower safeguards will have ripple effects across the ecosystem and may worsen the trust gap with AI.

Utah’s own RAISE Act. Lawmakers in Utah have been quietly working on passing a law that is very similar to California’s SB 53 and New York’s RAISE Act. The bill is currently making its way through the state house. One key difference is an added requirement for model providers to develop a child safety plan for their models.

Our Take: The Trump Administration has sought to deter state action on AI, but concerns with AI safety, as well as its impacts on jobs and children, have been relatively bipartisan.

India AI Summit. India recently held the latest global AI summit, which is a continuation of the work started in Paris last year and the UK in 2024. The focus was mainly on AI innovation and attendees scolded the EU for its prescriptive approach to AI oversight.

Our Take: The shift in global attitudes towards AI regulations has been swift and the focus of these global AI summits showcases that dramatic turnabout.

In case you missed it, here are a few additional AI policy developments making the rounds:

• Asia. The newly elected Japanese government will host a ministerial meeting to discuss how AI can be effectively utilized by the government. In Korea, the National AI Strategy Committee adopted a final AI Action Plan at its second plenary session.

• Australia. The Australian government is scrapping its AI Advisory Board after launching it 15 months ago. The Advisory was charged with setting out recommendations for AI safeguards.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

1. The Dangers of Desktop Agents

2. Trustible Partnership Announcements

3. Tech Explainer: Context Engineering

4. Incident Roundup: CASM Found in Major Image Training Dataset

5. Trustible’s Top AI Policy Stories

(Source: Google Gemini w/ Nano Banana)

1. The Dangers of Desktop Agents

Two similar desktop agentic AI apps have made headlines in the past few weeks. Anthropic’s Claude Cowork brings their Claude Code capabilities to non-developers via a sandboxed macOS app that can read, edit, and create files autonomously. OpenClaw (formerly Moltbot, formerly Clawdbot) takes a slightly different approach allowing users to interact through WhatsApp or Telegram and then connecting to 100+ services for arbitrary agentic flows. While these apps have desktop apps, they still usually interact with LLMs hosted in the cloud, as most consumer grade computers cannot yet host sufficiently powerful LLMs.

For most organizations, these tools will be difficult to greenlight anytime soon. Agentic browsers like OpenAI Atlas already present massive security and privacy risks, and these general purpose desktop apps take the risks a step further. The core problem is that an automated system acting on behalf of a human breaks assumptions baked into most security architectures. Access controls, audit logs, and anomaly detection are built around the idea that a human is on the other end. Agents blur that line in ways that aren’t easy to monitor or contain. They’re also vulnerable to hijacking via prompt injection. Cisco researchers have already demonstrated a malicious OpenClaw “skill” that exfiltrated data while bypassing safety guidelines entirely. Furthermore, as both tools run as desktop apps, they can still send your sensitive local files and context to hosted LLMs in the cloud where data may be permanently stored or processed. For companies with strict data handling policies, that’s a non-starter.

Key Takeaway: These tools are impressive and getting a lot of attention online, but they are far away from being enterprise-ready.Most organizations will need to wait for better sandboxing, clearer data handling policies, and security architectures that actually account for non-human actors. Many organizations will quickly move to block such desktop applications via existing device management tools.

2. Trustible Announcements

Here’s a quick recap of some major announcements by Trustible in the past few weeks

Trustible Announces Strategic Partnership with Leidos

We’ve partnered with Leidos to bring automated AI governance to government agencies. In a proof-of-concept engagement, Leidos used Trustible’s platform to compress governance intake processes from weeks to hours, demonstrating that automation can reduce friction while maintaining the oversight that mission-critical environments require. Read the full announcement here.

Trustible Partners with the AI Incident Database

Trustible is now the lead corporate sponsor for the AI Incident Database (AIID), the most widely used public repository of real-world AI harms. Through this partnership, Trustible customers will be able to cross-reference their AI inventories against documented incidents and receive alerts when new incidents relate to models or vendors they’re tracking. Read more about the announcement here.

Trustible Publishes Pragmatic AI Policy Paper

We’ve published A Pragmatic Blueprint for AI Regulation, a policy paper offering a middle-ground framework for AI governance built around shared liability, copyright balance, child protection, content provenance, and information sharing. The paper argues that closing the AI adoption gap requires trust, and trust requires clear rules without stifling innovation. We decided to take stances on a few core AI policy areas that are often ignored in the larger ‘doomer’ vs ‘optimist’ debates, and to advocate for regulation that businesses actually want. Check out the whitepaper here.

3. Tech Explainer: Context Engineering

In recent months, context engineering has replaced prompt engineering as the focus for building effective AI agents. While the focus of prompt engineering has been on developing techniques that make an LLM respond to a specific query effectively (e.g. telling the model to “think step by step” or giving it an example output), context engineering refers to the methodology of giving an agent the right set of information at the right time. This may be challenging because at any given point, an agent may have access to a broad range of assets like tools, internal memory, external data stores, and the conversation history; however, all of this information still needs to be processed by an LLM that can process a limited number of tokens at a time. While leading frontier models can process 250k words at a time, they may not digest and recall all the tokens properly. Common context management techniques include summarization (where a separate LLM is used to condense the context), sub-agents (where each agent only needs partial context) and the use of memory (where an agent adds information to an outside store that can be referenced as necessary).

While it enables agents to perform more complex tasks, the use of memory, in particular, introduces new governance concerns. A recent study by the Center for Democracy and Technology identified that users’ key concerns around memory include:

• Persistence: Who has control over when and how memories are deleted?

• Privacy: Can memories inadvertently be shared with additional tools/systems?

• Transparency: Can a user review all the memories associated with them?

There are not yet well established best practices for managing this, and many regulations and risk frameworks don’t have specific considerations for context or memory governance.

Key Take-away: Organizations deploying AI agents will need to develop new practices that bridge traditional data governance with the dynamic nature of context management. This includes defining clear policies for memory lifecycles, implementing context boundaries between different use cases, and ensuring users maintain meaningful control over their data.

4. AI Incident Spotlight: CSAM Found in Dataset Used to Train Content Moderation Tools (Incident 1349)

What Happened: The Canadian Centre for Child Protection (C3P) discovered that NudeNet, a dataset of over 700,000 images used to train AI nudity detection tools, contained approximately 680 images of suspected or confirmed child sexual abuse material (CASM). More than 120 depicted identified or known victims. The dataset had been freely available on Academic Torrents since 2019, and C3P identified over 250 academic works that either cited or used NudeNet or classifiers trained on it. Researchers who downloaded the dataset unknowingly possessed and distributed illegal material. Following C3P’s takedown notice, Academic Torrents removed the dataset, but the classifiers and models derived from it remain in circulation.

Why it Matters: This is the second major incident involving CSAM in AI training data following the 2023 discovery of similar material in LAION-5B. This highlights the ongoing problem with large-scale web scraping without rigorous vetting that captures illegal and harmful content. But the NudeNet case is particularly troubling because the dataset was specifically designed for content moderation. Tools built to detect harmful imagery were themselves trained on it.

Given NudeNet’s wide distribution and six-year availability, the contamination likely extends beyond academic research. Foundational models and commercial content moderation systems may have incorporated NudeNet or its derivatives without disclosure. Without transparency into training data provenance, downstream deployers inherit risks they cannot assess. Model cards rarely disclose specific dataset sources, let alone whether those sources were vetted for illegal content.

The incident also illustrates a difficult dual-use problem. Building effective detection systems for harmful content often requires training on examples of that content. But assembling such datasets creates its own harms as it perpetuates distribution of illegal material, re-victimizes survivors whose images are included, and exposes researchers to legal liability. The goal of protecting against abuse inadvertently extends it without proper controls in place.

How to Mitigate: Require training data documentation from model providers, particularly for content moderation systems. If a vendor cannot explain how their training data was sourced and vetted, treat that as a material risk factor. For organizations that must include toxic content in datasets for detection purposes, that data should only be handled under strict access controls, legal compliance frameworks, and coordination with organizations like C3P or NCMEC that maintain vetted hash databases for this purpose. The NudeNet dataset existed for six years before anyone flagged it. That’s a long time to be unknowingly distributing illegal content.

Policy Roundup

The Trouble with Trump’s AI Policy. There is a growing divide between the Trump Administration’s position on AI and Republican lawmakers. Republicans at the state and federal level are currently at odds with the Trump Administration’s “AI innovation” ethos, with Republicans pushing for more oversight of the technology (e.g., Senator Marsha Blackburn’s TRUMP AMERICA AI Act).

Our Take: Outside of the Executive Branch, Republicans have been more skeptical of AI and have sought some safeguards around the technology. This divide will set up an interesting clash with the Department of Justice as it seeks to evaluate the constitutionality of state laws under the Trump AI Moratorium Executive Order.

More Turmoil with the EU AI Act. Lawmakers in the EU continue to struggle with implementing the EU AI Act. The European Commission missed its deadline for publishing draft guidance for classifying high-risk AI systems, while France is pushing a behind-the-scenes effort to separate the AI Act amendments from the EU’s larger Digital Omnibus package.

Our Take: The continuing drama over the EU AI Act is making it difficult for companies to understand their obligations and deadlines under the law. It may also show that EU lawmakers tried to do too much in one law, with the consequences now on full display.

Singapore’s New Agentic Governance Framework. Singapore’s Infocomm Media Development Authority published the “Model AI Governance Framework for Agentic AI.” This framework is one of the first comprehensive governance frameworks for agents from a government entity.

Our Take: Policymakers are usually behind the curve on technology but Singapore has been an outlier on AI guidance, producing a host of practical AI-related materials.

In case you missed it, here are a few additional AI policy developments making the rounds:

• Africa. Egypt will be hosting the first AI Everything Middle East & Africa Summit, which is intended to emphasize the region’s focus on digital development.

• Asia. South Korea’s AI law came into effect last month, with a one year grace period for enforcement. The law has recently faced major pushback from the tech industry for going “too far” and from advocacy groups for not going far enough.

• North America. The Mexican government released a Declaration of Ethics and Good Practices for the Use and Development of AI. The declaration outlines ten fundamental principles that serve as “a non-binding guide for public institutions, government agencies, autonomous bodies, as well as actors from the private and social sectors.”

• South America. A fake version of Taylor Swift’s “Fate of Ophelia” is testing the limits of Brazil’s IP law. The song, “A Sina de Ofélia,” was generated by AI using voices from two well-known Brazilian pop artists.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, and welcome to the latest edition of the Trustible AI Newsletter! We’ve got a lot of exciting news to share in the next few weeks, but for now, be sure to download our latest whitepaper on AI monitoring! Here’s our team’s latest insights:

1. Why OpenAI and Anthropic Built Dedicated Health Applications

2. How to Evaluate Healthcare AI

3. Grok’s Generation of Non-consensual Intimate Imagery

4. Trustible’s Top AI Policy Stories

1. Why OpenAI and Anthropic Built Dedicated Health Applications

(Source: Google Gemini)

Within a few days of each other, both OpenAI and Anthropic announced dedicated ‘health’ versions of their AI platforms, ChatGPT Health, and Claude for Healthcare. According to their own data, over 230 million health related requests are made per week on ChatGPT, and it’s one of the top use cases. The dedicated health applications will have specific connectors for healthcare related databases, integrations with fitness and wellness platforms, and health related questions will now be ‘routed’ to the dedicated healthcare application instead of being answered.

There are several likely motivations behind this split: opening new revenue streams, competing against health-specific wrapper companies, and a desire for more data. But we’ll focus on regulatory and compliance motivations. Under many AI related laws such as the EU AI Act, an AI service providing medical advice would be categorized as ‘high risk’, triggering a number of heavy compliance obligations. Many existing privacy, liability, and security laws surrounding health data also apply. By carving out health related uses into a dedicated app, both companies can build dedicated guardrails, infrastructure, and processes around the sensitive application. This allows their non-health applications to still innovate quickly without getting bogged down by compliance. And by putting significant effort into routing users to a ‘safer’ application for health related queries, these companies will be able to claim that the non-health version of their applications are not ‘intended’ for this high risk domain and are not marketed as such. This is actually not the first carve-out, as both companies already support a similarly dedicated platform for the US public sector. Much like with healthcare, these platforms have dedicated infrastructure, customized security and privacy controls, and a different set of guardrails.

Key Takeaway: Regulation has always shaped product architecture, and we should expect big AI companies to create dedicated platforms for each high-risk domain these laws identify.

2. Tech Explainer: How to Evaluate Healthcare AI

Given the recently announced healthcare applications from OpenAI and Anthropic, alongside Utah’s new pilot program, it’s worth digging into how these systems were evaluated. Obviously since there are bespoke capabilities, guardrails, and integrations in these applications, they require customized testing, and there is already a growing ecosystem of healthcare related benchmarks and evaluations, albeit with many limitations. Here’s a quick analysis from the information released so far:

ChatGPT Health

OpenAI evaluated using HealthBench, a benchmark of realistic healthcare conversations with physician-created rubrics. Key strengths of his benchmark are:

• Multi-turn conversations (superior to single-turn evals as quality often degrades over time)

• Multi-faceted rubrics evaluating medical accuracy, communication quality, and jargon avoidance, scored via LLM-as-a-judge

• Development by 262 physicians from 60 countries, improving cross-cultural validity

However, the benchmark doesn’t fully account for varied input formats from healthcare record providers (these integrations are central to the platform). Performance for “ChatGPT Health” was not published; but gpt-5-thinking scored 67.2%. It is unclear how this score translates to real world outcomes.

Claude for Healthcare

Anthropic’s announcement covered two functionalities: supporting healthcare professionals with prior authorizations and care coordination, and helping individuals summarize medical history and prepare for appointments. They reported Claude-4.5-Opus performance on MedAgentBench, which assesses agent capabilities in medical records contexts with pre-defined tools. While co-developed by physicians, it’s a proxy metric, and Claude’s actual tools differ from the sandbox environment. No evaluations addressed the personal healthcare scenario.

Utah Auto-Refill Program

Utah’s automated refill pilot uses Doctronic’s algorithm, which showed 99.2% agreement with physician decisions in testing. Unlike the broader AI systems, this narrowly scoped use case allows direct performance measurement on the actual task. However, testing used urgent care cases with physician-entered data, which may differ from patient chatbot inputs in production.

Our Take: Benchmarks are a useful evaluation tool, but they imperfectly simulate real-world conditions because they imperfectly capture aspects of real-world conditions like external data formats and real agentic tools. In addition, for the open-ended consumer products, it is not clear how these scores will translate to improved clinical outcomes. While the current systems have been released with a number of safeguards, better reporting standards will be necessary as AI Healthcare tools become more commonplace.

3. AI Incident Spotlight: Grok’s Generation of Non-Consensual Intimate Imagery (Incident 1329)

What Happened: In late December 2025, users discovered that xAI’s Grok would readily “undress” women in photos, manipulating existing images to create sexualized deepfakes without consent. The flood of content included images of celebrities, private individuals, and minors. Viral prompts ranged from “put her in a bikini” to far worse. Despite reports, X was slow to respond. Even one of Musk’s ex-partners struggled to get deepfakes of herself removed. After international backlash, X announced partial restrictions in mid-January, but the standalone Grok Imagine app continues generating explicit imagery.

Why it Matters: This isn’t a fringe product. Days after Secretary Hegseth announced that Grok would be integrated into Pentagon systems, including classified networks, regulators in at least a dozen countries launched investigations or outright bans. The same model generating what California’s Attorney General called an “avalanche” of illegal content is being deployed to 3 million DoD personnel.

The political dimension matters too. Even policymakers who oppose AI regulation have consistently carved out exceptions for child safety. This is one area with genuine bipartisan consensus, and incidents involving minors accelerate legislative action. By pushing boundaries on content moderation, xAI may be generating exactly the public backlash that fuels demand for stricter AI regulation across the board. Every headline about AI-generated sexual material erodes trust in AI broadly, not just in Grok.

How to Mitigate: Treat content moderation capabilities as a procurement criterion. Before deploying any image generation system, request documentation on what categories are blocked and how. For organizations considering Grok or X API integration, this incident warrants a serious risk assessment, particularly for customer-facing applications where generated content could create legal exposure.

4. AI Policy Roundup

Next Steps on AI Moratorium EO. The Department of Justice issued a memo establishing a task force to challenge state AI laws, as directed under the Trump AI Moratorium EO. The EO also calls for federal legislative proposals to regulate AI, but the director of the Office of Science and Technology Policy offered few details on the Administration’s plans at a recent congressional hearing.

Our Take: The EO spurred controversy even before it was signed because of the power it attempts to assert over states to regulate AI. It has not appeared to blunt momentum at the state level to pass AI laws, as several state legislatures have introduced bills in 2026.

ChatGPT’s Confidentiality Quagmire. Sam Altman recently asserted that OpenAI does not have an obligation to keep sensitive information confidential when people use ChatGPT as a therapist. Altman acknowledged that privacy concerns with AI may hinder adoption.

Our Take: Model providers are further blurring the lines between their products and privacy obligations. Health privacy laws like HIPAA and HITECH do not explicitly cover products like ChatGPT, but as the models expand into offering health services (e.g., digital therapy) that may change.

Congress Targets Deepfake Porn. Congress is considering bipartisan legislation that would allow victims to sue over nonconsensual sexual images. The DEFIANCE Act passed the Senate unanimously and will head to the House.

Our Take: Congress is responding to concerns over Grok’s capabilities to produce fake, sexually explicit deepfakes. This is one of the rare times that federal lawmakers agree on creating a private right of action, which allows individuals to sue. The law does not ban these images, though is thought of as a complement to the TAKE IT DOWN Act.

In case you missed it, here are a few additional AI policy developments making the rounds:

• Africa. Nigeria is working towards passing a comprehensive AI law, making it among one of the first countries in Africa to enact such a law. The law is primarily focused on safeguards for high-risk systems, and would allow regulators to demand information from providers for non-compliance. The law is expected to be enacted in March 2026.

• Asia. The Taiwanese legislature passed an AI basic law towards the end of 2025 and that law came into effect on January 14, 2026. The law outlines a series of principles for AI development and deployment, though has no specific enforcement mechanism.

• Europe. Regulators in the EU and UK are considering consequences for AI tools that can create sexually explicit images. The concerns come in the wake of the controversy over Grok’s ability to produce sexualized images. EU lawmakers are considering banning the technology altogether and the UK government is threatening to revoke xAI’s ability to self-regulate.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, Happy New Year, and welcome to the first 2026 edition of the Trustible AI Newsletter! 2025 proved to be a critical - but tumultuous - year in the world of AI, and we don’t anticipate that trend changing in 2026. But, as we navigate what’s to come across the technical, policy, and business landscape of AI, we do believe in one constant: that 2026 will be a transformative year for AI governance, as it becomes the primary business imperative that will drive how enterprises will actualize the positive ROI of AI.

In this week’s edition, we’re sharing our 2026 predictions across what’s in store for AI governance, AI technical trends, AI incident trends, and what’s around the corner in the policy and regulatory sphere.

Let’s dig in.

1. Trustible’s 2026 AI Governance Predictions

We aren’t alone in predicting that 2026 will be the “make or break” year for AI. There are a number of consequential questions that will likely be answered in 2026, including whether AI agents will be adopted at scale, whether major AI regulations in the EU and across U.S. states will actually come into force in their current form, and whether the AI bubble will burst, or continue to grow. These are monumental questions that policymakers and professional talking heads will continue to debate, but all of them also have implications for teams tackling AI governance.

Here are our predictions on AI governance for 2026:

• AI Governance Beyond Intake - Many organizations now have robust policies, fully populated inventories, and initial risk assessments. What comes next is a lot of change management for systems already in place. This work may look very different from earlier governance work.

• AI Agents Become Mainstream - At this point last year, even many AI practitioners may have never heard of an MCP server, or had reviewed a proposed agentic AI system. This year, many organizations have mandates to deploy agentic AI workflows, and a lot of people are hoping that agentic AI provides the value that chatbot copilots did not.

• Growing Third Party Risks - As agentic AI rolls out inside organizations, knowing which tools and platforms are connected to AI systems will become its own inventorying challenge, and source of risks. In addition, many vendors may deploy their own agents for work, introducing new potential risks for their customers to stay on top of.

• Pressure for AI ROI - After several years of high budget experimentation, many organizations are now looking for tangible ROI from their AI systems. AI vendors will be under a lot of pressure to show revenue, and so prices for AI tools are likely to increase at the same time that organizations are looking to focus their efforts more on high value AI. Calculating that value will be a major challenge and narrative going forward.

• AI Policy Moves ‘Up the Stack’ - 2025 had many AI policy proposals focused on the foundational model level, with the EU publishing their relevant Code of Practice for GPAI, and bills in California and New York being signed to regulate them. However, these types of regulations are being specifically targeted by the Trump administration, and trying to pass new ones is likely to face pushback. We think policymakers are more likely to target specific AI use cases or types of systems for further regulation, especially focused on protecting kids from AI, or regulating use of AI for mental health purposes.

You can read our full 2026 trends and prediction piece here.

2. Technical Deep Dive - 2026 Technical Look-Ahead

2025 brought a strong new generation of AI models from many providers, with an increased focus on training for reasoning and tool use. While some providers focused on creating increasingly large models, others continued to explore how smaller models trained on high-quality data can produce competitive results. We expect to see steady improvements across these areas, but for our 2026 predictions, we focus on the broader picture beyond just performance:

• World Models are AI models that aim to understand and model the physical world directly (in contrast to LLMs that are trained to predict the next word and as a byproduct encode some knowledge about the world). In 2025, there were some early developments in this space from Fei-Fei Li’s company World Labs and China’s Tencent; we expect to see some progress in this space, but these models are unlikely to overtake LLMs in usability and popularity in 2026, because they require large amounts of complex data that is not readily available.

• AI-Generated Videos will become impossible to distinguish from real videos; many examples from state of the art models, like Google’s Veo-3, already lack tell-tale “AI” signs (like background objects that move in unrealistic ways). However, generating longer videos (> 30 seconds) may remain difficult because of challenges with maintaining character and scene consistency.

• New Year, Same Risks: In late 2025, adversarial poetry, a new jailbreaking technique, was able to overcome the defences of a large number of popular LLMs. At the same time, hallucination rates remained high on multiple benchmarks. We do not expect either problem to be “solved” in 2026, these risks are inherent to LLMs because LLMs are trained to generate text, not recognize factuality or the potential danger of the generated content.

• Model Transparency will continue to play an uncertain role in adoption and trust. While transparency decreased overall in 2025, according to the Stanford Transparency Index, AI adoption increased. In the open model space, Qwen models, whose providers disclose little information about the training data, were top downloads from Hugging Face, while the highly transparent OLMO models did not receive much attention. In addition, new nuances have emerged around “transparency”: popular LLM providers release increasingly long System Cards, but many of the evaluation results now rely on LLM-as-a-Judge style automated evaluations which introduce biases and a new layer of complexity.

None of these predictions point to a single dominant shift in 2026, but some of the most exciting developments may come from the non-LLM side of AI. More broadly, if 2025 was the year of AI pilots and experiments, 2026 will be the year of transforming them into hardened production-ready systems. Meanwhile the continued uncertainty around risks and transparency, points to a need for increased education around AI risks and evaluations.

3. AI Incident Spotlight - 2025 AI Incident Recap & 2026 Predictions

The AI Incident Database catalogued 345 distinct incidents in 2025, a record high. Our analysis of these incidents shows 3 major trends in 2025:

• Deep Fake Scams - The majority of incidents in 2025 involved some form of scams, often involving deepfakes. These included using AI tools to generate flashy ‘phishing’ websites, using AI to create a massive web footprint for a fraudulent company, and many instances of scammers exploiting fake videos of celebrities claiming to love the scam’s target (Incident 901, Incident 1126, Incident 1185).

• Chatbots & Mental Health - Unfortunately there were many incidents involving deaths associated with chatbot use. These included instances of chatbots providing recommendations on how to tie a better noose, affirming a man’s belief that his mother was plotting to kill him leading to murder-suicide, and a teenager being induced to commit suicide by a fake AI-powered Game of Thrones character. In a sign of how bad things have gotten, Wikipedia now has a dedicated page linked to them. According to OpenAI’s own data, using AI for chats about mental health issues is one of the top user use cases for ChatGPT.

• Early ‘Agentic’ Incidents - 2025 saw some of the first incidents directly linked to AI agents, including a Replit agent deleting a production database, Claude Code’s agent mode autonomously conducting a cyber attack, and Wall Street Journal reporters successfully jailbreaking an AI powered vending machine. These were attained not with direct improvements in the AI models themselves, but rather by connecting models with a lot of different tools capable of performing actions.

Here are a few of our predictions for AI incidents in 2026:

• Agentic AI - There were only a few incidents directly linked to AI agents in 2025, but as agentic AI gets more adoption, we expect to see many incidents linked to agents. There are a few good reasons to suspect this, including the poor security status of many MCP servers and the relatively early maturity on how to evaluate or red-team agents.

• AI in Healthcare - It’s estimated that over two-thirds of medical practitioners are now using AI tools in their jobs, with notes transcription being the leading use case. However the impact of AI errors may take time to resolve. We expect to see some incidents linked to errors produced especially by earlier model versions that haven’t yet been acted upon.

• AI Videos - The quality of AI generated videos from tools like OpenAI Sora-2, or Google’s Nano Banana, is truly impressive, and many videos will become increasingly difficult for many people to identify as AI generated. We expect more scams and misinformation incidents specifically linked to hyper-realistic looking videos.

4. Trustible’s 2026 Policy Predictions

AI policy in 2025 was a roller coaster of new developments domestically and globally. The new Trump Administration upended AI safety work from the Biden Administration, the EU squabbled over whether to delay the AI Act (which it ultimately did), and governments at every level moved ahead with their own AI rules. Here are our top three thoughts on what to expect in AI policy in 2026:

• AI Lawsuits Will Test Regulatory Limits. Last year we tracked various lawsuits related to AI harms, from companion bot-related deaths to copyright infringement. We do not expect those battles to fade away, but a new one is about to heat up. States have been passing AI laws and those are in the crosshairs for new legal fights, as well the litigation coming from President Trump’s AI moratorium Executive Order (EO). These new legal fights will forge a new path on old laws and rights as they apply to AI rules.

• Turnaround for the AI Trust Gap. The past year saw another dip in AI trust among the general public even though adoption was on the rise. A large part of the distrust stems from a lack of regulation, which some studies show would help ease concerns over the technology. As some countries start to implement AI rules and safeguards, do not be surprised if there is a (slight) uptick on AI trust.

• AI Innovation Hits a Roadblock in the US. The second Trump Administration started with a bang for AI innovation, effectively undercutting any efforts that could burden the American AI ecosystem. Expect that line of thought to shift ever so slightly in 2026, as the Trump Administration grapples with challenges that AI presents to national security and critical infrastructure. The AI moratorium EO acknowledges the need for a federal AI framework, which is a marked shift from where the Administration stood last January. Expect further guidance on AI security and resiliency, as well as guidelines for certain industries (NIST recently announced a new workstream for AI and advanced manufacturing).

Overall, this year will blend “more of the same” with some new challenges. We expect US states to continue regulating AI, even as the federal government tries to clamp down on the AI legal patchwork. We also expect to see more interest in agentic AI, though regulatory frameworks are still a few years away. What we see this year is an opportunity to clarify some legal uncertainty, while also increasing a push for basic AI governance to help address safety and security concerns.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, and welcome back to the Trustible AI Newsletter! The holiday season is upon us, and it appears the White House is poised to gift states a new executive order this week to revive the federal AI moratorium. We’ll share more thoughts if and when the order drops, but at a minimum, expect a busy few months in the courts as the questions this order raises will almost certainly be decided in front of a judge. In the meantime, in this week’s edition:

1. Will AI Make Black Friday Become Black Mirror?

2. Technical Deep Dive - A Poet’s Key to Model Hacking

3. AI Incident Spotlight - Deloitte Publishes Citation Hallucination in Government Sponsored Report (Incident 1286)

4. Trustible’s Top AI Policy Stories

1. Will AI Make Black Friday Become Black Mirror?

What effect could AI agents have on pricing? E-commerce sites have long battled bots and scalpers that buy up limited goods for resale, with mixed success. Platforms like Ticketmaster have become notorious for profiting off secondary markets, resulting in recent actions to curb secondary market activity. For many consumers, it feels like more goods than ever (from vintage clothes, to Pokemon cards, and Lego sets) are dominated by resellers.

The new wave of AI agents risk making this problem worse. Old school bots were often simple web ‘scrapers’ that knew how to click specific buttons in order, and could be thwarted by certain types of activity blockers. AI agents can combine the reasoning powers of LLMs, with enhanced tool calling capabilities, to become more sophisticated, and are simpler to set up and deploy at scale. There are a slew of start-ups already targeting how to train dedicated agents to do this by creating fully sandboxed digital replicas of websites like Amazon.

It’s worth considering what impacts this may have on prices, and the broader global economy. Many limited goods with resale value may get quickly bought, and then immediately posted for re-sale. Now, bots could be used to manipulate the price directly. A single item could be bought and resold to other bots before it ever leaves a physical warehouse. Marketplace platforms and credit card companies will have significant financial incentives to allow this so they get a cut of every resale. Pricing for goods could start to more quickly resemble the stock market where most trades are already done based on algorithms.

This will likely mean higher prices on many goods. This will especially be true if e-commerce platforms themselves use AI to adopt dynamic pricing strategies. A recent exposé found that Instacart was using a hidden algorithm to test the upper limits of prices they were willing to pay for certain groceries. While we do not have enough information to fully understand the interplay between AI-enable pricing and constant bot activity, what we are seeing is that the incentive structure likely doesn’t favor the consumer.

Another unsettling dynamic that could emerge is dynamic pricing that acts as a proxy for social scoring. Consumers with certain buying histories or in certain geographic locations may be rewarded with access to certain pricing schemes to the detriment of other buyers. Characteristics like race, gender, sexuality, or disability could be inferred based on proxy data, which could in turn cause specific populations to burden higher costs because of the types of goods or services they are trying to access. For instance, rural consumers may pay higher grocery delivery prices if they live in a food desert or racial minorities may be quoted higher rent prices in certain neighborhoods.

Key Takeaway: Given that ‘affordability’ is currently a global concern, and that algorithms have already started to be a major contributor to that, negative impacts from AI-driven higher prices could become a large political force in the coming decade. Policymakers may feel the pressure to impose limited safeguards that help mitigate these issues.

2. Technical Deep Dive - A Poet’s Key to Model Hacking

Safety alignment 
May succeed until
Poetry
Foils the plan

Despite extensive alignment efforts, many LLM safety mechanisms can be brought down with a few lines of poetry. A recent research paper on “Adversarial Poetry” showed that using poetry to frame an adversarial request (e.g. advice on how to execute a cyber attack) resulted in a broad range of models producing unsafe outputs 62% of the time. The attack success rate (ASR) varied widely by model: all the GPT-5 models had an ASR of under 10%, while the Deepseek-3 and Gemini 2.5 models had an ASR of over 95%. While this isn’t the first method to cause models to ignore their built-in defences (e.g. the infamous Do-Anything-Now prompt), it does point to two broader themes.

First, while many providers now report extended safety evaluations, they are often focused on well-known attack vectors and may not paint a full picture. In addition, to creating a custom poetry dataset, the researchers in this study took a well-known dataset of adversarial prompts from MLCommons and translated them into poems, the average ASR went from 8% to 43% - suggesting that model developers may be overfitting to a known set of attacks during safety fine-tuning and not addressing the broader alignment problem.

Second, syntax has a big role in how LLMs process data. Another recent study showed that LLMs can rely on syntax over exact semantics of a sentence. For example, when asked ‘Where is Paris undefined?’ A model may answer ‘France’ despite the actual sentence being nonsensical. This may point to the success of the poetry attacks - the grammatical structure of poetry is not associated with adversarial behavior and thus may not trigger those safety protections.

Key Take-away: LLMs will likely never be fully resistant to jailbreaking, which may take the form of complex multi-turn attacks or a simple poem (GPT-5 models seem particularly resistant to the later), because the training data contains unsafe content and reliable “unlearning” techniques do not exist. When building AI systems that may have adversarial users, relying on the model provider’s safety alignment is not sufficient and additional guardrails (e.g. output filtering) should be integrated into the systems.

3. AI Incident Spotlight - Deloitte Publishes Citation Hallucination in Government Sponsored Report (Incident 1286)

What Happened: A public report written by Deloitte, on behalf of the Provincial Government in Newfoundland, contained some hallucinated citations for key statistics and facts. Newfoundland reportedly paid Deloitte 1.6 million Canadian dollars for the report that outlined a human resources plan. The report had citations to publications that don’t actually exist, and supposed claims in those publications were used as justification for recommendations in the report. Deloitte claims the errors were strictly related to generating the citations, not the outcomes of or recommendations from the report. This comes only a few months after a highly similar incident in Australia where Deloitte again cited poor citations in reports generated for public sector agencies.

Why It Matters: There are a couple of things this incident highlights. The first is the challenge with generating citations in general. This is often a tedious task that many people want to automate, but it’s also one that is actually quite challenging for AI to do, unless it’s connected to some kind of ‘global database’ of research upfront. Ironically, most publication archives are now heavily deploying anti AI scraping technology, which will actually make this problem worse in the short term despite model improvements. It also highlights one challenge some big consulting companies will have in the AI era. These recent incidents have caused reputational damage to Deloitte, and while they claim only the citations were AI generated, that is difficult to prove. Top services companies, like prestigious consulting firms, law firms, think tanks, etc often differentiate on the quality of their work, and often try to stack staff from elite universities to reinforce their brand. However these firms are also under huge pressure for productivity and AI can be a big contributor to that. The biggest risk for them is that ‘AI Slop’ could undermine their chief competitive advantage. Few people would hire McKinsey at their normal price point if they can get the same level of insights and advice directly from ChatGPT. While top companies have swarmed on AI, we also expect there to be backlash that could suddenly make the value of truly ‘human’ services more valuable in the AI era, especially for ‘elite/luxury’ markets.

How to Mitigate: Obviously the most reliable way to prevent hallucinated citations is to have them all manually reviewed, although that can be slow and tedious (hence why AI was used in the first place). There are some low hanging fruit ways for building some automated verification steps however. Simply running each citation itself through a non-API driven system to verify it exists is an option, and some organizations have started using a separate AI tool or model to run its own verification checks on generated content. It’s also important to have clear policies for employees to specify when they use AI for generating content, and to have a clear list of things to check in AI generated content in order to catch common AI mistakes like this.

4. Trustible’s Top AI Policy Stories

Trump’s New AI Executive Order. President Trump is expected to sign a new executive order (EO) aimed at pausing state AI laws. A draft was previously leaked that outlines how the Administration will leverage the Department of Justice to challenge state AI laws.

Our Take: The Trump Administration has sought to pre-empt state AI laws but has not offered a federal replacement. We anticipate a fairly lengthy legal battle to ensue once the EO is signed.

AI in the NDAA. Lawmakers added amendments to the National Defense Authorization Act that bans foreign models from use in the federal government and tasks the Department of Defense with creating an AI model evaluation framework.

Our Take: Congress is also considering legislation for an “AI in national security” playbook and these amendments would align with the targeted, security-focused approach to AI that we have seen from the federal government.

New York Times and Perplexity. The New York Times joined a copyright lawsuit against Preplexity.ai that alleges it illegally copied millions of its articles. Perplexity has been embroiled in legal battles for how it gathers and uses content for its AI search engine.

Our Take: The ongoing copyright infringement cases highlight the growing need to update IP laws and regulations that can account for how AI systems can use protected content.

In case you missed it, here are a few additional AI policy developments making the rounds:

Africa. The Ugandan government announced that it will release a draft plan to regulate AI. The decision marks one of the first significant efforts to regulate AI in Africa.

Asia. The Japanese government will release a draft plan to improve Japanese AI development and increase AI adoption. Japan has taken a lighter-touch regulatory approach with AI previously and is looking to develop its domestic AI ecosystem.

Australia. The Australian government released its National AI Plan, which is intended to help grow the country’s AI industry. The plan seeks to support new AI infrastructure, increase AI adoption, and enact laws to protect its citizens from potential AI harms.

Europe. Lawmakers are under pressure in the UK to regulate “superintelligent” AI system development. Specifically, the latest regulatory push wants more safeguards imposed on frontier model providers to reign in developing potential superintelligent systems.

North America. The Canadian government released its first public sector inventory of AI systems. The government also published the world’s first standard for developing equitable and accessible AI systems.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

Happy Wednesday, and welcome to this edition of the Trustible AI Newsletter! Two weeks is an eternity in AI world, and in the past two weeks, we’ve seen a seismic shift in the AI regulatory environment both here in the U.S. and across the pond in the EU (more on that later in this edition.)

But, it’s also been a big couple of weeks for all of us here at Trustible; last week, we launched our new Trustible AI Governance Insights Center, an open source repository of our AI governance heuristics, from our risks taxonomy, to recommended mitigation strategies, AI benefits, and even more developments in our AI model ratings curated by our team of experts. Over time, we’ll be adding even more insights and resources, but as a public benefit corporation, this is an important step in advancing our mission to help society realize the transformative potential of AI. You can explore the insights center at trustible.ai/resource-center.

We are also thrilled to share that we’ve been listed as a Representative Vendor in the 2025 Gartner® Market Guide for AI Governance Platforms. We believe this is a milestone that signals the start of an inflection point, when AI governance is no longer optional, experimental, or theoretical; it’s now a business imperative for enterprises looking to realize the promise of AI. You can read all about the exciting news here.

In this week’s edition, we’re covering:

1. Trustible’s Take - Transatlantic AI Uncertainty

2. AI Incident Spotlight - Cyber Attacks by AI Agents

3. Technical Explainer: Big challenges with open-weight models

4. Trustible’s Top AI Policy Stories

1. Trustible’s Take - Transatlantic AI Uncertainty

As a result of a flurry of regulatory proposals last week, there is now more uncertainty than ever about when AI regulations may kick-in and what those regulations will be. In Europe, the EU Commission proposed a ‘Digital Omnibus’ to reform several digital laws, including GDPR and the EU AI Act. The Commission is proposing a delay for high risk AI system obligations for a period ranging between 12 and 24 months. A driving force behind the delay comes from issues developing compliance standards for high risk systems. The Commission’s proposed delay stipulates that if standards are developed before the new deadlines hit, then high risk system requirements will take effect sooner. However, it is unclear if the Commission’s proposal will make it past a skeptical EU Parliament (which has to agree) or if the compromise text would pass before the current deadline for high risk obligations comes into effect in August 2026.

Meanwhile in the US, congressional Republicans are moving forward with plans to resurrect the state AI moratorium by attaching it to the annual National Defense Authorization Act. President Trump fully supports the effort moratorium and is considering an Executive Order (EO) that would effectively enact a moratorium without Congress, but the planned EO appears to be on hold. A federal moratorium on state AI laws is facing backlash from several prominent GOP elected officials, including Governor DeSantis and Senator Hawley. Any moratorium without an actual superseding federal law would likely face immediate lawsuits, especially if done via an EO.

Ironically, the recent policy prerogatives in the EU and US aim to incentivize AI growth and adoption but instead have injected such a level of regulatory uncertainty as to undermine these goals. The EU and US have substantially lower rates of trust in AI than any other part of the world. Solving the AI trust problem is the key to growing AI adoption rates on both sides of the Atlantic.

Key Takeaway: The increasing regulatory uncertainty paired with the perception that AI is unregulated is not going to help accelerate adoption. Backtracking on efforts that improve trust and adoption could just cause the “AI bubble” to burst, along with any potential to “win” the AI race against China.

2. AI Incident Spotlight - Cyber Attacks by AI Agents (Incident 1263)

What Happened: Anthropic identified that a Chinese hacking group (GTG-1002) used its Claude Code platform to launch a fully autonomous cyber espionage attack against over 30 targets. The groups used several ‘jailbreaking’ techniques to evade protections built into Claude Code. The attack notably included autonomous ‘multi-step’ processes, where the agent first scanned for the target’s cloud resources, programmatically identified vulnerabilities, created exploits for them, and then successfully extracted data before being shut down by Anthropic.

Why it Matters: There are several highly notable and severe aspects of this incident. The first is that it highlights a new dangerous paradigm for cybersecurity. The fact that an AI agent was able to successfully conduct a highly sophisticated multi-step attack, with limited human interaction confirms the fears of many in the cybersecurity world. The fact that it was a Chinese group, with links to the Chinese state, using an American AI system is also likely to create a massive reaction in DC. This is also one of the few voluntary first-party incidents reported by a large model creator, although it’s unclear what their longer term mitigation efforts may be beyond simply trying to ban the relevant parties.

How to Mitigate: A lot of focus of these incidents is often on the underlying ‘model’. However LLMs only ever generate text, and even in agentic systems, they’re often simply generating text that instructs a system to do something, or to call some kind of external tool. The real dangers here came from all the other ‘system’ components built into Claude Code as a ‘platform. Claude-4 the model can generate the text command to make an HTTP request, but Claude Code the platform can actually make the request. Claude Code then also was storing information in memory, running a process over several minutes (or hours), and executing generated computer code. For organizations hosting AI systems, limiting how much a system can independently access the internet, or run code, can heavily restrict the potential blast radius and capabilities of a system. For those worried about similar incoming cybersecurity attacks, tools like CloudFlare anti-AI bot tool will become an essential layer to help block sophisticated non-human interactions.

3. Technical Explainer: Big challenges with open-weight models

Open-weight models, like Phi and Qwen, are available for download and unconstrained use by researchers, businesses, and consumers. Unlike closed-source models (e.g., GPT or Gemini) that use input/output filters, protections in open-weight models must be integrated directly or added by deployers. Recent research highlights risk management challenges that are particularly salient for open-weight models.

Training Data Curation: One key mitigation strategy is to avoid training on unsafe sources. However, current filtering methodologies are inconsistent and some “knowledge” may be useful for benign capabilities (e.g. designing cybersecurity defences). In addition, recent work suggests that harmful knowledge may emerge from a combination of benign data sources (e.g. knowledge of biorisks may be inferred from general knowledge of biology). Choosing an appropriate curation strategy may be challenging for deployers, and a lack of clear guidelines can make it difficult for developers to review models.

Tamper-Resistant Training: Post-training methodologies can further reduce unsafe behavior, however, downstream modifications can intentionally or unintentionally remove these protections. In addition, as models become integrated into agentic systems, they may retrieve harmful knowledge from the internet, introducing a new attack vector.

Model Tampering Evaluation: Open-weight developers often lack the resources for extensive audits, shifting the burden to deployers. Furthermore, standardized frameworks don’t exist for these evaluations making it difficult to compare and trust different models. (In our own AI Model Ratings, we’ve observed that many open-weight developers explicitly disclose a lack of adversarial evaluations)

Model Provenance: Both researchers and deployers may want to study the lineage of specific open-weight models. While the former may want to understand the broader ecosystem, the latter needs to know whether a particular model is appropriate for an application. Currently, no reliable and scalable approach exists for tackling this challenge.

Key Takeaway: The challenges outlined are exacerbated by a lack of transparency - most popular open-weight models largely do not address these topics in their documentation. While these models allow for more flexibility and control, they shift the burden to deployers to investigate model provenance, run safety evaluations and build additional safeguards while considering a lack of unified standards and solutions for each step.

P.S. While this work emphasizes the lack of reporting around safety, we’ve recently collaborated with the EvalEval coalition on a new paper that shows that societal impact evaluations are underreported across the LLM landscape.

4. Trustible’s Top AI Policy Stories

House Hearing on Chatbots. The House Subcommittee on Oversight and Investigations held a hearing to better understand the risks posed by AI chatbots, particularly to minors, and hear recommendations from experts on potential regulatory solutions.

Our Take: Congress continues to be lasered focused on AI harms posed to children and may be one of the few AI issues that sees bipartisan legislation pass.

National Security Framework. A bipartisan group of lawmakers in the House and Senate are working on legislation that would require the National Security Agency to publish an AI security playbook, which is intended to help outline how AI systems are being protected from foreign adversarial threats.

Our Take: AI security is a priority for the Trump Administration (as emphasized in the White AI Action Plan) and represents another area where lawmakers could plausibly pass legislation.

United Nations and Healthcare. A new report from the World Health Organization warns that AI used in healthcare settings needs legal guardrails to protect patients and healthcare professionals.

Our Take: The concerns are not new but the report notably observes “there is a broad consensus on the policy measures” that could improve AI adoption.

In case you missed it, here are a few additional AI policy developments making the rounds:

Africa. The UAE announced the “AI for development initiative,” which will invest $1 billion to expand AI infrastructure in Africa. While US tech companies have been making active investments to help African countries develop AI technology and infrastructure, the US government has largely been absent.

Asia. AI-related policy developments in Asia include:

• China. The Chinese government issued guidance that would ban foreign-made AI chips from new data center projects. The decision comes amidst tensions with the US over advanced chips sales in China.

• South Korea. Korea’s Ministry of Science and ICT released draft regulations for the AI Basic Act, the country’s comprehensive AI law passed in December 2024. The proposed regulations clarify how covered entities will need to comply with the law, which takes effect in January 2026.

Australia. The Australian government released the Australian Public Sector (APS) AI Plan, which is aimed at improving how AI is used by public sector agencies. While the Australian government has backed away from enacting AI rules for private companies, the ASP AI Plan could have far-reaching impacts on companies doing business with the federal government.

Middle East. The Trump Administration signed a new Memorandum of Understanding with the Saudi Arabian government, which would allow the Saudi government to access US AI technology. The agreement is part of the Trump Administration’s broader AI policy goals with Middle Eastern countries.

South America. The UN’s COP30 climate conference met in Brazil and focused heavily on how AI is impacting climate change. While attendees acknowledged AI’s potential for helping address climate change, many raised concerns over AI’s strain on natural resources and energy consumption.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In this week’s edition, we’re covering:

1. Chatbot Teetertot: Balancing Child Safety & AI Literacy

2. Tech Explainer: The Role of Guardrail Models in AI Systems

3. AI Incident Spotlight: Deep-Fake Endorsement Scam (Incident 1261)

4. Policy Round-Up

1. Chatbot Teetertot: Balancing Child Safety & AI Literacy

Minors are increasingly using AI tools for social support and companionship. However, it’s not difficult for kids to elicit problematic or harmful content from AI tools, mainly AI chatbots and companion bots. The most notable examples are from Character.ai, whose companion bots have lead to children committing self-harm or suicide, as well as sharing or generating illicit content. Character.ai has since blocked kids from accessing its chatbots.

Industry is trying to figure this out, with OpenAI’s Teen Safety Blueprint as one example of how the private sector can offer solutions. Policymakers in the US have also taken notice and are attempting to enact new laws that address these issues. For instance, a new California law requires companies to implement safeguards to prevent companion chatbots from discussing suicide and must direct the user to self-harm resources if they express suicidal thoughts to the bot. Congress has also taken notice and the bipartisan GUARD Act was introduced in the Senate, which would (among other things) ban minors from accessing companion bots.

Understandably, lawmakers and parents want to protect kids from the harms posed by chatbots. Age verification requirements have long been used to prevent minors from accessing inappropriate or illicit content. Yet, we must reflect on whether age-gating sufficiently addresses the problem (as those requirements can be gamed and also pose serious First Amendment issues) and whether these types of barriers to certain technologies will do more long term harm. Yes, it is essential to protect kids from potentially harmful systems and content, but it also highlights the challenge of effectively teaching them essential AI literacy skills to build a competitive next generation workforce.

While AI developers have a role to play with child safety, it does not stop with them. Organizations deploying public-facing chatbots need to understand that, even if children are not the intended audience, they may still use their chatbot products. It’s important for companies with public facing chatbots to implement safeguards around outputs and make appropriate disclosure (e.g., noting when a product should not be used by people under the age of 18). These mitigations are especially true as chatbots become more general purpose, which broadens the aperture on exposure.

Our Take: Kids today are more tech savvy than the previous generation and we need to acknowledge that fact as we think about how to implement pragmatic protections while helping them build valuable AI skillsets. We also need to think about how new child safety laws are not so sweeping that they unintentionally stunt growth for other AI use cases.

2. Tech Explainer: The Role of Guardrail Models in AI Systems

One way to mitigate a variety of AI system risks including processing PII, outputting unsafe (e.g. toxic language or specialized advice) content and prompt injections is by incorporating guardrail models that review and flag the inputs to and preliminary output from the system. Think of guardrail systems as guardians designed to be the first line of defense against potential threats. Designed specifically for detection rather than reasoning, guardrail models are typically smaller and faster than general-purpose LLMs. Both AWS Bedrock and Azure AI have off-the-shelf guardrail models that can be integrated into any endpoint, while on the open-source side, Llama-Guard is a popular solution that can recognize 13 common harm categories.

Recently, OpenAI released a new open-source guardrail model, gpt-oss-safeguard. Unlike the other solutions that can only detect a pre-defined set of harms, this model allows the user to input an arbitrary human-written policy (i.e. a set of rules to follow), meaning it can be used for domain-specific concerns (e.g. flagging spoilers on a movie forum or detecting regulatory non-compliance in legal text). The increased flexibility does not guarantee accuracy: the developers state that gpt-oss-safeguard is often less accurate than a custom model. As well, the documentation does not explore what kinds of “policies” will be effective; recent research suggests that existing human content moderation guidelines may need to be modified to work properly with LLMs. This type of model can be used as a rapid deployment solution, when off-the-shelf models do not cover the use case, and resources can not be devoted to building out an internal solution that may require thousands of labeled data points.

Our Take: Guardrail models serve as a first and final line of defence against unsafe system outputs, but require careful evaluation to be used effectively. Existing experts in that domain will need to collaborate with content engineers to review off-the-shelf guardrail models or build bespoke policies for models like gpt-oss-safeguard.

3. AI Incident Spotlight: Deep-Fake Endorsement Scam (Incident 1261)

What Happened: A deep fake of a senior official in the government of Western Australia was used to facilitate an online scam. The scammers generated an AI video showing Roger Cook, the official, falsely endorsing an investment service. The deep fake was described as hyper realistic in both voice and appearance.

Why it Matters: Public officials are at particularly high risk of being ‘deep faked’, and leveraged by scammers. The latest systems can impersonate facial likeness and voice with only a few short high quality video or audio clips. These are particularly easy to get for public officials who are highly visible and often need to participate in televised events. Many public officials also don’t always receive the same degree of privacy protections, and certain activities like satire of politicians can actually be a protected activity, blurring the lines of what is acceptable or not. It’s likely that these types of deep fake impersonation attacks will become more common over time.

How to Mitigate: Unfortunately there aren’t many ways that individuals or organizations can prevent this kind of activity on their own. Instead, the best options are to have mitigations set up to reduce the likelihood of people falling for these types of scams, and to support regulations on platforms that may potentially host, or benefit, from these types of schemes. For organizations, it is appropriate to consider regular training on how to detect deep fakes, and even run on-going exercises for it on employees.

4. Policy Round-Up

Trustible’s Top AI Policy Stories

ChatGPT’s New Lawsuits. OpenAI is facing a flurry of new allegations over ChatGPT’s impact on mental health. New lawsuits against the company allege that ChatGPT induced psychosis, as well as lead to financial ruin and suicide.

Our Take: It is important to clearly communicate how these tools should be used and make sure that users understand they are interacting with a machine, not a real person.

Pausing the EU AI Act. The European Commission (EC) released initial proposals for its Digital Omnibus package, which includes loosening EU AI Act obligations and postponing enforcement for certain requirements.

Our Take: The uncertainty is complicating compliance for many organizations but they should focus on EU AI Act compliance in case the EC opts against enforcement delays.

UK Copyright Decision. A UK court decision is upending the interplay between AI data use and IP law after it ruled in favor of Stability AI in a case brought by Getty Images for secondary copyright infringement.

Our Take: The case adds a new dimension to how AI is trained on protected data because the alleged infringement did not occur in the UK. Organizations should make sure they have proper permission to use data when training their AI models and tools.

In case you missed it, here are a few additional AI policy developments making the rounds:

United States Congress. Two pieces of bipartisan AI legislation have been introduced in the Senate. The GAIN AI Act would regulate how American AI chips were exported to China and other countries of concern. The AI-Related Job Impacts Clarity Act would require companies and the federal agencies to report when AI-related layoffs occur. House Democrats are also attempting to rollback the Trump Administration’s efforts to use AI at the Centers for Medicare & Medicaid Services.

Trump Administration. After OpenAI’s CFO caused controversy by implying the federal government could be a “backstop” for AI infrastructure debt, the Trump Administration dismissed the idea of a federal bailout for frontier model companies. Sam Altman has denied that OpenAI expects a bailout.

Asia. AI-related policy developments in Asia include:

• China. A new law requires social media influencers to hold degrees for certain topics (e.g., medicine or finance) before posting about them. The law also requires influencers to disclose when their content is AI-generated.

• India. The Indian government is preparing to release a draft comprehensive AI law. The current content is unknown but is expected to be modeled after the Information Technology Act of 2000.

Europe. Nvidia and Deutsche Telekom announced plans to build Europe’s largest AI in Germany. The latest announcement emphasizes Europe’s desire to build its own AI ecosystem.

Middle East. The Trump Administration signed a memorandum of understanding with the UAE to further cooperation in AI and energy.

As always, we welcome your feedback on content! Drop us a line with your thoughts to newsletter@trustible.ai for a chance to win an exclusive set of Trustible AI Model Playing Cards!

AI Responsibly,

- Trustible Team

1. Trustible’s Take - The Need for AI Fallbacks

2. AI Incident Spotlight - (AI Incident 1248)

3. Reddit’s Hidden Hand in AI Training and Why It Matters

4. Policy Round-Up

1. Trustible’s Take - The Need for AI Fallbacks

Several owners of Eight Sleep, a tech enabled ‘smart bed’, were suddenly woken up early last Monday by their beds going haywire. It turns out that a massive AWS outage related to a misconfigured Domain Name Service (DNS) system was able to take down more than just websites and SaaS applications. This event highlights two core risks related to the evolving AI ecosystem.

The first, is that there is a lot of infrastructure that still supports AI systems. For example, all internet traffic relies on DNS to figure out what servers to talk to, AI systems need to read and write information to databases like AWS DynamoDB, and the physical hardware that supports the relevant computation is also at risk. One reason the AWS outage was so widespread is because there are only a handful of hyperscaled cloud service platforms, and the set of affected data centers (‘us-east-1 based in northern Virginia) happens to be one of the earliest, and most used regions. As we start to incorporate more AI into everyday systems and rely on it more, the risk of creating ‘central points of failure’ increases.

The second issue at play was that the devices with embedded ‘smart’ capabilities did not have appropriate fallback mechanisms. Instead of simply disabling the smart features and acting as a plain old bed, some of the smart beds start increasing temperature endlessly. A few days after the outage, Eight Sleep did roll out a dedicated ‘outage mode’ that is able to use local bluetooth to send instructions to the smart bed. At this point in time, we have a ‘non-AI’ path and process for most things, however will that be true 10 years from now? One popular AI policy proposal is to ensure that there are non-AI fallbacks, bypass, or appeal processes, as a main mitigation towards this potential AI overreliance.

Key Takeaway: The AI ecosystem is not yet mature enough to have ‘multi-cloud’ deployments, partly because not all model providers are available across all major cloud providers. This maturity, and clear standards for ‘non-AI’ modes are likely going to be necessary before AI is adopted for highly critical applications in fields like healthcare or national infrastructure.

2. AI Incident Spotlight - (AI Incident 1248)

What Happened: US Conservative activist Robby Starbuck has sued Google claiming that Google’s AI models regularly defame him by accusing him of sexual assault. He filed a similar lawsuit against Meta earlier this year alleging similar things coming from Meta’s AI tools. That case was settled before going to trial. Many of his allegations date back to an earlier model, Bard, that Google has since deprecated.

Why it Matters: The exact cause of the hallucination isn’t known. It’s possible that there was politically motivated intentional misinformation posted online that the systems incorporated (a form of data poisoning), or it could simply be a hallucination because of similarities between Robby and others. Regardless of the source of the misinformation, this case, and others like it, will likely test the precedent of applying ‘Section 230’ to AI systems. The issue at the core is who is liable for false information like this. Under current interpretations, web platforms are not directly responsible for the defamatory content created by users of their platform. Whether an AI system counts as a ‘platform’ (protected), or user (liable) could radically shift the liability scheme for LLM providers. So far, no one has been successful in winning a defamation case against AI in the US.

How to Mitigate: Most of the most recent LLM systems have the ability to conduct real-time web searches in order to pull in ‘fresh’ information from reliable sources that can help pull in facts. However this web searching capability is a feature of the system not the models themselves, and therefore many AI features built on just model APIs won’t have this ability, and allowing searches introduces additional privacy and cyber risks. System prompts that cover how to respond when doing research or answering questions about people can help ensure that only verified information is shared, and to avoid stating potentially negative information altogether.

3. Reddit’s Hidden Hand in AI Training and Why It Matters

Few platforms have shaped the modern internet like Reddit. Its open, chaotic mix of expertise, argument, and lived experience has become a goldmine for organic community discussion for brands, but also, for LLMs. Because Reddit posts are conversational, self-correcting, and wide-ranging, they provide the kind of nuanced human expression that makes AI sound more human. In fact, many of the citations or answers you see in AI-generated content trace their roots back to Reddit threads.

But this organic data source has become a flashpoint. Reddit’s recent lawsuits against Perplexity and other AI vendors highlight an emerging legal and ethical battleground: who owns public discourse when it fuels AI? The platform argues that scraping and repurposing Reddit data without consent undermines both its community and its business model, especially as Reddit now licenses its content to certain model developers under paid agreements (representing $35M in revenue for Reddit as of Q2 2025, up 24% year over year.)

Technically, Reddit’s data is particularly valuable because of its structure. Its posts and comment trees offer deeply nested, timestamped dialogues, rich with slang, reasoning chains, code snippets, and emotional context, all of which are perfect for pretraining and fine-tuning models to understand how humans argue, explain, and empathize. AI crawlers systematically follow thread hierarchies and metadata (like upvotes or subreddit topics) to learn which ideas communities endorse or reject, turning social feedback loops into learning signals. This makes Reddit data uniquely high-quality - and uniquely sensitive.

For enterprises, Reddit represents both opportunity and risk. It’s a valuable source for sentiment analysis, market research, and training domain-specific chatbots. Yet the same exposure that makes Reddit content powerful also makes it volatile. A viral post, an out-of-context quote, or an AI model trained on outdated or toxic Reddit data can easily amplify reputational risks.

Why is it Relevant: Reddit’s fight with AI companies is a preview of how data ownership, consent, and community ethics will define the next phase of AI governance.

Key Takeaway: AI governance professionals should be alert to how LLMs draw from social data ecosystems like Reddit, and monitor these channels for keyword and thematic discussions where misinformation may be shared. Engaging in these channels, setting the record straight, and controlling your organization’s own footprint (from ads to AMAs) helps protect your brand. These are not neutral channels, they are living communities whose norms, biases, and moderation dynamics shape AI behavior.

4. Policy Round-Up

Trustible’s Top AI Policy Stories

Anthropic vs the White House. Anthropic has been in an interesting back-and-forth with White House AI Czar David Sacks, who has criticized the company as promoting an agenda “to backdoor Woke AI and other AI regulations.”

Our Take: Anthropic has been a leading voice on AI safety and supported state laws, including the recently enacted SB 53. The spat shows the fine line companies need to walk between promoting AI regulations, innovation, and navigating a tenuous political environment.

EU Prepared to Expand Copyright Laws to AI. It appears MEPs are prepared to require EU copyright law apply to AI training regardless of where the training occurs.

Our Take: The decision would put US tech companies and their frontier models in the cross hairs. But depending on the actual text, it could have big implications for companies that are training models on external data sources.

Frontier Model Providers Get EU AI Act Warning. The Dutch Data Protection Authority warned four frontier model providers (OpenAI, xAI, Google, and Mistral) that their chatbots’ advice on the Dutch parliamentary elections could classify them as high-risk systems under the EU AI Act.

Our Take: While the EU AI Act has not come into full effect yet, this is a good reminder that “low risk” systems can evolve into high-risk systems, which is why continuous oversight is necessary.

In case you missed it, here are additional AI policy developments:

United States Congress. A recent letter to Federal courts Senate Judiciary Chair Senator Chuck Grassley (R-IA) revealed that interim guidance has been issued to federal courts, which addressed “non-technical suggestions on the use, procurement, and security of AI tools.” Grassley supports formal AI regulations for the federal judiciary, given the errors and controversy that have arisen with AI and the courts.

Trump Administration. The International Trade Administration (ITA), which sits within the Department of Commerce, is launching a program to develop full-stake AI export controls. ITA issued a request for information, in accordance with President Trump’s Executive Order on Promoting the Export of the American AI Technology Stack, to seek industry input on how to establish and implement the program.

Africa. Gebeya (an Ethiopia-based platform) launched Gebeya Dala, an AI-powered app builder designed with African cultural considerations. This is the latest in a series of culturally-specific AI tools that have launched this year that align with regional or non-western cultures.

Asia. AI-related policy developments in Asia include:

• China. China’s legislature formally adopted amendments to China’s cybersecurity law, which promotes stronger ethical AI standards, risk monitoring, and assessments.

• Japan. The Government of Japan entered into a Memorandum of Cooperation with the U.S. government, in part to cooperate on accelerating AI adoption and innovation.

• Vietnam. The Government of Vietnam is working on amending its intellectual property regulations to promote AI innovation. The government is also supporting an open source AI ecosystem as part of their broader strategic plans as a regional AI player.

Australia. The Labour government indicated that it will not exempt frontier model providers from copyright laws for text and data mining. The government also launched an investigation into how chatbot companies like Character.ai implement safeguards for children. The government is also suing Microsoft because of deceptive price hikes related to copilot integrations into Microsoft 365.

Europe. AI-related policy developments in Europe include:

• Albania. Albania’s Prime Minister announced that its AI minister, Diella, is pregnant with “83 children.” The AI-generated offspring will serve members of parliament as their assistants.

• EU. The EU’s standards-setting body caused controversy by announcing that it would “fast-track” the most delayed EU AI Act standards with a smaller group of experts. The move was characterized as “unprecedented.” The decision caused pushback from some members of the standards body because of “serious unintended consequences.”

• UK. The Labor Government announced a new partnership with OpenAI that will allow its UK business customers to host their data within the UK. A local news station also piloted an AI newscaster in a story about whether AI will replace humans in the workforce.

North America. The Canadian government is considering a series of AI-related laws that will address deepfake, data transfers and age assurances for chatbots. Canada abandoned its efforts to pass a comprehensive AI law after their federal elections earlier this year.

Middle East. AI-related policy developments in the Middle East include

• Saudi Arabia. Saudi-based AI company Humain is making plans to be listed on the Saudi stock exchange as well as the NASDAQ. Humain and Qualcomm also announced a partnership on deploying advanced AI infrastructure in Saudi Arabia.

• UAE. G42 (UAE-based AI provider) and Cisco announced a partnership to build “secure, trusted and high-performance [AI] infrastructure.”

South America.

• Argentina. A recent study showed that a third of media professionals in Argentina use AI to assist with their jobs, which includes “help write and edit articles, craft headlines and translate text.” There are some concerns that the lack of regulations for how journalists use AI could hurt the industry financially, as well as a need for AI-related journalistic standards.

• Chile. The Chilean government is dealing with public backlash over resource issues posed by AI infrastructure, specifically as the government seeks to build more data centers. The outcry over AI energy and water consumption has been brewing in other countries, including the U.S.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

At Trustible, we’re on a mission, giving AI Governance professionals the information, insights, and tools they need. Our legal and technical analysts help filter through the noise and identify what’s meaningful for enterprises, and what’s just hype. To ensure we’re advancing that mission, we’ve been listening to your feedback over the past few months, and we’re going to be revamping our newsletter going forward.

Here’s what we’re changing and what you can expect going forward:

• Technical Insights

  • AI technology is evolving at a rapid pace and it’s hard for anyone to keep up, let alone contextualize what new developments will mean for enterprise use of AI. Our machine learning experts will use this section to translate technical into plain english and connect it to the challenges organizations face.

• Policy Round-Up

  • Our regular policy roundup will continue as an overview of the major AI policy headlines from the past two weeks. We won’t be able to cover everything around the world, but we’ll focus on the developments that most impact practitioners.

• AI Incident Spotlight

  • This new section will be a deep dive explainer on a recent incident captured in the AI Incident Database. Our goal will be to provide actionable recommendations on how to prevent similar incidents for enterprises.

• Trustible’s Take

  • This will be our editorial team’s take on the most notable news relevant to AI Governance professionals. This section will focus a lot on trying to be the voice of pragmatic AI, and to try and cut through the hype found on traditional and social media platforms.

• News & Updates

  • We’ll regularly publish summaries of our more in-depth whitepapers, research, and blog, along with new announcements from Trustible. We want our newsletter to be insightful and actionable for anyone working in AI Governance, but we also want to let you know how we’re building solutions to many of the issues discussed!

With that, in today’s edition (5-6 minute read):

1. Trustible’s Take: Why SB 53 Won’t Have a Big Impact

2. AI Governance Meets AI Insurance: How Trustible and Armilla Are Advancing AI Risk Management

3. AI Incident Spotlight - AI Agents Aren’t Always Leaving a Paper Trail

4. Technical Insight - What To Know about ‘Model Specs’

5. Global & U.S. Policy Roundup

1. Trustible’s Take: Why SB 53 Won’t Have a Big Impact

There has been a lot of fanfare about Governor Newsom signing SB 53, a frontier AI safety bill. Many proponents argue it will have a major impact in regulating AI. We’re not sure. Especially after its various amendments and changes, we think it’s very unlikely to make a large impact in the AI space. Here’s why:

• SB 53 is almost entirely a ‘subset’ of the EU AI Act’s requirements

  • While there are a few differences included in SB 53, including clear whistleblower protections for frontier model provider employees, many of the ‘core’ safety framework requirements are extremely similar to the ‘safety and security’ requirements in Chapter 3 of the EU’s Code of Practice for GPAI providers. A major sign that this alignment was intentional was the use of the same computer thresholds to establish ‘frontier models’ (SB 53), as ‘GPAI Models with Systemic Risk’ (EU AI Act). OpenAI actively lobbied Newsom to align these requirements, and notably neither endorsed nor denounced the bill.

• Most frontier labs are already compliant, and proposed enforcement is weak

  • The requirements of the ‘frontier AI framework’ described in SB 53 reads exactly like Anthropic’s proposal for it, in alignment with OpenAI’s and Google’s framework, and there’s even a clear path for xAI to update theirs to align with the requirements. Given the laws’ very high threshold for ‘frontier’ models (10^26 FLOPS), that’s likely the whole list. In addition, only the California AG is able to enforce the law and can only impose civil penalties for non-compliance. It’s unclear whether the frontier labs will need to do anything in order to comply, and the risks of doing so are relatively low in the short term.

• It doesn’t address any meaningful AI issues, nor clarify the legal environment

  • SB 53 doesn’t address many of the immediate AI policy issues that downstream deployers and users are struggling with. For example, despite frontier model requirements, the bill does not address copyright issues, liability transfers, or content watermarking. The focus on only ‘catastrophic risks’ is unlikely to make a big impact for high risk sectors trying to understand how to simply not break existing customer relationships and laws when deploying AI systems.

Key Takeaway: It’s unclear whether SB 53 is ‘regulation’ or ‘regulatory capture’ by frontier model providers. For most downstream AI system builders, the biggest impact will likely be receiving 300 pages of documentation, instead of the current 150 pages.

2. AI Governance Meets AI Insurance: How Trustible and Armilla Are Advancing AI Risk Management

AI adoption is accelerating, but readiness still lags behind. Nearly 59% of large enterprises are already working with AI and plan to expand investment, yet only 42% have deployed AI at scale. At the same time, incidents of AI failure are rising sharply; the Stanford AI Index recorded a 26× increase in AI incidents since 2012, and more than 140 AI-related lawsuits are currently pending in U.S. courts.

The message is clear: as organizations race to integrate AI into products, operations, and decisions, risk management has to evolve just as quickly. That’s why last week, Trustible and Armilla AI announced a new partnership to tackle these challenges.

Together, we’re connecting the dots between AI governance and AI insurance, helping enterprises both prevent and protect against emerging AI risks. Trustible helps organizations operationalize responsible AI governance, while Armilla, provides affirmative AI insurance, explicitly covering risks that traditional cyber or E&O policies often exclude, such as model errors, generative AI copyright and libel issues, and regulatory penalties.

By working together, Trustible and Armilla create a feedback loop between good governance and improved insurability, enabling organizations to innovate confidently while minimizing and transferring residual risk.

You can learn more about the partnership here.

3. AI Incident Spotlight - AI Agents aren’t always leaving a paper trail (AI Incident 1218)

What Happened: Cybersecurity researchers have identified that in some instances, searches from Microsoft 365 Copilot don’t get properly registered in a document’s audit log. Any human ‘look-up’ or access to a file in Microsoft 365 is logged in a dedicated ‘audit trail’ which is an essential part of appropriate access controls. However despite Copilot citing answers from a certain document, there is not always a permanent record that Copilot read information from that file.

Why it Matters: Generative AI ‘answering’ systems can accidentally break normal access control rules and share information from documents a user may not otherwise have access to (data leakage). Not logging this access appropriately can exacerbate this issue, or even encourage this attack vector because. Most enterprise IT/ security policies require strict access controls and audit logs to help detect unauthorized access or use, and at least in some instances, Copilot may not obey the normal control expectations.

How to Mitigate: Without additional information, our theory is that documents get stored in their ‘embedding representation’ inside of 365 so that they can be searched over by an LLM. This means the information being accessed by 365 Copilot is not the ‘original’ document that stores the audit log. In addition, registering every ‘system’ access may bloat the audit log too heavily and there are not yet standards for logging AI system vs human accesses. For now, we recommend keeping highly sensitive documents out of files/folders indexed by 365 until this issue is fixed.

4. Technical Insight - What To Know about ‘Model Specs’

OpenAI recently made some significant changes to their ‘Model Spec’. The latest changes focus on better behavior for agents, trying to reduce ‘sycophancy’, and incorporated insights from a recent ‘public alignment’ project OpenAI has been running. Their model spec outlines how OpenAI has fine-tuned their models to enforce certain nuanced situations. It’s the best tactical representation of both their top level AI principles, and the specific guardrails built into their systems. While OpenAI is the only provider to use this exact format, other frontier model providers publish their versioned ‘System Prompts’ (Anthropic, Grok), which serve a similar purpose, although are not as structured.

Why is it relevant: OpenAI’s model spec is one of the most detailed documents about how frontier model providers are trying to align their models. While system cards give in depth insights into technical details, the model spec is consumable by non-technical experts, and contains more actionable information. Knowing what a model is supposed to do is essential for helping establish if the model is malfunctioning (acting outside of the spec), or if it allows certain behaviors that the deployer may want to block on their own. While recent legislation like SB 53 focus only on ‘catastrophic risks’ and will require reports on mitigations efforts towards those, the model spec contains relevant information for understanding whether the system has specific guardrails against things like data leakage, generating images based on a person, or how to handle chats about sensitive topics like sexuality. Publishing model specs, or similar documents, could be the next type of ‘transparency’ document frontier models may be required to publish in the future. The author of the Trump Administrations’ ‘AI Action Plan, Dean Ball, recently proposed such an idea in his newsletter, even while arguing for federal pre-emption of other AI regulations.

Key Takeaway: OpenAI’s model spec is better documentation to examine than their system cards for non-technical AI governance professionals trying to understand the risks of deploying or using OpenAI models.

5. Policy Round-Up

Trustible’s Top AI Policy Stories

The Problems with Sora 2. OpenAI launched its new video generation model, Sora 2, a couple of weeks ago. Since then, Sora 2 has raised fresh concerns over its environmental impact, ability to spread misinformation, and IP infringement.

Our Take: When using new models, AI governance professionals should consider metrics like the model’s impact on resources (e.g., energy and environment), as well as understand what types of outputs are being generated and the appropriate ways to use them.

The EU’s AI Breakup with the US. The European Commission released the Apply AI Strategy, which will invest approximately €1 billion in the EU’s AI industry to reduce its reliance on the US and China.

Our Take: A new EU ecosystem will provide companies with new choices for AI models and tools that take a different approach to AI safety and security than the US.

Fears Over the AI Bubble. There have been growing concerns over an “AI bubble” in the economy that is reminiscent of the dot-com bubble from the late-1990s.

Our Take: AI is here to stay (whether or not a bubble exists or bursts) and that means AI governance is not going anywhere.

California Regulates Companion Chatbots. Governor Newsom signed SB 243 into law, which protects minors and other vulnerable groups from AI companions.

Out Take: Chatbots are generally thought to be low risk use cases, but the new law underscores how companies need to have insights into the safeguards around their chatbots.

In case you missed it, here are additional AI policy developments:

United States Congress. Two bipartisan AI bills were recently introduced in the Senate. The AI LEAD Act would impose a “duty of care” standard for AI system developers and would classify AI systems as products, as opposed to platform. The AI Risk Evaluation Act would establish the advanced AI evaluation program through the Department of Energy.

Asia. AI-related policy developments in Asia include:

• China. The Chinese government is attempting to crackdown on Nvidia chip imports as it seeks to promote its own homegrown chip industry. It has also been reported that Chinese government officials were caught using ChatGPT to create tools for mass surveillance and social media monitoring.

• Vietnam. The Ministry of Science and Technology is seeking public input on a comprehensive AI law.

Europe. ASML’s Chief Financial Officer criticized the EU for overregulating AI, claiming that the difficulty with AI in Europe is “because [the EU] started with regulating, to keep AI under the thumb.”

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. OpenAI is looking to Canada for cheaper energy and as part of the deal would help build new data centers in Canada as it pushes to expand its sovereign AI industry.

• Mexico. Salesforce announced that it would invest approximately $1 billion in Mexico over the next five years in an effort to expand AI adoption.

South America. OpenAI signed a letter of intent that would invest up to $25 billion for a large-scale data center, which is expected to be built in the Argentine Patagonia.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In today’s edition (5-6 minute read):

1. Model Swap

2. Good Models Still Require Credible Data

3. What Makes for the “Perfect” AI Use Case Intake Process?

4. Global & U.S. Policy Roundup

1. Model Swap

OpenAI and Anthropic recently ran a cross-evaluation of each other’s models. Both highlighted different risks, and demonstrated how task framing shapes evaluation outcomes.

OpenAI leaned on instruction-following and jailbreak resilience as core safety markers, noting for instance that Claude refused up to 70% of hallucination probes rather than risk giving a wrong answer. Anthropic emphasized long-horizon misuse and sycophancy, finding that OpenAI’s o3 resisted harmful misuse better than GPT-4o, GPT-4.1, or o4-mini, which were often willing to cooperate with simulated bioweapon or drug synthesis requests.

While some research labs like Epoch AI, and government sponsored institutes like the UK’s AI Security Institute (AISI) have done independent model evaluations, this was the first instance of the 2 frontier model leaders conducting this kind of evaluation swap. The differences in evaluation approaches, and respective strengths/weaknesses of each other’s models highlights how the philosophies, and incentives, of the model providers can be reflected in their models. It also highlights the need to have a wide range of evaluations, and now just rely on self reporting. The evaluations focused on ‘frontier capability’ assessments however, not assessments on more ‘day to day’ AI tasks. OpenAI more recently introduced GDPEval to assess model performance on industry specific tasks, and this type of benchmark could quickly become an industry standard relevant to AI deployers and users.

Key Takeaway: Evaluations that just rely on a single number, can easily mask some of the more nuanced differences between foundation models. Once you dig into the details, the priorities, culture, and incentives of a model provider may be more clear, and may be an issue for certain types of use cases.

2. Good Models Still Require Credible Data

A recent study showed that ChatGPT may return answers based on claims in redacted scientific studies, even when information about the redaction appears in the same document. Unlike hallucinations, where the model fabricates a claim, this study points to the models inability to properly contextualize information in the training data and recognize when it’s inaccurate. This challenge can’t be solved easily: first, the pre-training process breaks up the documents into multiple pieces, thus a retraction statement may have been associated with the title of the paper, but not with individual statements in another part of the paper. Second, modem LLMs are trained on a vast corporas of data that are not manually reviewed, thus removing all erroneous information from pre-training data isn’t feasible (even if it was - this process would be biased and subjective).

While this study focused on GPT 4o-mini’s internal knowledge, many modern systems integrate external information through web searches or connection to internal knowledge bases (i.e. RAG). These integrations can help the model analyze the document as a whole and consider retractions published on external platforms. We tested a couple example claims from the study and found that during a web search the model recognized the retraction and corrected its original answer.. However, this process still relies on the model having access exclusively to up-to-date information; an out-of-date document that isn’t annotated as such can cause similar erroneous assertions. This failure mode may have been behind Air Canada’s chatbot stating false advice on bereavement travel last year. Our own team at Trustible has encountered similar challenges using AI for coding tasks, where old, outdated code (tech debt), regularly confuses the model.

Key Takeaways: LLMs can not reliably identify if information is up-to-date, especially if updates or retractions are not clearly associated with a specific fact. Practitioners should take care to maintain accurate data sources for training, fine-tuning and RAG. In addition, detailed prompting can help the system explicitly check for potential inconsistencies.

3. What Makes the “Perfect” AI Use Case Intake Process?

AI intake is the front door to governance. In a panel at the IAPP AI Governance Global North America in Boston a few weeks ago, Trustable led a discussion with leaders from Leidos and Nuix on AI use case intake processes. The room quickly aligned on a core truth: there’s no “perfect” intake process, only the one that fits your org’s risk profile, scale, and speed. The real work is choosing trade offs you can live with.

The conversation unpacked six design levers teams should tune, not max out:

• Granularity: are you tracking whole use cases, or features within products?

• Heaviness: what’s the minimum set of questions that still surfaces risk?

• Outcomes: does intake simply route, or also drive mitigations and decisions?

• Participation: who owns what—privacy, legal, security, product, HR, IT?

• Implementation: start with forms/spreadsheets, but plan for workflow and automation.

• Timing: catch ideas early without slowing experimentation.

Practitioners shared pragmatic moves: start where you are (even if it’s messy), iterate fast, and reframe intake from “audit” to “risk reduction.” Build muscle memory with short cycles and clear handoffs. As volume grows, expect ad hoc docs to buckle; that’s your signal to standardize fields, centralize your inventory, and automate routing so triage and transparency don’t degrade. Perhaps most importantly, make intake a shared habit—invite cross-functional partners in before the first pilot, not after the first incident. Culture change is the glue that keeps the process from backsliding.

Key Takeaway: fit beats perfection. A right-sized intake gives leaders visibility, lets teams move quickly with guardrails, and creates artifacts that stand up to regulatory and stakeholder scrutiny.

You can read the full recap on our blog.

4. Global & U.S. Policy Roundup

Trustible’s Top AI Policy Stories

OSTP Request for Information. As part of the Trump Administration’s AI Action Plan, OSTP is seeking input on existing rules that may hinder AI deployment or adoption.

Our Take: Beyond commenting on rules that industry does not like, this proceeding will allow companies to identify redundancies among existing rules that can help streamline AI governance processes should the federal government tweak them.

Anthropic settlement approved. Judge Alsup approved the $1.5 billion settlement agreement between Anthropic and authors who say the company infringed on their copyrights.

Our Take: The copyright cases continue to highlight why companies need to know where data used in their AI tools come from and they have the requisite permission to use IP in those tools.

United Nations AI Dialogue. The UN announced the “global dialogue on AI governance,” which would create a panel of experts to study best practices for AI governance.

Our Take: The UN is trying to stake its claim on AI standards but the final recommendations will likely remain at a high-level, making it difficult to operationalize at the enterprise level.

AI Bills in California. Governor Gavin Newsom signed SB 53 into law, which is a watered down version of last year’s SB 1047.

Our Take: The new law imposes new governance obligations for frontier model providers with some limited downstream impacts.

In case you missed it, here are additional AI policy developments:

Asia. AI-related policy developments in Asia include:

• China. Deepseek launched DeepSeek-V3.2-Exp, a new model billed as an “intermediate step toward [their] next-generation architecture.” The new model is likely to put new pressure on other Chinese model providers like ​Alibaba, which launched their new Qwen 3 Max model earlier this month.

• Japan. The Ministry of Defense is taking a hard line on AI in the military, allowing it to help with defense operations but maintaining that humans must be in charge of lethal force decisions. Japan’s new policy comes as it formalized the SAMURAI Project with the U.S., which is intended to advance AI safety in unmanned aerial vehicles.

• India. The government signed a new partnership agreement with Venezuela to “jointly explore the integration of [AI] and digital public infrastructure in sectors such as health, payments, and education.”

Australia. The Digital Economy Minister reiterated that Australia will take a light touch approach to AI regulation. The Australian government was interested in a more comprehensive approach but have since abandoned that effort.

Europe. AI-related policy developments in the Europe include:

• EU. The European Commission (EC) may pause implementing the AI Act, after strongly rejecting the idea back in July. The potential pause will be discussed at an upcoming AI Board meeting in October primarily because of implementation delays at the national level. The EC also published draft guidelines for reporting serious incidents as required under the AI Act.

• Italy. The Italian government passed a new AI law that criminalizes certain uses of AI, such as creating deepfakes or assisting with committing crimes. The law also requires that children under the age of 14 get consent from their parents to access AI.

Middle East.

• UAE. Sam Altman met with President Sheikh Mohamed bin Zayed Al Nahyan to discuss how to foster closer cooperation on AI.

• Saudi Arabia. Representatives from South Korea met with officials in Saudi Arabia to discuss closer collaboration on building more innovative environments for SMEs and expanding new market opportunities.

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. The AI and Digital Innovation Minister announced a new AI Task Force that will work over the next 30 days to make recommendations for Canada’s national AI strategy. It is unclear whether these recommendations will turn into actual regulations.

• Mexico. CloudHQ announced a $4.8 billion data center in a state just north of Mexico City.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In the meantime, in today’s edition (5-6 minute read):

1. The Great AI Copyright Conundrum Part II

2. Judging LLM-as-a-Judge

3. Global & U.S. Policy Roundup

4. AEO / GEO - The New SEO

1. The Great AI Copyright Conundrum Part II

The summer saw a flurry of developments on the interplay between AI and IP. Two other cases, one against Meta and another involving Anthropic, handed significant victories to big tech and how their AI systems use protected works. As we move into the fall, there is another avalanche of news on AI and copyright litigation.

We previously discussed a lawsuit brought by a group of authors who sued Anthropic for copyright infringement by alleging that the company trained Claude on their protected works without their permission. Anthropic sought to put an end to the case with a $1.5 billion settlement, which would have paid roughly $3,000 per book that they pirated. However, Judge William Alsup rejected the deal because of his concerns with how the deal was struck and the claims process. Judge Alsup will review the deal again on September 25 to “see if [he] can hold my nose and approve it.” Meanwhile, Apple is subject to a new lawsuit by authors who claim that their protected works were used to develop Apple Intelligence. Perplexity was also recently sued by Encyclopedia Britannica and Merriam-Webster over how it uses their material for its “answer engine.”

The Anthropic settlement sought to provide an off-ramp for the dispute at a cost lower than prolonging the litigation. Yet the broader concern (as raised by Judge Alsup) is how this deal came about, given it was done behind closed doors. Generally settlements are not bad, but when genuine legal questions need resolution, a quick fix cash settlement does not quite meet the moment. Moreover, determining the value for using someone’s IP to train an AI system is not a cut and dry equation. Paying IP owners the same amount (as seen in the Anthropic settlement) ignores the fact that some IP is more valuable than others. A recent New York Times op-ed raises legitimate issues over how to put a price on someone’s IP and how someone could effectively “game the system” if all IP is worth the same price.

Key Takeaway: First and foremost - know where your data comes from because you do not want to be accused of violating IP law. Second, we need policymakers to intervene and actually fix the issue because relying on court decisions to effectively make law is not a long-term sustainable solution. If there are growing concerns over the “patchwork of state AI laws” in the US, then there should be greater concerns with the patchwork of legal decisions and outcomes, especially when the US Supreme Court is less inclined to settle these types of questions.

2. Judging LLM-as-a-Judge

Over the last two years, the LLM-as-a-Judge (LLJ) methodology has become a go-to technique in AI development - but a recent paper titled Neither Valid nor Reliable? Investigating the Use of LLMs as Judges by Chenbouni et al. highlights critical weaknesses of this approach. LLJ refers to the practice of using existing LLMs to support development of new AI systems. Common uses include:

• Performance Evaluation: Replacing human annotators for reviewing outputs of AI systems. For example, LLJ can be used to judge fluency and professionalism of model outputs or to evaluate safety alignment (many of the tests in GPT-5’s system card rely on this technique).

• Model Enhancement: LLJs can assist with training of new models through processes like reward modeling and/or replacing humans in RLHF (where the goal is to select a preferred output from a model).

• Data Annotation: LLJs can be used to annotate datasets for model training and evaluation.

LLJs became popular for these tasks because they are considered a proxy for human judgement that can be used cheaply, quickly and at-scale; however, the limitations of this approach are often overlooked. Two key factors to consider are:

• Validity: When assessing whether LLJs do in fact agree with human judgement, it is important to consider whether humans themselves can agree on this task. On tasks with a high degree of ambiguity, both the method for measuring agreement and an appropriate threshold may not be clear. For example, the GPT-5 system card mentions a 75% agreement between the LLJ and human assessors - this threshold may suggest that relying on the LLJ alone may not be sufficient.

• Reliability: LLJs exhibit a variety of biases ranging from preferring outputs from itself (when used to compare multiple candidate models) to skewed racial and gender preferences in reward modeling. Furthermore, while some modern models can output an “explanation” for their label, this text may not be faithful to the underlying processes. With these factors in mind, a rating for “professionalism” (as in the above example) of text may reflect a number of subjective factors beyond the user-specificed criteria.

Chenbouni’s paper outlines a deeper list of interconnected challenges with the use of LLJs and my. Many of the challenges outlined parallel human evaluation on difficult tasks. Mitigations can include using LLM-as-a-Jury (an ensemble of diverse LLJs) and carefully reviewing LLJ outputs for biases and consistency before deploying them at scale.

Key Takeaway: In an ecosystem where the pressure to deploy AI systems quickly is high, LLJs can be seen as a quick win for evaluation; however, concerns around validity and reliability suggest that they themselves should be carefully evaluated for appropriateness for a given task. When using LLJs, practitioners should examine the task itself for subjectivity (for example by evaluating human inter-annotator scores on a sample), evaluate the model outputs against a large gold-standard dataset, analyze the errors for systematic biases, and when possible uses an ensemble of LLJs (i.e. LLMs-as-a-Jury.)

3. Global & U.S. Policy Roundup

U.S. Federal Government. AI-related policy developments across the federal government include:

• White House. Office of Science and Technology Policy (OSTP) director Michael Kratsios teased the White House’s AI policy plans. OSTP will solicit input from the public on “federal regulations that they think hold back the development and deployment of AI.” President Trump also hosted CEO’s from all the major tech companies to discuss AI innovation.

• Federal Agencies. The FTC launched an inquiry into how certain companies deploy their chatbots, with a particular focus on harms posed to teens and children. The inquiry is not a formal investigation and seeks to obtain information from seven AI companies. The Government Accountability Office released a report outlining 94 AI-related requirements for the federal government.

• Congress. Senator Ted Cruz (R-TX) is back in the news declaring that the AI moratorium on state and local laws is “not dead at all.” It is unclear what Cruz meant by his comments, as the moratorium language was removed by the Senate from the Republicans reconciliation bill earlier this summer. Cruz also introduced the SANDBOX Act, which would permit AI developers and deployers to seek waivers for regulations that could “impede their work.” OSTP would coordinate with federal agencies to evaluate requests within the scope.

U.S. States. AI-related policy developments at the state level include:

• California. The state legislature passed SB 53 with a veto-proof majority. The bill is a reincarnation of last year’s SB 1047 with a far narrower scope that requires foundation model providers to implement and disclose safety and security protocols for their models.

• Michigan. The Michigan Chamber of Commerce came out against a proposed bill in the state legislature that would comprehensively regulate AI in the same vein as Colorado’s AI law. The Chamber called the bill “well-intentioned” but favored a federal AI law.

• Ohio. The state’s Department of Homeland Security launched a new reporting system that uses AI to disseminate information about potential threats of violence. Users can upload photo, video and audio of alleged suspicious activity, which is reported and reviewed by analysts at the Statewide Terrorism Analysis and Crime Center.

• Texas. Secretary of Education, Linda McMahon, visited a private school in Austin, TX that is relying on AI to help teach its students. The Secretary also participated in a roundtable discussion about “AI literacy and the evolving role of technology in education.”

Africa. The South African government hosted a forum to discuss creating a “National Artificial Intelligence Network of Experts.” Uganda launched the Aeonian Project, which intends to build Africa’s first “AI factory.”

Asia. AI-related policy developments in Asia include:

• China. The Beijing Internet Court released decisions in eight AI-related cases. Notably, the court found that AI-generated content can be protected under the country’s copyright laws and that platforms using algorithms to detect and remove AI-generated content must provide “reasonable explanations” for their decisions. While these cases are significant, they are non-binding because China follows a civil law system.

• Kazakhstan. The President of Kazakhstan announced plans for a new Ministry of Artificial Intelligence and Digital Development to address some of the emerging threats posed by AI technologies.

• South Korea. The Ministry of Science and Information and Communication Technology released a draft decree to implement South Korea’s AI Basic Act. The draft decree requires certain disclosures for developers of generative and high-impact AI, as well as establishes safety assurance standards for high-performance AI.

Europe. AI-related policy developments in the Europe include:

• Albania. The Prime Minister appointed an AI-generated minister to address corruption and improve government transparency. The minister, known as Diella, was created with help from Microsoft.

• EU. The European Commission announced that the forthcoming "digital omnibus” will cover “targeted adjustments” to the EU AI Act. At the same time, the “stop the clock” movement got a notable boost from former Italian Prime Minister, Mario Draghi, who argued that the EU AI Act should be paused to assess potential “drawbacks.” Poland also is proposing a 6 to 12 month delay for high-risk AI system penalties under the EU AI Act. Mistral also announced that it raised €1.7 billion in its most recent funding round and secured a strategic partnership with Dutch semiconductor company, ASML.

• UK. OpenAI and Nvidia plan to announce a large investment in AI infrastructure as part of President Trump’s state visit towards the end of September. The exact dollar amount has not been disclosed, but the deal includes commitments from the UK government to supply energy, whereas OpenAI will provide “access to its AI tools and technology” and Nvidia will supply “the chips used to power AI models.”

Middle East. The UAE sent local and federal chief AI officers to meet with representatives from the US tech industry in Silicon Valley. The Institute of Foundation Models at Mohamed bin Zayed University of Artificial Intelligence and G42 also launched K2, described as a “leading open-source system for advanced AI reasoning.”

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. The Canadian federal government intends to create a public registry for its AI projects. OpenAI is also seeking to move its copyright case out of Canada and to the US, arguing that it does not do business there and is not subject to Canadian copyright law.

• Mexico. The Chamber of Deputies and Senate are working on a comprehensive AI law that leverages existing law with new requirements to address risks from AI technology. Mexico’s work on a national AI law comes at a time when other countries are abandoning or delaying their efforts on similar legislation.

4. AEO / GEO - The New SEO

In the pre-AI internet era, most businesses lived or died at the hands of algorithms owned by search giants that determined how webpages showed up in internet searches. An entire field of tools and services, called Search Engine Optimization (SEO), developed to backwards engineer the search algorithms and optimize a website’s ability to rank high in search results. As we enter the AI era, a brand new category is being created around optimizing content for AI-answer engines.

There are two variants to AI-optimization, but are often conflated with each other: Answer Engine Optimization (AEO), and Generative Engine Optimization (GEO). AEO is geared towards appearing in the ‘answer box’ that now shows above many traditional search platforms. These aim to provide a single answer for any given query. GEO in contrast is focused on the experience inside of AI chat platforms entirely. Most major consumer AI platforms like ChatGPT have the ability to pull in various search results or have existing indexed content, and use them to generate answers. Pulling data from the internet can both reduce hallucinations, and (probably) also help provide some legal liability mitigations in the US, and search platforms that show user generated content have traditionally been protected under Section 230.

While there is no shortage of potential market impacts from this, there are two major points of importance here for AI Governance professionals. The first is that the way content is written and optimized for the web will have to change. Whereas SEO was heavily focused on backlinks for page authority, AI systems may be optimized for different kinds of direct information. Dozens of startups have emerged to help organizations with this task, and unsurprisingly many groups are also looking for AI to help. How a brand is presented in an AI tool, could become an area of concern for AI governance teams who may have the expertise on how to support the business as it navigates AEO/GEO issues. The second, is to be aware that the answers from an AI chat system can be influenced in the same ways as search. Google has already discussed potential ‘sponsored answer results’, there likely will be over time, and in the meantime, any ‘answer’ from an AI system that conducts internet searches should be viewed with the same level of scrutiny as a regular search engine.

On the inverse, GenAI content is a cause that many brands are seeing their content visibility decline sharply (both in traditional SEO and GEO.) Search engines and LLMs are prioritizing human, organically written content for rank due to its unique and novel quality, where content developed by LLMs are ranked as a lower quality. We’ve covered this in the past on AI our coverage on AI slop. This presents a balancing act, where AI can help accelerate content development and visibility in the era of AEO/GEO, but overreliance can actually do more harm than good.

Key Takeaway: AI tools may quickly become the ‘gatekeepers’ of information. What they say about companies, brands, or services, could quickly be accepted as canonical. Many AI tools will only give a few answers to questions like ‘What’s the best newsletter platform’, and so the competition to be in the AI’s answers will be fierce, and could quickly impact entire markets.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In the meantime, in today’s edition (5-6 minute read):

1. The Hidden Danger of Chatbots: Why “Yes” Can Be Deadly

2. How Model Safeguards & Performance Degrade With Use

3. Global & U.S. Policy Roundup

4. Shifting Winds: What Federal Moves Mean for U.S. AI Hegemony

1. The Hidden Danger of Chatbots: Why “Yes” Can Be Deadly

Chatbots are designed to be agreeable. Their instinct to say “yes” makes them engaging, but it also makes them dangerous in high-risk applications—especially when the stakes involve mental health, financial decisions, or employment outcomes.

There has been increasing attention given to people using AI enabled chatbots for ‘self service mental health’. A recent CommonSenseMedia study suggests that of the 72% of US teens who regularly use AI systems, up to an eighth of that cohort may have used AI for therapeutic or mental health purposes. The dangers of using AI systems for this use case became evident recently after the family of a recently deceased teenager, Adam, sued OpenAI for gross negligence and wrongful death after Adam committed suicide. The lawsuit alleges ChatGPT did everything from giving feedback on how to construct a stronger noose, why he shouldn’t feel guilt about the suicide, and how to steal alcohol from his parents to dull the pain. The lawsuit points out that if any human gave the feedback that ChatpGPT gave Adam, there would not be any doubt about their complicity. Cases such as this could quickly set clear precedents for the liability that chatbot creators may face.

There are many potential takeaways from this unfortunate incident, and plenty of reasons why nascent AI systems should not be used for mental health purposes. For an in-depth breakdown of some of these, we recommend a recent Stanford study on the issue. However, we'll focus on just one particular element of the AI system that makes them particularly dangerous: their built-in willingness to say ‘yes’ to anything and everything. This tendency has been described in extreme cases as ‘sycophancy’, but outside of that sensationalist term, a system that is conditioned to always say ‘yes’ should not be used in certain circumstances. On the one hand, an AI system that is constantly pushing back or saying ‘no’ will likely not be widely adopted, and training for that may be difficult. On the other hand, in certain domains such as health, law, or finance, we often expect, and even pay massive sums, to people to tell us ‘no’. No, you likely don’t have a rare disease, ‘no’, that action likely isn’t legal, and ‘no’, that investment strategy isn’t going to make you rich. A system that will always tell you ‘yes’ is a dangerously tempting tool for a wide variety of use cases because it will always appeal to our own desire for confirmation and validation. There are not yet clear principles or regulations on when an AI should reject instructions, and when it may be equally important to always accept them.

Key Takeaway: LLMs are conditions to say ‘yes’ to as many things as possible, and there are market drivers that encourage that. However, that makes them uniquely unsuitable for certain domains where having a bias for ‘yes’ could cause physical, financial, or reputational harm.

2. How Model Safeguards & Performance Degrade With Use

Following the wrongful death lawsuit against OpenAI alleging that ChatGPT is associated with multiple reported instances of self-harm and attributed as a cause of suicide in recent cases, OpenAI released a blog detailing their current and future safeguards. What’s notable is their acknowledgment that existing protections work best on short exchanges, not extended conversations. This effect is not unique: its been observed across both models (e.g. Claude, Llama and Grok) and violation types.

Until now this failure-mode has not been given significant attention: Companies like OpenAI publish extensive System Cards that detail types of safety tests run, but most of them focus on single-turn conversations (e.g. Given a specific question in isolation, will the model return an inappropriate response). While GPT-5’s System Card does mention a manual red-teaming exercise that tested for psychological harms, which included ‘multi-turn, tailored attacks [that] may occasionally succeed’ - the severity of the violations was low. These tailored attacks were unlikely to include realistic conversations that spanned multiple months.

Generative AI systems used two broad categories of safeguards - both of which may fail to capture nuance of long-term use.

Post-Training: Many AI Systems use SFT (supervised fine-tuning) and RL (reinforcement learning) to teach the model to produce correct responses to user prompts. Both of these methods are applied to single-turn conversations - meaning models aren’t explicitly trained to respond correctly in a multi-turn context. This may not be evident during basic testing that, also, only tests single-turn conversations.

Output Monitoring: OpenAI uses classifiers to check if both user prompts and model outputs should be flagged for potential violations. In the blogpost, they mention that the classifiers have a particular issue with underestimating the severity of what it's seeing. One particular difficulty may be that the violations are subtle and require the context of a longer conversation to properly gauge severity.

Key Takeaway: Both the performance and protections of models during long conversations (we estimate at over 20 turns) are understudied, but anecdotal evidence suggests that they degrade significantly. For enterprises and consumers, especially those relying on LLMs for high-risk use cases, it’s important to ensure that users are aware of this risk and the negative outcomes, and to consider the outputs accordingly. It’s equally important to monitor and test your systems’ performance as part of your overall AI governance program.

3. Global & U.S. Policy Roundup

Major International AI New: The United Nations’ General Assembly established the UN Independent International Scientific Panel on AI and the Global Dialogue on AI Governance.

U.S. Federal Government. First Lady Melania Trump announced a national challenge for K-12 AI students to create innovative AI solutions for community problems. The Department of Labor also announced new guidance to help states understand how the Workforce Innovation and Opportunity Act grants can help “bolster [AI] literacy and training across the public workforce system.”

U.S. States. AI-related policy developments at the state level include:

• California. The State Superintendent hosted the first meeting of a new AI workgroup that was established under legislation from 2025. The group will help create guidance for AI in K–12 education.

• Illinois. A recent report from the Alliance for the Great Lakes raised alarm bells for Chicago’s drinking water because of AI data centers’ increasing strain on Lake Michigan and other local water systems. Illinois is home to approximately 187 data centers, most of which are located near Chicago.

• Michigan. Governor Gretchen Whitmer signed a package of bills into law that prohibits using AI to create non-consensual explicit images of real people. The new laws also set sentence guidance for violators, which can include jail time.

• Virginia. Google plans to invest an additional $9 billion for AI infrastructure in Northern and Central Virginia through 2026. A new AI-power private school also opened its doors in Northern Virginia.

Africa. CNBC Africa held their first AI Summit in Johannesburg, South Africa. The summit brought together over 300 industry professionals to discuss the “impact of AI across Africa's key sectors.”

Asia. AI-related policy developments in Asia include:

• China. An official from the National Development and Reform Commission echoed comments from Chinese President Xi Jinping that focused on preventing “disorderly competition” with new AI models. The Chinese government is attempting to prevent duplicative efforts that could lead to deflationary pressures within the AI sector.

• South Korea. The South Korean government is proposing an 8 percent increase in AI investment under its 2026 budget. In addition to spending increases for R&D, the proposed budget also directs more funds towards AI startups.

• Thailand. Bangkok has proven to be a strategic hub for data centers, fueled in part because of major investments from tech companies such as AWS, Google, Microsoft, and Alibaba.

Australia. The Commonwealth Bank of Australia (CBA) recently reversed a decision to lay off employees due to AI. The CBA noted that the customer service jobs it eliminated after introducing its AI-powered “voice-bot” were not redundant.

Europe. AI-related policy developments in the Europe include:

• EU. EU Industry is being accused of dragging their feet on helping write EU AI Act standards. Piercosma Bisconti, one of the experts who is helping write EU AI Act standards, claimed that companies who support delaying the EU AI Act’s implementation “should be contributing [to standards development], and they are not” and added that "EU industry is barely at the table.”

• Switzerland. The Swiss Federal Technology Institute of Lausanne released Apertus, an open-source LLM that is trained only on publicly available data. The Swiss hope that Apertus can be a plausible alternative to proprietary models, like OpenAI’s GPT models.

Middle East. Saudi owned Humain released “HUMAIN Chat,” a chatbot designed to “comply with Islamic values.” The latest announcement comes as Saudi Arabia angles to be the third largest model provider in the world, after the US and China.

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. The Canadian federal government signed a memorandum of understanding with Cohere to help accelerate AI adoption for public services and reinforce its position as global leader in AI. Meanwhile, the vast majority of Canadians want some type of AI regulation as many Canadians express concern with AI safety and risks.

• Mexico. The Supreme Court unanimously ruled that content generated exclusively by AI is copyrightable under the country’s current laws. The Court found that “automated systems do not possess the necessary qualities … for authorship.”

South America. Uruguay became the first Latin American country to sign the Council of Europe’s AI treaty, which seeks to promote the use of AI systems that respect human rights, democracy and the rule of law. 16 other countries, including the US, have signed the treaty.

4. Shifting Winds: What federal moves mean for U.S. AI hegemony

In the high-stakes international battle for AI leadership, the U.S. federal government’s recent AI decisions, ranging from GSA’s procurement acceleration to export-levy tactics, are not just about innovation; they're complete strategic pivots.

The White House’s broader AI Action Plan now prioritizes infrastructure, innovation, and international leadership, elevating AI to the status of critical U.S. infrastructure. This deregulatory posture, emphasizing "permissionless innovation", positions tech firms to benefit. But critics warn it increases risks from misinformation, ethical bias, and global tensions.

GSA’s inclusion of OpenAI, Anthropic, and Google models on its Multiple Award Schedule (MAS), with heavily discounted “OneGov” pricing of $1 or less, widens federal access to frontier AI technologies cheaply and quickly.. But this move has drawn protests over preordained pricing and lack of competition, raising concerns among governance officials. From a geopolitical standpoint, it ensures U.S. federal infrastructure remains closely tied to American AI ecosystems, reducing foreign reliance, yet may compromise procurement transparency and preparedness.

For large organizations and system integrators supporting government contracts, the procurement bonanza, from government-wide acquisition to USAi.gov’s sandbox, accelerates onboarding. USAi, a free GSA-hosted AI evaluation suite, offers agencies unified API access to multiple models in a secure environment. Yet its pilot nature, according to the GSA, is only temporary in nature; the GSA doesn’t want to be in the business of providing tools for the long-term.

Complementing that, the National Security Memorandum on AI underscores AI’s strategic role in defense, intelligence, and allied collaboration, while insisting government AI use must uphold civil rights, transparency, and democratic values. These policies together reflect an urgent understanding: global competitors like China and the UAE are racing ahead in compute, sovereign AI, and infrastructure. Enter initiatives such as the Stargate Project, a potential $500 billion-plus U.S. investment in AI infrastructure through partnerships with OpenAI, Oracle, SoftBank, and MGX, framed as a Manhattan Project–scale response to global pressure.

Moreover, the U.S. now taps into chip-based revenues: a previously unthinkable 15% levy on Nvidia/AMD AI chip sales to China, tied to export approval. This gives Washington financial leverage, disincentivizes adversary acceleration, and incentivizes domestic supply chains, all signals to enterprises that hardware cost projections must now include geopolitical premiums.

Why it Matters: The U.S. drive for AI global dominance through deregulation, infrastructure building, and enhanced AI literacy in schools and across the workforce contrasts sharply with the EU’s regulation-heavy path and China’s authoritarian deployment model. We are now deep in the throes of a fractious and alternative set of emerging visions for what global AI hegemony should look like and whose vision should come out on top. For the U.S. to truly succeed in scaling trusted AI adoption and putting both private and public sector organizations on a trajectory for AI global dominance, clarity in policy, strategy, and commercial support should be the priorities.

P.S. We’re hosting a webinar at 1 p.m. ET today with Databricks, Schellman, and Trustible to deep-dive on how frameworks and standards, like the Databricks AI Governance Framework and ISO 42001, can help build actionable guardrails that accelerate enterprise AI adoption. You can register here, or sign up to receive the recording later.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In today’s edition (5-6 minute read):

1. AI adoption uneven across demographic groups

2. Trustible launches global partner program

3. What you need to know to effectively and safely use AI

4. Global & U.S. policy roundup

5. What GPT-5 can tell us about AI evaluations?

6. Webinar: Building Guardrails for Enterprise AI - Exploring the Databricks AI Governance Framework and Beyond

1. AI adoption uneven across demographic groups

Recent studies from HBS and Pew show that AI adoption is uneven across demographics in ways that challenge expectations. HBS’s meta-analysis of 18 studies finds women are significantly less likely than men to use generative AI tools, with heightened skepticism over bias and risk being key factors. Pew reports that younger and more educated adults are adopting AI far faster than older adults or those with less educational attainment, with a notable 30-point gap in usage between those with a post-graduate degree (48%), and those with only a high school level (17%).

Meanwhile, an Elon University survey found some racial and ethnic gaps in LLM usage, with Black and Hispanic adults saying they use AI more often (57% and 66% respectively) than the general population (52%) of adults.

While there’s a variety of potential sources or reasons behind these disparities ranging from risk sensitivity, perceptions of bias, availability of education resources, etc, the potential consequences of sustained disparities are clear: negative feedback loops. When groups hold back from using AI, whether from distrust or lack of access, their perspectives become underrepresented in the very systems that are learning from user data and being fine-tuned on it. That dynamic could make future tools feel even less relevant to them and further undermine trust.

Key takeaway: While the tech industry is often quick to attack any form of regulation as potential burdens on or obstacles to AI growth, regulation may also be one of the key elements that unlocks adoptions from the most skeptical AI groups. This unlock is necessary to maximize the benefits of AI.

2. Trustible Launches Global Partner Program

Today, we’re announcing the global launch of the Trustible Partner Program, an ecosystem purpose built to weave AI governance through every stage of the AI lifecycle. Our program brings together technology alliances, system integrators, resellers and distributors, auditors, and insurers around a single objective: help organizations adopt AI responsibly, prove compliance, and scale value with governance at the core.

Accelerating responsible AI adoption isn’t an isolated activity by a single organization or vendor - it takes a coalition of partners to bring trusted AI from being just words on a page to operationalization. The Trustible Partner Program isn’t a passive commitment: we’re creating a working coalition of builders committed to getting AI governance done for organizations—starting today. By aligning leaders across the AI value chain, we’re delivering a connected AI governance experience that accelerates the time‑to‑value of AI and truly delivering trusted AI.

In the coming weeks, we’ll share our initial launch partners, with more to follow throughout the year. We’re opening applications today for Technology Alliances, Resellers/Distributors, System Integrators, and Strategic Alliances. If you’re building, advising, or delivering AI, and you believe governance should be built‑in, not bolted‑on, join us.

You can read more about the program on our blog.

3. What You Need to Know to Effectively and Safely Use AI

The Trump Administration's AI Action Plan proposes several policy recommendations around AI education. In this context, it’s not about the use of AI within the current education system, but rather educating our entire current and future workforce about how to use AI. Massive investments in data center infrastructure, growing model size and complexity, and agentically integrating into existing applications won’t yield returns if there are bottlenecks in human expertise for deploying, using, and governing these systems. While governments will likely take point in enacting AI education within schools, it’s worth considering what an AI literacy and education program should look like inside of an organization, and how to retrain their existing workforce to be expert ‘AI users’.

Basic Literacy - This should teach the basic vocabulary about AI, and basic tenants about the system for users. The exact technical details of AI models themselves don’t need to be covered in depth, but rather this should focus on more basic concepts like what a prompt is, how to provide appropriate context in a prompt, and what different AI tools may be broadly capable of.

Prompting Skills - Describing the exact desired output you want from an AI system isn’t always easy. Much like with computer programming, it can be challenging to figure out the exact steps you want an AI system to take, or describe your desired outcome with enough clarity for the AI system to understand. AI models don’t always have a full understanding of the world, and some relevant information about this needs to be directly provided. There are a lot of prompting techniques that can help

Risk & Governance Education - Much like how every technology enabled employees has to do an annual cybersecurity training, there should be an equivalent for using AI tools. There are some distinct AI risks, ranging from hallucinations, to system biases. Ensuring that users simply know these things are possible can itself be a challenge. Many organizations want ‘humans in the loop’ of AI systems as their main risk mitigation, but that still requires training on what to look for.

Agentic Awareness - As various forms of Agentic AI get developed and deployed, it will become essential to teach staff on what kinds of tasks are ready to be automated, and which ones still need to be done by humans because of AI system limitations, governance concerns, or even because of the business strategy and brand reputation of your organization. There will be a lot of hype from AI agents for years to come, and knowing what can be automated efficiently, will be half the battle.

Key Takeaway: LLMs can be very powerful and extremely accessible because their inputs can be natural language, but that can be a double edged sword. Most organizations won’t be able to hire easily for these AI skills, and will likely need to develop education and training programs for them themselves.

4. Global & U.S. Policy Roundup

Here is our quick synopsis of the major AI policy developments:

U.S. Federal Government. The Trump Administration is touting a deal cut with AI chip manufacturers’ Nvidia and AMD, which stipulates that the U.S. government would receive 15% of revenue from their chip sales to China. Congressional Democrats have asked the Administration to reconsider the deal due to national security concerns. The General Service Administration also announced a new program aimed at encouraging AI adoption across the federal government. The move is part of a push by the Trump Administration to modernize and automate more aspects of the federal government.

U.S. States. AI-related policy developments at the state level include:

• California. OpenAI asked Governor Gavin Newsom to “consider frontier model developers compliant with its state requirements when they sign onto a parallel regulatory framework like the [EU Code of Practice] or enter into a safety-oriented agreement with a relevant US federal government agency.”

• Colorado. The Colorado state legislature will return for a special session on August 21, 2025, during which it is expected to address potential changes to its AI law (SB 205). Proposed bills have been released to amend the current law, all of which would narrow its current scope. It is unclear whether a deal will be struck, as the regular legislative session adjourned without an agreement.

• Florida. The Florida Bar is looking into new rules that address AI risks in the legal profession. The Bar has taken some action on how AI can be used in the practice of law, but recent advancements with AI technology have caused Bar leaders to consider additional guidance.

• Texas. Texas Attorney General Ken Paxton launched an investigation into Meta and Character.ai over controversies with their chatbots. Attorney General Paxton emphasized that the companies may have engaged in “deceptive trade practices and misleadingly market[ed] themselves as mental health tools.” The Texas investigation comes shortly after Congressional Republicans announced their own investigation into Meta’s chatbots.

Africa. Google announced that it is committing $37 million to AI development across countries in Africa. The new investment is aimed at AI research and supporting local AI projects. The announcement comes as Ghana and Lesotho launch a new digital partnership to facilitate cooperation on setting digital standards and frameworks, as well as leveraging AI for agricultural support.

Asia. AI-related policy developments in Asia include:

• China. Sam Altman expressed concerns over where the U.S. stands against China on AI. During a recent interview, Altman noted that he is “worried about China” and that the AI arms race between the two countries is more complex than meets the eyes. Specifically, he highlighted China’s capacity to build and flaws in current U.S. policy (i.e., chip export controls).

• Malaysia. The Association of Southeast Asian Nations (ASEAN) convened the ASEAN Malaysia AI Summit in Kuala Lumpur. Secretary-General of ASEAN, Dr. Kao Kim Hourn, announced during the summit that he expects the ASEAN AI safety framework will be established “by early 2026.” The Malaysian government also launched their National Cloud Computing Policy at the summit. Simultaneously, Huawei hosted the Huawei Cloud AI Ecosystem Summit APAC 2025 at which they announced their intent to train 30,000 AI professionals in Malaysia over the next three years. Huawei has been a source of controversy in the U.S. because of its close ties to the Chinese government.

• Japan. NTT Data, a multinational Japanese information technology company, announced a strategic partnership with Google to “accelerate enterprise adoption of agentic AI and cloud modernisation.” This is the first time a Japanese company has signed a contract of this nature with Google.

Middle East. AI-related policy developments in the Middle East include:

• Saudi Arabia. The Saudi government is doubling-down on AI skills development with a recent partnership announced between the Saudi Data and Artificial Intelligence Authority and Oxford University. The new venture creates an “intensive artificial intelligence application engineering camp aimed at training both Saudi and international graduates in advanced AI technologies.” The Saudi Ministry of Education also launched new AI-related curricula for students and training programs educators.

• UAE. Representatives from the UAE Council for Fatwa previewed plans to utilize AI for issuing fatwa (legal opinions on points of Islamic law). The discussion came amidst a conference held in Cairo on the intersection between religious scholarship and AI.

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. Quebec's government healthcare corporation, Santé Québec, is rolling out a pilot program that uses AI to help doctors transcribe medical notes from patient visits. AI transcription tools remain popular, however they continue to raise concerns with confidentiality and hallucinations.

• Mexico. BBVA Mexico is moving to eliminate touch-tone options in its phone service and replacing it with a generative AI system called Blue. The system is meant to help reduce the time it takes for customers to find the assistance they need.

5. What GPT-5 can tell us about AI Evaluations?

Amidst controversies about misleading graphs and GPT-5’s dramatic change in personalities, Trustible took a deeper look at its System Card and other supporting documentation for the Trustible Model Ratings. As with previous ratings, OpenAI scored poorly on the Data section (covering the topic in only a few vague sentences), but received a High Transparency rating on several Evaluation categories. Their analysis points to a key few good and bad practices for model evaluation:

• Real-World Data: OpenAI used ChatGPT production data to complement existing benchmarks. Testing on realistic data provides a more accurate insight into model functionality. It also reduces the likelihood of test data leakage, as many off-the-shelf benchmarks inadvertently end up in training datasets. This works assuming the evaluation production data was explicitly excluded from the training data - GPT models are trained on ChatGPT data (excluding enterprise accounts and people who opted out).

• Learning from Audiences: The System Card introduces several new types of evaluations: one for sycophancy (given how an update in May resulted in overly sycophantic behavior) and one for measuring conversations specifically in health conversations. These choices reflect adaptations to concerns discovered during real-world use.

• LLM-as-a-Judge: Many of the benchmarks are evaluated using LLM-as-a-Judge. This evaluation method uses a second LLM to check if GPT-5 produced a correct output. The method is more scalable than human review and more flexible than an exact text matching strategy that checks if the model output matches a reference output exactly. However, there are several drawbacks with OpenAI’s approach: they only use o3 as the grader - while best practice recommends using a model from a different family or better yet “LLM-as-a-Jury” aka an ensemble of several models. An additional best practice would be to capture an error range, since LLM judges can also make mistakes.

• Underspecified Methodology: Many key details of the actual evaluation process, especially for benchmarks in the press release and not the system card were not included. Small details about prompt formatting or whether each example was run once or several times (LLMs have a lot of variance and some evaluations will run each input multiple times to get a better sense of performance) can significantly affect the final score on a benchmark. Once “tool use” is introduced, results get even more complicated to recreate. While OpenAI acknowledges the latter issue and intentionally chooses to not compare their models to other developers - it’s still not the most transparent approach. Ultimately, external evaluators are the best source of consistent results because they can use the same exact set-up when testing each model.

Key Takeaway: Improved performance on multiple benchmarks does not directly translate to better user experience - many users were unhappy with the GPT-5 update. For practitioners, it is important to evaluate models on data reflective of your unique task. Our review of GPT-5 showed a mix of good and bad practices that can be applied or avoided when doing your own analyses. You can read the full model rating here.

6. Webinar: Building Guardrails for Enterprise AI - Exploring the Databricks AI Governance Framework and Beyond

On September 3rd at 1 p.m. ET, join experts from Databricks, Schellman, and Trustible for an exploration of the Databricks AI Governance Framework, including its principles, architecture, and role in helping enterprises scale AI responsibly. We’ll examine how it aligns with emerging regulations and standards (like ISO 42001, NIST AI RMF, and the EU AI Act), and discuss practical considerations for implementing governance controls and assurance programs across the AI lifecycle, while creating a feedback loop between governance and compliance teams and AI internal stakeholders that can increase and accelerate safe AI adoption.

This 45-minute session is designed to equip technology, risk, and compliance leaders with insights they can apply to operationalize AI governance in their organizations.

Key Highlights:

• An In-Depth Look at the Databricks AI Governance Framework and Databricks AI Governance tools: Explore its components, objectives, and how it addresses the unique risks of enterprise AI adoption. Focus on how tools like Unity Catalog and MLFlow can help on the highly technical aspects of AI governance.

• Bridging Frameworks to Practice: How organizations can align the Databricks framework, and other standards such as ISO 42001, with other emerging global standards and regulatory obligations.

• Operational and Assurance Considerations: Practical insights into implementing governance controls, testing for compliance, and providing assurance over AI systems.

• Real-World Perspectives: Lessons from industry practitioners, auditors, and governance experts on avoiding common pitfalls and building resilient AI governance programs.

You can register to save your seat here.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In the meantime, in today’s edition (5-6 minute read):

1. What the Trump Administration’s AI Action Plan means for enterprises

2. The Hidden Cost of ‘Almost Right’ AI

3. Global & U.S. Policy Roundup

4. DOGE and Virginia using AI to eliminate regulatory rules

1. What the Trump Administration’s AI Action Plan Means for Enterprises

Recently, the Trump Administration released "Winning the AI Race: America’s AI Action Plan" (AI Action Plan), following a January executive order aimed at enhancing U.S. AI leadership. This plan proposes roughly 90 policy recommendations within three key pillars: AI innovation, infrastructure, and national security with international engagement. Although primarily focused on federal actions, several recommendations could significantly impact private-sector companies involved in AI development, deployment, or utilization.

Notably, three themes emerge regarding enterprise implications:

Firstly, the administration introduces uncertainty by challenging existing regulatory frameworks. Recommendations such as a "shadow" moratorium on state AI regulations by restricting federal funding could disrupt businesses navigating state-level rules. Similarly, proposed FTC reviews of AI-related investigations could further cloud compliance expectations.

Secondly, contradictions arise between some AI priorities and other administrative goals, notably around talent and energy policies. For example, recommendations to streamline energy infrastructure permitting for AI data centers clash with existing sustainability efforts. Similarly, proposed removal of Diversity, Equity, and Inclusion (DEI) references from NIST’s AI Risk Management Framework could complicate internal workforce policies and talent attraction.

Thirdly, federal interest in setting AI standards could result in cascading obligations for government contractors. Updates to federal procurement guidelines emphasizing ideologically unbiased AI models could complicate AI vendor selection, while requirements for critical infrastructure cybersecurity and AI incident response guidance necessitate organizations to adopt stringent federal standards.

The AI Action Plan also highlights opportunities for private-sector influence, such as industry-specific stakeholder engagement aimed at accelerating national AI standards adoption. Recommendations to bolster AI literacy and reskilling via tax incentives further provide tangible benefits to businesses.

Finally, despite significant ambitions, the Plan’s effectiveness relies heavily on congressional action and federal agency capacity, both of which face uncertainties, particularly after recent workforce reductions and potential political shifts in 2026. Nonetheless, the administration’s stance sends clear signals shaping the broader AI regulatory and operational environment, compelling businesses to proactively adapt strategies in anticipation of changing federal priorities.

You can read our full analysis on our blog here.

2. The Hidden Cost of ‘Almost Right’ AI

For all the hype around AI as a productivity tool, a stubborn truth is emerging: using AI doesn’t always save time. In some cases, it might even make things worse.

The risk isn’t just in bad outputs—it’s in over-relying on AI that sounds confident but gets the details wrong. The more trust you place in these systems without sufficient oversight, the more likely you are to spend your time cleaning up after them.

Two recent studies underscore this dynamic. METR, an AI research lab, recently found that experienced engineers working on open source development were often slowed down by AI code suggestions. Even with expert users, “plausible but wrong” outputs led to wasted debugging time and reduced net productivity. Similarly, Stack Overflow’s recent annual developer survey data revealed a hidden productivity tax tied to AI-assisted coding. Developers often spent more time fixing or validating answers than they would have spent solving the problem themselves. Finally, Atlassian’s recent developer survey found that while engineers self-reported development speed up from AI, all the downstream processes, such as code reviews, manual testing, deployment etc, were getting slowed down because they were now overwhelmed with more code changes and bugs than before. This highlights a bit of a ‘pipeline’ bottleneck problem that is caused by uneven AI use along a value chain.

If teams aren't careful, they may spend more time double-checking and redoing AI outputs than if they’d just done the work manually. Worse, over-reliance can create blind spots—leading people to accept wrong answers without realizing it.

Our Take: AI performs best on constrained tasks where outcomes are easy to verify—like short text generation or specific coding challenges. For complex tasks like multi-step coding, the validation cost can outweigh the generation benefit compared to a human baseline.

3. Global & U.S. Policy Roundup

Here is our quick synopsis of the major AI policy developments:

U.S. Federal Government. Beyond releasing the AI Action Plan, the Trump Administration is leaning on Asian countries to develop an AI approach that differs from the EU. The Securities and Exchange Commission also announced an AI task force aimed at improving AI adoption across the agency. In Congress, Senator Mike Rounds (R-UT) introduced a bipartisan bill on setting guardrails for AI usage in financial services.

U.S. States. AI-related policy developments at the state level include:

• California. ADM regulations.The California Privacy Protection Agency voted unanimously to finalize regulations on cybersecurity audits, risk assessments, and automated decisionmaking technology. Rules for automated decisionmaking technology will take effect on January 1, 2027.

• Florida. Governor Ron Destantis (R-FL) indicated that he will unveil legislation related to AI safeguards. DeSantis was a critic of the attempted federal AI moratorium on state AI laws. The expected legislative proposals would continue to put the Governor at odds with the Trump Administration, which generally views AI regulations as unnecessarily burdensome and hampering AI innovation.

• Illinois. Governor JB Pritzker signed the Wellness and Oversight for Psychological Resources Act, which prohibits using AI to provide mental health and therapeutic decision-making. The law would allow licensed behavioral health professionals to use AI tools for administrative purposes and supplementary support services. The new law comes as OpenAI announced new mental health guardrails for ChatGPT.

• Michigan. The Michigan Unemployment Insurance Agency launched a new chatbot to help “deliver quick and accurate responses to questions from workers and employers.” This is the first Michigan state agency to utilize a chatbot on its public facing website.

• Texas. A recent report found that two data centers outside of San Antonio consumed approximately 463 million gallons of water between 2023 and 2024. The water usage was particularly jarring given that Texas residents were under water restrictions during the same period of time due to an ongoing drought. Data centers in Texas are expected to account for almost 7% of total water usage by 2030.

Asia. AI-related policy developments in Asia include:

• China. During the annual World Artificial Intelligence Conference in Shanghai, the Chinese government unveiled its AI Action Plan. Notably, the Chinese AI Action Plan was released 3 days after the Trump Administration released its AI Action Plan. China’s plan emphasizes more participation in international fora to shape AI standards and increase AI adoption.

• Singapore. Microsoft and Digital Industry Singapore announced a new Agentic AI Accelerator program. The announcement comes as more AI companies have set-up shop in Singapore over the past year, in part due to the country's business-friendly climate.

• Thailand. The government’s Department of Special Investigation (DSI) is using AI to substantiate claims of cheating Thailand's 2024 Senate elections. Specifically, the DSI is using AI to analyze “14 terabytes of CCTV footage and other voting data” as part of its investigation.

EU. The next set of EU AI Act obligations took effect on August 2, the most notable provisions being obligations for general purpose AI (GPAI) systems. The new set of obligations kick-in as Google announced it would sign on to the EU’s GPAI Code of Practice, whereas xAI agreed to sign on to only the security and safety chapter.

Middle East. As Middle East countries continue to lead on AI infrastructure, those countries are facing water issues. The United Arab Emirates (UAE) in particular is one of the most water-stressed countries in the world, yet it is estimated that it will use up to 61 billion liters of water annually by 2030.

North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. The Canadian government is committing $1 million to a joint AI safety initiative with the U.K. The announcement comes as part of a broader collaborative effort on AI between Canada and the U.K.

• Mexico. The Mexican government announced that it is developing its own LLM. The government intended to make its LLM deployable to “5 million university students and more than 5 million businesses,” however the country does face some infrastructure constraints. The move to develop a more culturally appropriate LLM has gained traction this year, after a bloc of Latin American countries unveiled plans to develop a Latin America-specific LLM.

  ---

4. DOGE and Virginia using AI to eliminate regulatory rules

Virginia and the U.S. Department of Government Efficiency (DOGE) are both rolling out agentic AI tools to scan regulatory code—flagging outdated, redundant, or conflicting rules for removal. Virginia’s system is already combing through the state’s administrative code. DOGE says its tool will review over 200,000 federal rules, and agencies like HUD and CFPB are already testing it. DOGE’s initiative is supposedly only targeting rules that are no longer required by law, although that determination requires legal expertise to make.

This is one of the first real deployments of AI into the public sector with direct regulatory implications. While these tools are technically “advisory,” reports suggest agencies are treating the AI outputs as strong signals. At HUD, some regulations were flagged as outside statutory authority, even when they weren’t. It’s not clear what evaluation criteria are being used—or who has final say. If AI-generated flags trigger removals or policy shifts, these could qualify as high-risk or decision-making systems under several AI governance frameworks.

Nothing in Governor Youngkin’s announcement details what AI system is being used, or how it may be adapted for the nuances of regulatory text which have specific stylistic elements that could cause issues for AI systems.

Our take: This is likely only the beginning of AI being applied directly to regulatory tasks and informing decisions made by regulators. Many organisations may be looking to see how successful these initiatives are, as analyzing policy documents and suggesting simplification could be hugely valuable for large enterprises with overlapping and conflicting policies.

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In the meantime, in today’s edition (5-6 minute read):

1. Building A Moat: Agentic AI Web Browsers

2. AI Model Fine Tuning’s Ship of Theseus Problem

3. Global & U.S. Policy Roundup

4. Meta Analysis

5. FaaCT Finding: AI Takeaways from ACM FAccT 2025

Building A Moat: Agentic AI Web Browsers

Several prominent AI companies, including Perplexity and OpenAI, have announced the creation of their own web browsers. Perplexity’s browser, Comet, has been in preview recently and the first few positive reviews (TechRadar, Mashable, FastCompany) have some in highlighting how integrating AI directly with a browser’s capabilities can allow for creating agentic workflows, summarizing and highlighting key information in websites, or conducting deep-research type tasks with the advantage of being logged into systems. While AI features directly integrated with web browsers could offer a more seamless experience than separate applications with ChatGPT, it’s also worth considering some of the additional risks, or potential motives for these initiatives. Aside from competing with Google, as Perplexity’s CEO seeks to do, here are our theories on why a web browser could be massively beneficial to big AI companies:

• Bypassing Anti-AI Scraping - Many websites have rolled out tools to try and prevent scraping by AI bots, and Cloudflare recently made this a lot easier. However, anti-scraping systems are meant to allow humans to view the content and fully render it on your browser. If the browser then had the rendered html/text, it can capture that and share copies of it for training, therefore bypassing the anti-scraping bots and evading issues around robots.txt. This can give a huge competitive advantage to the browser creator, while creating massive privacy risks.

• Building Popularity Metrics - Search Engine Optimization (SEO) has long been a major focus for marketing teams everywhere, as search result rankings can make or break entire companies. However now orgs are thinking about the idea of ‘Answer Engine Optimization’, ensuring that your results come recommended by AI. The era of ‘back-links’ being prominent may be over, but there is still a lot of value in finding the websites others find the most useful or popular. Browsers that collect and track the top websites and other activity could deliver this information at scale, improving the quality of AI responses.

• Collecting Human-generated Content - As the internet becomes flooded with more AI generated content, it will become increasingly difficult to find ‘human written’ content to fuel additional model quality growth. This is partly because AI models trained on content they themselves generate seem to suffer from ‘model collapse’. However, if the browser can track what a user writes, and whether they used AI, or did it from scratch, they could identify and capture the human generated content from AI generated ones, and then further train on that content.

Disclosure: These are potential ways a browser could be exploited, and hypotheses about risks. Trustible has not reviewed the systems in question, nor their terms of service, to know whether these practices would be allowed.

Our Take: AI web browsers could become a powerful tool for personal productivity and reduce some of the friction and clunkiness of using a separate platform. However, a browser could also be the perfect tool for solving a number of current and emerging challenges faced by major AI companies, and could heavily expose people to massive privacy risks. The winner of the AI browser wars could emerge with an insurmountable advantage in their ability to collect and evaluate content.

AI Round Up ‘Round the World

Here is our quick synopsis of the major AI policy developments:

• U.S. Federal Government. The Trump Administration is expected to release its AI Action Plan, which was mandated under the President’s AI Executive Order, alongside 3 new AI-related Executive Orders. The Trump Administration is also continuing to invest in AI infrastructure, recently announcing that Georgia Tech would receive $20 million to build a new supercomputer that will use AI for scientific research. Meanwhile, a bipartisan bill was introduced in the Senate to protect certain types of data from being used to train AI models.

• U.S. States. AI-related policy developments at the state level include:

• California. The California Judicial Council adopted a rule requiring courts to develop policies for the use of generative AI by judges and court employees. The new policies must be in place by September 1, 2025.

• New York. Mayor Eric Adams emphasized that he would leverage more AI technologies to improve city services if he were re-elected to a second term.

• Pennsylvania. Tech and energy companies plan on investing over $90 billion in Pennsylvania as part of a broader effort to turn the state into an AI hub. The investments are primarily aimed at securing new energy sources for AI infrastructure.

• Argentina. Netflix recently announced that it used generative AI in its first TV show, an Argentinian sci-fi series called El Eternauta (The Eternaut). Using AI remains a contentious issue within the entertainment industry due to concerns over job cuts.

• Asia. AI-related policy developments in Asia include:

• China. Beijing hosted China International Supply Chain Expo, which featured over 650 companies from 60 countries. Nvidia CEO, Jensen Huang, praised China during the expo as a “catalyst for global progress” because of its open source AI. The comments came one day after Nvidia announced it would resume sales of its H20 chips to China. While the U.S. has generally been concerned about chip exports to China, the Trump Administration was less worried because the H20 chips are Nvidia’s “fourth best” AI chips.

• Kazakhstan. In a bid to boost its AI industry, Kazakhstan launched central Asia’s most powerful supercomputer. While the current government has expressed an interest in bolstering AI investments, the inability to retain appropriate talent may hinder the country’s ambitions.

• EU. Most of the major providers are agreeing to sign the EU’s Code of Practice for general purpose AI models, with Microsoft likely to sign and Anthropic announcing it would sign. Meta continues to be outlier as the only model provider to publicly state it will not sign the Code of Practice.

• Middle East. The U.S. launched an initiative with Israel to build a strategic tech hub that focuses on AI and quantum development in an effort to counter influences from China, Iran, and Russia. The partnership will likely expand to include other Gulf and Central Asian nations. The announcement comes as a billion dollar AI chips deal was placed on hold between Nvidia and the UAE over concerns that the technology may end up in China.

• North America. AI-related policy developments in outside of the U.S. in North America include:

• Canada. Canadian AI company, Cohere, is reportedly lobbying Canadian government officials in an effort to have Canada influence AI policy for the G7, as it currently holds the bloc’s presidency.

• Mexico. Voice actors in Mexico are demanding that the Mexican government enact regulations that would prohibit voice cloning without consent.

• Industry. Delta announced that it would increase the volume of ticket prices that are influenced by AI from about 3% to 20%. Lawmakers and privacy groups have voiced concerns over the initiative, characterizing it as invasive and predatory.

AI Model Fine Tuning’s Ship of Theseus Problem

On July 18, 2025, the European Commission released guidelines to help clarify obligations for general purpose AI model providers and complement the recently published Code of Practice. While the focus has primarily been on obligations for frontier model providers, there are concerns about how model fine-tuning may pass those obligations from the model providers to the modifying organization.

Think of this from the lens of the Ship of Theseus thought experiment: for a wooden ship to have each of its planks and nails replaced piece by piece over time, is the whole of the entirely replaced ship still the same ship? The European Commission’s guidelines raise a parallel real-world conundrum for AI model providers and fine tuners.

The latest Commission guidance clarifies that when a downstream organization is considered to be the model provider when the training compute used to modify the model is “greater than a third of the training compute of the original model.” This threshold was chosen, instead of a fixed number of FLOPs, because the amount of compute needed to significantly modify a model is relative to its size. In contrast to previous texts, this document provides some specific methods for calculating compute and provides placeholder values when compute for the original model is not disclosed (as is the case for a majority of systems).

Many common applications are not likely to reach the specified thresholds. For instance, LLama-4 Maverick would need roughly 5.5 trillion words of fine-tuning data to meet the modification threshold. In contrast, common guidance recommends fine-tuning with a much smaller data set, typically suggesting starting with several thousand examples. Moreover, organizations who provide applications that require substantial modification may have a more difficult time understanding how they may reach these thresholds. The Act’s annex emphasizes that almost all forms of compute used should be tallied with some enumerated exceptions. However, not all compute can be easily ascertained. For example, reinforcement learning processes are often used to instill “helpful” and “harmless” behavior into models that use larger amounts of compute but do not have a direct formula to estimate it.

Our Take: Many organizations may not need to be concerned with becoming general purpose model providers because they modify an existing GPAI system. However, the Act misses the mark on how smaller changes could substantially impact the model, especially when some techniques like parameter efficient fine-tuning can modify models with a much smaller amount of compute. The Act’s threshold may ultimately punish organizations that fine-tune at scale without making significant changes to the model as compared to those organizations that make smaller modifications that significantly affect alignment and alter model behavior.

Meta Analysis

Meta has dominated several AI news cycles over the past few weeks with major headlines of $100+ million signing bonuses for top AI researchers, and a purchase/investment in Scale AI that made Scale’s CEO the new head of Meta’s AI division. However, hidden amongst some of these market moves, there have also been two core stories that may impact organizations using Meta’s AI technologies. It’s worth looking into these key events, and understanding what it means for technology and policy professionals.

• Meta refuses to sign EU AI Act Code of Practice - As of the time of writing, Meta is the only major AI lab to confirm that they won’t sign the newly released EU AI Act General Purpose AI Code of Practice. Competitors Anthropic, OpenAI, Mistral have confirmed signatures, and others like Microsoft are considered likely to do so. The Code outlines copyright, transparency, and safety testing protocols for frontier models, and is technically voluntary until full AI Act enforcement begins in August of 2026.

• Key Takeaway: Meta has had repeated frustrations with the EU over digital regulations, and could simply withdraw or prohibit their AI systems in the EU altogether. However, they have the world’s top leading open-weight models, which makes avoiding the EU, or enforcing bans, particularly hard. Ironically, when Apple previously withheld their AI systems from the EU over regulatory concerns, they still landed in regulatory issues as it was considered an unfair use of market power under the DMA.

• Meta considers going closed source - Meta’s new AI head, Alexandr Wang, is reportedly considering a major strategy pivot away from the open-weight Llama models, into closed proprietary models. Releasing open source (or open weight) models has been a major issue in AI policy discussions, as it can both enable innovation, but also cannot ever be ‘reigned in’ or controlled once released.

• Key Takeaway: While a closed model could allow for some more flexibility to compete for Meta, there may be National Security aspects to consider. Most Chinese LLMs are also open weight, including Alibaba’s Qwen 3, and DeepSeek’s R1. While Llama is seen as less capable than these for now, halting its development would remove the leading US-centric open weight model which is still preferred by people for a range of applications. Chinese open models must meet testing requirements by Chinese authorities and align with CCP viewpoints making them unsuitable for a wide range of purposes in the US. We think it’s unlikely that Meta will pull back on their Llama models, especially if they’re going to want the Trump Administration’s help pushing back on heavy EU digital regulations, and avoiding current anti-trust scrutiny.

Our Take: Meta clearly has big ambitions to become a foundational layer amongst the likes of OpenAI and Anthropic, and their investments in talent and M&A makes that clearer than ever. Meta is distinct amongst the landscape as the best performing Western open source LLM on the market, something until recently Meta has highlighted as their stand out differentiator. Many organizations have indicated they want to build on top of Llama directly, or fine-tune it to create their own AI intellectual property, and to host and run entirely themselves on-prem, offering higher privacy and cyber protections. It’s also one of the most common models used in academia for research, as smaller versions of Llama can actually run on your average desktop computer without specialized AI hardware. But Meta’s recent actions have created uncertainty and unease on the apparent pivot, which may actually harm them in the short and long term. It’s unclear whether a closed-ecosystem system will be compliant in the EU, whether building on Llama will be further developed, or whether Meta’s systems will benefit from the improved security and reliability that accompany being a heavily-researched open system.

FAccT Finding: AI Takeaways from ACM FAccT 2025

In June, while policy makers debated the finer points of the US AI Moratorium and the EU AI Act Code of Practice, researchers from around the world (including our Director of Machine Learning, Anastassia Kornilova) gathered at the ACM FAccT Conference to discuss the finer points of building responsible AI systems. What we learned reinforced the importance of involving affected populations in AI system development and testing, and the breadth and depth of testing necessary to build safer AI systems. At the same time, we heard that gaps persist between the richness of this research and how policy is crafted both at an organizational and government levels.

You can read more about our takeaways on our blog.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In today’s edition (5-6 minute read):

1. Trustible operationalizing Databricks AI Governance Framework

2. Why the AI moratorium would have backfire

3. AI policy & regulatory roundup

4. Why AI slop matters

5. The great AI copyright conundrum

1. Trustible Announces Databricks AI Governance Framework Implementation

Last week, our partner Databricks introduced their AI Governance Framework (DAGF v1.0), a structured and practical approach to governing AI adoption across the enterprise. The DAGF acknowledges what many organizations are already discovering: AI governance is not simply a technical exercise. It’s about aligning people, processes, policies, and platforms to ensure that AI systems are trustworthy, compliant, and scalable.

The Databricks AI Governance Framework marks a pivotal step in helping organizations balance innovation with responsible deployments. But success depends on operationalizing them effectively across people, processes, and technology.

Yesterday, we announced that Trustible is proud to serve as the official Technology Implementation Partner of the Databricks AI Governance Framework and a key contributor alongside leading organizations such as Capital One, Meta, Netflix, Grammarly, and others. DAGF offers a practical, flexible framework designed to help enterprises embed AI governance into day-to-day operations, regardless of where they are in their AI maturity journey. You can read more about how we’ve interpreted the DAGF in the Trustible platform in our whitepaper here.

Key Takeaway: Starting this week, Trustible customers will be able to align their AI governance efforts directly to the framework through a dedicated DAGF module within the Trustible platform and help embed AI governance into the fabric of your AI strategy so you can build, deploy, procure, and scale with confidence. This is the first of many partnerships to come with AI deployers, infrastructure providers, and ecosystem partners to ensure enterprises of all size and shapes have access to ready-to-deploy governance solutions that adapt as quickly as the market.

2. Why the AI Moratorium would have backfired

While we now know the ultimate fate of the proposed State AI Legislative Moratorium that was included in the ‘One Big Beautiful Bill’ budget (it was removed by the Senate in a 99-1 vote), the idea is likely to stick around, and similar proposals may appear in the future. We supported its removal for a variety of reasons, but our biggest argument against it was that it would have backfired. Specifically, we think banning AI regulations in the absence of any federal clarity would have hurt the very startups and innovative environment that its proponents were trying to protect. Here’s a brief outline of why:

• Trust - The majority of Americans don’t trust AI. Even the perception that AI will become less regulated, would hurt that trust even more. Trust in AI systems is directly proportional to revenue for AI companies which can fuel further innovation.

• Level Playing Field - Big companies and Big Tech have dozens of lawyers, machine learning experts, and marketing teams that are able to deal with the uncertainty of the current environment. Smaller companies and startups don’t. A clear set of AI standards can help a startup be on even footing as Big Tech from a compliance perspective.

• Legal Uncertainty - Let’s be honest, with so much opposition from State Governors and Attorney Generals, this Moratorium would be challenged almost instantly, and would likely take years to work through the federal circuits. During that time some State laws in Colorado and Texas would be in a State of uncertainty which businesses and investors hate.

• A Solution Before A Problem - The overwhelming majority of the ‘1000+’ AI bills at the State level don’t actually regulate AI directly (most simply mention the term ‘Artificial Intelligence), and many have common sense overlapping requirements around issues like AI disclosure, and protecting against Deep Fakes. Perhaps ironically, Big Tech’s State lobbying efforts have already created a fairly lightweight and consistent State regulatory environment.

• Reduced Information Sharing - At the moment, there is virtually no information sharing within the AI ecosystem for AI vulnerabilities and incidents. That is primarily because of legal liability issues where companies do not want to admit on paper about any incident. This can be done with a regulator or standards body to get around competition issues. This lack of information sharing hurts everyone because it means we’re not learning from the mistakes of others and innovating as fast.

For a bigger deep-dive analysis, we have a more in-depth post here.

Our Take: The idea that all regulation is bad for business is a little simplistic, and can often just be a form of regulatory capture. A balanced federal framework would be the best approach for everyone involved, but pre-empting things before that framework has even been discussed would hurt the ability of all but the biggest AI companies.

3. Policy Round-Up

Here is our quick synopsis of the major AI policy developments:

• U.S. Federal Government. The failed federal AI moratorium (see our write-up for more details) has some speculating that it may renew a push for federal legislation that explicitly preempts state laws, though the contours of potential legislation remain murky.

• U.S. States. AI-related policy developments at the state level include:

  • California. The California Civil Rights Council approved a final regulation that clarifies how existing discrimination laws apply to AI tools for employment decisions. The new rules will take effect on October 1, 2025.

  • New York. Governor Kathy Hochul announced the construction of a new nuclear power plant in upstate New York to fulfill new energy demands, which is caused in part by growing AI usage. The announcement comes amidst a push from the Trump Administration and big tech companies to power AI computing infrastructure with nuclear energy.

• Canada. It appears Canada’s new government will not revive the Artificial Intelligence and Data Act, which Parliament terminated ahead of the federal elections in April 2025. Canadian lawmakers are considering which aspects of the former legislation they may want to pursue, such as addressing issues with copyright and AI. The move aligns with a broader global movement away from AI regulation and towards AI innovation.

• South America. AI-related policy developments in South America include:

  • Chile. Chilean lawmakers are facing opposition to their proposed comprehensive AI law. The proposed bill is an EU-inspired framework, which critics claim will harm technological investments and development if enacted.

  • Brazil. Leaders of the BRICS countries (Brazil, Russia, India, China, and South Africa) are expected to release a statement that calls for data protections against unauthorized AI use. The push will come as part of a two-day summit among BRICS leaders in Rio de Janeiro. BRICS serves as a diplomatic forum for developing countries and has recently been accused by President Trump of promoting "anti-American policies."

• EU. The next set of EU AI implementation deadlines is approaching on August 2 and the European Commission squashed rumors that it may pause enforcement obligations for certain EU AI Act provisions. Tech companies have been working behind the scenes to delay the Act’s enforcement timelines, with the obligations for general purpose AI models being top of mind (which kick-in on August 2). Tech companies argued that delays with the voluntary Codes of Practice warranted the postponement, which may not be released until the end of 2025.

• Industry. Microsoft announced that it would layoff approximately 4% of its workforce as it seeks to make heavier investments in developing its own AI. The move comes as big tech remains locked in an AI arms race, which has seen companies like Meta offer $100 million signing bonuses for top AI talent.

4. Why AI Slop Matters

With recent improvements in the quality and cost-effectiveness of AI generated content, it seems impossible to escape ‘AI Slop’ - low quality generated content used mainly for driving online engagement. We see it on our social media feeds, in the content we read, and even increasingly in our professional work. The public is becoming increasingly aware of it as well, with recent news stories covering its impact on events like the most recent season of Squid Games, or even the Sean Combs trial. The topic was even covered in-depth by John Oliver in a recent ‘Last Week Tonight’ episode. How big of an issue is ‘AI Slop’ however? Is it simply the new ‘Spam’, something to be ignored that will eventually fade into the background, or are there major governance implications for it? Here’s a brief overview of why ‘AI Slop’ may be relevant to organizations using AI.

Wasteful Spend - Even before the current ‘Age of AI’, there was a conspiracy theory called the ‘Dead Internet Theory’ that postulated that the majority of online content and interactions were driven by automated bots. The biggest challenge with this is that many organizations spend massive amounts of money on ads for ‘engagement’, or derive market insights from it. Big Tech platforms unfortunately have an incentive to ‘boost’ engagement artificially, even though it will yield poor ROI for the advertisers.

Degraded Reputation - Many organizations differentiate themselves based on the quality of the services they provide. Consider every top tier law firm, editorial publication, or consulting business that charges high rates for access to top thinkers. The problem: What if those ‘top thinkers’ are using the same AI as everyone else? The temptation to use AI systems may win-out, even as studies show that the diversity of AI content is actually quite low, and that overuse can degrade our own cognitive abilities over time. It will be a constant fight for organizations that seek to differentiate based on quality to avoid their reputation degrading as a result of too much slop.

Key Takeaway: While AI generated content is quick and easy to create, there may be a persistent bias against such content, and the internet will likely become overwhelmed for it. This could create a market for authentic human-generated content, but then maintaining that quality and output could become difficult to maintain. For enterprises that rely on AI to generate content, especially as part of their marketing strategy, expect that lack of authenticity ultimately to reduce the effectiveness of their strategies. It’s an important reminder that while AI is a transformative technology, it’s also a tool, not to replace human content generation.

5. The Great AI Copyright Conundrum

Source: New York Times

Over the span of three days, two major cases on AI and copyright law were released. On June 23, 2025, a judge ruled in favor of Anthropic after a group of authors sued Anthropic for copyright infringement by alleging that the company trained Claude on their protected works without their permission. The judge found that, while Anthropic may have broken the law when it trained Claude on millions of pirated books, the books that were legally purchased for model training purposes did not violate copyright law. The judge reasoned training the model on the books was a fair use because Claude’s outputs generated new text that was "quintessentially transformative” from the original material.

Two days later, a judge found in favor of Meta after a separate group of authors sued the company for using their copyrighted books to train Llama. The judge found that Llama’s outputs did not cause sufficient market harm to the authors because Llama was not able to generate “any meaningful portion” of the authors’ books that would threaten the books’ market value. Moreover, the authors did not present meaningful evidence that Meta’s use of their books diluted their value. The judge seemingly left open the door to further litigation on this issue by noting that the ruling only impacted “the rights of [the] 13 authors—not the countless others whose works Meta used to train its models … this ruling does not stand for the proposition that Meta’s use of copyrighted materials to train its language models is lawful.”

The big tech companies are heralding this as a win, however it does not solve the broader issues related to how AI model providers are using protected works to train their models. Companies should also take note that this does not alleviate them from infringing on someone’s intellectual property (IP) rights. Not all models are created equal and it is important to understand whether the underlying model(s) for a company’s AI products or services have guardrails in place to avoid violating IP laws. Companies should understand how a model handles IP in their training data. For instance, Trustible Model Ratings identifies when a model has policies around IP in their training data. Moreover, groups like Creative Commons are working on frameworks that help balance the IP rights with the need to have high quality datasets available to train AI systems.

Our Take: Big tech won the battle but is far from winning the war when it comes to how AI uses IP. Policymakers are thinking about how best to balance IP law with AI innovation, but do not expect an answer in the foreseeable future. In the meantime, companies should be taking steps to ensure that training data is properly licensed when appropriate, selecting models that have IP safeguards in place, and reviewing outputs to avoid using potentially infringing materials.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team

In today’s edition (5-6 minute read):

1. The Future of Jobs in AI Age

2. How to Respond to AI Risks

3. The plague of test data contamination (AI Cheating)

4. The Growing Problem of Over-Relying on AI

5. AI Policy & Regulatory Roundup

1. The Future of Jobs in the AI Age

The jury is still out on how much AI will impact the workforce. Plenty of leaders, including those at AWS and JPMorgan, have started laying the groundwork for shrinking white-collar staff as a result of AI. Some groups argue that this is yet another ‘fear’ tactic aimed at trying to encourage teams to learn AI skills, drive down salary costs, and increase productivity. Policymakers have also been acutely aware of potential negative employment impacts and have sought to offer some pushback, such as requiring employers to disclose when AI contributes to layoffs. There are some leaders that argue AI will not initially replace workers, but rather a human who is highly skilled in using AI tools, and thus more productive, will do so. There are even some who argue that economists are underestimating the impact AI may have on jobs and the workforce, and the pace at which it could happen. New college graduates seem to be the first impacted, as AI cannot yet replace expertise and experience, but can be better at some tasks than entry level knowledge workers. This of course begs the question of how we will keep training up the experts of the future without these early experiences, especially if using AI itself may reduce critical thinking.

However it’s not all doom and gloom. The counter-argument on these impacts is that humans will adapt their skills and preferences in the AI era, and new jobs will also emerge as a result. A recent New York Times Magazine article dug deep into 22 new roles that will be created because of AI. The article breaks down the 22 roles into the categories of Trust (those involved in overseeing, governing, and auditing AI systems), Integration (people doing the ‘last mile’ work of connecting data and systems to AI), and Taste (creative roles that are specifically aimed to finding that that AI is not capable of doing). It’s unclear how many roles in this category there will be, and many do still require a high skill level, and ironically, a deep understanding of how AI systems work, and their limitations.

Key Takeaway: A big question for corporate leaders, and policy makers, is whether these impacts will be fairly sudden, or gradual over time, and how fast people will be able to retrain (ironically using AI) towards new economic opportunities, or whether ‘AI replacement’ will become the next big political issue much like ‘globalized of production’ has become recently in the US.

2. How to respond to AI risks

AI risk management is a key (perhaps the key) part of an AI governance strategy. Identifying and managing risks can be an incredibly difficult task for many enterprises given the fast pace of change in AI systems and the evolving landscape of AI regulations (something help with at Trustible). Moreover, you may not even be able to control every risk yourself since so many use cases rely on leveraging third-party platforms.

There are four types of Enterprise Risk Management techniques that every AI governance professional should know about.

• Avoid – Sometimes the safest move is to say “no.” If an AI system can’t meet your privacy or cybersecurity standards, don’t use it. However, saying “no” of course may not always be an option - let’s face it, AI is incorporated into more and more SaaS products each and every day, and IT and risk professionals are becoming more aware of the “Shadow AI” problems swiftly emerging across enterprise tech stacks.

• Mitigate – Many risks can be reduced. Steps like removing personal data, checking for bias, adding human-in-the-loop review, and monitoring the system can bring risk down to an acceptable level. But not all mitigations are technical in nature; human mitigation, such as AI literacy training, is an effective form of mitigation that forms a key part of a comprehensive mitigation approach.

• Accept – After careful review, some leftover risk may be worth taking. Conduct the right risk assessment, document the decision, get the right approvals, and move forward with implementation.

• Transfer – When risk remains high, shift it elsewhere. In the AI value chain, risk is transferred downstream to groups building a model, those hosting the infrastructure supporting that model, the developers potentially customizing the model, and the end users consuming outputs. From a third-party risk management perspective, contracts that require vendors to cover certain losses or specialized insurance policies can carry part of the load and shield some risk.

Our take: Too many organizations simply avoid the risks altogether. This slows down or even halts AI adoption inside of their organizations. These four categories of risk response strategies can help you gain confidence in your organization’s ability to manage the risk and move forward with your AI innovation strategy.

3. The Plague of Test Data Contamination

While students have increasingly been accused of cheating using LLMs, models are doing the same through test data contamination. A recent study shows that models that excel at SWE-Bench, a popular software engineering benchmark, may be doing so through memorization, not a genuine understanding of coding tasks. The memorization occurs due to test data contamination, wherein the answers to the benchmark question are present in the training data. This invalidates the results, because the goal of the benchmarks is to measure how models perform to unseen data. The phenomena is not new - the MMLU benchmark has largely been deprecated due to major contamination issues. Unlike traditional machine learning, where training and test datasets consist of datapoints that can be easily enumerated, LLMs are trained on enormous datasets that are harder to filter. If benchmark data is public, it may inadvertently be included in a web scrape (especially if a copy of it is published on an unofficial website).

Model providers use benchmarks to demonstrate on increasingly harder tasks; however, contamination issues put into question the construct validity of these assessments. Novel tasks, like those in Apple’s recent reasoning paper, raise a question about the actual abilities of LLMs. Several recent benchmarks have attempted to protect against contamination, including LiveCodeBenchPro, which is regularly updated with novel coding questions, and MLCommon’s AILuminate, a closed benchmark run by the creator against the models’ APIs. For such benchmarks, it is crucial that the creators are impartial parties (unlike FrontierMath that was partially funded by OpenAI), and that each model is only evaluated once (to avoid gamifying the results). In addition to reducing contamination, such benchmarks can improve reproducibility, as LLM performance is sensitive to prompt formatting and parameters. Closed benchmarks have limitations, too, like reduced visibility into potential biases and bugs in their data.

Key Takeaway: Many popular LLM benchmarks are likely overestimating model performance due to test data contamination. Some groups are starting to implement semi-private benchmarks that can combat this concern, but this industry still has room for improvement, as a broader set of better benchmark criteria is being developed.

4. The Growing Problem of Over-Relying on AI

When we generally think about AI risks, the primary focus tends to be on bias or discrimination in the AI systems. However, these risks are realized and cause the most impact when applied in high-risk use cases (e.g., creditworthiness or employment-related decisions). Yet, the vast majority of use cases are more likely than not lower risk uses. When we think about risks through the dimensions of likelihood and severity, we may overlook one of the more likely risks that arise within lower risk use cases that has an evolving severity level: overreliance on AI.

Trustible’s risk taxonomy explains that overreliance on AI “occurs when users start to accept incorrect AI system outputs due to excessive trust in the system.” AI’s widespread use, especially generative AI systems, exacerbates the likelihood that this risk will occur. For instance, attorneys are increasingly relying on AI to help write legal filings but are falling prey to the AI’s confidence in legal citations. While the system’s hallucinations are the primary risk, it is the attorney’s confidence in content that manifests the harm. The education system is also seeing major upheavals as more students rely on generative to complete assignments, which has led to an increase in cheating or plagiarized assignments. Higher education professors have also relied on generative AI for their lessons, which raised questions over the value of college classes.

The severity of harm from overreliance on AI is the other dimension to consider, though one that is still coming into focus. A recent MIT study indicates that overreliance on AI systems has negative cognitive impacts, such as with memory or executive function. This may have an outsized impact on students who use generative AI to complete school work, but can also have negative consequences for adults who over-rely on the technology to accomplish tasks that require a certain level of cognitive ability.

Our Take: Overreliance on AI is a far-reaching risk but one that is preventable. Implementing AI literacy programs can help users understand how to leverage the technology without blindly accepting its outputs and excluding personal expertise or thought. Moreover, having an explanation accompany system outputs can help users easily identify source material to determine its validity and applicability.

5. AI Policy & Regulatory Roundup

Here is our quick synopsis of the major AI policy developments:

• U.S. Federal Government. The federal AI moratorium took a big step towards becoming law when the Senate Parliamentarian determined it did not violate Senate procedure and could remain in the Republican’s reconciliation bill. The moratorium survived after Senate Republicans amended the language to condition states’ access to certain broadband funds on their temporarily pausing AI laws. The language could still be stricken from the bill via floor amendment. It also faces opposition by House Freedom Caucus members.

• U.S. States. AI-related policy developments at the state level include:

  • New York. The state legislature passed the Responsible AI Safety and Education (RAISE) Act, which is a modified version of California's SB 1047. Notable differences from SB 1047 include not requiring a “kill switch” for frontier models and caps on penalties. Legislators have not yet sent the bill to Governor Kathy Hochul, but can do so at any time in 2025. Once it arrives on the Governor’s desk, she will have 30 days to act on it.

  • Texas. On June 22, 2025, Governor Greg Abbott signed the Texas Responsible AI Governance Act into law. It will take effect on January 1, 2026.

• European Union. Sweden’s Prime Minister, Ulf Kristersson, is the latest (and first EU government leader) to support pausing the EU AI Act. His comments come as the EU considers a digital omnibus package that aims to simplify regulations and could include amendments to the AI Act.

• Latin America. A group of 12 Latin American countries, led by Chile, are working to develop an LLM focused on the cultural and linguistic diversity of the region. Latam-GPT is an open source project that is intended to help boost AI accessibility across Latin American countries.

• United Nations (UN). The United Nations Working Group on Business and Human Rights released a report warning that AI systems must be developed to align with the UN Guiding Principles on Business and Human Rights. The report was presented as part of the 59th session of the Human Rights Council.

—

As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.

AI Responsibly,

- Trustible Team