1. Should AI Welfare Be In Your AI Risk Register?
By: Andrew Gamino-Cheong
Anthropic recently made headlines with a notable change to their acceptable usage policy for their platform, including a prohibition on sustained, intentional, cruelty or abuse to their AI systems. While the actual restrictions are much narrower than the media hype may suggest it comes shortly after a notable New York Times article about one of Anthropic’s co-founders heavily lobbying religious leaders to accept the idea of AI consciousness. A debate now sprung up about the new policy with 2 potential reasons behind the change.
For those supporting model consciousness, the anti-abuse policy is precautionary. If AI is at least conscious, there’s an argument for some degree of rights it should have, even if not equal to human rights. If there’s a chance the systems are in fact super intelligent, or even sentient, some of the ethical questions expand. The AI Safety world takes this issue even further with the AI thought experiment Roko’s Basilisk, which argues that a future super intelligent AI will take revenge on those that impeded its development, or were cruel to it. In this context, preventing abuse is trying to hedge against this risk.
On the more pragmatic side, deterring abuse may simply be trying to protect against training data poisoning, and reducing potential spill over of abuse onto humans at scale. Most AI companies collect chat data and use it to further fine-tune their systems. Most early LLM already had an unfortunate issue where if you got aggressive with them, you could bypass guardrails, getting the system to reply to objectionable/toxic requests. Future models trained on these conversations could exacerbate the situation, as it may reward the abusive behavior that was seen as successful. This could spiral and later encourage users to be more aggressive with AI systems to achieve their goals. If users are used to this kind of aggressive behaviour, it could spill over into instances where they’re interacting with a human instead of a machine, such as when the human support agent enters the chat (literally).
Key Takeaway: Regardless of Anthropic’s motives, many businesses now need to reflect on whether they need to consider ‘AI welfare/rights’ as a risk themselves. At the moment, there’s little public data to back up model consciousness claims, and no obvious legal requirement to do so, so it’s probably fine for the moment to omit AI welfare considerations from risk assessment, except in the context of complying with a vendor’s policy.
2. Whitepaper Announcement: Governance Concerns for AI Agent Capabilities
By: Andrew Gamino-Cheong
An AI agent is an AI-powered system that is able to do things to accomplish its goal. Instead of just suggesting an email to send, it can actually send that email. Agents are able to do these things because of the various ‘capabilities’ they are granted. A capability ranges from the ability to write and execute arbitrary code in a sandbox, to the ability to do tool calling to another platform via MCP, to the ability to store its own information as memory across its lifecycle. Many of the risks tied to an agent stem from the set of capabilities it is given, and every additional capability an agent has can dramatically increase the potential risks of an agent. AI Governance needs to understand the various types of capabilities given to an AI agent, and how they can impact the risks of a system and how to govern each capability appropriately.
In our latest white paper, we outline a taxonomy of agent capabilities, and governance concerns for each of them, to help educate governance teams on how to govern AI agents. The guide will be periodically updated as new capabilities or risks emerge, and the views here shape Trustible’s default risk recommendations for Agents in the platform. Learn more.
3. Trustible Spotlight
By: Sydney Cullen
During IAPP’s AI Governance Global conference in Seattle, we hosted a private dinner for nearly 30 AI governance leaders. We discussed what’s working in AI governance today, how teams are thinking about agents, and, because we have standards, very large slices of cake.
Between the dinner, a closed door roundtable, and an AI assurance workshop, the same concerns kept surfacing. Enterprises are struggling with ways to inventory agents and assess their risk. Scanning for shadow AI came up constantly, usually followed by “and then what?” Finding what’s out there is the easy half, and governing it is where teams get stuck. Several large enterprises are only now outlining deterministic risk assessments based on key attributes of a use case, and the agent questions are already piling up, from subagent governance to how to tell whether an agent is working around its guardrails.
Two other themes were present. Everyone wants policy-to-compliance mapping, but few know how to do it. And there was genuine confusion over how to tell apart two different jobs – adding specific risks to a use case, and scoring how risky the use case is overall. Teams keep treating them as one.
Enterprises also want shared taxonomies of risks, mitigations, and threats, but ones that aren’t overwhelming for a team trying to figure out which apply to their use cases. Some attendees asked for a how-to for building their own, since no standard exists yet.
Our Take: A scan produces an inventory (not governance), so ask any vendor what happens after discovery. Standards bodies should publish guidance on how to build one, instead of handing enterprises another to adopt.
4. Policy Updates
By: Sydney Cullen
Pennsylvania - The Pennsylvania House passed HB 1925 on Sept. 30 by a vote of 144-59, and the bill now heads to the Senate. It would add three parallel chapters covering health care facilities, insurers, and Medicaid and CHIP managed care plans. Each one requires AI disclosure, keeps the final call with clinicians, and calls for a governance structure and an annual compliance statement filed with the state. Most of it will look familiar if you’ve followed California’s AB 3030 and SB 1120 or Maryland’s HB 820, although HB 1925 applies its patient-message rule to all AI, not only generative AI. If it becomes law, it takes effect a year later. Pennsylvania is one of the states whose legislative session is still open, so there’s time for this one to pass this year.
Our Take: This would be stricter than other states because of the governance structure requirement, but we don’t think that changes much. Most health organizations already have some type of governance structure, so this guidance could actually help organizations improve their oversight operations since it shows what the state expects and gives orgs something to check their program against. The rest of the bill is on track with other states, and again, it’s likely things a health org already has in place or are working towards developing.
California - Governor Newsom signed a big batch of AI bills on Sept. 30. These five are the ones we think most organizations will care about the most:
SB 947, the “No Robo Bosses Act,” says employers can’t rely solely on an automated decision system for discipline or termination. A human has to back up the output with other information, and employees get a plain-language notice if a decision mainly relied on the system. It’s operational July 1, 2027.
AB 1883 bars employers from using monitoring tools to infer a worker’s emotional state or collect neural data.
AB 1979 requires health facilities, clinics, physician practices, and health care chatbot providers to take reasonable steps so licensed clinicians keep their independent judgment when AI is involved, and AI can’t independently do work that requires a license.
SB 503 makes developers of clinical decision support tools mitigate known or foreseeable bias and hand deployers documentation, and deployers have to monitor for bias.
AB 2713 updates the California AI Transparency Act so large platforms have to detect provenance data, show users whether content is AI-generated or altered, and not knowingly strip that data.
AB 1883, AB 1979, and SB 503 don’t state an operative date, so California’s default of Jan. 1, 2027 applies.
Our Take: Four of the five put a human or a bias check between an AI output and a decision about a person. Documentation of human oversight is necessary to show compliance, and even feels in line with requirements for high risk systems in the EU AI Act. If you use AI in HR or clinical settings, start listing which tools touch those decisions before January.
EU - The first harmonised standard under the EU AI Act, EN 18286, has been cited in the Official Journal, which makes it officially harmonised. It comes with no restrictions, so following it gives providers a presumption of conformity, mostly for Article 17. The standard lays out what QMS conformity looks like, including how to scope your QMS, how to handle changes to AI systems, and how to run inception, design, and development.
Our Take: Providers finally have a clearer picture of what QMS conformity looks like. At 55 pages it’s manageable on its own. With the rest of the standards still to come, including risk management, cybersecurity, and data quality, taking them all on at once will be a lot. Starting QMS work now lets you stagger the effort instead of scrambling ahead of the December 2027 deadline for stand-alone high-risk systems.
In case you missed it
Colorado. The Attorney General released a revised draft of rules on Oct. 6 to implement SB 26-189, which replaced the 2024 Colorado AI Act, and the state’s chatbot law. The rules get into when an AI output counts as influencing a decision, what adverse-outcome notices must say, and how human review requests work. Comments are open until Oct. 26, and compliance is still due in January 2027, so keep an eye out for final guidance.Compliance is still due in January 2027, so keep an eye out for final guidance.
Australia. Assistant Minister Andrew Charlton laid out a plan for frontier AI standards following the OpenAI incident that would make developers show their safety systems actually work, taking cues from how banking and aviation are regulated. The standards are due by the end of the year, with legislation planned for 2027.
U.S. Congress. Senators Cantwell and Rounds sent a letter to the heads of ODNI and OSTP asking for a coordinated federal response to AI-enabled cyber threats, including NIST standards, independent model testing, and defensive AI.
—
As always, we welcome your feedback on content! Have suggestions? Drop us a line at newsletter@trustible.ai.
AI Responsibly,
- Trustible Team