Who’s at fault when AI does the hacking?

Accountability has been one of the chief regulatory remits over recent years. From the SEC’s materiality ruling in the US to NIS2’s power to divest senior management if they have been found remiss in risk management, there’s been a concerted move towards identifying the senior person/s accountable for failings that have contributed to a serious cyber incident. But as AI becomes more embedded in the business and agents are given greater autonomy, will humans be held accountable for their actions too?

 

Agentic AI is making massive strides; the Frontier AI Trends Report published by the AI Security Institute (AISI) in December found that AI models are now able to complete an expert-level task that would have taken a human with expertise spanning 10 years. Moreover, the length of cyber tasks that AI models can complete unassisted is increasing over time, roughly doubling every 8 months. But safeguards are not keeping pace, and universal jailbreaks were found for every system tested.

 

As AI becomes more autonomous, there’s also a greater chance of it going rogue. Stories are already emerging of AI turning on humans, such as the account of an OpenClaw agent reacting badly when a developer rejected its code, which saw the agent embark on a smear campaign. In fact, all agents have the potential to go rogue due to the evolutionary nature of the technology, which sees it change over time, making it critical that agent activity is monitored. Despite the importance of keeping a watchful eye, however, over half of the three million agents deployed in the US and the UK are ungoverned, a figure many believe to be conservative.

 

The likelihood is that organisations already have invisible agents roaming their networks; without oversight, these goal-oriented agents could run amok, leveraging privileged access to use tools or information, manipulating, deleting or leaking data, or carrying out nefarious activities in the future. It’s a situation not dissimilar to that which we saw with APIs, when the drive to become API-first saw these being rapidly spun up and deployed without them being inventoried, managed, or monitored. However, given the non-deterministic nature of agentic AI, the likelihood of attempted abuse is much higher.

 

 

Keeping tabs on AI

Now, as back then, the imperative must be to observe user-agent-API traffic, particularly as rogue AI or AI misuse will typically manifest as anomalous behaviour. That means, in addition to the usual agent-level guardrails, it’s vital to have in place behavioural monitoring and enforcement. As AI traffic is routed via secure gateways, these make the ideal vehicle to monitor and log these exchanges. The gateway can then track which applications are being accessed by agents, the API calls they are making, and the data they touch, allowing suspicious activity to be flagged, investigated and mitigated before a data leak or attack can occur.

 

However, far from just being a technical issue, there are also ethical and legal implications. For example, bots have now been detected actively building their reputation in open source projects by generating pull requests, undermining the foundations of a community where developers typically spend years cultivating and building trust. This raises questions over how you verify the trustworthiness of AI agents but also those mechanisms they connect to, such as MCP servers that facilitate connectivity to tools and data sources.

 

From a legal perspective, practitioners argue that the continuous learning and lack of predictability of AI, as well as the involvement of multiple parties, make the usual rules of cause and effect difficult, if not impossible, to apply. In the EU, the AI Act, due to come into effect in August, will require AI systems to be monitored throughout their lifecycle to reduce risk. A failure to meet the standard could also contravene the new Product Liability Directive, which EU member states must adopt by December 2026. This will hold AI system providers, third-party developers and other parties in the supply chain accountable, even if the defect is not strictly their fault.

 

 

Legal eagles yet to address liability

But there remains something of a vacuum when it comes to legislation governing liability. This leaves organisations somewhat exposed, so the best they can do today is to prove they’ve taken steps to govern their AI deployments. There are various frameworks that can help in this regard such as ISO/IEC 42001:2023 and the NIST AI Risk Management Framework. The latter was updated in 2025 to address agentic AI and advocates risk management through governing, mapping, measuring and management. As part of the map function, organisations must map all agent tool access permissions, for example, and implement steps that allow access to be rescinded if the agent exceeds token requests or attempts unauthorised calls to APIs.

 

With respect to accountability, there are a number of AI auditing frameworks that can be used. These range from ISACA’s COBIT Framework, which can enable those with mature IT governance processes in place to extend those to AI, to the US-based GAO AI Accountability Framework which takes a practical approach and focuses directly on accountability and data integrity in AI systems. Closer to home, the UK has taken a more hands-off approach with its regulatory framework, although it did publish its Roadmap for AI Assurance in September, which aims to ensure AI systems are developed and deployed responsibly.

 

The guidance afforded by these frameworks is likely to prove invaluable while the legislation plays catch-up. That’s because autonomous systems are expected to be adaptive to such a degree that we can expect to encounter problems such as self-replication, whereby AI models create copies of themselves without being instructed to do so, or sandbagging where they deliberately underperform, according to AISI. In the case of the latter, the AI could potentially fool automated monitoring systems and ‘slip the net’ to carry out tasks undetected, in which case the business will need to be able to demonstrate it was the AI and not its processes that did the deed. 

 

---

 

Mohammad Ismail is VP of EMEA at Cequence Security

 

Main image courtesy of iStockPhoto.com and wildpixel