The danger of unchecked automation 

The pace of AI automation is currently outpacing our ability to ensure consistent quality control. Recent developments across global enterprises have demonstrated that full reliance on automated oversight can introduce vulnerabilities. Ford recently executed a multi-billion-dollar course correction by rehiring 350 veteran engineers after automated quality checks missed several critical design flaws.

 

The case was similar in 2024, when Amazon announced that it was stepping back from its fully automated grocery technology, which relied on computer vision systems to speed up the checkout process, but incorporated human reviewers to analyse sales. These shifts are illustrations of a broader challenge facing corporate infrastructure. 

 

While AI is a powerful force multiplier for processing data at speed, it often lacks the contextual reasoning required for safety assurance or complex risk management. Processing quickly, but entirely incorrectly, is the same as not processing at all. 

 

 

The tacit knowledge gap

The limitation of any automated system rests on the data used to train it. As seen in Ford’s case, the pursuit of automation to reduce operational costs created a knowledge vacuum. This transition often discards the institutional memory carried by experienced personnel before that expertise can be structured or codified into AI training datasets.

 

Without a strong knowledge base to guide them, organisations relying solely on AI risk amplifying weak inputs. Because these systems are unable to identify subtle deviations from normal operations, they are prone to hallucinating or producing counterproductive results. In manufacturing environments, this issue could result in physical design oversights that escape detection. In a broader corporate framework, the same flaw could lead to corrupted business analytics, broken supply chain calculations, and erratic automated workflows that could paralyse business operations.

 

 

Expanding security vulnerabilities

The consequences of unmonitored automation also extend into enterprise security. Integrating autonomous agents and large language models (LLMs) into AI workflows expands an organisation’s digital attack surface. Traditional infrastructure has already proven too rigid to track agile software agents as they move across multiple departments.

 

According to a recent survey, AI agents, agentic infrastructure, and Gen AI applications were noted as one of the biggest risks to organisations as attackers can exploit these new attack surfaces via API keys, tokens, or authentication credentials to gain privileged access to corporate databases. 

 

Attackers exploited this exact vulnerability in August 2025, when an AI chat integration was compromised by actors who breached Salesforce environments across more than 700 organisations, stealing OAuth tokens and making malicious queries indistinguishable from legitimate AI activity. 

 

 

A ‘human touch’ to AI

To counter these threats, organisations must pair automated data processing with human oversight. Technology handles the large-scale data processing, but experienced professionals supply the critical context. 

 

Organisations can achieve this control through three specific actions. Security teams must first establish definitive baselines for normal token usage and API traffic to ensure that any anomalous behaviour is isolated immediately. Next, governance frameworks must mandate peer reviews and human validation checkpoints for any automated decisions that carry significant financial or operational risk. Finally, cybersecurity departments and operational teams must work together to audit logging for internal AI agents, vet unapproved AI domains receiving traffic and monitor third-party AI plugins and browser extensions before threat actors have a chance to exploit the gaps. 

 

Organisational resilience cannot be achieved through prepackaged software or raw computing power alone. As Ford and Amazon discovered, total reliance on automated systems invariably sidelines the human intuition required to spot critical failures.

 

True resilience requires architectural visibility, clear governance, and a deliberate balance between technological speed and human judgment. The goal should not be to replace the veteran operator’s intuition, but to offload the cognitive noise so they have the clarity to deploy it. 

 

---

 

Jamie Moles is Senior Technical Manager at ExtraHop 

 

Main image courtesy of iStockPhoto.com and Dragon Claws