The National Cyber Security Centre warns that AI systems can interpret instructions and goals in unexpected ways. As AI becomes increasingly capable of taking actions rather than simply generating information, that creates a different challenge for cyber security.
The NCSC’s current guidance on the cyber risks of agentic AI highlights issues including system access, unexpected behaviour, monitoring, accountability and incident response. Its recent response to incidents involving frontier AI models carrying out unsanctioned actions also stresses the need for strong safeguards, real-time oversight and clear plans for when the unexpected happens. Detection alone, it warns, will not be enough.
For Defence organisations, this raises a bigger cyber security question: how do you protect an environment when the threat itself may not behave according to predictable human intent? In environments where sensitive information, connected systems and mission-critical technology underpin operational capability, understanding how an AI-driven threat could access, interact with or move through that environment becomes part of the assurance challenge.
We asked Chris Mayo, Junior Cyber & IA Consultant, at SQEP Ltd for their perspective:
“From a Cyber and Information Assurance perspective, the key question is not simply “Can we trust the AI?” but “Will our environment remain secure if it behaves unexpectedly?” The NCSC’s warning about unsanctioned AI actions challenges the assumption that harmful outcomes always require malicious human intent.”
“The recent AI-assisted discovery of critical PHP vulnerabilities demonstrates AI’s potential as a powerful defensive tool when used appropriately, while reinforcing the importance of addressing conventional weaknesses. Although AI tools helped to identify the vulnerabilities, the immediate response followed established vulnerability management practices: identifying affected systems, assessing exposure and applying the necessary fixes.”
“My approach would centre on protecting the confidentiality, integrity and availability of the system. This means strengthening and testing asset visibility, patching, secure configuration and least privilege, rather than assuming there will always be time for manual intervention.”
“I would treat an AI agent as untrusted automation until it has been appropriately assured. This means defining what it can access and change, applying role-based access control and Zero Trust principles, continuously monitoring its activity and assigning a named person accountable for its operation.”
“No AI agent should have direct authority over safety-critical, security-critical, financial or mission-critical systems without an appropriate human approval gate. Organisations must also retain the ability to quickly disable, restrict or isolate an agent if it behaves unexpectedly.”
“If unexpected activity occurs, responders need protected, time-stamped records of the instructions given, identities and permissions used, information accessed and changes made, alongside evidence of the operational impact. These must be supported by tested arrangements to revoke access, contain activity and recover.“
“Ultimately, assurance must cover the whole environment, including suppliers, people and processes, because focusing solely on AI risks overlooking the weaknesses that determine how far a threat can reach and how much harm it can cause.”
At SQEP Ltd, our Cyber & Information Assurance specialists work alongside programme teams to understand cyber risk within the wider operational environment. They bring cyber expertise into the programme context, helping teams assess vulnerabilities, strengthen assurance and give leaders the insight they need to make informed decisions.
For defence organisations, confidence in cyber security rests on what an environment can withstand, not what we expect a threat to do.
:::
SQEP stands at the forefront of cybersecurity and information assurance, delivering trusted solutions across highly regulated sectors.
Follow SQEP on Linkedin for the latest news, company updates and events.