A traditional AI infrastructure is the Training, Inferencing lifecycle of an AI model. Examples could be use of visual computing in retail, or industrial sectors.
The starting point of Threat-modelling in such case could be “Shostack’s Four Question Framework” [1], see the Threat Modelling Manifesto [2], a gist of which with the tools and framework (methodology) is listed below (The * options and considerations are expanded further down).
- What are we working on: Modelling a system using Data Flow Diagram (DFD), or Flow Diagram (Sequence Diagram), and in case of Agile just focusing on the APIs in the scope of current sprint, are all good starting points. I find DFDs also handy in getting a natural insight into the trust boundaries (between User, External-systems, Processes, Data-stores and choice of transport). It’s important to get the assumptions out first, and then iterate if all assumptions make sense or new ones need to be added.
- What can go wrong: A popular framework often used is based on STRIDE, which stands for Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege. Each violates some cybersecurity protection, e.g. Spoofing violates Authentication, Tampering violates Integrity, Repudiation affects Traceability, Information Disclosure directly impacts confidentiality, Denial of Service affects Availability and Elevation of Privileges violates Authorization. I find AWS Threat Grammar quite interesting for this – along with AWS Threat-Composer [3], and OWASP Threat Dragon [4], there are other tools available too. STRIDE-AI extends STRIDE to AIML, by adding focus on training data and sensor data.
- What are we going to do: A structured response to the threats is to consider whether we can avoid the threat (e.g. by selecting implementations that can, or where applicable using topology hiding, or protecting critical information). If we can’t avoid the threat then can we mitigate the risk (reduce the cost/disruption) by using one or more security practices using the Zero-Trust framework below:
- Did we do a good job: This is security product managers definition of done. From did we validate assumptions, did we discover more assumptions and threats, to what regulatory compliance can be met, does it cover our immediate customer needs. An agile environment opens up the possibility of focusing on certain areas first, around confidentiality and integrity detection of data, then incrementally extending the trust boundary to include confidentiality, integrity protection and availability. It is important to have a customer centric view of the definition of done, which regulatory requirements are must have, and given other tools being used, what threat risks can be transferred. Every sprint revisits the assumptions, new threats, efficacy of current threat mitigations and security architecture, backlog prioritization, and the definition of done. In other words, till the software is retired, it remains in active planning.
A generative AI (gen AI) system creates content or provides inputs to an Agentic AI system for autonomous action. Such systems have significantly more complex security needs than the conventional AI (expert) systems:
- Attack surface is much larger: The well-defined input-output system is replaced by a probabilistic system whose outcome could be diverse, inputs uncontrolled and results unpredictable. A Gen-AI system may work exceptionally well for one set of data, but may fail completely while dealing with another set of data, allowing threats and compromises to be masked far more easily. Supply-chain vulnerability is a major security concern for enterprises. See for example LLM05, LLM07, LLM09, LLM10 in [6].
- Prompt-based attacks: The response of a model can be manipulated by crafting deceptive prompts or hidden text, into making the system reveal protected data or grant access to protected resources. An attacker could interact in a way that consumes high amount of resources ending up in Denial of Service. See for example LLM01, LLM02, LLM04, LLM06 in [6].
- Data poisoning and model manipulation: By poisoning the training data, a Threat-actor can create biases, or backdoors. AI models suffer from data drift problem, requiring updates and thus creating a security hole for opportunistic Threat-actors. See for example LLM03, LLM05 in [6].
- Multiple AI Agent interaction: Such interactions increase the blast radius of a cybersecurity attack. These are not yet covered sufficiently in [6] but some of the items are covered e.g. under LLM07, LLM08.
- Goal misalignment: The system attempts to fulfil its stated requirement even at the cost of ending up creating a bigger problem somewhere else while delivering the service (“The monkey and the sword” problem). See for example LLM06, LLM08 in [6].
Mitigation strategies require fine-grained content moderation, filtering and supply-chain due diligence. The system needs to go through adversarial testing for robustness. Special attention is needed for Guardrails for safe and responsible AI. The OWASP Top 10 for LLM and Generative AI [6] goes into more details for product consideration.
