AI security in 2024

There we are, celebrating our first full year of massively-adopted generative AI. Like the rest of the IT industry, cybersecurity has been taken aback by the enormous new capabilities brought to human productivity and creativity.

Business lines have been swift to understand the benefits and to imagine countless sustainable use cases, development teams and platforms like Hugging Face have been equally swift to support use cases with impressive frameworks and toolkits, vendors have proposed aggressively competitive pay-as-you-go closed APIs, the open source has responded with many smart foundation models in just a few months...

2023, a "wasted year" for AI security?

... IT security, standing at the intersection of all the turmoil, has been struggling to catch up with innovation.

But security is not a new kid on the block: for several decades, it has learnt to adapt and to be resilient to waves after waves of innovation. The public Cloud demonstrated this, and before that, mass-datacenters virtualization, and, before that, the Internet revolution.

In 2023, the initial pace in AI security might appear sluggish to certain observers, possibly deeming it a 'wasted year': experts spent a lot of time and efforts to explore many different directions, trying to identify risks and solutions to features which turned out to be short-lived or only business-as-usual-in-disguise. The OWASP top 10 for LLMs is perhaps the best example of a deliverable that was published under pressure and which falls into these two traps, to some extent.

However, I firmly believe that the cautious outset and the exploratory phase were essential. They constituted a pivotal stage inherent in the typical security process, especially when faced with innovative advancements. As I love to say, the best way to learn is through tinkering and experimenting. Architects and systems designers, in particular, must have greasy hands. As Gregor Hohpe brilliantly explains in his blog, this is was distinguishes them from ivory tower magnates.

Reflecting on our hands-on experience in 2023, what can we glean, and what might we anticipate for 2024?

Allow me to share my perspectives.

2024, the year of AI security focus

This year will be a year of focus: by "collimating" efforts, security will become impactful and business-enabling.

The fall of GPTs

GPTs are like Internet-of-things: they make for cool standalone ready to use "devices", but their creators tend to not give them the second thought they deserve. On the long term, GPTs are doomed to become abandonware, thereby turning into dormant vulnerability bombs.

With the industrialization of LLM training processes and the handling of models versioning and models lifecycle management like any left-shifted project, GPTs should quickly disappear from the corporate landscape.

In a nutshell: with business-made GPTs quickly fading off, security teams are going to dedicate their time on IT-made (and IT-maintained!) implementations.

The end of corporate foundational models

Building foundational models is not sustainable, except for GAFAMs and unicorns. Even if more GPUs were available on the market, at a much cheaper cost, many observers have concluded that, for the majority of corporations, the associated challenges outweigh the benefits

As a result, money and workforce are going to shift from building foundational models to fine-tuning or training versatile ones. This trend was anticipated by Abu Dhabi's Technology Innovation Institute, which open sourced its Falcon LLM. Things accelerated EOY 2023 when France's Mitral AI open sourced its own LLM.

In a nutshell: corporate IT departments will ditch building foundational models pretty fast. For security teams, it means dedicating time to expert models built on top of foundational models (see below).

The race between open and closed models takes off

The choice of closed LLMs versus open LLMs is a business and an architecture decision, not a security one. Ultimately, it's not even a business choice, but a purely financial matter: the pay-per-use model may be very expansive for completion and very cheap for fine-tuning, or the other way around. It depends so much on the use case.

For that very reason, both models are here to stay.

So, as usual, CISOs will have to adapt to external forces, which doesn't mean that they shouldn't drive the design and the integration!

One might think that since closed models are usually hosted off premises, they pose more compliance and privacy concerns than their open counterparts. That's true, but if we look at the models under the lenses of security alone, then closed or open doesn't make much a difference. CISOs have learnt from the Cloud how to deal with multi-tenant providers, regional constraints, and privacy requirements. The real deal has to to with integration.

In a nutshell: for security, the choice of a closed or an open LLM won't matter much. Integration will. When embracing a model, its secure integration is what CISOs should prioritize.

The race between RAG and fine-tuning takes off

We said that building foundational models is a thing of the past for most corporation. What's going to fill-in the gap? Expert models (or even mixture of several expert models) built on top of foundational models.

As of now, there are two main ways to build expert models:

• Retrieval Augmented Generation (RAG): a business-dependent corpus of document is ingested into LLM-searchable databases,
• Fine-tuning: foundational LLMs are further trained to perform business-dependent tasks, by providing painfully curated samples and testing their predictions.

If fine-tuning requires much more machine learning expertise than RAG, both techniques pose their own security challenges:

• At first glance, RAG looks better for security because it relies on an easy-to-govern, easy-to-integrate corpus of putatively well know corporate data. But it depends on the agency of a plug-in for performing searches. This plug-in is a big deal to secure.
• Fine-tuning is said to lack explainability and to be prone to bias. As I keep saying, explainability and bias are extremely important, but they are not security concerns. Integrity and traceability are. Many applications and data sources are subjected to bias, statistician know how to deal with this. As for explainability, it might be a legal requirements, but for security forensics, all we need is love... errr, no! Actually, all we need are redo logs.

In a nutshell: for security, fine-tuning is going to by pretty run-of-the-mill, because it amounts to ensuring integrity and traceability. Securing RAG is going to be part of the challenge of 2024, because RAG relies on plug-ins (see below).

The integration "triad": read, write, act!

2024 is said to be the year when LLMs are going to take live decisions - what I call "action ops". Actions ops stand just one step above "write ops", that many LLMs already do.

Both kind of operations modify application states and data stores, they have a direct impact on the business of the information system. Thus, actions ops and write ops are both very sensitive.

Should action ops and write ops eclipse "read ops"? Not at all!

The core of LLM integration rests on the read, write, action triad.

To understand why the whole triad is important, one must recall the one single, outstanding risk when we talk LLMs: prompt injection.

• Prompt injection can happen upstream (when an LLM reads from a compromised producer) or downstream (when an LLM contaminates a consumer).
• LLM have no memory, their conversations are transient, but LLMs can remain corrupted over unlimited periods of time by re-poisoning themselves. Re-poisoning happens when downstream corrupted data are fed back upstream.

Whenever an LLM interacts with the information system to perform a "triad" operation, it cannot do this on its own: it must resort to an agent (or pluging).

Securing plugins to prevent prompt injection is going to be "the" challenge of 2024, in my opinion. Properly handling this challenge through enterprise patterns ought to be the job of security architects.

In a nutshell: the standardization of secure LLM integration is unlikely to happen before 2025(?). In the meantime, local security architects will have to define bulletproof enterprise patterns, centered on plugins, to deal with upstream (read) and downstream (write, execute) prompt injection prevention.

Securing General Artificial Intelligence, anyone?

So much has been rumored about the imminent wake of "general artificial intelligence", but we have so little evidence...

We all know about openAI's Q-star drama. If their experimental AI is supposed to be capable of solving simple mathematical problems in a much smarter way than GPT4, the methodology followed by openAI researchers is quite novel and groundbreaking: it might (or might not?) lead to the GAI El Dorado. Other, longer-term approaches are being envisioned, like Yann LeCun's world model module departing from the notion of "language" which is so crucial to LLMs.

In the unlikely event that GAI comes to fruition in 2024, my only recommendation, the one which is stuck to when I pioneered Cloud security in 2015, is this: ensure readiness. Concretely, it means that even if GAI is expected to be immensely valuable to speed up the cure of many human or environmental ailments, security pros should:

1. see GAI as a Black Swan
2. think about appropriate/commensurate countermeasures, which I call Taleb fuses
3. consider this Taleb fuse as a starting point: the first law of generative AI
4. imagine more fuses, maybe more context-dependent. And adjust your risk analysis accordingly.