Privacy Engineering & AI Threat Modeling: Model and Data Theft

Imagine a relentless adversary targeting your company’s crown jewels: all the sensitive information and personal data guarded by your AI system. One technique are attacks directly targeting the AI model itself.

These attacks aim to achieve two goals:  First, to replicate the functionality of the AI system. This grants the adversary access to the same capabilities that your company leverages from the AI model, potentially stealing your competitive edge. Second, to uncover the training data. This exposes the sensitive information used to train the model, even if anonymized, potentially leading to privacy violations and data breaches.

There are several related attacks:

• Membership Inference: Attempts to determine if a specific data element was used to train the model. Attackers analyze the model’s response to understand whether or not the specific individual data element is present in the model, even if not directly disclosed.
• Attribute Inference: Aims to infer specific attributes of the training data. By observing the model’s behavior, attackers might learn characteristics like age, location or other demographics.
• Inversion Attack: Attackers attempt to manipulate their inputs into generative models to reveal information in its training data. An example of this would be an attack to generate an image of a specific individual. Exploited with specific AI types, particularly generative models (e.g., face generation). Attackers manipulate inputs to force the model to reveal information present in its training data. For example, generating an image resembling a training data point.
• Model Stealing: These reverse engineering attacks involves querying the model extensively with carefully crafted inputs to understand its internal logic. By analyzing the responses, attackers can potentially build a replica model, effectively stealing its functionality.

While security and engineering resources are more likely to take the lead protecting against these attacks, here are some questions a Privacy Engineer should be asking during the AI product lifecycle:

Plan/Design

• What is the minimum amount of data, and especially personal data, required to achieve the AI system’s purpose?
• Is the personal data anonymized, and is the anonymization providing sufficient protection against model and data theft attacks?
• Can synthetic or anonymized data by used instead of real personal data?
• How will the model itself be protected from suspicious prompts or queries, attempts to access unauthorized data, or reverse engineering?
• Are noise added during training (such as through differential privacy) making it harder to determine or infer individual data points?

Implement/Test

• Are mechanisms put in place and working to monitor for suspicious activity, unauthorized data access or other attempts to reverse engineer the model or data?

Maintain

• Who has ongoing access to the AI model and the training data?
• Is the security principle of least privilege applied to minimize the attack surface?
• Are results from ongoing operations monitoring integrated into the process of updating and improving the AI model