LLM Jacking: How Hackers Are Exploiting Large Language Models

Since the Sysdig Threat Research Team (TRT) initially discovered LLM jacking in May 2024, the attack methodology has evolved substantially. As large language models (LLMs) develop further and find new uses, hackers are becoming more nimble and learning new ways to exploit these sophisticated systems.

The rapid targeting of emerging LLMs like DeepSeek is especially notable. In September 2024, Sysdig TRT highlighted the increasing frequency of these cyberattacks, and this trend has only become more pronounced. According to TRT, the swift targeting of DeepSeek shortly after its public introduction was not unexpected. 

Major corporations, including Microsoft, have also become victims. They recently initiated legal action against cybercriminals who misused stolen login credentials. The accused reportedly utilized DALL-E to create inappropriate and offensive materials.

What is LLM Jacking?

LLMjacking refers to cyberattacks involving unauthorized access, manipulation, and misuse of LLMs. The TRT introduced this term to describe scenarios where attackers gain unauthorized entry into victims’ LLM resources using compromised credentials. 

Organizations relying on cloud-based LLMs are particularly susceptible to these threats.

Attackers exploit LLMs for various purposes, from seemingly harmless personal interactions and creative tasks like image creation to more malicious activities, including malicious code optimization and developing tools intended for cyberterrorism. 

In severe cases, attackers may employ LLMjacking to poison AI models or extract sensitive data, facilitating ransomware attacks against major organizations.

As cyber threats evolve, attackers continuously adapt their techniques and extend attacks to additional AI services and platforms.

Video source: YouTube/Cyber&Tech

How does LLM Jacking Work?

LLMjacking targets large AI models by exploiting security gaps either in the model itself or in the infrastructure that hosts it. Unlike typical cyber threats, which usually focus on manipulating input data, LLMjacking directly interferes with the internal workings of AI systems—often without triggering any alarms.

Attackers can carry out LLM jacking in several different ways:

• Command manipulation: Cybercriminals embed special commands into the model’s normal interactions or its API calls, tricking the AI into providing outputs that help achieve the attackers’ goals. Examples include secretly leaking confidential information or installing hidden backdoors into applications.

• Exploiting APIs: Since many AI models connect to external services through APIs, poorly secured APIs become easy entry points. Attackers exploit weak authentication or stolen API keys to access, control, or misuse sensitive data or even disrupt services altogether.

• Tampering with model training: Attackers might also secretly inject malicious data into the AI’s training process, gradually altering the model’s responses. This approach subtly changes the AI’s behavior or creates hidden biases to favor attacker-controlled outcomes, all while remaining undetected.

• Manipulating prompts: Sometimes attackers bypass AI safety checks by cleverly wording the prompts or requests sent to the AI. For instance, attackers can craft prompts that trick the model into ignoring built-in safeguards, causing it to produce potentially harmful or illegal content.

Sysdig’s Threat Research Team first noticed an LLM jacking case where attackers ran scripts to find unsecured AI credentials in cloud environments. After obtaining valid credentials, attackers gained complete access and control over the targeted AI models, enabling them to steal data, alter model behavior, or cause service outages.

Video source: YouTube/Medusa

High Costs and Motives Behind LLM-Jacking

Running LLMs involves substantial costs, sometimes reaching hundreds of thousands of dollars each month. These high costs create a strong incentive for attackers to hijack LLMs and 

capitalize on stolen credentials rather than pay for legitimate access. Services like OpenAI, Anthropic, AWS Bedrock, and Azure become prime targets precisely because of their significant operational value.

Attackers waste little time exploiting newly launched models. For example, shortly after DeepSeek-V3’s release on December 26, 2024, attackers had already set up unauthorized proxies to exploit the model on HuggingFace. The situation escalated even faster with DeepSeek-R1, which attackers accessed within just a day. This rapid exploitation reflects attackers’ clear understanding of the financial benefits tied to early access.

Beyond technical exploits, attackers increasingly use advanced social engineering tactics powered by AI itself. Using personal details from potential victims, like preferred restaurants, shopping patterns, or workplace specifics, they create highly convincing phishing messages tailored to each individual. This targeted, AI-enhanced approach significantly boosts the effectiveness of their initial breaches.

In short, substantial financial rewards, combined with ease and speed of exploitation and sophisticated social-engineering methods, explain why LLM jacking continues to grow rapidly.

Preventive Measures Against LLM Jacking

Addressing LLM jacking requires structured and layered defense efforts across multiple domains:

• Ethical model design and integrity checks: Language model developers must integrate safeguards during the design and training stages. This includes mechanisms to detect and respond to malicious prompts, along with routine evaluations of both models and datasets. Regular audits help identify structural vulnerabilities and maintain reliability.

• Restricted access and continuous oversight: Robust authentication systems and clearly defined user roles should limit access to LLMs. In addition to access control, systems should include constant activity monitoring to detect anomalies or signs of unauthorized use, allowing for swift containment and response.

• Policy development and regulatory enforcement: Legislative bodies need to establish concrete legal standards for deploying and using LLMs. Regulations should define misuse, outline penalties, and provide mechanisms for identifying and holding offenders accountable. Legal clarity helps reinforce safe practices and discourages abuse.

• User-level education and reporting systems: End-users must be informed about the risks related to language model misuse. Awareness initiatives should focus on common signs of compromised systems and encourage proactive reporting. An informed user base can function as an additional layer of defense.

• Advancement of protective technologies: Security research must continue focusing on tools that prevent unauthorized model behavior. This includes refining content filtering, implementing stronger output constraints, and developing systems resistant to prompt injection and manipulation. Investment in these areas is necessary to adapt to ongoing and future threats.

LLM Jacking: Key Takeaways

LLM jacking has moved fast from a fringe threat in 2024 to a real, growing problem. Attackers aren’t just stealing keys; they’re manipulating prompts, poisoning training data, and using AI to craft social engineering attacks that actually work.

Any public-facing LLM is now a target. DeepSeek was exploited within a day, and Microsoft’s tools were hijacked for offensive use. This isn’t a theory. It’s happening in production environments right now.

Security isn’t optional. Lock down access, monitor activity, test for prompt injection, and educate your teams. 

LLM jacking isn’t slowing down. If you’re building or deploying LLMs in any serious way, security can’t be a bolt-on. It has to be part of your design language.

Thank you for reading and engaging, and consider subscribing to my Substack channel to discover A New Space for Transformative AI Insights and Practical Strategies.