PART 2 // Understanding LLM Behavior in Production // Why Your AI Feature Works in Dev and Breaks in Prod

Every now and again, I get calls from founders who I advise on AI-native products saying: “It was amazing in testing. But now it’s giving weird results in prod.”

What works in staging often fails in prod. Not because the model changed, but because you didn’t understand how it thinks.

The Paradigm Shift: From User to Co-Creator

Here’s the truth: LLMs don’t behave like traditional APIs. They operate probabilistically. That means you're managing variance, not just logic. This shift from deterministic logic to probabilistic variance is a core paradigm shift for product leaders entering the AI space to grasp.

When you ship an LLM-powered product, your users aren’t just “using a feature”, they’re co-authoring the outcome with the model. This co-authorship model necessitates a move beyond traditional user journey mapping to anticipate a wider range of potential interactions and outcomes.

That makes your product inherently:

• Non-deterministic
• Context-sensitive
• Dynamic across time and model versions  (And by the way, this inherent dynamism requires continuous monitoring and adaptation, unlike the more static nature of traditional software features.)

LLMs behave differently depending on:

• Prompt phrasing
• Input length
• Fine-tuning or system prompt changes
• Rate limits and load balancing on the model provider’s end
• Even model updates you didn’t control (for example when OpenAI silently updates GPT-4). Note, this external dependency on model providers introduces a new layer of uncertainty and the need for proactive awareness of model updates and their potential impact.

So if you’re not logging and analyzing usage at scale, you’re operating blind. Scalable logging and sophisticated analytics are no longer optional; they are the essential sensors for understanding and managing your AI product in the real world.

Real World Example

Duolingo Max launched with GPT-4 to power Explain My Answer. Before launch, their team battle-tested thousands of edge cases, because they knew LLMs behave differently when exposed to messy, real-world language. Their proactive focus on real-world edge cases, rather than just ideal scenarios, highlights a mature understanding of LLM deployment.

They didn’t just test prompts. They studied model behavior patterns at scale. This shift from individual prompt testing to understanding systemic behavioral patterns is a key differentiator in building robust AI products.

That’s what let them ship something robust, not gimmicky.

The Technical Jargon

• Token Sampling: LLMs generate responses by sampling token probabilities. Understanding that output is a result of probabilistic sampling, not deterministic selection, is crucial for managing expectations around consistency.
• Top-k / Top-p Sampling: Controls how random the generation is. Strategic adjustment of these parameters allows PMs to fine-tune the balance between creativity and predictability based on the specific user need.
• Context Window: Each model has a max token limit for input + output (e.g., 128k tokens for GPT-4 Turbo). Designing user interactions and prompts within these limits is essential for maintaining coherence and preventing unexpected behavior.
• Prompt Injection: Users can override or manipulate instructions, which presents a major risk. Viewing prompt injection not just as a security vulnerability, but as a fundamental challenge in designing trustworthy AI interactions, elevates the PM's strategic role.

Practical Actions for Product Managers

• Monitor your AI feature post-deploy, not just pre-launch. Post-launch monitoring should be a continuous, iterative process, not a one-time check.
• Set up instrumentation: prompt, input, output, latency, model version. Comprehensive instrumentation provides the data needed for informed decision-making and proactive issue resolution.
• Run behavioral regressions before switching model versions. Behavioral regression testing should be a standard practice to ensure a consistent user experience across model updates.
• Build evaluation sets with real user data, not just synthetic tests. Prioritizing real-world data in evaluation sets provides a far more accurate understanding of production performance and potential failure points.

Suggested Resource for Deeper Learning

OpenAI’s API Behavior Guide covers temperature, max tokens, and managing behavioral drift. Read the documentation (and encourage your teams too as well) to actively experiment with these parameters to develop an intuitive understanding of their impact on model behavior.

If you don’t understand LLM behavior in the wild, you’re not managing a product, you’re simply crossing your fingers. Proactive understanding and management of LLM behavior is a core differentiator for successful AI product leaders.

Treat your model like a team member:

• Track its behavior, 
• give it feedback (through prompt iteration and fine-tuning strategies),
• and build guardrails around its quirks ((through careful prompt design, input validation, and output filtering)

Viewing the LLM as a dynamic team member requiring ongoing guidance and management reflects a mature approach to AI product leadership.