Humanloop

MCP is rapidly becoming the universal adapter for AI. Since its release in November, developers and teams have raced to adopt the standard, giving agents the tools they need to interface with the real world, from APIs to internal systems. Our latest explainer breaks down MCP: what it is, how it works, and how to get started. 🔗 Read here: https://lnkd.in/e7FM_Xhg

Shout out to Abhinav B. and the team at Boardy 🙌 They released an open-source wrapper that syncs and type-safes prompts from Humanloop into code. This makes prompt collaboration between subject matter experts and developers way faster 🔥 Check out Abhi's explanation of why this is important below ⬇️ Try it out: https://lnkd.in/ezCJNvrv

🔥 Just solved one of prompt engineering's biggest headaches with an open-source tool I built. A Major Pain Point in Prompt Engineering Solved ⚙️ The Problem: 🧩 Prompt engineers update prompts with new variables, developers miss updates, things break in production. An endless, frustrating cycle. The Solution: 💡 I built a type-safe prompt management system that treats prompts like functions with structured inputs/outputs. My tool generates types (inspired by Prisma) that give developers real-time IDE support while letting prompt engineers work independently. No more coordination headaches. Shoutout to Conor Kelly and Raza Habib from Humanloop whose work inspired this project! This is already accelerating our work at Boardy and ready to share with others. Repo in comments! Thoughts? 👇 #AI #PromptEngineering #OpenSource

Awesome explainer from Abhinav B. (Boardy AI) on the challenges of collaboration with subject matter experts on prompt engineering. ...and an awesome open source package to sync and strongly type your prompts from Humanloop into your code. Nice one Abhinav B.

Here are my six takeaways. We had 30+ speakers and thousands of live viewers at the AI in Production virtual event yesterday. Between all the technical snafus I managed to learn something! 1. Human evaluation remains critical for LLM and RAG system development. The difference between great AI products and annoying ones lies in proper human evaluation. The best teams integrate domain experts throughout the development process and use a combination of code-based assertions, LLM judges, and human feedback to evaluate system performance. 2. Guardrails are essential for reliable AI systems and should be benchmarked like any ML model. Effective guardrails implement multiple layers of protection (rules/heuristics, specialized ML models, and secondary LLM calls) to handle PII detection, jailbreak attempts, content moderation, hallucination prevention, and off-topic conversations. 3. Graph-based retrieval significantly enhances RAG capabilities but requires careful design. GraphRAG approaches can combine document-centric graphs with entity relationships, offering more contextual understanding. Multiple implementation options exist (predefined ontologies vs. LLM-extracted entities), each with different tradeoffs between upfront effort and retrieval quality. 4. Agent design requires breaking complex behaviors into manageable subtasks. Modern AI agents need orchestration between memory (short-term, episodic, semantic), reasoning components, and specialized tools. The architecture should support reflection, task planning, and potentially delegation to other agents. 5. Response time and cost management are crucial for production AI. Users won't wait for slow responses (unless you are deep research). Successful systems implement smart caching, parallel delegation, session management, and conversation summarization to manage both latency and token costs. 6. Brand safety and mitigation strategies must be built into the system architecture. Production AI requires comprehensive guardrails. API-level safety filters, input/output sanitization, PII redaction, and protection against prompt hacking to avoid brand-damaging failures. Huge shout out to Rafay and Humanloop for supporting the event. Without them this kind of stuff wouldn't be possible. And a big shoutout to Stefan Ojanen for helping me put these takeaways together. We just released the first batch of recordings. I would love to hear what your takeaways were.

Calm before the storm. I want to create the best online conference experience folks can have. This means I have been recording intermission videos, practicing my guitar skills and making sure all the talks are full of value. But I am still super nervous about the event today. So, what do NVIDIA, Meta, Google, Uber, Adobe, Microsoft, and Stripe all have in common? They will have speakers presenting at today's event (in case my random singing out of tune misses its mark) Huge shout out to Rafay and Humanloop for their support making this conference happen! It wouldn't be possible without them.

🗓️ Wednesday March 12th at 10:00 PT Our CEO Raza Habib will be speaking at the MLOps Community’s 'AI in Production 2025' about Eval-Driven AI Development. What to expect: • Learn how top AI teams use evaluation-driven development to guide model improvements and avoid common pitfalls. • Discover how to leverage code-based, LLM-as-judge, and human evaluators to optimize LLM performance. • Gain insights from Brianna Connelly, VP of Data Science at Filevine, on how their AI team uses evals on Humanloop to refine AI applications and RAG systems. Register to take part virtually: https://lnkd.in/gM-KHcYw

Last year, I was speaking to an engineering leader at a publicly traded technology company when she said something that really surprised me. I asked how important prompts were to AI applications.  “Very”, she said, “they’re the core of the application”.  “How do you handle the process of prompt engineering?" I asked. Her response wasn’t what I expected.

ProductCon 2025 did not disappoint! 🚀 Our team was in London 🇬🇧 this week speaking to product managers and leaders about developing and scaling AI products and agents. By detaching prompt management and evaluation workflows from core software development, Humanloop allows product experts to more easily steer AI performance pre and post product/agent development. Thanks to Carlos Gonzalez de Villaumbrosia and the amazing team at Product School for putting together an incredible event! Where should we go next? 👀

📍PMs in AI Meetup, London 🇬🇧 Yesterday we held a Meetup in the UCL Centre for Artificial Intelligence for product managers working on AI agents and applications. Huges thanks to all who turned up (it was a full house!) and to our speakers: • Sam Stephenson (Founder, Granola) - who advised on making your 1 AI feature extremely effective before trying to add any more. • Alberto Rizzoli (Co-founder, V7) - who said to listen to user problems, not their proposed solutions (this is more true than ever with AI). • Raza Habib (Co-founder, Humanloop) - advised to bring domain experts into the prompt engineering and evaluation process as early as possible to drive differentiated and effective AI performance. The London AI community is next level 🚀 What should be the theme of our next meetup? 👀