Building a Better AI Support Helper (OpenAI + Intercom)

Building a Better AI Support Helper with OpenAI + Intercom

Ever been in that situation where you're juggling 15 support tickets and can't remember what you promised to do next on each one? Yeah, me too.

Especially when looking at someone else's ticket I've spent countless hours scrolling through ticket histories trying to figure out what was promised to the customer for next steps. Did I say I'd write up a ticket for the dev team or was I waiting for them to send me logs? It's mentally exhausting and a huge time sink.

The Problem with "Please Follow Up"

Our team had been using a basic bot that would ping agents after a few hours of ticket inactivity with the incredibly helpful message: "Please follow up."

Sure, it had some intelligence, like it wouldn't bug you about snoozed tickets, and it knew when to hand off to our closing bot. However, it wasn't actually helping us remember what needed to be done next.

From Nudges to Actual Help

I wanted something better: an assistant that could read the conversation, understand where things left off, and suggest the next logical actions.

The solution combines Intercom workflows with Zapier integration and OpenAI. When a ticket has been inactive for a certain period, the system:

1. Sends the conversation transcript to OpenAI (with some formatting fixes)
2. OpenAI then analyzes what's been discussed and what was promised
3. Adds an Intercom note that provides specific follow-up suggestions to the agent

For example, instead of just saying "Please follow up," it might tell you: "Please create the Linear ticket for the dev team about the redirect bug."

The Technical Challenge

While integrating Intercom, Zapier, and OpenAI sounds straightforward, getting everything to work smoothly was surprisingly tricky:

• Intercom's custom actions needed to trigger a Zapier workflow, not a Google App Script like I usually do
• The Zap included custom JavaScript code to call Intercom's API for the conversation transcript, since its not a built in Zap action.
• More Javascript custom code to format the transcript into AI optimized text, so it's not just a complex array of non-relevant data.

But the real challenge? Prompt engineering. Getting the AI to provide genuinely helpful suggestions took a few weeks, and is still ongoing.

Lessons in Prompt Engineering

If you're considering using AI for similar projects, here's what worked for me:

• Start simple, then add complexity: My first prompts were basic, focusing on just identifying if a reply was needed. As those worked well, I added more nuance.
• Include specific examples: I added examples of what is a good follow-up suggestion to guide the model.
• Structure matters: Breaking the prompt into clear sections (analysis, context consideration, suggestion formulation) dramatically improved results.
• Set the right tone: I specifically directed the AI to be concise and actionable rather than apologetic or overly detailed. I wanted it to use "Please" and to keep it to a single short sentence. It wouldn't save us time if we had to read a paragraph just to find out what to do next.
• Iterate, iterate, iterate: The prompt that worked best was version 14. Each iteration incorporated feedback from real usage.

Current Status & Results

I've been piloting this with my own tickets for a few weeks, and it's already saving me significant time. Instead of spending 5-10 minutes per ticket remembering context, I can now process my queue much faster.

Our senior support agent has been testing it this past week, and while we're still gathering comprehensive data, the early feedback is positive. The suggestions are targeted and actually helpful.

The Bigger Picture

This project isn't revolutionary, its a simple concept and really only needed a few connections to existing tools. But it addresses a real pain point that support teams face daily.

The future of AI that interests me the most is how it's going to connect with everything. I think we all know that AI will do amazing things, but getting it in the right place to do that is a fascinating problem in and of itself.

If you're working on similar support automation projects or want to chat about prompt engineering for practical applications, I'd love to connect!