Why did we choose to build our own custom AI Support Bot? – Journey and Results

This project has been a big win for our team at Instapage , and I wanted to share the journey and results. As AI continues to evolve, many companies are exploring how they can improve their operational efficiencies, and we decided to do the same.

Early last year, with the AI craze taking off, we decided to explore where AI could make an impact in our company, so we figured customer support might be the perfect starting point for us. 

After months of reviewing third-party solutions, we realised none offered the quality and flexibility we needed. So, we decided to build our own custom chatbot that could reply to customer email tickets. We learned a lot along the way and ended up achieving some pretty exciting results that I’d love to share with you.

The Testing and Comparison: Before building our solution, we tested nearly 20 different AI tools. Our evaluation criteria included quality of replies, customization, ease of integration, and, of course, cost. Most third-party solutions started at from $20k per year - to requiring a full migration of our ticketing system—so naturally, we thought we could do better while spending a lot less.

That’s when we decided to develop our own custom solution. 

Early on, we tested multiple large language models (LLMs) and ultimately chose OpenAI’s ChatGPT API. It checked most of our boxes—ease of implementation, robust online documentation, and consistent improvements that would allow our solution to evolve alongside it.

Throughout the process, we constantly compared our custom bot’s performance against the top solutions on the market. Our bot ended up delivering better-quality replies, a more customizable experience for our customer operations team, and of course a solution that can be easily molded for our needs over time—all at a fraction of the cost. 

Our solution was built by one developer over one quarter and not even working full time on this project, our Customer Operations team ensuring reply quality and internal rating of tickets for future bot improvements. Keep in mind we never had a resource working full time on this project, it was just an exciting side-project for all of us involved at the time.

The Journey and Results: When we first launched the AI bot, we knew it wasn’t going to be perfect. Our initial customer satisfaction (CSAT) score for the AI specifically started off pretty low—at 22%. It was nowhere near the CSAT of our human agents, who consistently hit 90+%. But as we iterated and improved the bot, we saw the AI CSAT grow steadily, eventually reaching 71.5%, which is pretty impressive for a chatbot.

We also saw some great results over time as the bot “got smarter”:

• 27% of  our email support tickets were fully handled by the bot.
• significantly speeding up first reply times by 65%.
• 83% reduction in AI tickets bad ratings, going from 12 to just 2 from the quarter it got launched to the next one.

General Learnings

• Evaluate Quality, Tone, and Accuracy: Before deciding whether to go with a custom or third-party solution, ensure the chatbot's quality, tone, and accuracy of replies align with your internal customer support standards. Having a robust quality check process in place from the beginning is key—this way, you can compare the AI’s performance against your current and ever evolving standards. Additionally, be mindful of the full cost of implementation, including fees like per-ticket reply costs, potential platform migrations, and the retraining of your staff if necessary if you are considering a migration.

• Iterate and Improve Constantly: One of the most valuable practices we implemented was regularly reviewing the quality of the bot’s replies. We made significant improvements by refining the data we fed it, in our case, our Help Center content. Over time, we even rewrote several articles to help the bot "understand" them better and provide more accurate responses. Constantly update your bot’s data sources, especially as new features or use cases arise.

• Scalability and Resource Management: If resources are limited, like they were for us, starting small and scaling gradually is a good strategy. This allows you to test and iterate without overwhelming your team. Prioritize regular reviews of the bot’s performance and focus on key areas that have the most impact, such as accuracy and customer satisfaction.

Conclusion: This project has been one of the most fun and rewarding ones we’ve worked on, and it wouldn’t have been possible without the collaboration of our amazing team. Big thanks to our one man army developer Bogdan Mastalos and our Customer Operations team, lead by Buleandră Iulian Florin — your hard work made this happen.

We’re really proud of what we built, and excited about how this solution will allow our support team to focus more on our customers, take a more proactive approach in their engagement, and ultimately bring more value to their overall experience.