Using AI Assistants Powered by Azure OpenAI Realtime API

AI Assistants driven by Large Language Models (LLMs) are revolutionizing the enterprise landscape by enabling users to interact naturally and seamlessly over data and knowledge repositories. These assistants answer complex queries, making enterprise processes more efficient. However, until now, most implementations have been limited by a few crucial aspects of user experience:

• The need for separate Speech-to-Text (STT) and Text-to-Speech (TTS) calls, adding latency to the system.
• The quality of audio responses often lacked the natural, human-like nuance.
• Users had to wait for the system to finish its response before providing further input, as it couldn’t process interruptions smoothly.

With the recent release of the OpenAI Realtime API, these challenges have been addressed, creating a more immersive and interactive AI assistant experience. Let’s explore how this game-changing technology works through a practical use case.

Enhanced Interactivity with Realtime API

Imagine a scenario where a user interacts with an AI assistant powered by Azure OpenAI Realtime API. The user has subscribed to the games from a Company and is interacting with an AI Assistant to have a variety of queries answered. The user issues a variety of voice commands, and each query triggers different backend actions based on the context:

• Query 1: Retrieving game status from a cloud database (Azure SQL Database).
• Query 2: Investigating game crashes by searching a knowledge repository of manuals using AI search.
• Query 3: Raising a grievance by invoking a REST API call to Jira in the cloud.

With the Azure OpenAI gpt-4o Realtime API, the system not only takes direct audio input but also performs function calling to various backend systems. Based on the user’s query, it can call external systems like databases or APIs and generate a tailored response, which is delivered back to the user via high-quality, natural-sounding neural voices.

Key Features and Improvements

The Azure OpenAI Realtime API elevates the user experience in multiple ways:

1. Reduced Latency: By integrating audio input/output directly within the API, the need for separate STT and TTS calls is eliminated, thus reducing the latency significantly.
2. Natural Audio Quality: The neural voice technology creates audio responses that sound lifelike and engaging, providing users with a more human-like interaction.
3. Intelligent Interrupt Handling: Unlike traditional systems, the Realtime API can detect when a user is interrupting the conversation. It can pause its audio output to listen for further input, creating a more fluid, natural conversational flow.
4. Seamless Tool Invocation: The AI assistant seamlessly interprets user queries and triggers backend processes like API calls or database lookups, without needing manual intervention.
5. Multimodal Interaction: While the API supports audio input and output, users can also choose to interact via a chat interface. The API generates real-time transcripts of the conversation, making it easy for users to read responses if needed.

The Future of AI-Powered Conversations

The Realtime API unlocks the next generation of voice-based AI assistants, where speed, interactivity, and human-like audio quality converge to create powerful user experiences. Whether for enterprise tasks like looking up game statuses, investigating system issues, or interacting with customer support, this API offers a unified, highly responsive platform for real-time interactions.

Watch the video below for a demonstration of these capabilities in action.

Acknowledgements:

Many thanks to Manoranjan Rajguru who ported the JS implementation of the Realtime API client to Python, which has been used to create the demo here.