Building Custom AI App Experiences with Azure AI

Introduction

Azure AI Studio is now generally available, providing developers with the tools needed to build custom AI applications. This guide will show how to use Azure AI Studio in combination with your code to create AI-powered apps grounded in your data. Additionally, we'll cover orchestrating models across multi-step processes, setting up automated deployments, running evaluations, and monitoring production apps as part of GenOps.

Azure AI Studio: More Than Just a User Interface

The Azure AI Studio is more than just a UI—it allows you to harness Azure AI resources directly from your code. It’s also an excellent starting point for familiarizing yourself with AI app development. Without signing in, you can access the model catalog at ai.azure.com, which includes over 1,700 models from OpenAI, Microsoft, Meta, Mistral, and more.

Choosing the Right Model

The model benchmarks section helps you compare models based on various factors:

• Accuracy: Performance of different models across tasks.
• Coherence: Measures how well the model generates natural, smooth responses.
• Groundedness: Assesses how well the model refers to provided source materials.
• Fluency: Evaluates language proficiency.
• Relevance: Determines how well the model meets expectations based on prompts.

Accessing AI Services

Azure AI offers pre-built services to develop multimodal applications, incorporating cognitive skills such as:

• Speech and language translation
• Vision and OCR
• Content safety for detecting harmful or inappropriate inputs/outputs

Building AI Apps with Custom Data

Once signed in with an Azure account, developers can create a project to securely connect to Azure resources. This project organizes assets and provisions necessary resources.

Deploying a Model

1. Select a model from the Model Catalog (e.g., GPT-4.0).
2. Deploy it with default settings.
3. Test the model in the Playground.
4. Define a System Message to guide the AI’s responses (e.g., "Be cheerful and use emojis for context").

Enhancing Model Responses with Retrieval-Augmented Generation (RAG)

To improve the AI’s responses, you need to ground it with reference data:

1. Navigate to Add Your Data in the Playground.
2. Upload relevant data files or connect to an existing Azure AI search index.
3. Enable vector and keyword-based hybrid search.
4. Use vector search to match prompts with the closest data points.

Integrating AI into Your Code

Setting Up the Development Environment

1. Install the OpenAI SDK and Azure Identity Library.
2. Use VS Code to integrate the AI model into your application.
3. Copy code from the Playground and run it in VS Code.
4. Capture responses in JSON format for seamless integration.

Orchestrating Multi-Step AI Processes

Real-world applications require multiple steps with different models and prompts. Using an orchestrator, you can:

• Research trends to gather information.
• Search product catalogs.
• Generate articles based on AI-driven insights.
• Continuously refine the AI-generated content.

Running the Orchestrator

1. Deploy resources using AZD UP Command.
2. Use Prompt Templates for refining AI-generated content.
3. Run an AI-assisted workflow to generate an article based on curated research.

Debugging and Monitoring

Using Traces to Debug

Traces help visualize model interactions and pinpoint errors. Developers can:

• Review step-by-step AI responses.
• Analyze model calls and prompt completions.
• Fine-tune prompts for better performance.

Evaluating Model Performance

Azure AI evaluators assess:

• Relevance
• Fluency
• Coherence
• Groundedness

Performance scores (1–5 scale) help refine the model iteratively.

Production Monitoring

Application Insights provides:

• Evaluation scores across runs
• Token usage statistics
• Model performance metrics
• Transaction searches for tracing errors

Conclusion

With Azure AI, developers have everything needed to build custom AI experiences directly from their code. To get started:

• Visit ai.azure.com
• Explore code samples at aka.ms/apppp-templates
• Subscribe to Mechanics for further updates

Start building today and unlock the full potential of AI-powered applications!