LLMOps with Azure Machine Learning prompt flow

LLMOps with
Azure Machine Learning
prompt flow
SATO Naoki (Neo)
Senior Software Engineer, Microsoft

Access to thousands of
LLMs from OpenAI,
Meta, Hugging Face
Azure Machine Learning for Generative AI
Prompt engineering/
evaluation
Built-in safety and
responsible AI
Continuous monitoring
for LLMs
Purpose-built AI
infrastructure

The paradigm shift (MLOps vs LLMOps)
Traditional MLOps LLMOps
Target audiences
Assets to share
Metrics/evaluations
ML models
ML Engineers
Data Scientists
ML Engineers
App developers
Model, data,
environments, features
LLM, agents, plugins,
prompts, chains, APIs
Accuracy
Accuracy, fairness,
groundedness, relevance,
coherence
Build from scratch Pre-built, fine-tune

Operationalize LLM
app development
with prompt flow
LLMOps is a complex process.
Customers want:
• Private data access and controls
• Prompt engineering
• CI/CD
• Iterative experimentation
• Versioning and reproducibility
• Deployment and optimization
• Safe and Responsible AI
Design and development
Develop flow based on prompt
to extend the capability
Debug, run, and evaluate
flow with small data
Modify flow (prompts and
tools etc.)
No If satisfied
Yes
Evaluation and refinement
No
Evaluate flow against large
dataset with different metrics
(quality, relevance, safety, etc.)
If satisfied
Yes
Optimization and production
Optimize flow
Deploy and
monitor flow
Get end user
feedback

Streamline prompt engineering projects
Azure Machine Learning
prompt flow
Customer Benefits
• Create AI workflows that connect various language models,
APIs, and data sources to ground LLMs on your data.
• One platform to design, construct, tune, evaluate, test, and
deploy LLM workflows
• Evaluate the quality of workflows with rich set of pre-built
metrics and safety system.
• Easy prompt tuning, comparison of prompt variants, and
version-control.
Documentation: https://aka.ms/prompt_flow

Azure Machine Learning prompt flow (1/7)
Capabilities Overview
• Develop workflows
• Develop flows that connect to
various language models, external data sources,
tools, and custom code
• Test and evaluate
• Test flows with large datasets in parallel
• Evaluate the AI quality of the workflows with
metrics like performance, groundedness, and
accuracy
• Prompt tuning
• Easily tune prompts with variants and versions
• Compare and deploy
• Visually compare across experiments
• One-click deploy to a managed endpoint for
rapid integration

Prompt flow authoring
Develop your LLM flow from scratch
• Construct a flow using pre-built tools
• Support custom code
• Clone flows from samples
• Track run history

Connections
Manage APIs and external data sources
• Seamless integration with pre-built LLMs like
Azure OpenAI Service
• Built-in safety system with Azure AI Content
Safety
• Effectively manage credentials or secrets for
APIs
• Create your own connections in Python tools

Variants
• Create dynamic prompts using external
data and few shot samples
• Edit your complex prompts in full screen
• Quickly tune prompt and LLM
configuration with variants

Evaluation
• Evaluate flow performance with your own
data
• Use pre-built evaluation flows
• Build your own custom evaluation flows
Tune Variant 0
Tune Variant 1
Tune Variant 2
Flow variants
Evaluation
Bulk Test

Evaluation
• Compare multiple variants or runs to pick
best flow
• Add new evaluations to a finished run
• Ensure accuracy by scaling the size of data
in evaluation
Tune Variant 0
Tune Variant 1
Tune Variant 2
Flow variants
Evaluation
Bulk Test

Deploy
• Seamless transition from development to
production with AzureML’s managed online
endpoints
Production
Tune Variant 0
Tune Variant 1
Tune Variant 2
Flow variants
Test App

What is prompt flow code experience ？
Use code to define flow
File based flow, organized in a well-defined folder structure
Support CLI/SDK
Smooth transition between cloud and local
Download flow to local, import flow to cloud
Develop, test, debug, deploy on local
Submit run from local to cloud
From local deploy to cloud
Manage runs/evaluation in cloud
Integrate with your CI/CD automation
SDK/CLI to init, execute, evaluate, visualize flow and metrics
VS Code Extension
Flow editor
Local connection management
Run history
Collaboration
or share cross
workspace
Submit flow
runs to cloud
from your repo
(anywhere)

Demo - Azure Machine Learning prompt flow
1. Upload PDF files and create a vector search index in Azure AI Search
2. Create a new chat flow in prompt flow
3. Configure Azure OpenAI Service (LLM) and Azure AI Search (vector search)
4. Run the chat flow
5. Evaluate the chat flow
6. Deploy the chat flow as a REST API

LLMOps with Azure Machine Learning prompt flow

Learn More
• What is Azure Machine Learning prompt flow - Azure Machine Learning |
Microsoft Learn
• Prompt flow — Prompt flow documentation (microsoft.github.io)

LLMOps with Azure Machine Learning prompt flow

Recommended

More Related Content

What's hot (20)

Similar to LLMOps with Azure Machine Learning prompt flow (20)

More from Naoki (Neo) SATO (20)

Recently uploaded (20)

LLMOps with Azure Machine Learning prompt flow