Looking into Azure AI Foundry: Building a Smart Response System

Over the past few weeks, I had the opportunity to dive deep into Azure AI Foundry and work on poc that combined prompt flow, large language models (LLMs), and database indexing. The idea was to build a system that evaluates different LLMs, retrieves relevant information from a database, selects the right email template, and generates a personalized email response. 

Azure AI Foundry

Azure AI Foundry is a powerful platform designed to simplify the development of AI applications by orchestrating flows involving LLMs, prompts, and Python tools. The best thing about the AI foundry is the visualized graph-based interface for creating flows and its ability to test and debug seamlessly. After setting up my project in the Azure AI Foundry portal, the major part was the setting of Prompt flows.

Designing the Prompt Flow

The first step was to create a Prompt Flow that could evaluate multiple LLMs based on specific input parameters. Here’s how I structured it:

Input Parameters: The flow began by taking user inputs such as query type, historical data context, and additional metadata.

LLM Evaluation: Using Azure OpenAI GPT models, I evaluated several LLMs for their performance on these inputs. This step involved crafting multiple prompt variants using Jinja templating and comparing their outputs.

Index Lookup: Once the best-performing model was selected, I integrated an Index Lookup tool to query a vector database for relevant historical data.

Template Selection: Based on the retrieved data, the system dynamically chose one of several pre-uploaded email templates.

Database Indexing & Retrieval

The Index Lookup tool in Azure AI Foundry made it easy to search through my vector database for relevant results. This tool uses embeddings generated by LLMs to find the most contextually appropriate matches for a given query.

For example:

If the input query was related to customer feedback, the system would retrieve historical feedback records.

For support-related queries, it would fetch relevant support ticket summaries.

This indexing mechanism ensured that every email response was grounded in accurate and relevant data.

Generating the Response

Once the right template was selected, I used the chosen LLM to fill in placeholders in the template with dynamic content. The final email response was not only accurate but also personalized based on historical interactions.

For instance:

A customer asking about delayed shipping would receive an email referencing their previous order details.

A user requesting technical support would get an email tailored to their issue history.

This seamless integration of templates with real-time data retrieval made the system highly effective.

There main areas to look into when turning the POC into a proper use case.. 

1. LLM Evaluation: Comparing multiple LLMs required careful tuning of prompts and iterative testing. What was frustrating at times was that different LLM gave different results for the same query.

2. Data Integration: Ensuring that database indexing worked smoothly with diverse datasets took some effort.

3. Template Management: Designing flexible templates that could adapt to various contexts required creativity. Since I had limited data this impacted the output result.

Final Thoughts

Azure AI Foundry I think can revolutionize workflows by combining retrieval-augmented generation (RAG) techniques with powerful LLMs. The ability to evaluate models dynamically, retrieve relevant data efficiently, and generate personalized outputs has immense potential across different industries —from customer support to marketing automation.