The Microsoft.Extensions.AI.Evaluation libraries

Microsoft.Extensions.AI.Evaluation is a set of .NET libraries defined in the following NuGet packages that have been designed to work together to support building processes for evaluating the quality of AI software.

• Microsoft.Extensions.AI.Evaluation - Defines core abstractions and types for supporting evaluation.
• Microsoft.Extensions.AI.Evaluation.Quality - Contains evaluators that can be used to evaluate the quality of AI responses in your projects including Relevance, Truth, Completeness, Fluency, Coherence, Retrieval, Equivalence and Groundedness.
• Microsoft.Extensions.AI.Evaluation.Safety - Contains a set of evaluators that are built atop the Azure AI Foundry Evaluation service that can be used to evaluate the content safety of AI responses in your projects including Protected Material, Groundedness Pro, Ungrounded Attributes, Hate and Unfairness, Self Harm, Violence, Sexual, Code Vulnerability and Indirect Attack.
• Microsoft.Extensions.AI.Evaluation.Reporting - Contains support for caching LLM responses, storing the results of evaluations and generating reports from that data.
• Microsoft.Extensions.AI.Evaluation.Reporting.Azure - Supports the Microsoft.Extensions.AI.Evaluation.Reporting library with an implementation for caching LLM responses and storing the evaluation results in an Azure Storage container.
• Microsoft.Extensions.AI.Evaluation.Console - A command line dotnet tool for generating reports and managing evaluation data.

Install the packages

From the command-line:

dotnet add package Microsoft.Extensions.AI.Evaluation
dotnet add package Microsoft.Extensions.AI.Evaluation.Quality
dotnet add package Microsoft.Extensions.AI.Evaluation.Safety
dotnet add package Microsoft.Extensions.AI.Evaluation.Reporting

Or directly in the C# project file:

<ItemGroup>
  <PackageReference Include="Microsoft.Extensions.AI.Evaluation" Version="[CURRENTVERSION]" />
  <PackageReference Include="Microsoft.Extensions.AI.Evaluation.Quality" Version="[CURRENTVERSION]" />
  <PackageReference Include="Microsoft.Extensions.AI.Evaluation.Safety" Version="[CURRENTVERSION]" />
  <PackageReference Include="Microsoft.Extensions.AI.Evaluation.Reporting" Version="[CURRENTVERSION]" />
</ItemGroup>

You can optionally add the Microsoft.Extensions.AI.Evaluation.Reporting.Azure package in either of these places if you need Azure Storage support.

Install the command line tool

dotnet tool install Microsoft.Extensions.AI.Evaluation.Console --create-manifest-if-needed

Usage Examples

For a comprehensive tour of all the functionality, concepts and APIs available in the Microsoft.Extensions.AI.Evaluation libraries, check out the API Usage Examples available in the dotnet/ai-samples repo. These examples are structured as a collection of unit tests. Each unit test showcases a specific concept or API, and builds on the concepts and APIs showcased in previous unit tests.

Feedback & Contributing

We welcome feedback and contributions in our GitHub repo.