Skip to content

Make recorded test base class behavior configurable per server#1938

Open
hallipr wants to merge 6 commits intomainfrom
users/pahallis/fabric-livetest
Open

Make recorded test base class behavior configurable per server#1938
hallipr wants to merge 6 commits intomainfrom
users/pahallis/fabric-livetest

Conversation

@hallipr
Copy link
Member

@hallipr hallipr commented Mar 5, 2026

What does this PR do?

Generalizes the live test infrastructure to support multiple servers:

  • Added AzureLiveTestSettings and AzureRecordedTestsBase to encapsulate Azure-specific test configuration and common test logic, enabling easier maintenance and extension for other products.
  • Updated test harnesses and fixtures to use generic LiveServerFixture<AzureLiveTestSettings>
  • Added a new test base library for Fabric MCP (Fabric.Mcp.Tests) with its own live test settings class (FabricLiveTestSettings).
  • Refactored tests to remove direct dependencies on McpTestUtilities.GetAzMcpExecutablePath(), replacing it with an override in each server-specific settings class.

GitHub issue number?

Partially addresses #289

Pre-merge Checklist

  • Required for All PRs
    • Read contribution guidelines
    • PR title clearly describes the change
    • Commit history is clean with descriptive messages (cleanup guide)
    • Added comprehensive tests for new/modified functionality
    • Updated servers/Azure.Mcp.Server/CHANGELOG.md and/or servers/Fabric.Mcp.Server/CHANGELOG.md for product changes (features, bug fixes, UI/UX, updated dependencies)
  • For MCP tool changes:
    • One tool per PR: This PR adds or modifies only one MCP tool for faster review cycles
    • Updated servers/Azure.Mcp.Server/README.md and/or servers/Fabric.Mcp.Server/README.md documentation
    • Validate README.md changes using script at eng/scripts/Process-PackageReadMe.ps1. See Package README
    • Updated command list in /servers/Azure.Mcp.Server/docs/azmcp-commands.md and/or /docs/fabric-commands.md
    • Run .\eng\scripts\Update-AzCommandsMetadata.ps1 to update tool metadata in azmcp-commands.md (required for CI)
    • For new or modified tool descriptions, ran ToolDescriptionEvaluator and obtained a score of 0.4 or more and a top 3 ranking for all related test prompts
    • For tools with new names, including new tools or renamed tools, update consolidated-tools.json
    • For new tools associated with Azure services or publicly available tools/APIs/products, add URL to documentation in the PR description
  • Extra steps for Azure MCP Server tool changes:
    • Updated test prompts in /servers/Azure.Mcp.Server/docs/e2eTestPrompts.md
    • 👉 For Community (non-Microsoft team member) PRs:
      • Security review: Reviewed code for security vulnerabilities, malicious code, or suspicious activities before running tests (crypto mining, spam, data exfiltration, etc.)
      • Manual tests run: added comment /azp run mcp - pullrequest - live to run Live Test Pipeline

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR generalizes the live/recorded test infrastructure to support multiple MCP server products (Azure, Fabric, Template) by making the test base classes generic over a settings type parameter. This is in partial support of issue #289 which aims to extract Azure-agnostic types from Azure.Mcp.Core into Microsoft.Mcp.Core.

Changes:

  • Introduced LiveTestSettingsBase abstract class with server-agnostic settings and AzureLiveTestSettings, FabricLiveTestSettings, TemplateLiveTestSettings as server-specific subclasses that each define their own ServerExecutableName.
  • Made CommandTestsBase, RecordedCommandTestsBase, and LiveServerFixture generic over TSettings : LiveTestSettingsBase, and created AzureRecordedTestsBase to encapsulate Azure-specific sanitizer logic.
  • Migrated all ~40+ existing test files to use the new generic types and removed the old LiveTestSettings, LiveTestSettingsFixture, and McpTestUtilities.GetAzMcpExecutablePath().

Reviewed changes

Copilot reviewed 76 out of 76 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
core/Microsoft.Mcp.Core/tests/.../LiveTestSettingsBase.cs New abstract base class for server-agnostic test settings
core/Microsoft.Mcp.Core/tests/.../LiveTestSettings.cs Deleted old non-generic settings class
core/Microsoft.Mcp.Core/tests/.../LiveTestSettingsFixture.cs Deleted old fixture class (logic moved into LiveTestSettingsBase.LoadTestSettingsAsync)
core/Microsoft.Mcp.Core/tests/.../CommandTestsBase.cs Made generic over TSettings, removed old PlaybackSettings and TryLoadLiveSettingsAsync
core/Microsoft.Mcp.Core/tests/.../RecordedCommandTestsBase.cs Made generic over TSettings, moved default sanitizers to virtual PopulateDefaultSanitizers
core/Microsoft.Mcp.Core/tests/.../LiveServerFixture.cs Made generic over TSettings, uses Settings.GetMcpExecutablePath()
core/Microsoft.Mcp.Core/tests/.../McpTestUtilities.cs Removed GetAzMcpExecutablePath(), made params non-optional
core/Azure.Mcp.Core/tests/.../AzureLiveTestSettings.cs New Azure-specific settings class
core/Azure.Mcp.Core/tests/.../AzureRecordedTestsBase.cs New Azure-specific recorded test base with Azure sanitizers
core/Fabric.Mcp.Core/tests/.../FabricLiveTestSettings.cs New Fabric-specific settings class
core/Fabric.Mcp.Core/tests/.../Fabric.Mcp.Tests.csproj New test library project for Fabric
core/Template.Mcp.Core/tests/.../TemplateLiveTestSettings.cs New Template-specific settings class
core/Template.Mcp.Core/tests/.../Template.Mcp.Tests.csproj New test library project for Template
Microsoft.Mcp.slnx Added Fabric and Template test projects
servers/Fabric.Mcp.Server/Fabric.Mcp.Server.slnx Added Fabric test project
servers/Template.Mcp.Server/Template.Mcp.Server.slnx Added Template test project
servers/Azure.Mcp.Server/docs/new-command.md Updated assembly attribute namespace
core/Microsoft.Mcp.Core/.gitignore Removed (moved to more specific location)
core/Microsoft.Mcp.Core/tests/Microsoft.Mcp.Tests/.gitignore New gitignore for Generated directory
~40 test files across tools/* Mechanical migration to AzureRecordedTestsBase/CommandTestsBase<AzureLiveTestSettings>

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copy link
Contributor

@scbedd scbedd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 nit. Check a livetest run of a couple sample ones manually before merge.

@hallipr
Copy link
Member Author

hallipr commented Mar 5, 2026

/azp run mcp - pullrequest - live

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

hallipr and others added 6 commits March 9, 2026 11:31
…rs/LiveTestSettingsBase.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…stSettings.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…rs/LiveTestSettingsBase.cs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Untriaged

Development

Successfully merging this pull request may close these issues.

3 participants