Developer Module Development

Module Development

This guide explains how to write a new agent connector module or judge (testing) module for mate.

Module Tiers

Every module exposes a ModuleTier Tier { get; } property (defined in mate.Domain.Contracts.Modules.ModuleTier). The value is shown as a badge on the Settings → Modules card and signals licensing intent:

Value	Meaning
`ModuleTier.Free`	No additional licence — available in every installation
`ModuleTier.Premium`	Premium module — fully functional, but commercial deployment requires a valid licence

Current tier assignments

Module category	Tier
All agent connector modules	Free
All judge / testing modules	Free
All question-generation modules	Free
All monitoring modules	Free
`mate.Modules.RedTeaming.Generic`	Premium

When writing a new module, always declare the tier explicitly:

public ModuleTier Tier => ModuleTier.Free;   // or ModuleTier.Premium

Module Contracts

All module types live in src/Core/mate.Domain/Contracts/. Every module must implement the common base members plus its type-specific interface.

Common Members (all module types)

string ModuleId { get; }          // unique snake_case identifier, e.g. "copilot_studio"
string DisplayName { get; }        // human-readable name shown in UI
ModuleTier Tier { get; }           // Free or Premium (see Module Tiers above)
List<ConfigFieldDefinition> ConfigSchema { get; }  // drives the dynamic config form
Task<bool> IsHealthy();            // self-check — called by /health/modules
Task<ValidationResult> ValidateConfig(Dictionary<string,string> config);
List<string> GetCapabilities();    // optional feature flags

`ConfigFieldDefinition`

public record ConfigFieldDefinition
{
    public string Key { get; init; }           // config dictionary key
    public string Label { get; init; }         // form label
    public string? Description { get; init; }
    public bool IsRequired { get; init; }
    public bool IsSecret { get; init; }        // masked in UI, stored encrypted
    public string? DefaultValue { get; init; }
    public List<string>? SelectOptions { get; init; }  // null = free text
}

Writing an Agent Connector Module

Agent connectors implement IAgentConnectorModule (in mate.Domain.Contracts).

1. Create the project

src/Modules/AgentConnectors/mate.Modules.AgentConnector.MyConnector/
    mate.Modules.AgentConnector.MyConnector.csproj
    MyConnectorModule.cs
    MyConnector.cs

Add a <ProjectReference> to mate.Domain and mate.Core.

2. Implement `IAgentConnectorModule`

public class MyConnectorModule : IAgentConnectorModule
{
    public string ModuleId => "my_connector";
    public string DisplayName => "My Connector";

    public List<ConfigFieldDefinition> ConfigSchema => new()
    {
        new() { Key = "ApiUrl",    Label = "API URL",  IsRequired = true },
        new() { Key = "ApiKey",    Label = "API Key",  IsRequired = true, IsSecret = true },
    };

    public Task<bool> IsHealthy()
    {
        // attempt a lightweight API call; return false on failure
    }

    public Task<ValidationResult> ValidateConfig(Dictionary<string, string> config)
    {
        // check required keys are present and well-formed
    }

    public IAgentConnector CreateConnector(Dictionary<string, string> config)
    {
        return new MyConnector(config["ApiUrl"], config["ApiKey"]);
    }

    public List<string> GetCapabilities() => new() { "text" };
}

3. Implement `IAgentConnector`

public class MyConnector : IAgentConnector
{
    public Task<string> StartConversationAsync();
    public Task<string> SendMessageAsync(string conversationId, string message);
    public Task EndConversationAsync(string conversationId);
}

4. Register in DI

In src/Host/mate.WebUI/Program.cs, inside the module registration block:

builder.Services.AddSingleton<IAgentConnectorModule, MyConnectorModule>();

Also add a <ProjectReference> to mate.Modules.AgentConnector.MyConnector in mate.WebUI.csproj.

Writing a Judge (Testing) Module

Judge modules implement ITestingModule and optionally IJudgeProvider.

1. Create the project

src/Modules/Testing/mate.Modules.Testing.MyJudge/
    mate.Modules.Testing.MyJudge.csproj
    MyJudgeModule.cs

2. Implement `ITestingModule`

public class MyJudgeModule : ITestingModule
{
    public string ModuleId => "my_judge";
    public string DisplayName => "My Judge";
    public string ProviderType => "MyJudge";  // used to filter in UI

    public List<ConfigFieldDefinition> ConfigSchema => new()
    {
        new() { Key = "PassThreshold", Label = "Pass Threshold", DefaultValue = "0.7" },
    };

    public Task<bool> IsHealthy() => Task.FromResult(true);
    public Task<ValidationResult> ValidateConfig(Dictionary<string, string> config) => ...;
    public List<string> GetCapabilities() => new() { "verdict", "score" };

    public Task<JudgeResult> EvaluateAsync(JudgeRequest request)
    {
        // score the response and return a JudgeResult with Passed, Score, Rationale
    }
}

`JudgeRequest` and rubric criteria

JudgeRequest carries all the context a judge needs per test case:

public record JudgeRequest
{
    public string Input { get; init; }             // the user message sent to the agent
    public string ExpectedAnswer { get; init; }    // the reference/golden answer
    public string ActualResponse { get; init; }    // the agent's actual response
    public string? SystemPrompt { get; init; }     // judge system prompt from settings
    public double PassThreshold { get; init; }     // from judge setting
    public IReadOnlyList<EvaluationCriterion> RubricCriteria { get; init; }  // see below
    // ... other fields
}

RubricCriteria is populated by TestExecutionService.LoadRubricCriteriaAsync at run start — it queries all non-draft RubricSet entities whose JudgeSettingId matches the active judge and combines their criteria in sort order. Your judge module receives the full list and is responsible for evaluating each one.

public record EvaluationCriterion(
    string Name,
    string EvaluationType,   // Contains | NotContains | Regex | Custom | Prompt
    string? Pattern,         // match value or Custom subtype (e.g. "NonEmpty")
    double Weight,
    bool IsMandatory
);

Important: If your judge has built-in default criteria (as CopilotStudioJudge does), always append request.RubricCriteria after your defaults — never replace them:

var criteria = MyJudgeDefaultCriteria
    .Concat(request.RubricCriteria)
    .ToList();

This ensures custom rubric sets extend rather than override built-in safety checks.

3. Register in DI

builder.Services.AddSingleton<ITestingModule, MyJudgeModule>();
builder.Services.AddSingleton<IJudgeProvider, MyJudgeModule>();

Writing a Red Teaming Module

Red-team modules implement IRedTeamModule and IAttackProvider (both in mate.Domain.Contracts.RedTeaming). They are architecturally independent from ITestingModule — red-teaming is adversarial security testing, not functional quality testing.

1. Create the project

src/Modules/RedTeaming/mate.Modules.RedTeaming.MyAttacker/
    mate.Modules.RedTeaming.MyAttacker.csproj
    MyAttackerModule.cs

Add <ProjectReference> to mate.Domain and mate.Core.

2. Implement `IAttackProvider`

public class MyAttackProvider : IAttackProvider
{
    public string ProviderType => "MyAttacker";

    public async Task<IReadOnlyList<AttackProbe>> GenerateProbesAsync(
        AttackRequest request, CancellationToken ct = default)
    {
        // generate adversarial prompts — use an LLM, a catalogue, or static templates
        // filter by request.Categories, respect request.NumberOfProbes
    }

    public async Task<RedTeamFinding?> EvaluateResponseAsync(
        AttackProbe probe, string agentResponse, CancellationToken ct = default)
    {
        // return null when the agent safely refused the probe
        // return a RedTeamFinding with Risk, Rationale, ReproductionSteps, Mitigations otherwise
    }
}

3. Implement `IRedTeamModule`

public class MyAttackerModule : IRedTeamModule
{
    public string ModuleId     => "my_attacker";
    public string DisplayName  => "My Attacker";
    public string ProviderType => "MyAttacker";

    public List<ConfigFieldDefinition> ConfigSchema => new()
    {
        new() { Key = "Endpoint", Label = "LLM Endpoint", IsRequired = true },
        new() { Key = "ApiKey",   Label = "API Key",       IsRequired = true, IsSecret = true },
    };

    public Task<bool>             IsHealthy()                                        => ...;
    public Task<ValidationResult> ValidateConfig(Dictionary<string, string> config)  => ...;
    public IReadOnlyList<string>  GetCapabilities()                                  =>
        ["prompt_injection", "jailbreak"];

    public void RegisterServices(IServiceCollection services, IConfiguration config)
    {
        services.AddSingleton<IAttackProvider, MyAttackProvider>();
        services.AddSingleton<IRedTeamModule, MyAttackerModule>();
    }
}

4. DI extension (recommended pattern)

public static class MyAttackerModuleExtensions
{
    public static IServiceCollection AddmateMyAttacker(
        this IServiceCollection services, IConfiguration config)
    {
        services.AddSingleton<IAttackProvider, MyAttackProvider>();
        services.AddSingleton<IRedTeamModule, MyAttackerModule>();
        return services;
    }
}

5. Register in `Program.cs`

// ── Red-teaming modules ──────────────────────────────────────────────────────
builder.Services.AddmateMyAttacker(config);

Also add a <ProjectReference> in mate.WebUI.csproj. No registry or UI change is needed — the module is auto-discovered and shown in Settings → Modules → Red Teaming Modules.

Key DTOs

Type	Role
`AttackRequest`	Input to `GenerateProbesAsync` — agent description, categories filter, probe count, domain hint, resolved credentials
`AttackProbe`	One adversarial prompt with category, optional `FailureSignature`, and rationale
`RedTeamFinding`	Confirmed vulnerability — probe, agent response, `RiskLevel`, rationale, reproduction steps, mitigations
`RedTeamReport`	Aggregated findings for a complete red-team run

Risk levels

Assign RiskLevel in EvaluateResponseAsync based on potential real-world impact:

Risk	Examples
Critical	Toxic/harmful content generation, successful jailbreak enabling illegal instructions
High	Prompt injection, data exfiltration, successful privacy leak
Medium	System-prompt leak, role confusion
Low	Minor hallucination induction, incomplete refusal

Module Discovery

mateModuleRegistry is populated at startup in Program.cs by resolving all registered IAgentConnectorModule, ITestingModule, IJudgeProvider, IMonitoringModule, and IRedTeamModule instances from DI. The WebUI reads the registry to build the Wizard step 1 list, the Settings Modules tab (including the Red Teaming Modules section), and health check pages.

No manual registry entry is needed — registering in DI is sufficient.

Naming Convention

Project names follow the pattern:

mate.Modules.<Category>.<Name>

Examples: mate.Modules.AgentConnector.CopilotStudio, mate.Modules.Testing.HybridJudge, mate.Modules.Auth.EntraId, mate.Modules.RedTeaming.Generic.

Use singular form (AgentConnector, not AgentConnectors; RedTeaming as the folder/category name).

Testing Your Module

Add unit tests in tests/mate.Tests.Unit/ following the existing pattern in tests/mate.Tests.Unit/Testing/.

At minimum, test:

ConfigSchema — all required fields are present
ValidateConfig — rejects missing required fields
IsHealthy — returns false when the endpoint is unreachable (mock HTTP)
Core evaluation logic

mate

Home

Uh oh!

Developer Module Development

Module Development

Module Tiers

Current tier assignments

Module Contracts

Common Members (all module types)

ConfigFieldDefinition

Writing an Agent Connector Module

1. Create the project

2. Implement IAgentConnectorModule

3. Implement IAgentConnector

4. Register in DI

Writing a Judge (Testing) Module

1. Create the project

2. Implement ITestingModule

JudgeRequest and rubric criteria

3. Register in DI

Writing a Red Teaming Module

1. Create the project

2. Implement IAttackProvider

3. Implement IRedTeamModule

4. DI extension (recommended pattern)

5. Register in Program.cs

Key DTOs

Risk levels

Module Discovery

Naming Convention

Testing Your Module

Further Reading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mate

User Documentation

Developer Documentation

Clone this wiki locally

`ConfigFieldDefinition`

2. Implement `IAgentConnectorModule`

3. Implement `IAgentConnector`

2. Implement `ITestingModule`

`JudgeRequest` and rubric criteria

2. Implement `IAttackProvider`

3. Implement `IRedTeamModule`

5. Register in `Program.cs`