feat(helm): add metrics endpoint and ServiceMonitor support

[optimus-fulcria]

Summary

Closes #1369

Adds first-class Helm values for enabling the controller metrics endpoint and optionally deploying a Prometheus Operator ServiceMonitor.

Changes

• values.yaml: New controller.metrics section with enabled, port, secure, and serviceMonitor sub-values
• controller-configmap.yaml: Sets METRICS_BIND_ADDRESS and METRICS_SECURE env vars when metrics are enabled
• controller-deployment.yaml: Adds metrics container port when enabled
• controller-service.yaml: Adds metrics service port when enabled
• controller-servicemonitor.yaml (new): Optional ServiceMonitor with configurable labels, interval, scrapeTimeout, and relabeling

Usage

controller:
  metrics:
    enabled: true
    port: 9093
    secure: false
    serviceMonitor:
      enabled: true
      additionalLabels:
        release: prometheus
      interval: 30s
      scrapeTimeout: 10s

Test plan

• Helm unit tests added for deployment, service, and ServiceMonitor templates
• Verified default values don't change existing behavior (metrics disabled by default)
• Deploy to Kind cluster with metrics enabled and verify Prometheus scraping

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds first-class Helm configuration for exposing the controller’s Prometheus metrics endpoint and optionally deploying a Prometheus Operator ServiceMonitor to scrape it (closes #1369).

Changes:

• Introduces controller.metrics values (enablement, port, secure mode, and serviceMonitor config).
• Updates controller Deployment/Service/ConfigMap templates to expose the metrics port and set METRICS_* env vars when enabled.
• Adds a new controller-servicemonitor.yaml template plus Helm unit tests for the new behavior.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
helm/kagent/values.yaml	Adds controller.metrics and controller.metrics.serviceMonitor values to configure metrics + ServiceMonitor.
helm/kagent/templates/controller-configmap.yaml	Conditionally sets METRICS_BIND_ADDRESS / METRICS_SECURE when metrics are enabled.
helm/kagent/templates/controller-deployment.yaml	Conditionally exposes a metrics container port.
helm/kagent/templates/controller-service.yaml	Conditionally exposes a metrics Service port.
helm/kagent/templates/controller-servicemonitor.yaml	Adds optional ServiceMonitor resource for Prometheus Operator scraping.
helm/kagent/tests/controller-deployment_test.yaml	Adds unit tests for deployment port + configmap env vars behavior.
helm/kagent/tests/controller-service_test.yaml	Adds unit tests for conditional Service metrics port exposure.
helm/kagent/tests/controller-servicemonitor_test.yaml	Adds unit tests for ServiceMonitor rendering and configuration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

ServiceMonitor rendering is only gated by values. This chart already guards optional CRD-based resources with .Capabilities.APIVersions.Has (e.g., openshift-route.yaml) to avoid install/upgrade failures when the CRD is not installed. Consider also gating this template on .Capabilities.APIVersions.Has "monitoring.coreos.com/v1" so controller.metrics.serviceMonitor.enabled=true doesn’t hard-fail on clusters without Prometheus Operator CRDs.

Suggested change

	{{- if and .Values.controller.metrics.enabled .Values.controller.metrics.serviceMonitor.enabled }}
	{{- if and .Values.controller.metrics.enabled .Values.controller.metrics.serviceMonitor.enabled (.Capabilities.APIVersions.Has "monitoring.coreos.com/v1") }}

[Copilot]

When controller.metrics.secure is true, the controller enables controller-runtime’s authn/authz filter on the metrics endpoint. This ServiceMonitor config sets scheme: https but does not configure any bearer token/auth settings, so Prometheus scrapes are likely to be rejected. Add an endpoint auth configuration (e.g., bearerTokenFile) when secure metrics are enabled (and/or allow configuring the full endpoint auth block via values).

Suggested change

	insecureSkipVerify: true
	{{- end }}
	insecureSkipVerify: true
	{{- with .Values.controller.metrics.serviceMonitor.auth }}
	{{- toYaml . | nindent 6 }}
	{{- else }}
	bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
	{{- end }}
	{{- end }}

[EItanya]

Do we need to expose these on the service? The controller service can potentially be a LoadBalancer because of the MCP + A2A server so I think metrics should either be scraped from the pod directly, or there should be a different service

[optimus-fulcria]

Good point. Moved the metrics port to a dedicated ClusterIP service (controller-metrics-service.yaml) so it stays internal regardless of the main service type. Updated the ServiceMonitor selector to target the new metrics service.