staging clusters to use envoy gateway api

[anish-mudaraddi]

contingent on this PR being merged: stfc/cloud-helm-charts#188

This deploys envoy gateway onto our staging clusters. Using the charts that will be released from the above PR.

changelog:

• setup envoy gateways service - which will deploy envoyproxy and setup 2 gateways for internal and user-facing services on clusters. Deprecate ingress nginx

• argocd to use envoy gateway and backendtlspolicy for full encryption for external traffic. ArgoCD to use an internal TLS certificate generated by cert-manager for encrpyting east-west traffic

• use cert-manager chart to setup clusterissuers on staging clusters. For worker, this will setup a letsencrypt issuer that can make http01 requests using gateway api. This cert-manager will also setup an internal TLS CA to sign internal certs for encrypting east-west traffic

• setup a separate monitoring stack and setup httproute with basicauth securitypolicy to match ingress annotations, deprecate addon method of installation

• setup httproute for longhorn

• Setup httproute for Harbor, setup internal TLS certs and backendTLSPolicy for encrypting east-west traffic gateway->pod and pod->pod

• Setup httproute for docker registry, use internal TLS and backendTLSPolicy for encrypting east-west traffic gateway->pod

• Setup httproute, securitypolicies and internalTLS for opensearch - managed via cert-manager and signed with internal CA

• Deprecate materials-galaxy and victoria metrics and remove those services from staging as we don't use them

[jacob-ward]

Approved but definitely requires a second reviewer

General comment that this should definitely have been split into multiple commits, a lot of the changes are quite simple but still.

If each item in your changelog in your commit message had been it's own commit this would have worked a bit better in my view