Add TCP startup checks

[vrutkovs]

When TLS / mTLS is enabled existing healthchecks would fail as they expect the response to be plain text.

After a discussion with the team we've figures out that the healthchecks need to be reworked:

• These are not checking if the application is running, they are checking whether web server is running. As a result, these can be checked via TCP probe
• The checks are not updated throughout the lifecycle of the app, so instead of liveness / readiness check these can be converted into startupProbe

Plan:

• Document current workaround for TLS users - custom startup probe, unset liveness/readiness checks
• Add TCP probe startup checks
• Announce deprecation in liveness/readiness checks
• After 3 releases liveness / readiness checks will be removed - this will cause pod rollout
• Document healthcheck strategy

---

Original issue

Is your feature request related to a problem? Please describe

With mTLS enabled in k8s environment kubelet has to have a valid certificate for liveness / readiness checks. This makes pod setup cumbersome

Describe the solution you'd like

Provide an unauthenticated /healthz endpoint, which doesn't require mTLS or TLS at all

Describe alternatives you've considered

Running a heathcheck via a command in the pod
Operator instrumenting a pod with correct certificates

Additional information

No response

[vrutkovs]

cc @valyala as it need to be implemented for VM / VT as well

[vadimalekseev]

Provide an unauthenticated /healthz endpoint, which doesn't require mTLS or TLS at all

@vrutkovs it is not easy to expose only a single endpoint for access without TLS. A workaround is to launch an additional HTTP server without authorization. See the example below:

./victoria-logs -httpListenAddr :9428 \
  -tls -tlsKeyFile ./key.pem -tlsCertFile ./crt.pem \
  -httpListenAddr :8080 -tls=false

-- this command will start VictoriaLogs, which will start two servers: the first on port 9428 with TLS, and the second on port 8080 without it.

For the HTTP server serving requests without TLS and authorization, it makes sense to restrict the listening interfaces, for example: -httpListenAddr 100.100.100.1:9428.

But there is an risk in this setup: if the listening interface is restricted incorrectly, the HTTP server might serve requests without authorization. To avoid this, we can provide a new flag to the HTTP server so that it only exposes debug endpoints such as /debug/pprof, /health, /metrics, without the ability to process application-specific requests.

[vrutkovs]

Discussed this over the meeting: our current healthz checks are set on webserver setup, so these are equivalent to a startup probe. We'll update the operator to use TCP checks, so that both plain text and TLS servers would be covered

[vrutkovs]

Note that with TCP checks these checks will be logged as TLS handshake error from <kubelet IP>, so the solution is not ideal

To avoid this, we can provide a new flag to the HTTP server so that it only exposes debug endpoints such as /debug/pprof, /health, /metrics, without the ability to process application-specific requests.

I'll open a new issue with a request to implement it

[sudharsans]

@vrutkovs, I would like to work on this. Here is the PR #1966

Let me know if this works or if there's anything I need to check.

Thanks!