Potential issues with services/replicas?

[danelowe]

I have replicas for some containers which should be internally load-balanced by docker swarm, and allow for rolling updates. For production, I currently have a separate cloudflared container running for each hostname, pointing to the service names (rather than IP address). I'm not sure if cloudflared caches the IP address.

When using hera instead of the cloudflared containers, I'm seeing an issue where upon restarting the hera container, only 2-3 of the 5 expected tunnel config files are being created.

I check the logs and see e.g.:

[INFO] Hera v0.2.5 has started
[INFO] Found certificate: mydomain.co.nz.pem
[INFO] Container found, connecting to 430c25080893...
[INFO] Registering tunnel sub1.mydomain.co.nz
[INFO] Container found, connecting to d805ef7e8d27...
[INFO] Registering tunnel sub2.mydomain.co.nz
[INFO] Container found, connecting to 2f91eca07de6...
[INFO] Container found, connecting to 09e5ccb1465f...
[INFO] Registering tunnel sub3.mydomain.co.nz
[INFO] Container found, connecting to 33ebdf48b35d...
[INFO] Restarting tunnel sub1.mydomain.co.nz
...

And a bunch of other logs from s6.

So it looks like hera is seeing a second container from the same service and deciding to restart the tunnel. Because there's always only one log entry about restarting a tunnel, and its always the last from hera, I'm figuring it also causes hera to stop creating the tunnels.

Quickly looking through the code, I'm also thinking that having services and replicas might lead to more unexpected behaviour, such as removing or restarting tunnels during rolling updates, and pointing to a singular container.

Also, should the service check take into account the health status of the containers?

What are your thoughts on supporting services with replicas?