Health Check notification - Send failed

[naji-abdulla]

Consistently see healthcheck notification failed to send in some of the deployments

Client has the following errors in the log

E0301 16:41:25.941121  157041 core.go:616] Send failed 1 times: read unix @->/var/run/seesaw/engine/engine.sock?backlog=8192: i/o timeout
..........
E0301 16:45:41.944832  157041 core.go:616] Send failed 9 times: read unix @->/var/run/seesaw/engine/engine.sock?backlog=8192: i/o timeout
E0301 16:46:13.945305  157041 core.go:616] Send failed 10 times: read unix @->/var/run/seesaw/engine/engine.sock?backlog=8192: i/o timeout

Server has the /var/run/seesaw/engine/engine.sock connections are piling up

ss | grep engine.sock | wc -l
15905

Any pointers to why this error happening will be helpful

[hazaelsan]

Those errors happen when the engine didn't respond quickly enough (10 seconds). The fact the last line shows this failed 10 consecutive times is a bit concerning.

A few questions:

• What version are you running?
• Was the load particularly high at that time?
• Was there some network event? (e.g., convergence)
• Anything useful in the engine logs?

[naji-abdulla]

This turned out to be because when the server call back returns an error to RPC machinery , RPC connection seems to be timing out.