Update the Cloudwatch Logs proposal

[ryanjjung]

I did a slightly deeper dive into our use cases and made a couple slight adjustments to the CloudWatch Logs proposal. A few facts that led to these adjustments:

• Fargate services that log to CloudWatch automatically use the container ID as part of the log stream name, and you must use the awslogs-stream-prefix option when defining the Fargate LogConfiguration. There is no way to configure this to use pre-created log streams.
• Stalwart servers (prior to the move to Fargate containers) will use a local installation of fluent-bit to forward the logs to CloudWatch. That plugin allows for both kinds of configuration, one where you specify a log stream as well as one where you specify a prefix.
• Log groups and log streams are both organizational units of sorts, and we can be a lot more terse in our naming than what I originally wrote up.

[aatchison]

Thanks for digging in!

[ryanjjung]

There was some talk about using the CloudWatch Agent instead of a fluent-bit installation. If this is do-able (extracting logs from Docker/systemd and shipping straight to CloudWatch Logs with that agent), then that's a step easier and we should do that. I'll do a little more research and update the proposal.

[ryanjjung]

I'm not sure the CloudWatch Agent will do the trick for us.

The main blocker will be that it only supports shipping logs out of files (it's hard to link to the specifics in this document, but you can look under the "CloudWatch agent configuration file: Logs section" part to see that this is the only ingestion option). Stalwart runs in Docker containers controlled by a systemd service unit. Its logs go into the system journal, which is not a plaintext file. In order to get at these logs, we either have to do a docker logs (or equivalent) or a journalctl command of some kind. fluent-bit has a plugin that does that for us.

The CloudWatch Agent is not installed by default (this doc has installation instructions, and I tested on a bastion server to see that it is not pre-installed with Amazon Linux). So getting it on the system would be the same process we already have a half-baked solution of for fluent-bit: install a package, slap a config file in place, write a systemd unit file to run the right agent invocation, enable and start the service.

But even if we did that, we wouldn't be able to capture this log for the reasons mentioned above. I think we ought to leave it at fluent-bit.