Add application logging proposals

[ryanjjung]

This PR comes after a discussion in an Infra Team meeting recently in which we discussed defining our logging purpose and parameters. I threw this at the wall, so let's talk about it and see what sticks.

[malini]

this approach looks good to me!

[aatchison]

I have been looking at this and have a few thoughts.

Let's create different categories of policy and consolidate those?

Configuration

• type of data collected
• end to end encryption
• etc

Access

• who is allowed to access
• how do they access
• roles and policies
• how do we audit access

Lifecycle

• retention settings
• where do we aggregate logs and how do we manage them

Auditing

• how do we keep track of what is configured, and where it lives
• how do we ensure new applications are following the polices
• alert on out of spec resources

Implementation

• inventory of resources
• types of resources using logging
• how do we deliver a unified configuration to all services
• etc

[aatchison]

Very verbose, and it is well categorized. Thanks for taking my suggestion!

[ryanjjung]

I redesigned a lot of this to include some level of detail in response to @aatchison's commentary. I've changed the proposal into two files. One is "guidelines" - a set of rules to guide any logging solution we may have to come up with. (What if we launch in Azure? With Kubernetes? Get rich and buy a Splunk license? These things change the specifics of our logging systems, and we will have to come up with new solutions. I wanted a generic set of guidelines to steer those future decisions as well.)

The other is a proposal for how those guidelines can be implemented using AWS services with CloudWatch Logs as the destination. It explains how we should configure encryption, retention, etc. for that particular solution, and how we can retrofit our existing projects to align with the guidelines.

[mzeier]

Please no Splunk. I still have scars.

[aatchison]

We might think about using:

• profile roles for applications that allow Cloudwatch write/least access for the application resources themselves
• read roles with read access to the cloudwatch logs
• user roles such as a read only role for access via the read roles

We could create these roles and resources based on the service and environment names like service-env-fdsaf-Profile for example.

[ryanjjung]

Yeah, my thinking here is to leave the implementation at the Policy level so that you can attach those Policies to any permissions model you want to run. You can do the users -> user groups -> policies model or the policy -> role -> oidc auth model, whatever works for the org, and it's flexible.

[aatchison]

Sure, that makes sense. I'm just advocating for defining both security profiles, the ones emmiting logs to cloud watch and any entity reading them can be separate. That way readers can't modify and apps can't read.

Actually, I would throw in a third role for lambdas or other actions that need to write to clear logs as an example. Some resources require permissions on both ends.