#sre (2020-10)
Prometheus, Prometheus Operator, Grafana, Kubernetes
Archive: https://archive.sweetops.com/monitoring/
2020-10-15
Datadog is pretty expensive and it will be a few months till I get possible progress on this.
Turns out InfluxDb and Elastic cloud have aws marketplace saas subscriptions now.
I’m more familiar with InfluxDB and use telegraf. However cost for elastic is crazy low $130-200 with multinode and 80gb or so storage initially.
Here’s what I’d like to know.
I want to provide AWS compliance inventory list. Some of this is from athena some would be aws level info like from config.
Despite being less familiar I’m think Elastic cloud will be better quick value for some inventory and basic SLO stuff. Agree? Is there any equivalent to an agent less or single agent aws integration that does auto discovery and all like Datadog for example or am I looking at agent installs on every box to get even basic info?
While InfluxDB is my personal favorite in the past and can now query athena too, if Elastic has any aws integration out of the box that saves me time in building some inventory/state based reports (aws config + aws ssm inventory in athena) I’d like to know.
Turns out they have a serverless beats so the integration I was talking about can be done it seems without any server to run agent. Super cool.
2020-10-16
I have an enhanced Cloudwatch Event for changes to IAM Policies. I want to send this to PagerDuty.
I’ve got the event… but need to know best way to send a nice message to pagerduty for this. Do I create an SNS endpoint, send it to there, and then use EventTransformers to turn the message into a structured APIv2 message or what?
I really hate sending a wall of json at folks, so turning a Cloudwatch event into a nicely formatted message with all the context needed would be great, just not sure of the next transformation steps or if something is already automatically able convert this structured input into fields in pagerduty without my effort.
I found they have cloudtrail integration. Not sure yet what level of formatting it applies or if just a json message
Lambda would be good for that use case
2020-10-18
2020-10-20
2020-10-30
anyone have a good query to mimic kubectl top node in grafana/prometheus?