#prometheus (2019-06)

Has anyone here used https://github.com/weaveworks/prom-aggregation-gateway for aggregating metrics from Lambda functions? Curious if anyone has field notes to share.

@Igor Rodionov deployed something like that. not specifically for lambdas though.

@Igor Rodionov has joined the channel

@Erik Osterman (Cloud Posse) Hi. Are you planning to talk about prometheus anytime soon during office-hours?

#office-hours topics are really driven by who ever attends

is there something specific you’re interested in?

nothing specific, we are thinking about using prometheus

I’d be happy to give a demo

Our next office hours is tomorrow

time please

https://zoom.us/j/684901853

Every Wednesday at 11:30 am PST

bad timing for me(i am in CST), will try to join

thanks

Hello, I have a question about https://github.com/cloudposse/prometheus-to-cloudwatch

How do deal with Prometheus and Cloudwatch’s different model of metrics gathering? Prometheus assumes reported metrics are summed whereas Cloudwatch assumes each reflects current values.

That leads to funny looking graphs like this in Cloudwatch when a container is restarted.

@Andriy Knysh (Cloud Posse) can probably help. But it probably comes down to using something like counters vs gauges. Prometheus supports multiple types of metrics whereas I am not sure if CloudWatch does (if so they don’t call it gauge). From working with other monitoring systems it is common to support both. I’d be surprised if there isn’t a way to achieve it.

Thanks. I think gauge type metrics (e.g. current memory usage) work OK but not all metrics can be tracked that way. For instance “total number of requests” needs a counter because it tracks events rather than a value. The problem arises because the Prometheus client in my container reports “123 requests have occurred since the container was restarted” but when this gets forwarded to Cloudwatch it is interpreted as “123 requests occurred right now” and in the next update 30 seconds later it thinks there have been another 123 requests whereas there have been none.

@Tamlyn Rhodes I’m not sure how to change the metric types between prometheus and CloudWatch. https://github.com/cloudposse/prometheus-to-cloudwatch is just a proxy that scrapes prometheus URLs, converts the format, and sends the metrics to CloudWatch. It does not assume anything. It might be possible to change the module to do some logic.

also take a look at these releases, it might help

https://github.com/cloudposse/prometheus-to-cloudwatch/releases/tag/0.2.0

https://github.com/cloudposse/prometheus-to-cloudwatch/releases/tag/0.3.0

OK, thanks for you help. I’ll investigate other approaches.

Have a good weekend

if you have any improvements, PRs are welcome