#prometheus (2019-06)

prometheus

Archive: https://archive.sweetops.com/prometheus/

2019-06-28

Tamlyn Rhodes avatar
Tamlyn Rhodes
cloudposse/prometheus-to-cloudwatch

Utility for scraping Prometheus metrics from a Prometheus client endpoint and publishing them to CloudWatch - cloudposse/prometheus-to-cloudwatch

Tamlyn Rhodes avatar
Tamlyn Rhodes

How do deal with Prometheus and Cloudwatch’s different model of metrics gathering? Prometheus assumes reported metrics are summed whereas Cloudwatch assumes each reflects current values.

Tamlyn Rhodes avatar
Tamlyn Rhodes

That leads to funny looking graphs like this in Cloudwatch when a container is restarted.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) can probably help. But it probably comes down to using something like counters vs gauges. Prometheus supports multiple types of metrics whereas I am not sure if CloudWatch does (if so they don’t call it gauge). From working with other monitoring systems it is common to support both. I’d be surprised if there isn’t a way to achieve it.

Tamlyn Rhodes avatar
Tamlyn Rhodes

Thanks. I think gauge type metrics (e.g. current memory usage) work OK but not all metrics can be tracked that way. For instance “total number of requests” needs a counter because it tracks events rather than a value. The problem arises because the Prometheus client in my container reports “123 requests have occurred since the container was restarted” but when this gets forwarded to Cloudwatch it is interpreted as “123 requests occurred right now” and in the next update 30 seconds later it thinks there have been another 123 requests whereas there have been none.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Tamlyn Rhodes I’m not sure how to change the metric types between prometheus and CloudWatch. https://github.com/cloudposse/prometheus-to-cloudwatch is just a proxy that scrapes prometheus URLs, converts the format, and sends the metrics to CloudWatch. It does not assume anything. It might be possible to change the module to do some logic.

cloudposse/prometheus-to-cloudwatch

Utility for scraping Prometheus metrics from a Prometheus client endpoint and publishing them to CloudWatch - cloudposse/prometheus-to-cloudwatch

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

also take a look at these releases, it might help

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/prometheus-to-cloudwatch

Utility for scraping Prometheus metrics from a Prometheus client endpoint and publishing them to CloudWatch - cloudposse/prometheus-to-cloudwatch

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/prometheus-to-cloudwatch

Utility for scraping Prometheus metrics from a Prometheus client endpoint and publishing them to CloudWatch - cloudposse/prometheus-to-cloudwatch

Tamlyn Rhodes avatar
Tamlyn Rhodes

OK, thanks for you help. I’ll investigate other approaches.

:--1:2
Tamlyn Rhodes avatar
Tamlyn Rhodes

Have a good weekend

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you have any improvements, PRs are welcome

:--1:1

2019-06-25

rohit avatar
rohit

@Erik Osterman (Cloud Posse) Hi. Are you planning to talk about prometheus anytime soon during office-hours?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

#office-hours topics are really driven by who ever attends

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is there something specific you’re interested in?

rohit avatar
rohit

nothing specific, we are thinking about using prometheus

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’d be happy to give a demo

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Our next office hours is tomorrow

rohit avatar
rohit

time please

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Every Wednesday at 11:30 am PST

rohit avatar
rohit

bad timing for me(i am in CST), will try to join

rohit avatar
rohit

thanks

2019-06-20

tamsky avatar
tamsky

Has anyone here used https://github.com/weaveworks/prom-aggregation-gateway for aggregating metrics from Lambda functions? Curious if anyone has field notes to share.

weaveworks/prom-aggregation-gateway

An aggregating push gateway for Prometheus. Contribute to weaveworks/prom-aggregation-gateway development by creating an account on GitHub.

Igor Rodionov avatar
Igor Rodionov

I just deployed push gateway on kubernetest

weaveworks/prom-aggregation-gateway

An aggregating push gateway for Prometheus. Contribute to weaveworks/prom-aggregation-gateway development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov deployed something like that. not specifically for lambdas though.

Igor Rodionov avatar
Igor Rodionov
06:42:06 AM

@Igor Rodionov has joined the channel

    keyboard_arrow_up