#sre (2018-11)

Prometheus, Prometheus Operator, Grafana, Kubernetes

Archive: https://archive.sweetops.com/monitoring/

2018-11-28

pecigonzalo avatar
pecigonzalo

What are you guys using with ECS and prometheus for SD?

pecigonzalo avatar
pecigonzalo

We had our own SD in python, but im always afraid of hitting the api limits as we scale, as we did before

pecigonzalo avatar
pecigonzalo

(we had a DNS SD based on lambda and ECS events)

joshmyers avatar
joshmyers

At previous client who were so big they ended up having to pay for AWS API requests a tool was written to do a single lookup a la ec2_sd, write it to a file and the file gets mounted inside k8s prom, where k8s prom was multi team and if each team did their own lookups, would bust the limit

pecigonzalo avatar
pecigonzalo

Yeah, we did a tool for that in Lambda, tbh is not that complicated

pecigonzalo avatar
pecigonzalo

but I was looking for a “simpler” solution

pecigonzalo avatar
pecigonzalo

I have a similar situation for uploading new prometheus configs, without doing a docker deployment, since albeit incorrectly that was the “easy” start for us but it sort of sucks

mrwacky avatar
mrwacky
gliderlabs/registrator

Service registry bridge for Docker with pluggable adapters - gliderlabs/registrator

1
mrwacky avatar
mrwacky

@pecigonzalo

tamsky avatar

what’s wrong with Consul for SD ?

joshmyers avatar
joshmyers

You need consul?

1
joshmyers avatar
joshmyers

Maybe not what you want if that is all you are going to use Consul for

tamsky avatar

@joshmyers so what are your reasons for not using Consul if used strictly for SD ?

mrwacky avatar
mrwacky

ease of use, setup, deficiencies in AWS SD options, yeah, Consul is great

joshmyers avatar
joshmyers

I don’t have any. I’m just saying folks may not want to run a 3 node etcd cluster when they have been using AWS API as cheap service discovery

1
mrwacky avatar
mrwacky

Good news, Consul is not etcd

joshmyers avatar
joshmyers

hah, oops, same thing. It is a thing you need to manage?

tamsky avatar

of all the services I’ve operated/managed since 2014, consul is the least needy service I’ve met

joshmyers avatar
joshmyers

Nice

tamsky avatar

self-bootstrapping EC2 ASG cluster FTW

joshmyers avatar
joshmyers

Have used with Nomad before and not had any issues with it, but it isn’t a managed type service, was my only point

tamsky avatar

managed type services are good for getting started – one should have a plan for when your org’s needs or skills outgrow a managed service offering from anyone

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…such as multi-cloud

joshmyers avatar
joshmyers

Aye, multi cloud is hard though

1
2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, the all elusive multi-cloud strategy

pecigonzalo avatar
pecigonzalo

@mrwacky yeah we know about consul and registrator, but as explained by @joshmyers that is ofc the option if you have Consul, we dont

pecigonzalo avatar
pecigonzalo

and while it is a good easy discovery once you have that, the question would be you do if you dont hve consul

pecigonzalo avatar
pecigonzalo

at the moment our services dont use mesh, as we dont need/want that yet

pecigonzalo avatar
pecigonzalo

so consul will be there ONLY to support prometheus discovery, and that seemed overkill to me, but maybe its the only option

tamsky avatar

there are a lot of options. all of them that end in *_sd_config are candidates. let us know what you pick and why:

Configuration | Prometheus

An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

2018-11-29

mrwacky avatar
mrwacky

I’m sure there’s other options..

2018-11-30

pecigonzalo avatar
pecigonzalo

Tamsky, I know the options, as I mentioned we are already using this, and we had our own lambda for SD

pecigonzalo avatar
pecigonzalo

I was trying to ping/pong how others were doing it

tamsky avatar


(we had a DNS SD based on lambda and ECS events)
but I was looking for a “simpler” solution

I guess I was trying to help out re: “simpler” solutions.

tamsky avatar


I have a similar situation for uploading new prometheus configs, without doing a docker deployment, since albeit incorrectly that was the “easy” start for us but it sort of sucks

how do you handle persistent storage for prometheus in your docker setup – that answer might guide us toward an easy process that can update your prometheus configs.

    keyboard_arrow_up