#sre (2019-07)

Prometheus, Prometheus Operator, Grafana, Kubernetes

Archive: https://archive.sweetops.com/monitoring/

2019-07-11

GFox (someTXcloudGuy) avatar
GFox (someTXcloudGuy)

Hi, anyone have any tips and tricks for a fast way to go through a hundred config.json files in a repo pulling out the threshold values of metrics for monitoring config???

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Something more than jq?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
1
GFox (someTXcloudGuy) avatar
GFox (someTXcloudGuy)
07:38:15 PM

looking for changes and pulling out thresold data points to alarm evaluation period name period statistic to compare, maybe extract to spreadsheet for comparison ??

GFox (someTXcloudGuy) avatar
GFox (someTXcloudGuy)

will give jq a shot, thank you

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yep, can totally use jq for that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The syntax is a little bit funky

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But once you get it, very powerful

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

They have a lot of examples

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you get stuck let me know

GFox (someTXcloudGuy) avatar
GFox (someTXcloudGuy)

Thank you!

GFox (someTXcloudGuy) avatar
GFox (someTXcloudGuy)

Hi Erik, do you have an ex or could point me to an example, not strong in running code query, nor parsing. Want to run see if there’s a way to run jq against my repo or local folder to see any differences in the objects for these monitoring config.json files …and I don’t know if I’m saying asking it correctly, kind of know what I want but fuzzy at the same time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
jq -r '.metrics[] | [.name, .evaluationPeriod, .comparisonOperator]| @csv' < example_snstopic_aws_json.js
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

produces

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
"CPUUtilization","1","GreaterThanOrEqualToThreshold"
"StatusCheckFailed","1","GreaterThanOrEqualToThreshold"

2019-07-15

JMC avatar

Has anyone found a way to configure alerting for Kibana with the latest version, without using the pricey Watchers functionality ? Seems ElastAlert and SentiNL is not working really great with ES 7.x.

    keyboard_arrow_up