#sre (2020-11)
Prometheus, Prometheus Operator, Grafana, Kubernetes
Archive: https://archive.sweetops.com/monitoring/
2020-11-17
How are folks managing their Grafana dashboards? Templating JSON isn’t the one.
I had intended to go the path of version controlled dashboards w/ infra-as-code methodology but @Jeremy G (Cloud Posse) made some good points on why that route is pain (specifically around updating dashboards). at the moment we’re just doing click-ops dashboard creation in the ui saved to a remote RDS db.
@joshmyers Grafana dashboards are things where we do not want drift detection to undo ClickOps changes: we want people to be able to customize the dashboards via the Grafana UI.
We have a script in Geodesic called grafana-db we use to upload dashboards from various sources to Grafana via the Grafana API, and have Grafana store them in a remote database. This solves the problem of importing published dashboards in a way that makes them editable. For our installation, we maintain a script that loads the Grafana API key and runs grafana-db
for every dashboard source we want.
Geodesic is a cloud automation shell. It's the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…
Yeah, that isn’t gonna cut it for us. 100+ microservices that we want dashboards for so want something to generate via code
Using https://github.com/grafana/grafonnet-lib + https://github.com/alxrem/terraform-provider-jsonnet is interesting, get a nice diff of the plans etc
Jsonnet library for generating Grafana dashboard files. - grafana/grafonnet-lib
Terraform provider for generating JSON documents from Jsonnet templates - alxrem/terraform-provider-jsonnet
https://github.com/grafana/grafonnet-lib doesn’t feel ready for prime time quite yet
Jsonnet library for generating Grafana dashboard files. - grafana/grafonnet-lib
heh, this is quite nice actually
Hi @joshmyers, what did you end up doing in the end? clickops or dashboard as code?
Dashboard as code
cools, jsonnet + terrafrom?
Yup
any pitfalls?
Nope
It’s still defining a dashboard as code, which is hard. They are inherently visual things so wipping up something in code before seeing what you want the thing to look like it the biggest gotcha
Thanks Josh, I will most likely convince my team to follow same route
yeah, exporting json is fine, but then maintaining json is world of pain. no easy choices here
https://github.com/mpostument/grafana-sync is interesting
Keep your Grafana dashboards in sync. Contribute to mpostument/grafana-sync development by creating an account on GitHub.
There is no silver bullet here, until you can export Grafana dashboards as Grafonnet
exactly
Example
##################################################
# Terraform
##################################################
terraform {
required_version = "= 0.13.5"
required_providers {
grafana = {
source = "grafana/grafana"
version = "~> 1.7.0"
}
jsonnet = {
source = "alxrem/jsonnet"
version = "~> 1.0.1"
}
}
backend "s3" {
bucket = "BADGERS"
region = "BADGERS"
key = "BADGERS"
dynamodb_table = "BADGERS"
role_arn = "BADGERS"
}
}
##################################################
# Providers
##################################################
provider "jsonnet" {
jsonnet_path = ["${path.module}/.terraform/modules/grafonnet-lib/"]
}
provider "grafana" {
url = local.grafana_url
auth = var.grafana_auth
}
##################################################
# Variables
##################################################
variable "grafana_auth" {
type = string
description = "Grafana Auth"
default = null
}
variable "namespace" {
type = string
description = "Team namespace"
}
variable "team_name" {
type = string
description = "Display friendly team name"
}
variable "jenkins_master_build_repos" {
type = list
description = "List of service repos to fetch master build duration for"
default = []
}
variable "jenkins_prod_deploy_jobs" {
type = list
description = "List of jobs to fetch prod deploy duration for"
default = []
}
locals {
grafana_url = "us-east-1.BADGERS.net"
}
##################################################
# Grafonnet-lib
##################################################
module "grafonnet-lib" {
source = "git::<https://github.com/grafana/grafonnet-lib.git?ref=356bd73e4792ffe107725776ca8946895969c191>"
}
##################################################
# Dashboard folder
##################################################
resource "grafana_folder" "default" {
title = "${var.team_name} Dashboards"
}
##################################################
# Dashboards
##################################################
resource "grafana_dashboard" "jenkins" {
folder = grafana_folder.default.id
config_json = data.jsonnet_file.jenkins.rendered
}
data "jsonnet_file" "jenkins" {
source = "${path.module}/dashboards/jenkins.jsonnet"
ext_code = {
tags = jsonencode(list("userservices", "jenkins"))
jenkins_master_build_repos = jsonencode(var.jenkins_master_build_repos)
jenkins_prod_deploy_jobs = jsonencode(var.jenkins_prod_deploy_jobs)
}
ext_str = {
namespace = var.namespace
cloudwatch_namespace = "${var.namespace}/jenkins"
service = "jenkins-jobs"
}
}
Then you just need your dashboard grafonnet
looks neat
thanks once more