#sre (2020-11)
Prometheus, Prometheus Operator, Grafana, Kubernetes
Archive: https://archive.sweetops.com/monitoring/
2020-11-17

How are folks managing their Grafana dashboards? Templating JSON isn’t the one.

I had intended to go the path of version controlled dashboards w/ infra-as-code methodology but @Jeremy G (Cloud Posse) made some good points on why that route is pain (specifically around updating dashboards). at the moment we’re just doing click-ops dashboard creation in the ui saved to a remote RDS db.

@joshmyers Grafana dashboards are things where we do not want drift detection to undo ClickOps changes: we want people to be able to customize the dashboards via the Grafana UI.
We have a script in Geodesic called grafana-db we use to upload dashboards from various sources to Grafana via the Grafana API, and have Grafana store them in a remote database. This solves the problem of importing published dashboards in a way that makes them editable. For our installation, we maintain a script that loads the Grafana API key and runs grafana-db
for every dashboard source we want.
Geodesic is a cloud automation shell. It's the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Yeah, that isn’t gonna cut it for us. 100+ microservices that we want dashboards for so want something to generate via code

Using https://github.com/grafana/grafonnet-lib + https://github.com/alxrem/terraform-provider-jsonnet is interesting, get a nice diff of the plans etc
Jsonnet library for generating Grafana dashboard files. - grafana/grafonnet-lib
Terraform provider for generating JSON documents from Jsonnet templates - alxrem/terraform-provider-jsonnet

https://github.com/grafana/grafonnet-lib doesn’t feel ready for prime time quite yet
Jsonnet library for generating Grafana dashboard files. - grafana/grafonnet-lib

heh, this is quite nice actually

Hi @joshmyers, what did you end up doing in the end? clickops or dashboard as code?

Dashboard as code

cools, jsonnet + terrafrom?

Yup

any pitfalls?

Nope

It’s still defining a dashboard as code, which is hard. They are inherently visual things so wipping up something in code before seeing what you want the thing to look like it the biggest gotcha

Thanks Josh, I will most likely convince my team to follow same route

yeah, exporting json is fine, but then maintaining json is world of pain. no easy choices here

https://github.com/mpostument/grafana-sync is interesting
Keep your Grafana dashboards in sync. Contribute to mpostument/grafana-sync development by creating an account on GitHub.

There is no silver bullet here, until you can export Grafana dashboards as Grafonnet

exactly

Example

##################################################
# Terraform
##################################################
terraform {
required_version = "= 0.13.5"
required_providers {
grafana = {
source = "grafana/grafana"
version = "~> 1.7.0"
}
jsonnet = {
source = "alxrem/jsonnet"
version = "~> 1.0.1"
}
}
backend "s3" {
bucket = "BADGERS"
region = "BADGERS"
key = "BADGERS"
dynamodb_table = "BADGERS"
role_arn = "BADGERS"
}
}
##################################################
# Providers
##################################################
provider "jsonnet" {
jsonnet_path = ["${path.module}/.terraform/modules/grafonnet-lib/"]
}
provider "grafana" {
url = local.grafana_url
auth = var.grafana_auth
}
##################################################
# Variables
##################################################
variable "grafana_auth" {
type = string
description = "Grafana Auth"
default = null
}
variable "namespace" {
type = string
description = "Team namespace"
}
variable "team_name" {
type = string
description = "Display friendly team name"
}
variable "jenkins_master_build_repos" {
type = list
description = "List of service repos to fetch master build duration for"
default = []
}
variable "jenkins_prod_deploy_jobs" {
type = list
description = "List of jobs to fetch prod deploy duration for"
default = []
}
locals {
grafana_url = "us-east-1.BADGERS.net"
}
##################################################
# Grafonnet-lib
##################################################
module "grafonnet-lib" {
source = "git::<https://github.com/grafana/grafonnet-lib.git?ref=356bd73e4792ffe107725776ca8946895969c191>"
}
##################################################
# Dashboard folder
##################################################
resource "grafana_folder" "default" {
title = "${var.team_name} Dashboards"
}
##################################################
# Dashboards
##################################################
resource "grafana_dashboard" "jenkins" {
folder = grafana_folder.default.id
config_json = data.jsonnet_file.jenkins.rendered
}
data "jsonnet_file" "jenkins" {
source = "${path.module}/dashboards/jenkins.jsonnet"
ext_code = {
tags = jsonencode(list("userservices", "jenkins"))
jenkins_master_build_repos = jsonencode(var.jenkins_master_build_repos)
jenkins_prod_deploy_jobs = jsonencode(var.jenkins_prod_deploy_jobs)
}
ext_str = {
namespace = var.namespace
cloudwatch_namespace = "${var.namespace}/jenkins"
service = "jenkins-jobs"
}
}

Then you just need your dashboard grafonnet

looks neat

thanks once more