#sre (2020-11)

Prometheus, Prometheus Operator, Grafana, Kubernetes

Archive: https://archive.sweetops.com/monitoring/

2020-11-17

joshmyers avatar
joshmyers

How are folks managing their Grafana dashboards? Templating JSON isn’t the one.

btai avatar

I had intended to go the path of version controlled dashboards w/ infra-as-code methodology but @Jeremy G (Cloud Posse) made some good points on why that route is pain (specifically around updating dashboards). at the moment we’re just doing click-ops dashboard creation in the ui saved to a remote RDS db.

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

@joshmyers Grafana dashboards are things where we do not want drift detection to undo ClickOps changes: we want people to be able to customize the dashboards via the Grafana UI.

We have a script in Geodesic called grafana-db we use to upload dashboards from various sources to Grafana via the Grafana API, and have Grafana store them in a remote database. This solves the problem of importing published dashboards in a way that makes them editable. For our installation, we maintain a script that loads the Grafana API key and runs grafana-db for every dashboard source we want.

cloudposse/geodesic

Geodesic is a cloud automation shell. It's the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

1
joshmyers avatar
joshmyers

Yeah, that isn’t gonna cut it for us. 100+ microservices that we want dashboards for so want something to generate via code

joshmyers avatar
joshmyers
grafana/grafonnet-lib

Jsonnet library for generating Grafana dashboard files. - grafana/grafonnet-lib

alxrem/terraform-provider-jsonnet

Terraform provider for generating JSON documents from Jsonnet templates - alxrem/terraform-provider-jsonnet

joshmyers avatar
joshmyers

https://github.com/grafana/grafonnet-lib doesn’t feel ready for prime time quite yet

grafana/grafonnet-lib

Jsonnet library for generating Grafana dashboard files. - grafana/grafonnet-lib

joshmyers avatar
joshmyers

heh, this is quite nice actually

Lukasz K avatar
Lukasz K

Hi @joshmyers, what did you end up doing in the end? clickops or dashboard as code?

joshmyers avatar
joshmyers

Dashboard as code

Lukasz K avatar
Lukasz K

cools, jsonnet + terrafrom?

joshmyers avatar
joshmyers

Yup

Lukasz K avatar
Lukasz K

any pitfalls?

joshmyers avatar
joshmyers

Nope

joshmyers avatar
joshmyers

It’s still defining a dashboard as code, which is hard. They are inherently visual things so wipping up something in code before seeing what you want the thing to look like it the biggest gotcha

Lukasz K avatar
Lukasz K

Thanks Josh, I will most likely convince my team to follow same route

Lukasz K avatar
Lukasz K

yeah, exporting json is fine, but then maintaining json is world of pain. no easy choices here

joshmyers avatar
joshmyers
mpostument/grafana-sync

Keep your Grafana dashboards in sync. Contribute to mpostument/grafana-sync development by creating an account on GitHub.

joshmyers avatar
joshmyers

There is no silver bullet here, until you can export Grafana dashboards as Grafonnet

Lukasz K avatar
Lukasz K

exactly

joshmyers avatar
joshmyers

Example

joshmyers avatar
joshmyers
##################################################
# Terraform
##################################################
terraform {
  required_version = "= 0.13.5"

  required_providers {
    grafana = {
      source  = "grafana/grafana"
      version = "~> 1.7.0"
    }
    jsonnet = {
      source  = "alxrem/jsonnet"
      version = "~> 1.0.1"
    }
  }

  backend "s3" {
    bucket         = "BADGERS"
    region         = "BADGERS"
    key            = "BADGERS"
    dynamodb_table = "BADGERS"
    role_arn       = "BADGERS"
  }
}

##################################################
# Providers
##################################################
provider "jsonnet" {
  jsonnet_path = ["${path.module}/.terraform/modules/grafonnet-lib/"]
}

provider "grafana" {
  url  = local.grafana_url
  auth = var.grafana_auth
}

##################################################
# Variables
##################################################
variable "grafana_auth" {
  type        = string
  description = "Grafana Auth"
  default     = null
}

variable "namespace" {
  type        = string
  description = "Team namespace"
}

variable "team_name" {
  type        = string
  description = "Display friendly team name"
}

variable "jenkins_master_build_repos" {
  type        = list
  description = "List of service repos to fetch master build duration for"
  default     = []
}

variable "jenkins_prod_deploy_jobs" {
  type        = list
  description = "List of jobs to fetch prod deploy duration for"
  default     = []
}

locals {
  grafana_url = "us-east-1.BADGERS.net"
}

##################################################
# Grafonnet-lib
##################################################
module "grafonnet-lib" {
  source = "git::<https://github.com/grafana/grafonnet-lib.git?ref=356bd73e4792ffe107725776ca8946895969c191>"
}

##################################################
# Dashboard folder
##################################################
resource "grafana_folder" "default" {
  title = "${var.team_name} Dashboards"
}

##################################################
# Dashboards
##################################################
resource "grafana_dashboard" "jenkins" {
  folder      = grafana_folder.default.id
  config_json = data.jsonnet_file.jenkins.rendered
}

data "jsonnet_file" "jenkins" {
  source = "${path.module}/dashboards/jenkins.jsonnet"

  ext_code = {
    tags                       = jsonencode(list("userservices", "jenkins"))
    jenkins_master_build_repos = jsonencode(var.jenkins_master_build_repos)
    jenkins_prod_deploy_jobs   = jsonencode(var.jenkins_prod_deploy_jobs)
  }

  ext_str = {
    namespace            = var.namespace
    cloudwatch_namespace = "${var.namespace}/jenkins"
    service              = "jenkins-jobs"
  }
}
joshmyers avatar
joshmyers

Then you just need your dashboard grafonnet

Lukasz K avatar
Lukasz K

looks neat

Lukasz K avatar
Lukasz K

thanks once more

2020-11-18

2020-11-19

    keyboard_arrow_up