#office-hours (2021-07)

Meeting password: sweetops

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

Meeting password: sweetops

2021-07-29

SweetOps avatar
SweetOps
04:31:49 PM
[Terraform State Security Cloud Posse Explains](https://www.youtube.com/watch?v=xw0iPkYGKQM)

2021-07-28

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:54 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

oskar avatar
oskar

no questions here today so far so i’m asking something a bit off-topic maybe: has anyone test-run opstrace as an alternative to other saas “o11y” offerings? if so how did it go and did you go further into production with it? thanks!

1
1
oskar avatar
oskar

addendum for people who haven’t read about it yet. it’s very much centred around k8s - relying on cortex and loki. blog articles have a strong docker ~2013-ish vibe so beware - could be too good (marketing) for their own sake (yet).

https://github.com/opstrace/opstrace https://opstrace.com/blog/cloud-provider-integrations https://opstrace.com/blog/a-giant-leap-for-alerts

GitHub - opstrace/opstrace: The Open Source Observability Distribution attachment image

The Open Source Observability Distribution. Contribute to opstrace/opstrace development by creating an account on GitHub.

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

this looks like a whole [grafana.com> backend stack assembled in one project </i](http://grafana.com)

oskar avatar
oskar

yeah it would be awesome if only for helping run and maintain prometheus/grafana stacks but apparently can be much more and manageable (at least that seems to be the promise). pulling in cloud level events etc.

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

we’ve been doing cortex in-house for 1 year. A ton of efforts but it pays off. Grafana pricing for us was 10k+ month, while we packed it in under 1k

1
1
Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

I’d love to see if that thing really gives a working opinionated setup

cool-doge1
oskar avatar
oskar

same same. very excited to learn about this project (again don’t know if it really is as good as it looks at first sight!)

Zoom avatar
Zoom
06:29:20 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:24 PM

Thayne Trevenen has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:41 PM

Brandon vh has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:54 PM

Andy Miguel (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:56 PM

Oskar Maria Grande has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:34 PM

David Hawthorne has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:01 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:26 PM

Tim Gourley has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:37 PM

Andrew Thompson has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:54 PM

Jonas Steinberg has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:08 PM

Blaise Pabon has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:18 PM

Andy Roth has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:20 PM

Nick James has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:21 PM

Clayton Olley has joined Public “Office Hours”

Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)
Mitchell's New Role at HashiCorp attachment image

Mitchell Hashimoto takes on a new individual contributor role at HashiCorp.

GitHub - Cigna/confectionery: A library of rules for Conftest used to detect misconfigurations within Terraform configuration files attachment image

A library of rules for Conftest used to detect misconfigurations within Terraform configuration files - GitHub - Cigna/confectionery: A library of rules for Conftest used to detect misconfiguration…

GitHub - jckuester/awsls: A list command for AWS resources attachment image

A list command for AWS resources. Contribute to jckuester/awsls development by creating an account on GitHub.

weekly.tf - Revue

Terraform Weekly

Kubernetes Monitoring, Application Debug Platform | Pixie

A simple and robust monitoring and live-debug platform for distributed environments, designed for developers.

1
Zoom avatar
Zoom
06:34:24 PM

Yusuf Adeyemo has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:27 PM

Mykola Lev has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:20 PM

Mykola Lev has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:36 PM

PePe Amengual has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:47 PM

Fernando Castillo has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:53 PM

Luis Masaya has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:16 PM

Fernando Castillo has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:55 PM

Winson Chan has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:39 PM

Marc Slayton has joined Public “Office Hours”

Zoom avatar
Zoom
06:40:16 PM

Joel Caceres has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Infra App - Simplest Kubernetes Desktop Client attachment image

Instantly manage and monitor Kubernetes from your desktop.

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

reminds me the most: https://www.kubernetic.com/

Kubernetic - The Kubernetes Desktop Client

Kubernetic is a brand new Desktop Client for Kubernetes that lets developers and ops manage their Kubernetes cluster(s) through a UI interface in a very simple way.

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

used for a long time - positive feedback

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Lens | The Kubernetes IDE attachment image

Lens IDE for Kubernetes. The only system you’ll ever need to take control of your Kubernetes clusters. It’s open source and free. Download it today!

Zoom avatar
Zoom
06:45:16 PM

Andrew Thompson has joined Public “Office Hours”

Blaise Pabon avatar
Blaise Pabon

This week in Jonas’ world

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Abbot: Make Chat Your Command Center

Abbot is the easiest way to add ChatOps to your Slack or Discord team. Add skills from our directory or create your own in C#, JavaScript, or Python.

oskar avatar
oskar

probably not as extensible as this open source chatbot framework though - have not put much time into it myself yet and i’m biased as a clojure fan. still: https://github.com/yetibot/yetibot

GitHub - yetibot/yetibot: :robot_face: Extreme chatops bot for Slack and IRC :wrench: New contributors welcome :building_construction: attachment image

Extreme chatops bot for Slack and IRC New contributors welcome - GitHub - yetibot/yetibot: Extreme chatops bot for Slack and IRC New contributors welcome

1
Blaise Pabon avatar
Blaise Pabon

wow… yeti looks amazing

oskar avatar
oskar

yeah it totally is. if you are only a little into lisp you should check it out. and if you are into babashka (https://github.com/babashka/babashka) definitely check it out!

GitHub - babashka/babashka: Native, fast starting Clojure interpreter for scripting attachment image

Native, fast starting Clojure interpreter for scripting - GitHub - babashka/babashka: Native, fast starting Clojure interpreter for scripting

Blaise Pabon avatar
Blaise Pabon

I was raised on Lisp …but that was a very long time ago.

oskar avatar
oskar

haha it’s like riding the bicycle - never forget it really

oskar avatar
oskar

there is not much to forget actually ;D

Zoom avatar
Zoom
06:46:54 PM

Hao Wang has joined Public “Office Hours”

Zoom avatar
Zoom
06:48:35 PM

Sheldon Hull has joined Public “Office Hours”

Zoom avatar
Zoom
06:48:44 PM

Mosh has joined Public “Office Hours”

managedkaos avatar
managedkaos

Question: would like to know if anyone has used PGP to encrypt AWS access keys generated by the TF module for secret keys… what’s your experience been like?

managedkaos avatar
managedkaos

related to secrets discussion

Zoom avatar
Zoom
06:51:48 PM

uwaila adams has joined Public “Office Hours”

roth.andy avatar
roth.andy

I wouldn’t be much of a fanboy if I didn’t bring up K9s when there’s talk about K8s management GUIs

https://github.com/derailed/k9s

GitHub - derailed/k9s: :dog: Kubernetes CLI To Manage Your Clusters In Style! attachment image

Kubernetes CLI To Manage Your Clusters In Style! - GitHub - derailed/k9s: Kubernetes CLI To Manage Your Clusters In Style!

2
Zoom avatar
Zoom
06:53:35 PM

David Hawthorne has joined Public “Office Hours”

Jonas Steinberg avatar
Jonas Steinberg

question: how are people terraforming alerts specifically in the case where there are many identical underlying resources spread across numerous accounts or environments that require different threshold values? So to articulate a bit more: imagine you have 20 EKS clusters and you want to monitor something like host IO (or whatever): something that is going to vary from environment-to-environment and that you can’t just set defaults for. How are people handling this case? My approach is as follows, but I’d love to learn about something better or alternatives in general?

1. have a directory with yaml files each containing a single "alert"
2. in each yaml alert file have some non-environment specific values like "query" which will always be the same from cluster-to-cluster
3. have an "options" section which, basically/unfortunately, contains all the environmental specification, mainly the threshold values *for every environment*
4. loop through all these files using terraform fileset
5. yamldecode every file
6. put the name of the alert in a local tf object
7. add the non-environment specific options as a list
8. get all the environment specific stuff and add it to that object
9. iterate over all that with terraform each spinning up alerts in the process with names and environmental config 

# example
name: istio-host-latency-alert
type: metric alert
query: |
  avg(last_5m):top(top(avg:istio.mesh.request.duration.milliseconds.sum{cluster_name:${stage}} by {host}, 10, 'mean', 'desc')/top(avg:istio.mesh.request.duration.milliseconds.count{cluster_name:${eks_cluster}} by {host}, 10, 'mean', 'desc'),10,'mean','desc') > ${critical_threshold}
message: |
  ({{event.tags.cluster_name}}) Detected Host Latency Greater than 1 Second
escalation_message: ""
tags: []
options:
  - dev:
      notify_no_data: false
      renotify_interval: 60
      notify_audit: false
      timeout_h: 60
      include_tags: true
      require_full_window: true
      threshold:
        critical: 1000
        warning: 50
  - qa:
      notify_no_data: false
      renotify_interval: 60
      notify_audit: false
      timeout_h: 60
      include_tags: true
      require_full_window: true
      threshold:
        critical: 1000
        warning: 50
  - client-test:
      notify_no_data: false
      renotify_interval: 60
      notify_audit: false
      timeout_h: 60
      include_tags: true
      require_full_window: true
      threshold:
        critical: 1000
        warning: 50
...

oskar avatar
oskar

looks sane to me. anything specific that bugs you there? if you look for other perspectives, i’m in the same mindspace unfortunately.

oskar avatar
oskar

very good question thanks for sharing. i really also like the “dynamic” view @ has on this as well. and to paraphrase maybe what you got at for the “static” approach you painted here jonas - not too many layers with the parametrization as @Erik Osterman (Cloud Posse) put it. and maybe also disregarding some of those differences to even simplify further as @roth.andy has said. hope i got this right. either way, thanks again very interesting topic to me as well.

oskar avatar
oskar

sound like good rules of thumb for the “static” view on this kind of “o11y” (charity majors probably disagrees indeed btw ).

Jonas Steinberg avatar
Jonas Steinberg

ha, yeah.

oskar avatar
oskar

sorry to hijack this thread again but @ somebody was asking about which APM solution you are using before the zoom closed. i’m also curious

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

Any chance to have a global multiplier for an environment? Maybe a set of multipliers, say one for timeouts, another for RPMs. They you’d customize only multipliers. Of course you still should have a bypass to override a single value, when needed.

mholttech avatar
mholttech

I missed the office hours yesterday so didn’t see this until I was watching the recording but this is actually exactly the rabbit hole I started down just yesterday for AWS Cloudwatch Alarms. I’m approaching this using the CloudPosse YAML Config module and the built in Parameters variable.

mholttech avatar
mholttech

The only thing I haven’t figured a solution out for yet is setting default parameter values if not set in var.parameters

mholttech avatar
mholttech

This is what my YAML File looks like:

ConsumedReadCapacityUnits:
  metric_name: "ConsumedReadCapacityUnits"
  metric_namespace: "AWS/DynamoDB"
  treat_missing_data: "ignore"
  comparison_operator: ${ConsumedReadCapacityUnits_comparison_operator}
  description: ${ConsumedReadCapacityUnits_description}
  metric_value: ${ConsumedReadCapacityUnits_metric_value}
  evaluation_periods: ${ConsumedReadCapacityUnits_evaluation_periods}
  period: ${ConsumedReadCapacityUnits_period}
  statistic: ${ConsumedReadCapacityUnits_statistic}
  threshold: ${ConsumedReadCapacityUnits_threshold}
  dimensions: 
    TableName: ${ConsumedReadCapacityUnits_TableName}

And I pass this into the YAML Config module:

  parameters = {
    ConsumedReadCapacityUnits_TableName           = "terraform-registry-touching-gorilla"
    ConsumedReadCapacityUnits_threshold           = "1"
    ConsumedReadCapacityUnits_period              = "300"
    ConsumedReadCapacityUnits_evaluation_periods  = "1"
    ConsumedReadCapacityUnits_metric_value        = "1"
    ConsumedReadCapacityUnits_description         = "Alarms when ."
    ConsumedReadCapacityUnits_statistic           = "Average"
    ConsumedReadCapacityUnits_comparison_operator = "GreaterThanOrEqualToThreshold"
  }
Zoom avatar
Zoom
06:56:14 PM

Florain Drescher has joined Public “Office Hours”

sheldonh avatar
sheldonh

@Erik Osterman (Cloud Posse) this is what I have automatically initialized on each repo. I also have the CI checks run this too. It’s not perfect but it helps.

Gitleaks with lefthook (I’ve stopped using pre-commit framework as lefthook is super fast). https://github.com/sheldonhull/ci-configuration-files/blob/06f539315716d7a04fbf4ffbfd2e53e906729ef1/lefthook.secrets.yml#L4-L7

ci-configuration-files/lefthook.secrets.yml at 06f539315716d7a04fbf4ffbfd2e53e906729ef1 · sheldonhull/ci-configuration-files attachment image

This repo will contain some general configuration files for usage with pre-commit, linting, and others that I might want to drop into a new repo. - ci-configuration-files/lefthook.secrets.yml at 06…

Zoom avatar
Zoom
06:58:23 PM

Vicken Simonian has joined Public “Office Hours”

Jonas Steinberg avatar
Jonas Steinberg
GitHub - awslabs/git-secrets: Prevents you from committing secrets and credentials into git repositories attachment image

Prevents you from committing secrets and credentials into git repositories - GitHub - awslabs/git-secrets: Prevents you from committing secrets and credentials into git repositories

sheldonh avatar
sheldonh

Git secrets is good too! It’s another layer and can be a global hook. I think “shift left” by having it checked locally is ideal. However, the final source is really the CI action so no one can bypass by accident

1
roth.andy avatar
roth.andy
Push Rules | GitLab

Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.

1
1
sheldonh avatar
sheldonh

All my projects have a go-task/make command to start work. task init. This also installs the hooks

2
1
sheldonh avatar
sheldonh

Not perfect, but this ensures any new repo has the required policies “self setup”

Zoom avatar
Zoom
07:03:33 PM

Mohammed Yahya has joined Public “Office Hours”

sheldonh avatar
sheldonh

This is what my init command looks like. I never leave projects “barebones” I use a way less pretty version like build harness that ensures tools are setup with no fuss/complexity.

  init:dev:
    desc: initialize tools for a developer, but not required for CI
    cmds:
      - |
        dotnet --list-sdks  || echo -e "{{ .red}} :small_red_triangle: dotnet-tools not available. Please install manually the first time here: <https://dotnet.microsoft.com/download> and then run command again {{.nocolor}}"
        mkdir -p {{ .TOOLS_DIRECTORY }} || echo -e "{{ .dark_gray}}:arrows_counterclockwise: skipped creating {{ .TOOLS_DIRECTORY }} directory per already exists {{.nocolor}}"
        # Uninstall manually
        export PATH="$PATH:{{ .HOME }}/.dotnet/tools"
        dotnet tool install --global GitVersion.Tool || dotnet tool update --global GitVersion.Tool # && echo -e "{{.green}} :white_check_mark: gitversion tool installed{{.nocolor}}"
        # If gitversion gives problems with docker commands then evaluate just running as dotnettool
        lefthook install

see last line. The init = always sets up whatever i know is important

sheldonh avatar
sheldonh

That’s from https://taskfile.dev/#/ which I use wherever I’d use make (cross platform, parallelism, file watcher, very simple structure, and basic templating). This is nice for basics. It’s my current goto though I’m exploring other things like Atmos too

Task

A task runner / simpler Make alternative written in Go

oskar avatar
oskar

if you remove the cross-platform requirement you seem to be having would you be able to unbiasedly recommend it over make still? even if you / your team knows make well?

Task

A task runner / simpler Make alternative written in Go

oskar avatar
oskar

thanks for sharing btw. i didn’t know taskfile.dev

sheldonh avatar
sheldonh

Personally I think Make is the default simply because it’s always been there. I don’t see any advantage in it. Wasn’t intended as a devops task runner tool, was meant for C development, so lots of clunky work arounds for devops usage.

Imo, a single curl bootstrap of task means it’s super easy to get going and if I had to pick it would be a no brainer for me

oskar avatar
oskar

got it, so to paraphrase to you a more modern take. i would agree.

sheldonh avatar
sheldonh

You get basic templating too and cross platform support can’t be discounted even if no one uses it yet, cause you never know!

version: '3'

includes:
  build: ./Taskfile_{{OS}}.yml

You can use templating and fingerprint work for incremental builds. https://taskfile.dev/#/usage?id=by-fingerprinting-locally-generated-files-and-their-sources

The CI jobs in 3 seoncs bootstrap with snap install –classic task or whatever it is and you are good.

Task

A task runner / simpler Make alternative written in Go

1
oskar avatar
oskar

that is nice. thanks, will check it out.

sheldonh avatar
sheldonh

Give this a shot as a starter sometime.

It’s NOT perfect. If you have Python devs then use a python task runner, Go use mage, etc.. If you need something instead of Make though, it’s fantastic.

I did a starter write up (have more in the ci-configuration repo i linked in main room), but as I started I tried to note some of the nice base configurations I setup. I plan on improving too with vars.yml instead of embedding the color formatting into the main taskfile.

sheldonh avatar
sheldonh
task attachment image

A cheatsheet with snippets for Task a cross-platform task runner alternative to Make.

sheldonh avatar
sheldonh

Cheers

1
oskar avatar
oskar

very nice - coming up on my reading list - thanks for sharing again

sheldonh avatar
sheldonh

@Erik Osterman (Cloud Posse) If we get any time at the end of call, would love to know…

  • Any starter repo for using variant to run workflow of terraform stacks?

Doing a quick day of work to try and see if I can get away from Terragrunt without a big huge effort on my pilot project and can flip over to this instead.

Zoom avatar
Zoom
07:07:19 PM

Mykola Lev has joined Public “Office Hours”

Zoom avatar
Zoom
07:11:32 PM

Andrew Thompson has joined Public “Office Hours”

roth.andy avatar
roth.andy

Huge +1 from me and my team for taskfile.dev. We use it extensively across almost all of our projects

1
sheldonh avatar
sheldonh
atmos/workflows.yaml at master · cloudposse/atmos attachment image

Universal Tool for DevOps and Cloud Automation (works with terraform, helm, helmfile, istioctl, etc) - atmos/workflows.yaml at master · cloudposse/atmos

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
GitHub - cloudposse/tutorials attachment image

Contribute to cloudposse/tutorials development by creating an account on GitHub.

Blaise Pabon avatar
Blaise Pabon

TIL Spacelift is a new CI tool.

2

2021-07-27

SweetOps avatar
SweetOps
04:32:06 PM
[Waypoint Demo 2 of 3 (Kubernetes) Cloud Posse Guest Speaker: Taylor Dolezal](https://www.youtube.com/watch?v=8z852c0wtHY)
Thayne Trevenen avatar
Thayne Trevenen

Hey Y’all, I am going to start putting TF state files in S3, whats the best policy to be as safe as possible because TF hasn’t given us anything to help with secrets in FT state?

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Don’t put secrets in TF :)

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

More specifically, move to using other mechanisms for secrets, such as AWS’s Secrets Manager.

1
SweetOps avatar
SweetOps
07:27:16 PM
[Waypoint Demo 3 of 3 (Minecraft on EKS!) Cloud Posse Guest Speaker: Taylor Dolezal](https://www.youtube.com/watch?v=GQrWmGvGOP4)

2021-07-26

Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)

@here August 18th we will have another special edition of Office Hours!

@Taylor Dolezal will be returning to demo HashiCorp’s [Boundary> project. Please queue up your questions here and hope to see you there </i](https://www.boundaryproject.io/)

Boundary by HashiCorp attachment image

Boundary is an open source solution that automates a secure identity-based user access to hosts and services across environments.

2
SweetOps avatar
SweetOps
09:47:46 PM
[Waypoint Demo (AWS Lambda) Cloud Posse Guest Speaker: Taylor Dolezal](https://www.youtube.com/watch?v=Kl2d2JMtnOo)

2021-07-21

Adedapo Ajuwon avatar
Adedapo Ajuwon

Hi all, new here. Looking forward to my first live office-hour talk. Cheers.

1
Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)
@here friendly reminder we have @Taylor Dolezal from HashiCorp demoing [Waypoint> today in <https://cloudposse.com/office-hours/ office hours](https://www.waypointproject.io/)!

Hope to see you there

Waypoint by HashiCorp attachment image

Waypoint is an open source solution that provides a modern workflow for build, deploy, and release across platforms.

4
2
4
2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:50 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

1
oskar avatar
oskar


How are people in our community handling drift In the Real World? Did anyone try driftctl? What “homebrew” solutions have community members in use at the moment?

oskar avatar
oskar

Also found out about clairvoyance the other day but I haven’t personally been able to test run it myself.

https://github.com/reulan/clairvoyance

Via corresponding hashicorp talk here: https://www.youtube.com/watch?v=zlwhw3YGlUc

GitHub - reulan/clairvoyance: Drift detection and reporting for Terraform. attachment image

Drift detection and reporting for Terraform. Contribute to reulan/clairvoyance development by creating an account on GitHub.

Jonas Steinberg avatar
Jonas Steinberg

@ my team is using spacelift which has native drift detection. there is also an interesting connection between increasing drift and lack of environmental progression automation; in other words: the less automation between deployments from environment-to-environment…the more drift!

oskar avatar
oskar

thanks for chipping in jonas i’ve heard good things about spacelift, thanks for recommending it. very much agree on the automation side - yet some contexts / apis / providers can at least run into some “transition time” and in some cases do simply provide better ux through their respective native gui interface in some maybe (?) rarer cases (specific example would be setting up an alert in newrelic or datadog). yet again i agree - eg a “gitops” cicd workflow definitely should best practice and a default (i.e. atlantis). by that there should - as a strong guideline - very seldom be any outside-of-terraform manipulation of infrastructure.

oskar avatar
oskar

i’m just asking to be looking at it from this “exceptional” perspective. maybe the alert example would be a good one to talk about!

Jonas Steinberg avatar
Jonas Steinberg

@ you lost me lol. what now?

oskar avatar
oskar

tl;dr there (unfortunately) are scenarios where even over a longer time we can regularly expect drift - e.g. setting something up through a very user friendly web interface for a specific resource comes to mind (e.g. complex alert plus rule in datadog). how do people in the community handle those kind of cases?

Jonas Steinberg avatar
Jonas Steinberg

that is literally what spacelift does.

Jonas Steinberg avatar
Jonas Steinberg

well that’s one feature.

oskar avatar
oskar

that’s awesome. will definitely look further into it for sure

Jonas Steinberg avatar
Jonas Steinberg

right now we’re working on altering when drift_time > x.

oskar avatar
oskar

ok what does that mean exactly? are you talking about “auto-syncing” things after a while?

oskar avatar
oskar

(or semi-auto)

Jonas Steinberg avatar
Jonas Steinberg

it is not configuration management or eventually consistent: no.

oskar avatar
oskar

i see, will look into it thanks for the rec again

Zoom avatar
Zoom
06:29:57 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:03 PM

Rupinder Dhariwal has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:07 PM

Taylor Dolezal has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:07 PM

Adam Blackwell has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:12 PM

Scott Mathson has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:13 PM

Ian Bartholomew has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:14 PM

David Scott has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:17 PM

Andy Miguel (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:19 PM

Jim Park has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:23 PM

Jonas Steinberg has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:26 PM

Patrick Joyce has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:57 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:06 PM

Mikael Fridh has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:21 PM

Benjamin Smith has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:49 PM

Ray Myers has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:57 PM

Joel Castillo has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:57 PM

Miles Monteleone has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:20 PM

Thayne Trevenen has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:37 PM

Antarr Byrd has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:18 PM

Michael Genrich has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:45 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:48 PM

Andrew Thompson has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:32 PM

17866946419 has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:05 PM

Andrew Thompson has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:07 PM

Oskar Maria Grande has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:09 PM

David B has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:13 PM

Luis Masaya has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:26 PM

PePe Amengual has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:40 PM

Jim Antoniou has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:34 PM

Mazin Ahmed has joined Public “Office Hours”

Zoom avatar
Zoom
06:37:24 PM

Murali Krishna Koppuravuri has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:09 PM

Felipe Sakatauskas has joined Public “Office Hours”

Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)
GitHub - onlydole/waypoint-gitops: KubeCon EU 2021 Deep Dive Session attachment image

KubeCon EU 2021 Deep Dive Session. Contribute to onlydole/waypoint-gitops development by creating an account on GitHub.

1
Zoom avatar
Zoom
06:41:03 PM

Satish U has joined Public “Office Hours”

Mazin Ahmed avatar
Mazin Ahmed

Question for today’s office hours: How do you use Git pre-hooks for identifying secrets on organization-level? I’m looking for ideas to detect sensitive commits before it’s committed and pushed to Github. Any ideas how would you approach this on org level?

Zoom avatar
Zoom
06:43:00 PM

Winson Chan has joined Public “Office Hours”

Zoom avatar
Zoom
06:43:21 PM

Neil Gealy has joined Public “Office Hours”

Zoom avatar
Zoom
06:43:38 PM

Arjun Venkatesh has joined Public “Office Hours”

Zoom avatar
Zoom
06:43:47 PM

Julian Severino has joined Public “Office Hours”

Zoom avatar
Zoom
06:44:32 PM

James Haughey has joined Public “Office Hours”

Zoom avatar
Zoom
06:47:18 PM

Murali Krishna Koppuravuri has joined Public “Office Hours”

Zoom avatar
Zoom
06:49:53 PM

Scott Mathson has joined Public “Office Hours”

Zoom avatar
Zoom
06:52:44 PM

Adeoye R has joined Public “Office Hours”

Zoom avatar
Zoom
06:57:18 PM

Blaise Pabon has joined Public “Office Hours”

Jonas Steinberg avatar
Jonas Steinberg

@David regarding your yaml question from earlier: I doubt waypoint natively reads YAML, but coincidentally that is precisely what cloud posse’s Atmos does. I assume you know that, but if not check it. It’s awesome.

Jonas Steinberg avatar
Jonas Steinberg

So if you had the time you could get a yaml –> atmos(terraform) –> waypoint workflow going on. pretty sure.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andy Miguel (Cloud Posse) power outage

Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)

ack

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I won’t return probably

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Cc @matt

Zoom avatar
Zoom
07:10:32 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
07:11:50 PM

Zachary Loeber has joined Public “Office Hours”

Zoom avatar
Zoom
07:12:53 PM

Yusuf Adeyemo has joined Public “Office Hours”

Zoom avatar
Zoom
07:21:53 PM

Jim Park has joined Public “Office Hours”

Blaise Pabon avatar
Blaise Pabon

@Erik Osterman (Cloud Posse): waypoint looks like a serverless deployment tool (like OpenFAAS)… am I right?

1
Mazin Ahmed avatar
Mazin Ahmed

From what I understood, waypoint is like serverless, but 10x more dynamic, can be broader for K8S and other deployments

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

heh, well, this is the enigma of waypoint, it’s different things to different people

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

IMO, it’s honestly flat-out a CI/CD platform. It’s self hosted. It offers some “providers” (e.g. jenkins plugins). It handles build. It handles deploy. It handles release.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So in otherwords, waypoint would be used to build, deploy, and release serverless apps, k8s apps, etc.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it presents a clean HCL DSL for defining it (as opposed to the YAML approach favored by circle, github actions, gitlab ci, etc)

Mazin Ahmed avatar
Mazin Ahmed

It’s really interesting!! I’m definitely giving it a try, hopefully will be able to use it for production

Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)
Using Waypoint Runners To Enable GitOps Workflows attachment image

Waypoint runners perform builds, deployments, poll for Git repository changes, and allow deployments for any platform.

Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)
HashiTalks: Build

An event for the HashiCorp community of programmers, developers, and builders on Thursday, 22 July 2021.

zadkiel avatar
zadkiel

Hey there, just chiming in to react to last week episode. Terragrunt knows how to provision S3 (and gcs) buckets by itself before triggering terraform. I love these podcasts, don’t stop!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, but at the same rate, it’s kind of awkward that we’re using terraform for IaC and yet the statebucket is not controlled by terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

modifying that state bucket (e.g for compliance) in Terragrunt requires a PR to terragrunt, vs managing it in terraform https://github.com/cloudposse/terraform-aws-tfstate-backend

GitHub - cloudposse/terraform-aws-tfstate-backend: Terraform module that provision an S3 bucket to store the `terraform.tfstate` file and a DynamoDB table to lock the state file to prevent concurrent modifications and state corruption. attachment image

Terraform module that provision an S3 bucket to store the terraform.tfstate file and a DynamoDB table to lock the state file to prevent concurrent modifications and state corruption. - GitHub - c…

1

2021-07-19

2021-07-14

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:54 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Heath Snow avatar
Heath Snow

I’m curious what the test workflow looks like in the cloudposse Terraform repositories. Rather selfishly I want to finish up this PR and the feedback loop + testing methodology is keeping me from completing it (and thus I’m using a fork in the mean time). Not sure if this is the place to go about it.

Remove provider config by heathsnow · Pull Request #49 · cloudposse/terraform-aws-vpc-peering-multi-account

what Remove the requester/accepter provider configuration from the module. Update minimum Terraform version to 0.15.0 (using configuration_aliases wasn&#39;t working with 0.14.x) why Provider co…

Zoom avatar
Zoom
06:28:13 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:39 PM

Vicken Simonian has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:43 PM

Ossie Botu has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:44 PM

Emile Fugulin has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:49 PM

David Scott has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:44 PM

Yuri Lima has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:55 PM

Andy Miguel (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:57 PM

Sam C has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:35 PM

Jailson Silva has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:54 PM

Yusuf Adeyemo has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:09 PM

Anere Faithful has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:12 PM

Michael Holt has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:12 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:26 PM

Mohammed Yahya has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:57 PM

Thayne Trevenen has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:53 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:10 PM

Neil Gealy has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:37 PM

imran.hussain has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:53 PM

Brad Janke has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:01 PM

Denys has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:25 PM

Andy Roth has joined Public “Office Hours”

Zoom avatar
Zoom
06:34:50 PM

Heath Snow has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:24 PM

Neil Gealy has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:37 PM

Marc Slayton has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:41 PM

Felipe Sakatauskas has joined Public “Office Hours”

Zoom avatar
Zoom
06:36:50 PM

Jailson Silva has joined Public “Office Hours”

Zoom avatar
Zoom
06:37:33 PM

Stevan Arychuk has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:14 PM

shreenu kumar has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:31 PM

Graziele Vasconcelos has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:34 PM

Charles Sperbeck has joined Public “Office Hours”

Zoom avatar
Zoom
06:40:26 PM

Soham Dutta has joined Public “Office Hours”

Zoom avatar
Zoom
06:41:19 PM

Fernando Sanz has joined Public “Office Hours”

Zoom avatar
Zoom
06:42:51 PM

Tim Gourley has joined Public “Office Hours”

roth.andy avatar
roth.andy

Has anyone heard any updates on when Kubernetes will fix the issue with Jobs and InitContainers?

Zoom avatar
Zoom
06:46:18 PM

Eric Berg has joined Public “Office Hours”

Zoom avatar
Zoom
06:46:54 PM

Nick James has joined Public “Office Hours”

Zoom avatar
Zoom
06:48:49 PM

Mohammed Yahya has joined Public “Office Hours”

Zoom avatar
Zoom
06:50:41 PM

Denys has joined Public “Office Hours”

Zoom avatar
Zoom
06:53:31 PM

emem umoh has joined Public “Office Hours”

Zoom avatar
Zoom
06:55:52 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
06:55:56 PM

Blaise Pabon has joined Public “Office Hours”

Zoom avatar
Zoom
07:06:34 PM

Othman Musleh has joined Public “Office Hours”

Zoom avatar
Zoom
07:07:29 PM

Neil Gealy has joined Public “Office Hours”

Zoom avatar
Zoom
07:07:30 PM

PePe Amengual has joined Public “Office Hours”

matt avatar
2
sytten avatar
sytten

If I remember correctly it had trouble with path based, but host based works great

Mohammed Yahya avatar
Mohammed Yahya
nginx-proxy attachment image

nginx-proxy has 4 repositories available. Follow their code on GitHub.

Mohammed Yahya avatar
Mohammed Yahya

thanks, yes I’m looking for path based

Zoom avatar
Zoom
07:11:31 PM

Rizky Ramadhan has joined Public “Office Hours”

Zoom avatar
Zoom
07:16:56 PM
matt avatar
Best practices for tagging your systems in Datadog attachment image

Learn how you can make the most of your tags in Datadog.

1
SweetOps avatar
SweetOps
08:40:02 PM
[Managing Customer KMS Keys Cloud Posse Explains](https://www.youtube.com/watch?v=t3Ecgx8NYkM)
1
SweetOps avatar
SweetOps
09:50:18 PM
2
Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

RE: aws-controllers-k8s - IAM is not supported, this was the biggest bummer for us and we gave up, moved all to terraform

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

also if you read the thread - there’s really no way to implement it securely. You will end up giving a controller iam:* which is a huge hole

Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

I think this is the main showstopper there

Tim Birkett avatar
Tim Birkett

A team i work with has been using crossplane.io with some success.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wow, pretty useless without the ability to create IAM. In the end, we’re back to using something like terraform.

1

2021-07-12

Mohammed Yahya avatar
Mohammed Yahya

I have docker-compose to mange many solutions like gitlab, vault, jenkins, nexus, awx, selenium, nifi, spark, sonarqube, custom apps, pgadmin, portainer, minio, and I need a solid reverse proxy to replace apache httpd:

  1. Nginx
  2. Consul
  3. Traefik What you think about this?
jose.amengual avatar
jose.amengual

you forgot Varnish

I have docker-compose to mange many solutions like gitlab, vault, jenkins, nexus, awx, selenium, nifi, spark, sonarqube, custom apps, pgadmin, portainer, minio, and I need a solid reverse proxy to replace apache httpd:

  1. Nginx
  2. Consul
  3. Traefik What you think about this?
1
jose.amengual avatar
jose.amengual

which was built as a reverse proxy

Mohammed Yahya avatar
Mohammed Yahya
Very excited to announce [@AquaSecTeam> has acquired [@tfsec_dev>! I will be joining Aqua along with <https://twitter.com/owenrum @owenrum](https://twitter.com/tfsec_dev) to work full-time on the project - watch this space! <https://www.aquasec.com/news/aqua-security-acquires-tfsec/](https://twitter.com/AquaSecTeam)
1

2021-07-10

2021-07-09

Matt Gowie avatar
Matt Gowie

I believe we’ve talked about this before on office hours but I’m going to bring it up again as it’s still a topic that I feel doesn’t get enough attention:

How to manage Terraform dev/stage/prod releases when you’re utilizing terraform workspaces > the directory structure approach?

2
Matt Gowie avatar
Matt Gowie

For my largest terraform project (couple dozen root modules), we utilize develop , master , and release branches to add some release process around promoting changes. This works well in some regards, but it’s also difficult in that upgrading our automation to tf v1.0 for example requires that we roll everything forward as fast as possible because that change isn’t held in isolation on the branch.

Matt Gowie avatar
Matt Gowie

Anyway, looking to refresh on this topic. And particularly hear about how CloudPosse handles it.

Zach avatar

My company was using branches the same way when I arrived and I got us off it as fast as I could for those same reasons and moved to directory separation. It was an exhausting process managing promotion of changes across branches.

2
Mohammed Yahya avatar
Mohammed Yahya

the most asked question, I would move away from branches and use folder separation, we also used dev,qa, and prod branches before, it was a nightmare. suggest to only use features short-life branches and PRs for review the code added. everyone is talking about mono-repo mono-branch as single source of truth, with daily PRs.

Mohammed Yahya avatar
Mohammed Yahya

now the magic happen in your CICD, it should be smart to know which folder to apply in which order, you can control releasing by promoting changes through environments, with simple testing at the end of the delivery to make a perfect deployment.

Mohammed Yahya avatar
Mohammed Yahya

I like the idea to apply git-flow into Terraform, but I guess only small part of it could be helpful in IaC

Zach avatar


it was a nightmare
heavy emphasis on this

Matt Gowie avatar
Matt Gowie

Yeah interesting that both of you have this position. I hear ya… though I don’t exactly know how I would make that fit into the SweetOps methodology. Will be a good topic for discussion.

2021-07-08

Eric Berg avatar
Eric Berg

I’d like to talk about approaches to distinguishing between (datadog) metrics from our internal services (EKS) and the infrastructure services, like DD agent, k8-cni, etc.

Obviously, one approach is to add a tag to each resource that identifies it as internal. Interested in hearing others’ experiences

2021-07-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:24 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Zoom avatar
Zoom
06:28:20 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:30 PM

Eric Berg has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:34 PM

Vlad Ionescu has joined Public “Office Hours”

Zoom avatar
Zoom
06:28:36 PM

Thayne Trevenen has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:28 PM

Frank Scalzo has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:41 PM

Ossie Botu has joined Public “Office Hours”

Zoom avatar
Zoom
06:29:59 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:08 PM

David Hawthorne has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:15 PM

Ian Bartholomew has joined Public “Office Hours”

Zoom avatar
Zoom
06:30:35 PM

Ossie Botu has joined Public “Office Hours”

Zoom avatar
Zoom
06:31:31 PM

Florain Drescher has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:22 PM

Antarr Byrd has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:26 PM

Luis Masaya has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:31 PM

Evan Pitstick has joined Public “Office Hours”

Zoom avatar
Zoom
06:32:34 PM

Yusuf Adeyemo has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:40 PM

Jeremy (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
06:33:50 PM

vikram yerneni has joined Public “Office Hours”

Zoom avatar
Zoom
06:35:51 PM

Oliver Schoenborn has joined Public “Office Hours”

Zoom avatar
Zoom
06:38:02 PM

PePe Amengual has joined Public “Office Hours”

Zoom avatar
Zoom
06:39:47 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Sorry everyone - my connection is too bad we had to end early.

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

No worries! It happens to everybody!

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

#hugops

1
Mohammed Yahya avatar
Mohammed Yahya

no issue, although I’m glad because I missed it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks guys

    keyboard_arrow_up