#office-hours (2021-01)

Probably better to open an issue for this one.

@jose.amengual have you run into this?

not exactly

We enabled a capacity provider which caused and issue similar

maybe you have a capacity provider at the cluster level @Weston Platter?

Checking ….

I don’t have a capacity provider configured. Do I need that?

no no you do not

Capacity providers are a completely different animal

gotcha. I’ll go ahead and open a github issue.

Thank you for announcing these! I totally forgot what day it was

https://cloudposse.com/devops/office-hours/

I just find it haha thanks!

Related thread: https://sweetops.slack.com/archives/CCT1E7JJY/p1609797027221600

This is a good question. Will pin it for next week.

Excellent. I have been wrestling with this for a while. Obviously there are exceptions. But I like the multi-account member account approach to help isolate and demarc things, but I also worry about sprawl and have seen for some plans of a 100 or more accounts under an org. Myself I see it being more approachable somewhere in the 10 - 25 range.

I don’t have any news yet on this. We’re waiting probably until > Q2 before taking a serious look at it.

Very excited about what it is aspiring to do.

I am too.. but… it’s always tricky with projects which try to do “allthethings”. If you start on a clean slate it can be absolutely wonderful.

Going all-in on something, regardless of what it is though… can be amazing.

I wasn’t in office hours. Is this in reference to a particular tool, or the DevOps world in general?

Overall I totally agree, but I think there have been a couple of things that should be considered a best practice and used by all/almost all

1. Use Git
2. Use a modern Git-based VCS like GitHub/GitLab/BitBucket/etc
3. Do CI, with an automated testing pipeline
4. Containerize

Other more controversial ones that are, in my opinion, best practices

1. Kubernetes in all but the most basic of use cases
2. Throw out your style guide and automate it with hooks/CI/etc
3. …I’ll think about some more. I really like going over stuff like this 

IaC, probably in the top list…

This was a general statement. The point I was trying to make is that “Best Practice” today feels less like “use drbd, pacemaker, corosync and this my.cnf” and more like the following picture:

But I agree, there are definitely some practices that are mature and broadly adopted, like your list above.

https://youtu.be/KV9R9rB5J8U

AHA!

The 4 layers of infrastructure! They’re explained starting here: https://youtu.be/fVRy3qpTxME?t=2249

Yes! that was it.

I’ve also added the lucid chart here: https://cloudposse.com/big-picture/

but I think i’ll change it to a image so it’s easier to share

Updated diagram:

Love it, and an awesome visualisation, but no package management? Possibly with increasing awareness, as more orgs adopt protection against things like the package name squatting fiasco, or utilising isolation to protect from third-parties.

There is a second edition of this book - https://www.amazon.com/gp/aw/d/1492046906/ref=dp_ob_neva_mobile

Yes. I am reading this one!

interesting, I think they will splits the PRs

awesome to use

here is a use case in my ecs-cluster module https://github.com/mhmdio/terraform-aws-ecs-cluster/blob/master/variables.tf

Looks like AWS SSO assignment resources are dropping today in v3.24.0 — https://github.com/hashicorp/terraform-provider-aws/issues/15108#issuecomment-760421304

wow Matt you’er fast

Not using it, recommending it or promoting it any more for the same reasons we don’t recommend github actions

use a purpose built platform. don’t roll your own using a general purpose CI/CD solution.

So neg on the Gitlab and GitHub. But you still like and use Codefresh right? I ask as we have enterprise Bitbucket which has CI/CD capabilities. Im not sure I like it for more than a repo though.

Yes we still use a lot of Codefresh.

But based on some of our engagements last year, I just don’t recommend building your own terraform CD solution. The problem I think is that teams who want to do it are solving the wrong problem and haven’t yet practiced gitops enough to know the challenges. If after having used TACOS, the team still believes it can do a better job, then they can try it.

https://youtu.be/4MLBpBqZmpM

We also use a lot of GitHub actions. Love them. But we don’t use them for terraform .

(Other than for automated testing)

I tried using develop and master and found it hard to manage. So, I just do PR’s off of master. I create tags for each module as well similar to this source = git@<name-of-repo?ref=v1.0.0 so that when the module changes it’s not pointing to a local reference of my modules. There could be better ways to do it though. We are running terraform locally though and have a small team. I’d be interested in what others are doing as well.

I tried lot of approaches, let me list them first, then discuss them

• one repo - a branch per ENV

• multiple repos - a repo per env

• one repo - a folder per ENV all of these has pros and cons, but I move to something called stacks

So for any account I have stacks, one for network, others for Data, compute, app, CICD, so any account use one repo, and in that repo I have stacks each stack points to folder. then using a makefile I can have the sequence I want, and using remote state data source I can any output I want.

• one account reflect an ENV in one repo, and within multiple stacks with a makefile If I need to do any change I would create aPR on target account repo, run TF plan, approve it and merge, then run terraform apply, branch name cloud be issue-X or feat-X that match our backlog

some PR require changes on multiple stacks, and that can be easily done with this approach

Hi Mohammed, I use one repo - a branch per ENV after I tried every one of those solutions. And using seperate organizational accounts for envs. This solved a lot of problem for me like configuration differentiation.

https://youtu.be/xih7VqDe91I

Catalogs is a general concept

Here’s our catalog of datadog monitors:

https://github.com/cloudposse/terraform-datadog-monitor/tree/master/catalog/monitors

Here’s our catalog for AWS Config (with of the rules for CIS 1.2)

https://github.com/cloudposse/terraform-aws-config/tree/master/catalog

…so now we are extending our catalog approach to infrastructure using “Stacks”

and since we’ve abstracted the concept of a stack as YAML, now we can support that with virtually any TACOS provider.

Like scalr

Ah what did the DD monitor module not have that you were looking for? I was just in there, so I’m curious.

1 - preset messages already built (maybe you had that too) 2 - 10-20 prebuilt monitors for different services already ready. I wanted to convert but didn’t have time. https://registry.terraform.io/modules/claranet/monitors/datadog/latest

I would like to use cloudposse, but need to have time to convert the checks into yaml to use that.

@sheldonh did you see the catalog?

https://github.com/cloudposse/terraform-datadog-monitor/tree/master/catalog/monitors

There’s a lot there, but maybe not for your services

e.g. we have 20 monitors just for EKS

We’ll gladly accept PRs for more monitors that we can add to our catalog.

If you compare our module to the one by claranet, it’s a pretty big difference. The claranet one requires a submodule for every one.

While using the YAML config pattern, we add monitors very easily:

k8s-deployment-replica-pod-down:
  name: "(k8s) Deployment Replica Pod is down"
  type: query alert
  query: |
    avg(last_15m):avg:kubernetes_state.deployment.replicas_desired{*} by {cluster_name,deployment} - avg:kubernetes_state.deployment.replicas_ready{*} by {cluster_name,deployment} >= 2
  message: |
    ({{cluster_name.name}}) More than one Deployments Replica's pods are down on {{deployment.name}}
  escalation_message: ""
  tags: [ "ManagedBy:Terraform" ]
  notify_no_data: false
  notify_audit: true
  require_full_window: true
  enable_logs_sample: false
  force_delete: true
  include_tags: true
  locked: false
  renotify_interval: 0
  timeout_h: 0
  evaluation_delay: 60
  new_host_delay: 300
  no_data_timeframe: 5
  threshold_windows: { }
  thresholds:
    critical: 2

Yeah I’m going to convert over today.

Here’s the catch, I need to be able to copy json from building a manual monitor then codify.

I’m thinking I copy json to use tool to flip to yaml. If the schema is the same as the datadog json then I just solved my problem . Will look I don’t like the module verbosity of the clarinet one but I did appreciate prebuilt monitors for rds etc. I can build those myself those if I can flip to yaml quickly. If that works I’ll probably ride up a quick blog post on how I did it for folks and see about adding some more monitors to the library. Thanks!

@sheldonh Cool you’ll be using it! Interested in hearing how it goes.

Building out that catalog would be really cool. I haven’t added anything myself yet as the monitors I’ve added to client projects are too narrow in scope, but I hope to do so over the coming months for sure.

yep, thanks sheldon for the update. if there’s any generic monitors (e.g. for RDS), we’d welcome those! If you’re struggling with how to do something, do reach out and we’ll probably be able to answer how we handle it (or htink about it)

I have a suggestion, not sure if makes sense, but here it goes.

Slight schema modification to the yaml structure for monitors. With a slight change you could basically take a manually created monitor and use the export json to “flip” to yaml and plug it in without many tweaks. Any extra properties not included in the default json should be optional.

Ie

{jsoncontent} | yq/cfn-flip Now i have a formatted yaml snippet I could plug into the monitor. I had to work through problems though as “options” is used to group the settings and not in the cloudposse yaml.

What do you think? Worth a PR/open issue? Would make it easier to add monitors by doing manually, confirming they work then codifying i think.

(The schema came from https://github.com/FairwindsOps/astro)

can you share what the schema should look like?

maybe open issue for documentation purposes.

Would be glad to. It’s minor changes very minimal but would make it super easy to pull in any new monitor. I only had to make a few tweaks and make a few variables optional and it works. I’ll try to get something up there soon. Cheers

yes, strongly agree.

there’ll necessarily be complexity with time as we’re doing more. there’ll be different ways to address it that depend on our world view. we’re only shifting it around.

I want to talk more about this later this quarter, once we have more documentation on our latest strategy. I also want to compare it in matrix

It’s a master class in atmos.

I’d like to say it was “Easy”, but it’s been quite challenging. We’ve had to work a lot with Mumoshu to get to where we are on it today.

We’re also developing a companion cli in pure-go

still deciding on name of that one. that cli is for working around limitations in terraform providers in order for us to provde SOC2 compliance for customers.

Tuning in a bit late, but is there something specifically I could review regarding the SOC 2 Compliance issues your customers are having and how that relates to TF? Very similar goals on this end so trying to head off any frustration before audits begin

Our workarounds are going into https://github.com/cloudposse/posse-cli (which will be renamed to turf)

addresses those things there’s no way to do in pure terraform

You all are amazing as usual. I’m excited to see where this project goes. Would you say I could easily replace “runner scripts” now with this tooling or a bit difficult as designed for your specific workflow?

This is the quintessential essay by PG: http://www.paulgraham.com/makersschedule.html

This just occurred to me, so it may not quite be fully baked. This falls under psychological safety, although the context for that is usually regarding feeling safe to share opinions, ask questions, etc. But people need to feel safe to be silent and singly focused, too. The pressure to always have an opinion or answer to share or to context switch frequently creates a cognitive overload.

did this get talked about already? I joined a little late

Nope

(coming at this topic from a TPM perspective) one issue I found with daily stand ups is they are ideal for a team working towards similar goals, where the team shares responsibility on the same stories/tasks. DevOps teams usually are supporting things in silos (shared services), so stand ups become less meaningful since each person isn’t always very invested in what everyone else is doing. the opportunities for collaboration are there, but are arguably better handled over Slack as one-offs.

there’s also something to be said about expectations of availability. if your team expects you to respond to slack messages and such in a timely manner, this can pose a similar problem as the meetings where you’re in a good rhythm on your task and then you get taken out of it to context switch.

@Doug Lane (he/him) definitely agree.

It provides a way to deploy kubernetes yaml description files through out TF. It solves another problem that TF kubernetes provider was following new K8s releases from the back. That latency will be part of the history after this featue.

Note: For now, I am using https://github.com/leiarenee/terraform-kubernetes-yaml to deploy bare K8s yaml files under TF

I’ll bring this up next week

https://geekbot.com/

Ah interesting, thank you

https://sweetops.slack.com/archives/CHDR1EWNA/p1611779488027900

via @matt https://standuply.com/

yes

Hi Andrew, Which tools are using for async communication?

Standup bot and Dixi, am I right?

Dixi is the async bot

It DMs you in slack and asks you the typical 3 questions

/cc @Erik Osterman (Cloud Posse) @matt

Listing your blockers as soon as you start working has the potential for people to help with the blockers before Standup even happens. I regularly will list a blocker and a teammate will reply in the Thread and offer to help with it right away

Cool @roth.andy Thnks.

Hi @matt My real blocker is the meeting itself. That’s the paradox.

Anyway, next week I’d like to get the rest of Erik’s response about how CloudPosse does meetings before we got derailed on standups

Some of the problems I want to solve:

• what’s the relationship of what you’re working on and JIRA. is there a ticket? if not, why and let’s make sure there is one. JIRAs are how we “manage up”

• why has the same jira been “in progress” for 2 weeks. What’s the delta, what’s evolved?

• Is each of these issues in jira really still blocked? If so, what’s the next action. Who is that next action assigned to? How do we track that without creating even more process.

• How do we identify that the approach being taken is the right approach. Sprint planning will help identify what we need to get done. But frequently we don’t know what we don’t know; it’s not until we start working on the problem. THen we see the scope change. Assessing these changes in scope are what I think get missed in async standups. From the developer perspective, it’s very transactional: i’m assigned work, i get that work finished. From the business perspective, it’s different, not all work needs to get finished. Waiting a week or to the end of a sprint is too late. From the developer’s perspective, they are not blocked: they know what the next steps are. That’s not sufficient though.

• If the standups are just reflecting the status of what’s in jira, it’s not useful - we already get that from jira

• If the standups are not reflecting the status of what’s in jira, why not? (it’s a catch 22) I guess my point is there’s this subtle, hard to automate process in our standups.

I like the notion of what standuply does. It integrates with Jira and GitHub so it can pull in a lot of tedious data entry and reflection of “what did I do?”. What I’d like to see is how do we produce meaningful data that is not just a regurgitation of what we already know.

Yes that was me actually who was responsible for derailing.

I agree on you and I’m keen to listen Erik’s meeting experiences as a manager next week.

Poking fun at meetings is the stuff of Dilbert cartoons—we can all joke about how soul-sucking and painful they are. (from HBR)

I thing the problem is about multi tasking. Developers most of the time are expected to work with multiple tasks in parallel. This inevitably creates an uncertainty and disrupts the estimation. Only one task per developer should be on developmentstage at a time, so that estimations can be achieved at least approximately.