#office-hours (2021-01)

Meeting password: sweetops

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cpco.io/slack-office-hours

Meeting password: sweetops

2021-01-21

2021-01-20

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:00:26 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Mohammed Yahya avatar
Mohammed Yahya
Will this affect Terraform ?
<https://aws.amazon.com/about-aws/whats-new/2021/01/aws-sdk-for-go-version-2-now-generally-available/>
Matt Gowie avatar
Matt Gowie

Does anyone follow a git flow (i.e. develop + master branches) pattern for larger Terraform repositories? How has that worked out for you?

Dahs81 avatar
Dahs81

I tried using develop and master and found it hard to manage. So, I just do PR’s off of master. I create tags for each module as well similar to this source = [email protected]<name-of-repo?ref=v1.0.0 so that when the module changes it’s not pointing to a local reference of my modules. There could be better ways to do it though. We are running terraform locally though and have a small team. I’d be interested in what others are doing as well.

1
Mohammed Yahya avatar
Mohammed Yahya

I tried lot of approaches, let me list them first, then discuss them

• one repo - a branch per ENV

• multiple repos - a repo per env

• one repo - a folder per ENV all of these has pros and cons, but I move to something called stacks

So for any account I have stacks, one for network, others for Data, compute, app, CICD, so any account use one repo, and in that repo I have stacks each stack points to folder. then using a makefile I can have the sequence I want, and using remote state data source I can any output I want.

• one account reflect an ENV in one repo, and within multiple stacks with a makefile If I need to do any change I would create aPR on target account repo, run TF plan, approve it and merge, then run terraform apply, branch name cloud be issue-X or feat-X that match our backlog

some PR require changes on multiple stacks, and that can be easily done with this approach

PePe avatar

So this just happened to me: working on a module for a project using a dev environment but I have another co-worker working in another branch and then she did TF apply and I just got a Your query returned no results and I thought I broke something ( we use atlantis for other projects so this that do not happen) is there a way to check the state if it was changed ( like doing a git pull) ?(keep in mind in this case was a data. resource so is not going to be recreated)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Is there anything other than tfenv that provides that smooth experience for various terraform versions. Maybe a docker driven approach that’s not hideous to look at with something like whalebrew or the like?

Also I kinda wanted on installing a new version for it to prompt me to set as default instead of having to do 2 commands so before I dive into exploring submitting a PR or something on that, would like to know if it’s still the best tool to use for managing various versions of terraform

Zoom avatar
Zoom
07:28:54 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:05 PM

David Lundgren has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:12 PM

Phil Hershkowitz has joined Public “Office Hours”

btai avatar

any tips for improving global s3 upload speed? (think india, hong kong, etc) what other optimizations could I possibly make after turning on s3 transfer acceleration and using multipart uploads?

Zoom avatar
Zoom
07:29:19 PM

mb Branski has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:19 PM

Sam C has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:19 PM

Colton Wrisner has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:27 PM

James Thalacker has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:45 PM

Brian Tai has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:54 PM

Jim Park has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:02 PM
Zoom avatar
Zoom
07:30:25 PM

Cosmin Drimba has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:29 PM

Andy Miguel (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:57 PM

Patrick Jahns has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:16 PM

Tim Gourley has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:23 PM

Sebastian Stadil has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:27 PM

Mansoor Ebrahim has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:31 PM

Cosmin Drimba has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:53 PM

James Haughey has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:56 PM

Bill Clark has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:07 PM

15139103984 has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:36 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:22 PM

Srikar Ananthula has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:40 PM

Neil Gealy has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:48 PM

Leia Renée has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:53 PM

Andy Roth has joined Public “Office Hours”

Zoom avatar
Zoom
07:34:14 PM

Mikael Fridh has joined Public “Office Hours”

Zoom avatar
Zoom
07:34:34 PM

PePe Amengual has joined Public “Office Hours”

Zoom avatar
Zoom
07:34:49 PM

venkatamutyala has joined Public “Office Hours”

Zoom avatar
Zoom
07:34:58 PM

Todd Thomas has joined Public “Office Hours”

Zoom avatar
Zoom
07:35:06 PM

Guelor Emanuel has joined Public “Office Hours”

Zoom avatar
Zoom
07:35:33 PM

Loren Gordon has joined Public “Office Hours”

Zoom avatar
Zoom
07:36:18 PM

Mohammed Yahya has joined Public “Office Hours”

Zoom avatar
Zoom
07:38:48 PM

Eric Berg has joined Public “Office Hours”

Zoom avatar
Zoom
07:39:03 PM

Oskar Maria Grande has joined Public “Office Hours”

Zoom avatar
Zoom
07:40:01 PM

mike dizon has joined Public “Office Hours”

Zoom avatar
Zoom
07:40:01 PM

Vicken Simonian has joined Public “Office Hours”

Zoom avatar
Zoom
07:40:47 PM

Petros Kolyvas has joined Public “Office Hours”

Zoom avatar
Zoom
07:41:31 PM

Robert Jackson has joined Public “Office Hours”

Sebastian Stadil avatar
Sebastian Stadil

Question re: office hours. What types of internal users have you seen typically use these prebuilt stacks / catalog?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Catalogs is a general concept

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s our catalog of datadog monitors:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s our catalog for AWS Config (with of the rules for CIS 1.2)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-config

This module configures AWS Config, a service that enables you to assess, audit, and evaluate the configurations of your AWS resources. - cloudposse/terraform-aws-config

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…so now we are extending our catalog approach to infrastructure using “Stacks”

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and since we’ve abstracted the concept of a stack as YAML, now we can support that with virtually any TACOS provider.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Like scalr

Zoom avatar
Zoom
07:45:45 PM

Blaisep has joined Public “Office Hours”

Zoom avatar
Zoom
07:46:10 PM

Jose Franco has joined Public “Office Hours”

Zoom avatar
Zoom
07:46:28 PM

Sheldon Hull has joined Public “Office Hours”

Michael Dizon avatar
Michael Dizon

Are you guys going to update examples in atmos? I’d love to get it going!

sheldonh avatar
sheldonh

Whatever cloud provider, a registry for easy review and calling of modules is key. I checked and Env0 didn’t have it at this time. Scalyr i think did. Spacelift i haven’t checkout out, but will look too.

I really want to towards that curated high quality modules for teams to use this way.

Zoom avatar
Zoom
07:54:24 PM

Dennis Lipovsky has joined Public “Office Hours”

Zoom avatar
Zoom
07:56:51 PM

Miguelangel Freitas has joined Public “Office Hours”

sheldonh avatar
sheldonh

Cloudposse modules are epic I’m always looking to leverage them. I just painfully used a new datadog monitor project that wasn’t Cloudposse as it was a bit more extensive and it felt like pulling teeth.

Much

Matt Gowie avatar
Matt Gowie

Ah what did the DD monitor module not have that you were looking for? I was just in there, so I’m curious.

sheldonh avatar
sheldonh

1 - preset messages already built (maybe you had that too) 2 - 10-20 prebuilt monitors for different services already ready. I wanted to convert but didn’t have time. https://registry.terraform.io/modules/claranet/monitors/datadog/latest

I would like to use cloudposse, but need to have time to convert the checks into yaml to use that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@sheldonh did you see the catalog?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-datadog-monitor

Terraform module to configure and provision Datadog monitors from a YAML configuration, complete with automated tests. - cloudposse/terraform-datadog-monitor

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There’s a lot there, but maybe not for your services

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. we have 20 monitors just for EKS

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’ll gladly accept PRs for more monitors that we can add to our catalog.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you compare our module to the one by claranet, it’s a pretty big difference. The claranet one requires a submodule for every one.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

While using the YAML config pattern, we add monitors very easily:

k8s-deployment-replica-pod-down:
  name: "(k8s) Deployment Replica Pod is down"
  type: query alert
  query: |
    avg(last_15m):avg:kubernetes_state.deployment.replicas_desired{*} by {cluster_name,deployment} - avg:kubernetes_state.deployment.replicas_ready{*} by {cluster_name,deployment} >= 2
  message: |
    ({{cluster_name.name}}) More than one Deployments Replica's pods are down on {{deployment.name}}
  escalation_message: ""
  tags: [ "ManagedBy:Terraform" ]
  notify_no_data: false
  notify_audit: true
  require_full_window: true
  enable_logs_sample: false
  force_delete: true
  include_tags: true
  locked: false
  renotify_interval: 0
  timeout_h: 0
  evaluation_delay: 60
  new_host_delay: 300
  no_data_timeframe: 5
  threshold_windows: { }
  thresholds:
    critical: 2
sheldonh avatar
sheldonh

For what it’s worth I’ve read that this type of concept:

You want to simplify, that’s great! Simplifying complex systems though often is basically shifting around complexity. In this case… do you want Terragrunt to manage the complexity and it’s own flow, or do you shift it to the user, or a yaml config for example.

I think it’s a case of where do you feel the complexity should be best moved.

Food for though.

1
Zoom avatar
Zoom
08:01:11 PM

Jeremy Branham has joined Public “Office Hours”

Zoom avatar
Zoom
08:01:35 PM

Udit Dave has joined Public “Office Hours”

Zoom avatar
Zoom
08:03:47 PM

Jose Franco has joined Public “Office Hours”

Zoom avatar
Zoom
08:05:13 PM

Hao Wang has joined Public “Office Hours”

sheldonh avatar
sheldonh

WOOT WOOT. I didn’t see atmos. Is this built on top of variant2?

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s a master class in atmos.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’d like to say it was “Easy”, but it’s been quite challenging. We’ve had to work a lot with Mumoshu to get to where we are on it today.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re also developing a companion cli in pure-go

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

still deciding on name of that one. that cli is for working around limitations in terraform providers in order for us to provde SOC2 compliance for customers.

Zoom avatar
Zoom
08:07:12 PM

Andrew Thompson has joined Public “Office Hours”

sheldonh avatar
sheldonh

Seriously you guys are on FIRE

Matt Gowie avatar
Matt Gowie
TFSwitch

A command line tool to switch between different versions of terraform (install with homebrew and more)

sheldonh avatar
sheldonh

asdf is about the ultimate expression of lazy typing i’ve ever seen. Absolutely love it!

sheldonh avatar
sheldonh

Go is HUGE on backward compatibility. They have a promise even on “deprecated” features to never break functionality.

I’d say even if they move to SDK v2, it’s not going to impact us overall.

sheldonh avatar
sheldonh

Pretty sure SDK v2 has been out for years before this announcement too, so it’s not “new”. It’s been in usage from 2018 i believe. It simplifies a lot of code too.

Zoom avatar
Zoom
09:48:57 PM

New Zoom Recording from our Office Hours session on 2021-01-20 is now available.

2021-01-14

2021-01-13

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:00:30 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

2
1
Zoom avatar
Zoom
07:30:12 PM

Andy Miguel has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:23 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:29 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:39 PM

Ian Groff has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:39 PM

Mohammed Yahya has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:40 PM

James Haughey has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:43 PM

Hemanth Kumar has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:43 PM

Bircan Bilici has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:46 PM

Tim Gourley has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:55 PM

Andy Miguel has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:04 PM

Patrick Joyce has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:07 PM

mb Branski has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:36 PM

15139103984 has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:36 PM

Patrick Jahns has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:01 PM

Udit Dave has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:26 PM

Mazin Ahmed has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:49 PM

Michael Martin has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:28 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
07:34:17 PM

Kenji Nakamura has joined Public “Office Hours”

Zoom avatar
Zoom
07:34:44 PM

Isa Aguilar has joined Public “Office Hours”

Zoom avatar
Zoom
07:35:19 PM

Bill Clark has joined Public “Office Hours”

Zoom avatar
Zoom
07:37:15 PM

Dave Lundgren has joined Public “Office Hours”

Patrick Jahns avatar
Patrick Jahns

Working with the SSO resources - but would be great to also define at least Groups via code

Zoom avatar
Zoom
07:39:27 PM

PePe Amengual has joined Public “Office Hours”

Zoom avatar
Zoom
07:44:00 PM

Kenji Nakamura has joined Public “Office Hours”

Mike Martin avatar
Mike Martin

@ You mentioned you were getting started with Terraform - this is a GREAT book for “getting up and running”. https://www.amazon.com/Terraform-Running-Writing-Infrastructure-Code/dp/1491977086

Andrey Nazarov avatar
Andrey Nazarov

There is a second edition of this book - https://www.amazon.com/gp/aw/d/1492046906/ref=dp_ob_neva_mobile

1
Bill Clark avatar
Bill Clark

Yes. I am reading this one!

Zoom avatar
Zoom
07:48:41 PM

Oskar Maria Grande has joined Public “Office Hours”

Matt Gowie avatar
Matt Gowie

@ I think this is one of the outstanding SSO resources that’s still in draft: https://github.com/hashicorp/terraform-provider-aws/pull/15322

I’ve been following that PR and it seems like that might be what you were missing.

[WIP] r/ssoadmin_account_assignment: new resource; d/identitystore: new data sources by burck1 · Pull Request #15322 · hashicorp/terraform-provider-aws

Update 2020/11/03 To help us to continue to move forward, please go give a thumbs up on #15808. We&#39;ve completed most of the work for supporting the AWS SSO and AWS SSO Identity Store resources …

1
Mohammed Yahya avatar
Mohammed Yahya

interesting, I think they will splits the PRs

[WIP] r/ssoadmin_account_assignment: new resource; d/identitystore: new data sources by burck1 · Pull Request #15322 · hashicorp/terraform-provider-aws

Update 2020/11/03 To help us to continue to move forward, please go give a thumbs up on #15808. We&#39;ve completed most of the work for supporting the AWS SSO and AWS SSO Identity Store resources …

1
Andy Miguel (Cloud Posse) avatar
Andy Miguel (Cloud Posse)
DevOps Engineer - Blue Pisces Consulting Inc | Built In Los Angeles attachment image

Blue Pisces Consulting Inc is hiring for a DevOps Engineer in Los Angeles. Find more details about the job and how to apply at Built In Los Angeles.

Mohammed Yahya avatar
Mohammed Yahya

https://www.terraform.io/docs/configuration/types.html#experimental-optional-object-type-attributes

variable "with_optional_attribute" {
  type = object({
    a = string           # a required attribute
    b = optional(string) # an optional attribute
  })
}
Type Constraints - Configuration Language - Terraform by HashiCorp

Terraform module authors and provider developers can use detailed type constraints to validate the inputs of their modules and resources.

Mohammed Yahya avatar
Mohammed Yahya

awesome to use

Type Constraints - Configuration Language - Terraform by HashiCorp

Terraform module authors and provider developers can use detailed type constraints to validate the inputs of their modules and resources.

Mohammed Yahya avatar
Mohammed Yahya
mhmdio/terraform-aws-ecs-cluster

Terraform module for AWS ECS Cluster. Contribute to mhmdio/terraform-aws-ecs-cluster development by creating an account on GitHub.

Zoom avatar
Zoom
07:51:02 PM

Shouky Dan has joined Public “Office Hours”

Zoom avatar
Zoom
07:51:56 PM

Marc Tamsky has joined Public “Office Hours”

Zoom avatar
Zoom
07:52:18 PM

mb Branski has joined Public “Office Hours”

Mohammed Yahya avatar
Mohammed Yahya
Book

Exploring better ways to build and manage cloud infrastructure

1
Zoom avatar
Zoom
07:53:16 PM

Andrew Thompson has joined Public “Office Hours”

Zoom avatar
Zoom
07:55:44 PM

rajiv ranjan has joined Public “Office Hours”

Mohammed Yahya avatar
Mohammed Yahya

use this CFN templates until TF AWS SSO is ready

Matt Gowie avatar
Matt Gowie

Looks like AWS SSO assignment resources are dropping today in v3.24.0https://github.com/hashicorp/terraform-provider-aws/issues/15108#issuecomment-760421304

Support for Managing AWS SSO Permission Sets · Issue #15108 · hashicorp/terraform-provider-aws

Community Note Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request Please do not leave &quot;+1&quot; or other comme…

1
1
Mohammed Yahya avatar
Mohammed Yahya

wow Matt you’er fast

1
Mohammed Yahya avatar
Mohammed Yahya
AWSTemplateFormatVersion: 2010-09-09

Description: Configure AWS SSO

Parameters:

  AwsSsoInsanceArn: 
    Type: String
    Default: 'arn:aws:sso:::instance/ssoins-XXXXXXXXXXXXX'
    Description: 'AWS SSO Instance ARN.'

Mappings:

  Groups:
    Admins:
      Id: 'XXXXXXX-xxxxx-xxxx-xxxx-xxxx-XXXXX'
    Developers:
      Id: 'XXXXXXX-xxxxx-xxxx-xxxx-xxxx-XXXXX'
    Developers-CodeCommit:
      Id: 'XXXXXXX-xxxxx-xxxx-xxxx-xxxx-XXXXX'
  Accounts:
    master:
      Id: '111111111111'
    shared:
      Id: '222222222222'
    dev:
      Id: '333333333333'
    prod:
      Id: '444444444444'

Resources:

  adminsPermissionSet:
    Type: AWS::SSO::PermissionSet
    Properties: 
      Description: Admins Group Administrator Access Permission Set
      InstanceArn: !Ref AwsSsoInsanceArn
      ManagedPolicies: 
        - arn:aws:iam::aws:policy/AdministratorAccess
      Name: Admins
      Tags: 
        - Key: 'CFN'
          Value: 'Yes'
        - Key: 'Project'
          Value: 'Awesome'
  
  developersPermissionSet:
    Type: AWS::SSO::PermissionSet
    Properties: 
      Description: Developers Group Read Only Access Permission Set
      InstanceArn: !Ref AwsSsoInsanceArn
      ManagedPolicies: 
        - arn:aws:iam::aws:policy/ReadOnlyAccess
      Name: Developers
      Tags: 
        - Key: 'CFN'
          Value: 'Yes'
        - Key: 'Project'
          Value: 'Awesome'

  developersCodeCommitPermissionSet:
    Type: AWS::SSO::PermissionSet
    Properties: 
      Description: Developers Group CodeCommit Access Permission Set
      InstanceArn: !Ref AwsSsoInsanceArn
      ManagedPolicies: 
        - arn:aws:iam::aws:policy/AWSCodeCommitPowerUser
      Name: Developers-CodeCommit
      Tags: 
        - Key: 'CFN'
          Value: 'Yes'
        - Key: 'Project'
          Value: 'Awesome'

  # masterAssignmentAdmins:
  #   Type: AWS::SSO::Assignment
  #   Properties:
  #     InstanceArn: !Ref AwsSsoInsanceArn
  #     PermissionSetArn: !GetAtt adminsPermissionSet.PermissionSetArn
  #     TargetId: !FindInMap [ Accounts, master, Id ]
  #     TargetType: 'AWS_ACCOUNT'
  #     PrincipalType: 'GROUP'
  #     PrincipalId: !FindInMap [ Groups, Admins, Id ]

  sharedAssignmentAdmins:
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref AwsSsoInsanceArn
      PermissionSetArn: !GetAtt adminsPermissionSet.PermissionSetArn
      TargetId: !FindInMap [ Accounts, shared, Id ]
      TargetType: 'AWS_ACCOUNT'
      PrincipalType: 'GROUP'
      PrincipalId: !FindInMap [ Groups, Admins, Id ]

  prodAssignmentAdmins:
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref AwsSsoInsanceArn
      PermissionSetArn: !GetAtt adminsPermissionSet.PermissionSetArn
      TargetId: !FindInMap [ Accounts, prod, Id ]
      TargetType: 'AWS_ACCOUNT'
      PrincipalType: 'GROUP'
      PrincipalId: !FindInMap [ Groups, Admins, Id ]

  devAssignmentAdmins:
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref AwsSsoInsanceArn
      PermissionSetArn: !GetAtt adminsPermissionSet.PermissionSetArn
      TargetId: !FindInMap [ Accounts, dev, Id ]
      TargetType: 'AWS_ACCOUNT'
      PrincipalType: 'GROUP'
      PrincipalId: !FindInMap [ Groups, Admins, Id ]

  devAssignmentDevelopers:
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref AwsSsoInsanceArn
      PermissionSetArn: !GetAtt developersPermissionSet.PermissionSetArn
      TargetId: !FindInMap [ Accounts, dev, Id ]
      TargetType: 'AWS_ACCOUNT'
      PrincipalType: 'GROUP'
      PrincipalId: !FindInMap [ Groups, Developers, Id ]

  prodAssignmentDevelopers:
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref AwsSsoInsanceArn
      PermissionSetArn: !GetAtt developersPermissionSet.PermissionSetArn
      TargetId: !FindInMap [ Accounts, prod, Id ]
      TargetType: 'AWS_ACCOUNT'
      PrincipalType: 'GROUP'
      PrincipalId: !FindInMap [ Groups, Developers, Id ]

  sharedAssignmentDevelopers:
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref AwsSsoInsanceArn
      PermissionSetArn: !GetAtt developersPermissionSet.PermissionSetArn
      TargetId: !FindInMap [ Accounts, shared, Id ]
      TargetType: 'AWS_ACCOUNT'
      PrincipalType: 'GROUP'
      PrincipalId: !FindInMap [ Groups, Developers, Id ] 

  sharedAssignmentDevelopersCodeCommit:
    Type: AWS::SSO::Assignment
    Properties:
      InstanceArn: !Ref AwsSsoInsanceArn
      PermissionSetArn: !GetAtt developersCodeCommitPermissionSet.PermissionSetArn
      TargetId: !FindInMap [ Accounts, shared, Id ]
      TargetType: 'AWS_ACCOUNT'
      PrincipalType: 'GROUP'
      PrincipalId: !FindInMap [ Groups, Developers-CodeCommit, Id ] 
1
Bill Clark avatar
Bill Clark

If you have time and interest talk about your Codefresh GitOps method. And did you look and consider the Gitlab terraform automation? Pros and Cons

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not using it, recommending it or promoting it any more for the same reasons we don’t recommend github actions

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

use a purpose built platform. don’t roll your own using a general purpose CI/CD solution.

Bill Clark avatar
Bill Clark

So neg on the Gitlab and GitHub. But you still like and use Codefresh right? I ask as we have enterprise Bitbucket which has CI/CD capabilities. Im not sure I like it for more than a repo though.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes we still use a lot of Codefresh.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But based on some of our engagements last year, I just don’t recommend building your own terraform CD solution. The problem I think is that teams who want to do it are solving the wrong problem and haven’t yet practiced gitops enough to know the challenges. If after having used TACOS, the team still believes it can do a better job, then they can try it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We also use a lot of GitHub actions. Love them. But we don’t use them for terraform .

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(Other than for automated testing)

Zoom avatar
Zoom
08:05:46 PM

Rohit Koimattur has joined Public “Office Hours”

Zoom avatar
Zoom
08:10:19 PM

Phil Hersh has joined Public “Office Hours”

Zoom avatar
Zoom
08:21:46 PM

PePe Amengual has joined Public “Office Hours”

Zoom avatar
Zoom
08:25:13 PM

hari b has joined Public “Office Hours”

Zoom avatar
Zoom
09:51:42 PM

New Zoom Recording from our Office Hours session on 2021-01-13 is now available.

2021-01-11

2021-01-09

Vlad Ionescu avatar
Vlad Ionescu

I know y’all explained the CloudPosse way of defining envs and propagating changes many times, but for the life of me I can’t find that recording. Anybody know where I can find it? That part of Office Hours where @Erik Osterman (Cloud Posse) explained the base Terraform infra, and then the DBs, and then the apps thing. With a flowchart with arrows and nice diagram of things building on top of the previous stages

1
Vlad Ionescu avatar
Vlad Ionescu

AHA!

The 4 layers of infrastructure! They’re explained starting here: https://youtu.be/fVRy3qpTxME?t=2249

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes! that was it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ve also added the lucid chart here: https://cloudposse.com/big-picture/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but I think i’ll change it to a image so it’s easier to share

2021-01-07

2021-01-06

Eric Berg avatar
Eric Berg

Question for today’s discussion: When deploying via helm, do you use hacked versions of the full values.yaml files or a file with just diffs? Which is better for managing upgrades of things like the datadog agent, going forward?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
07:00:29 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

1
Vlad Ionescu avatar
Vlad Ionescu

Thank you for announcing these! I totally forgot what day it was

Miguel Zablah avatar
Miguel Zablah

I just find it haha thanks!

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
By any chance, does anyone here have a multi-region kubernetes setup that still uses wildcard DNS? I have a single cluster with hundreds of ingresses like [foo.example.com> or bar.example.com and I had been thinking about moving to a multi-region setup where half of the ingresses would live in us-east and half in us-west, but would like to keep the wildcard dns setup as to not need to create a bunch of route53 records. I can’t use Route53 geo-based routing as users that have their site hosted in us-east could be accessing their site from a different location (i.e. california). To clarify, the reason that I want to add a cluster in a second region is to minimize blast radius and not for redundancy (<http://foo.example.com foo.example.com](http://foo.example.com) would only live on the us-east cluster OR the us-west cluster but not both)
Mohammed Yahya avatar
Mohammed Yahya

@Erik Osterman (Cloud Posse) Q : In https://github.com/cloudposse/reference-architectures#3-delegate-dns Can some one explains An available domain we can use for DNS-base service discovery (E.g. [ourcompany.co](http://ourcompany.co)). This domain must not be in use elsewhere as the master account will need to be the authoritative name server (SOA).

cloudposse/reference-architectures

[WIP] Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Zoom avatar
Zoom
07:28:09 PM

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Zoom avatar
Zoom
07:28:12 PM

Tarlan Isaev has joined Public “Office Hours”

Zoom avatar
Zoom
07:28:22 PM

charles pogi has joined Public “Office Hours”

Zoom avatar
Zoom
07:28:23 PM

Weston Platter has joined Public “Office Hours”

Zoom avatar
Zoom
07:28:24 PM

Vlad Ionescu has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:06 PM

Jesse Cafarelli has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:12 PM

uri unger has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:26 PM

Ken Y.y has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:37 PM

Raymond Mazurik has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:39 PM

Andy Miguel has joined Public “Office Hours”

Zoom avatar
Zoom
07:29:48 PM

Shouky Dan has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:07 PM

Adam Crown has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:20 PM

Julian Severino has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:40 PM

Mohammed Yahya has joined Public “Office Hours”

Zoom avatar
Zoom
07:30:42 PM

Brian Tai has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:06 PM

Shouky Dan has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:12 PM

Mikael Fridh has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:17 PM

Randy Bridges has joined Public “Office Hours”

Zoom avatar
Zoom
07:31:36 PM

Florain Drescher has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:30 PM

Matt Calhoun has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:33 PM

Bill Clark has joined Public “Office Hours”

Zoom avatar
Zoom
07:32:57 PM

Michael Jenkins has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:37 PM

Neil Gealy has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:38 PM

Matt Gowie has joined Public “Office Hours”

Zoom avatar
Zoom
07:33:44 PM

Brandon vh has joined Public “Office Hours”

Zoom avatar
Zoom
07:34:07 PM

Tim Gourley has joined Public “Office Hours”

Zoom avatar
Zoom
07:35:04 PM

15139103984 has joined Public “Office Hours”

Zoom avatar
Zoom
07:36:02 PM

Hyejin Song has joined Public “Office Hours”

Zoom avatar
Zoom
07:37:07 PM

Eric Berg has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add outputs id16, id32, id64, id128 by alexjurkiewicz · Pull Request #118 · cloudposse/terraform-null-label

Truncated forms of id_full which are always available. This is useful when you want to use the same label for several resources with different length restrictions. Closes #117.

Jim Park avatar
Jim Park

Is this per field?

Jim Park avatar
Jim Park

or is this total?

Zoom avatar
Zoom
07:39:06 PM

Troy Taillefer has joined Public “Office Hours”

Jim Park avatar
Jim Park

@Erik Osterman (Cloud Posse) how to preserve most significant digit being at the end of the id?

Zoom avatar
Zoom
07:39:30 PM

James Haughey has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
feat: add possability to use lowercased context tags by SweetOps · Pull Request #107 · cloudposse/terraform-null-label

what add possability to use lowercased context tags why not all cloud-providers supports the uppercased keys for tagging/labeling resources

Jim Park avatar
Jim Park

yes, thanks!

Zoom avatar
Zoom
07:43:15 PM

Martin Mazurik has joined Public “Office Hours”

Zoom avatar
Zoom
07:43:44 PM

Rashid Boyko has joined Public “Office Hours”

Zoom avatar
Zoom
07:44:35 PM

Abisoye Olaomi has joined Public “Office Hours”

Zoom avatar
Zoom
07:45:03 PM

Kareem Shahin has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Moving on from Amazon RDS for PostgreSQL Versions 9.4 | Amazon Web Services attachment image

Historically, the PostgreSQL community releases a new major version yearly, and with that, has a defined end of life (EOL) policy of older major versions. This allows version and upgrade decisions to be made on dates known well into the future. The community EOL policy is to support a major version for 5 years after […]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

theres no db slack channel, so I’m asking here since I’m using RDS (and theyre deprecating support for my postgres version). Anyone thats done the postgres 9 -> postgres 10/11 migration have any gotchas we should be concerned about when doing it?

Zoom avatar
Zoom
07:47:17 PM

Vicken Simonian has joined Public “Office Hours”

Zoom avatar
Zoom
07:48:51 PM

Petros K has joined Public “Office Hours”

Zoom avatar
Zoom
07:50:25 PM

Anere Faithful has joined Public “Office Hours”

Uri Unger avatar
Uri Unger

Question for me: We are adopting the terraform-aws-jenkins infra and are pretty impressed with it. I have noticed some issues with the use of EFS though which I wanted to ask about. I am not sure if we’re somehow doing it wrong or if this is a genuine issue. 1- throughput in terms of single file operations: dsl jobs (as well as master startup time) are quite longer compared to what we experienced with using local store (which we have done in the past). benchmarking on the master node indeed shows a round trip for a single operation is 5-10ms. 2- normal “nfs pain”- network “hickups” results with locked threads in jenkins master that never unlocks.

Zoom avatar
Zoom
07:52:40 PM

Rohit Koimattur has joined Public “Office Hours”

Bill Clark avatar
Bill Clark

Has anyone looked at and/or considered Terraspaces as a framework for terraform? I have setup a quick and dirty environment in a Cloud9 instance and so far am loving it.

Zoom avatar
Zoom
07:57:06 PM

Michael Jenkins has joined Public “Office Hours”

charlesz avatar
charlesz

hello, what do you guys use for something like constant configuration changes? i was leaning towards ansible but just wanted to see if there is anything out there that works ok too

mfridh avatar
mfridh

I am still on bit.y/oauth2_proxy … it seems like maybe the project itself has moved on and is active still… (I’m on a very old version on the “legacy” services I front with oauth2 proxy still) – https://github.com/oauth2-proxy/oauth2-proxy

oauth2-proxy/oauth2-proxy

A reverse proxy that provides authentication with Google, Github or other providers. - oauth2-proxy/oauth2-proxy

Bill Clark avatar
Bill Clark

f5 BIG-IP 3DNS?

Bill Clark avatar
Bill Clark
08:07:59 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
kubernetes-sigs/external-dns

Configure external DNS servers (AWS Route53, Google CloudDNS and others) for Kubernetes Ingresses and Services - kubernetes-sigs/external-dns

Zoom avatar
Zoom
08:14:17 PM

Abisoye Olaomi has joined Public “Office Hours”

Mohammed Yahya avatar
Mohammed Yahya
Multi Account Setup - OpenDocs

DevSecOps OpenDocs - Document Everything!

Zoom avatar
Zoom
08:16:28 PM

Abisoye Olaomi has joined Public “Office Hours”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-components

Catalog of reusable Terraform components and blueprints for provisioning reference architectures - cloudposse/terraform-aws-components

Zoom avatar
Zoom
08:19:13 PM

Abisoye Olaomi has joined Public “Office Hours”

Bill Clark avatar
Bill Clark

I have not heard of anyone using a dedicated DNS account before

Bill Clark avatar
Bill Clark

Would you talk a little about your thoughts limits around multi-account setup. I think a limit of 10 or perhaps up 20 is manageable, but beyond that I think you should think of creating a new org for more member accounts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is a good question. Will pin it for next week.

Bill Clark avatar
Bill Clark

Excellent. I have been wrestling with this for a while. Obviously there are exceptions. But I like the multi-account member account approach to help isolate and demarc things, but I also worry about sprawl and have seen for some plans of a 100 or more accounts under an org. Myself I see it being more approachable somewhere in the 10 - 25 range.

mfridh avatar
mfridh

It certainly helps because spreading the zones out is horrible. You might end up needing to use the “prod” DNS from a non-prod account etc…

1
Bill Clark avatar
Bill Clark

ruby gem

organicnz avatar
organicnz

Any good news about Waypoint?)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I don’t have any news yet on this. We’re waiting probably until > Q2 before taking a serious look at it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Very excited about what it is aspiring to do.

mfridh avatar
mfridh

I am too.. but… it’s always tricky with projects which try to do “allthethings”. If you start on a clean slate it can be absolutely wonderful.

mfridh avatar
mfridh

Going all-in on something, regardless of what it is though… can be amazing.

organicnz avatar
organicnz

Oh you’re already talking about)

Jim Park avatar
Jim Park

No such thing as “Best Practice.” There are only tendrils of innovation that become increasingly adopted.

1
1
1
roth.andy avatar
roth.andy

I wasn’t in office hours. Is this in reference to a particular tool, or the DevOps world in general?

Overall I totally agree, but I think there have been a couple of things that should be considered a best practice and used by all/almost all

  1. Use Git
  2. Use a modern Git-based VCS like GitHub/GitLab/BitBucket/etc
  3. Do CI, with an automated testing pipeline
  4. Containerize

Other more controversial ones that are, in my opinion, best practices

  1. Kubernetes in all but the most basic of use cases
  2. Throw out your style guide and automate it with hooks/CI/etc
  3. …I’ll think about some more. I really like going over stuff like this 
3
roth.andy avatar
roth.andy

IaC, probably in the top list…

Jim Park avatar
Jim Park
09:15:22 PM

This was a general statement. The point I was trying to make is that “Best Practice” today feels less like “use drbd, pacemaker, corosync and this my.cnf” and more like the following picture:

Jim Park avatar
Jim Park

But I agree, there are definitely some practices that are mature and broadly adopted, like your list above.

Bill Clark avatar
Bill Clark

What I like about Terraspaces is that you can still go an do it with Terraform, but gives some better structure and easier safeties for a small group.

Jim Park avatar
Jim Park

I think one of the most important questions to ask is where on the adoption curve does my organization want to be?

Bill Clark avatar
Bill Clark

Thanks all! I appreciate the insights.

Zoom avatar
Zoom
09:50:06 PM

New Zoom Recording from our Office Hours session on 2021-01-06 is now available.

2021-01-04

Weston Platter avatar
Weston Platter

A question I have about https://github.com/cloudposse/terraform-aws-ecs-cloudwatch-sns-alarms … I’m using it to scale up/down the number of Fargate Tasks for an ECS Service. My issue is that it the scale down action continues to scale the task count below the min desired count. What I’m trying to achieve is for the scale down process to not scale below the min desired count.

cloudposse/terraform-aws-ecs-cloudwatch-sns-alarms

Terraform module to create CloudWatch Alarms on ECS Service level metrics. - cloudposse/terraform-aws-ecs-cloudwatch-sns-alarms

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Probably better to open an issue for this one.

cloudposse/terraform-aws-ecs-cloudwatch-sns-alarms

Terraform module to create CloudWatch Alarms on ECS Service level metrics. - cloudposse/terraform-aws-ecs-cloudwatch-sns-alarms

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@PePe have you run into this?

PePe avatar

not exactly

PePe avatar

We enabled a capacity provider which caused and issue similar

PePe avatar

maybe you have a capacity provider at the cluster level @?

Weston Platter avatar
Weston Platter

Checking ….

Weston Platter avatar
Weston Platter

I don’t have a capacity provider configured. Do I need that?

PePe avatar

no no you do not

PePe avatar

Capacity providers are a completely different animal

Weston Platter avatar
Weston Platter

gotcha. I’ll go ahead and open a github issue.

1
Weston Platter avatar
Weston Platter

Asking this here so I can ask this during this week’s office hours.

    keyboard_arrow_up