#general (2020-09)

General conversations related to DevOps/Automation

General Discussions

2020-09-23

OliverS avatar
OliverS

question for today: what is proper way of ensuring that kubectl command called in terraform (via local_exec) will succeed? I often (not all the time) find the command runs before the EKS cluster API server is ready so terraform aborts. If I re-run it again, that 10-20 seconds is sufficient for the server to be ready so terraform then completes the apply. I tried a few things, without success. Any docs on this would be awesome.

kubernetes1
roth.andy avatar
roth.andy
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

roth.andy avatar
roth.andy
cloudposse/terraform-aws-eks-cluster

Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

OliverS avatar
OliverS

Thanks Andrew, I thought I had tried this but I just noticed that there are a couple of places in our module where we do the local exec directly, not within the null resource. I’ll have a closer look.

SweetOps avatar
SweetOps
07:00:12 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

1
Charles Kim avatar
Charles Kim

Hi everyone

Fernando Castillo avatar
Fernando Castillo

Hey guys

2020-09-22

SweetOps avatar
SweetOps
07:00:26 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

3

2020-09-21

voron avatar
voron

https://github.com/docker/cli/issues/267#issuecomment-695149477 that’s all you need to know about docker development speed

Allow `docker push' to push multiple/a subset of tags · Issue #267 · docker/cli

Allow the following usage: docker push org/image-a:some_tag org/image-b:other_tag This would make docker push simpler to use in systems that need to build and push multiple images to dockerhub by o…

Richard Quadling avatar
Richard Quadling

Hello. Just looking to use https://github.com/cloudposse/terraform-aws-elasticache-redis. Part of the task is to create users on the redis server that are essentially read only users. Is this possible with this module, or terraform in general? We already have a bastion SSH tunnel in place that only allows tunnelling to specific destinations, so no issue with connecting to the redis instances.

My guess is that unless there’s a specific resource to monitor, terraform isn’t going to be involved.

But any suggestions would be appreciated.

cloudposse/terraform-aws-elasticache-redis

Terraform module to provision an ElastiCache Redis Cluster - cloudposse/terraform-aws-elasticache-redis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

better to use #terraform

cloudposse/terraform-aws-elasticache-redis

Terraform module to provision an ElastiCache Redis Cluster - cloudposse/terraform-aws-elasticache-redis

1
SweetOps avatar
SweetOps
07:00:15 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @

Good to have you here =)

3

2020-09-20

Alban avatar
Alban

Hi folks ! thanks and happy to be here

SweetOps avatar
SweetOps
07:00:05 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @

Good to have you here =)

:--1:2

2020-09-19

SweetOps avatar
SweetOps
07:00:17 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

2020-09-18

rei avatar

Moin Moin

Alan Kis avatar
Alan Kis

Moin ^^^

Daniele avatar
Daniele

SweetOps avatar
SweetOps
07:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

:--1:4
Majan Paul avatar
Majan Paul

2020-09-17

SweetOps avatar
SweetOps
07:00:26 PM

Hey everyone, give a warm welcome to our newest members!

  • @

Good to have you here =)

Jurgen avatar
Jurgen

Hello

2020-09-16

SweetOps avatar
SweetOps
07:00:21 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

:--1:1
1
2

2020-09-15

SweetOps avatar
SweetOps
07:00:12 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

2

2020-09-14

Vugar avatar
Vugar

Thank you guys! Glad to be here!

Patrick Sodré avatar
Patrick Sodré

Hello everyone, just joined. I’m looking forward to migrate my workloads to using cloudposse’s way.

:100:1
SweetOps avatar
SweetOps
07:00:07 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

2020-09-13

SweetOps avatar
SweetOps
07:00:07 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @

Good to have you here =)

party_parrot2
Maciek Ciach avatar
Maciek Ciach

Hello everyone !

2

2020-09-12

Rhenusone Rosalia avatar
Rhenusone Rosalia

I’m here! What’d I miss?

SweetOps avatar
SweetOps
07:00:17 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @

Good to have you here =)

2020-09-11

Simon Emms avatar
Simon Emms

hello all, great to be here.

1
sadbarkhattak avatar
sadbarkhattak

Hello everyone, just joined, looking forward for great new techi friends

1
SweetOps avatar
SweetOps
07:00:28 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

fiesta_parrot1
:--1:1

2020-09-10

SweetOps avatar
SweetOps
07:00:24 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @Kelvin Tan
  • @

Good to have you here =)

2

2020-09-09

Christopher avatar
Christopher

Could someone help me work out what’s wrong with this nginx rule? ..

If I hardcode values in place of the $subdomain variable, it works.. But it doesn’t seem to like the interpolation that’s done here. It throws a 502 error… The header X-Subdomain-Proxy has the correct subdomain value in it.

server {
    server_name ~^(?<subdomain>.+)\.example\.com$;
    add_header X-Subdomain-Proxy $subdomain;
    location / {
        proxy_set_header Host $[subdomain.newdomain.com](http://subdomain\.newdomain\.com);
        proxy_pass https://$[subdomain.newdomain.com](http://subdomain\.newdomain\.com);
    }
}
Drew Davies avatar
Drew Davies

Have you tried wrapping the $subdomain variable in curly braces? Eg.

proxy_set_header Host "${subdomain}.[newdomain.com](http://newdomain\.com)";
proxy_pass "https://${subdomain}.[newdomain.com](http://newdomain\.com)";
Christopher avatar
Christopher

Yep … Although just figured it out, I needed a resolver apparently! Seems to work if I add

resolver 1.1.1.1;

above the server block

:--1:1
SweetOps avatar
SweetOps
07:00:20 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @

Good to have you here =)

1
1

2020-09-08

SweetOps avatar
SweetOps
07:00:20 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

2020-09-07

muhaha avatar
muhaha
siemens/sentry-auth-oidc

OpenID Connect SSO provider for Sentry. Contribute to siemens/sentry-auth-oidc development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No, in our case we have it behind gatekeeper/keycloak

siemens/sentry-auth-oidc

OpenID Connect SSO provider for Sentry. Contribute to siemens/sentry-auth-oidc development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, sentry has a massive/painful update that we don’t support yet.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

v9 → v10

muhaha avatar
muhaha

Thanks.

Is possible to configu sentry agains claimed roles from jwt/oidc token, or is it working only as single user application ?

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

1

2020-09-05

SweetOps avatar
SweetOps
07:00:17 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @

Good to have you here =)

2020-09-04

SweetOps avatar
SweetOps
07:00:26 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

jober avatar
jober

Do you guys have any documentation for setting up a cold start using the reference architectures?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Current docs for that are in flux. Big things are happening but it won’t be ready for a couple months probably.

jober avatar
jober

Any suggestion for spinning up a new account. I really liked the setup

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we have revamped our strategy around a something that works well with terraform cloud and a cli for operations

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We have departed from the one repo per account strategy because it’s not gitops friendly and heavy to setup

jober avatar
jober

Ok, so you wouldnt suggest going down the route of setting up the reference architectures?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not that refarch, but we have a new one - just not public yet. We still :100: use geodesic, and our terraform modules. We don’t use the terraform-root-modules repo for new engagements and we have tons of stuff we still need to open source which we use in our engagements.

jober avatar
jober

Ok ill hold off and just setup a interim architecture for now

jober avatar
jober

I used it a year or so ago but cant seem to find the documentation around setting up an account from scratch

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I can show you how to get started

jober avatar
jober

Thanks so much

sheldonh avatar
sheldonh

@Erik Osterman (Cloud Posse), gave variant2 a try today. Pretty cool, though definitely a huge difference from what I’m used to with powershell. I see the biggest draw is all the the functions and effort in terraform are avialable. This means it already is starting with a huge library of well tested functions.

I think i’ll blog on this soon as i test further. I had to write up a bash bootstrap script to download into my docker “codespaces”, so beyond that and lack of formatter right now it started to make sense.

job "deploy" {
    description = "Runs serverless command to deploy to AWS"

    option "bin_path" {
      type = string
      description = "The location of the build output for go build"
      default = "bin"

    }
    option "project_dir" { 
      type = string
      description = "build folder for the project"
      default = "functions/tacos"
    }

    exec {
        command = "serverless"
        dir = opt.project_dir
        env = {
            AWS_SDK_LOAD_CONFIG=1
            AWS_DEFAULT_REGION="eu-west-1"
            AWS_PROFILE="nonprod"
        }
        args = ["deploy"]
  }
variant run build

produced

[email protected]_012633:~/workspace/devops-serverless$ variant run deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service tacos.zip file to S3 (10.25 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: tacos
stage: dev
region: eu-west-1
stack: tacos-dev
resources: 6
api keys:
  None
endpoints:
  None
functions:
  tacoops: dev-tacocops
layers:
  None
2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also remember you can now compile this and discard variant :-)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So you have a single binary executable artifact

sheldonh avatar
sheldonh

Here’s my challenge in reviewing this.. I know you haven’t worked with PowerShell, so we are coming from different backgrounds. i’m new to go, and here’s what I observed so far.

Variant2 requires much much more work to get basic build/task automation up compared to PowerShell (which is cross platform, I’m running in Linux (docker) and macOS). Variant2 while awesome with hashicorp’s functions, is way less featured than PowerShell, which provides a rich ecosystem and is much much better for argument parsing and arg strings.

Note: I love trying new things. I really love the concept and am trying to give it a shot, but coming from a framework that is “make on steroids” with rich objects and all, it’s hard for me to fully grasp the value of this yet.

If you are ever bored, maybe you’d be willing to do a 30min comparison of my current “runner” vs your experience with variant2 for some basic operations and we could compare? I could write up a blog post on InvokeBuild to show how it works in comparison if that would be of interest as well.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So variant2 is not a replacement for your powershell scripts, makefiles, python clis, go tools, jenkin clis, etc.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the point is more about how to combine a dozen cli UIs together. it’s definitely a niche problem, and maybe not a problem you need to solve.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but the way you call your InvokeBuild scripts might be different than the interface for helm, and helmfile and terraform and kubectl and make and …. so if you want to present to your users a single cli that feels like it was written from the ground up to provide a consistent interface, but don’t actually want to write a cli from the ground up, then variant is the right tool for that.

sheldonh avatar
sheldonh

Interesting. I was taking it as have a cross platform “make” replacement using a robust hashicorp library etc. The examples of running args for examples with exec is a replacement for a make file.

I’ll be eager to see more. I’m a new Gopher so anything to help gradually get me more immersed in Go is great.

sheldonh avatar
sheldonh

Got you. So the use case would be something perfect for you then, as you might have helm, terraform, terraform-docs and more all in y our tooling, and it would give you a single cli.

sheldonh avatar
sheldonh

So I’ve always defined this as “functions/libraries” and “runners”. The runners are the interface to tested cli/libraries.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, so variant is a runner

sheldonh avatar
sheldonh

If I can chain together a simple build like

build.ps1 -task clean, build, deploy

And it is running cleanup commands, remove commands, go build, terraform deploy/serverless etc…. I’m confused on what variant would offer over that i guess.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s opinionated as the rest and it deliberately does not allow many things you would find if you were writing it natively in Go/Python/C/etc. It’s doing that to force business logic into the cli tools, rather than into the cli. Keep the cli as minimal as possible.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


build.ps1 -task clean, build, deploy

the user needs to know that they should pass clean, build deploy and not build, deploy, clean

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

when they type build.ps1 --help do they see the full menu of options

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how do i pas arguments to build but not deploy?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how do i pas different arguments to build than to deploy?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how do i know what arguments I can pass to clean?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can I run [build.ps](http://build\.ps) -task clean --help?

sheldonh avatar
sheldonh
06:16:43 PM

This is how I instruct others to normally run. It populates prebuilt build tasks in vscode automatically (not required)

sheldonh avatar
sheldonh

And it generates a full asciidoc help file from a few commands or can give you a full task /job list with build.ps1 ? for example.

The arguments is a good point. I drive all this from parameters or environment variables (powershell is very robust on arg parsing)

sheldonh avatar
sheldonh
06:18:52 PM

This is the raw output without any nice formatting, though I have a task that generates a formatted asciidoc readme too.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so make is as well very environment driven. and (by my opinion) a very weak cli interface when compared to a modern cli. However, this is why variant is nice. So variant handles all the arg parsing. You can pass that to you invoke build. You can pass it to helm, terraform, bash scripts, etc. The user has a standard way of using the cli that doens’t depend on wiki ops.

sheldonh avatar
sheldonh

Got you. Ok for input standardization.

This is useful. PowerShell argument parsing is object based so it can be extremely rich, that’s why I’ve struggled to see some of these differences. Do you have a variant2 example from the repo they provide that demonstrates your favorite use case for this?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Ok for input standardization.

sheldonh avatar
sheldonh

PowerShell is object based vs text based like most cli tooling, so that’s where probably my paradigm is different (not better, just different).

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

bingo. input / cli standardization / modernization without rewriting it.

sheldonh avatar
sheldonh

For instance I can provide a full object like, perform terraform fmt on files for example

$files = Get-ChildItem -Recursive -Filter *.tf
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Isn’t powershell more comprable to something like bash or fish than to a task runner? It’s a full fledged language.

sheldonh avatar
sheldonh
06:23:23 PM

This can be passed into a build like build.ps1 -task format -Files $files

but the difference is powershell easily works with this object, it’s not just a file list. Each $file in the loop would give you access to a full range of properties that are actual dotnet objects, not parsed text.

sheldonh avatar
sheldonh

And PowerShell is more comparable to Python, but with a better “REPL” you can use just like bash/fish at the same time.

sheldonh avatar
sheldonh

The difference is you are working with objects, not text. Not saying it’s better, but for complex things like argument parsing it means you pass actual bool values, not text 1/0 that have to be converted to boolean in your tool.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, exactly - so this is a total apple to oranges comparison. variant in no way strives to replace python/perl/ruby/bash/go/golang/invokebuild./etc

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s more of a DSL to express workflows

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can definitely write workflows in python/perl/ruby/bash/go/golang/invokebuild

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

gotta go

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

office hours

sheldonh avatar
sheldonh

Got you. Maybe we can chat again sometime if you are up for it. Paradigm is different for my background ( i blogged on this monday) and I admit there are major benefits to everything as text/files, but there are also incredibly rich “task runner” things like this that make me question cli tooling anyway

sheldonh avatar
sheldonh

yep! for sure.

sheldonh avatar
sheldonh

thank you

2020-09-03

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @

Good to have you here =)

2020-09-02

SweetOps avatar
SweetOps
07:00:05 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

1
1

2020-09-01

SweetOps avatar
SweetOps
07:00:23 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

    keyboard_arrow_up