#general (2020-09)
General conversations related to DevOps/Automation
General Discussions
2020-09-01
Hey everyone, give a warm welcome to our newest members!
- @moos3
- @Ray Pasko
- @jtdoepke
- @Lior Rabin
- @Sehox
- @K H
- @Daniel Schopper
- @Paul Spurlock
Good to have you here =)
2020-09-02
Hey everyone, give a warm welcome to our newest members!
- @Cody Moore
- @Tayfun
- @stefan
- @Jeff Wozniak
Good to have you here =)
2020-09-03
Hey everyone, give a warm welcome to our newest members!
- @Alinafe Matenda
- @Repa Ferula
Good to have you here =)
2020-09-04
Hey everyone, give a warm welcome to our newest members!
- @bigwheel k
- @Ryan Pei
- @Maxime Sibellas
- @Jason
- @Denis Nguyen
- @Evgeni Enchev
- @Olha
- @Diamon
- @Jon Naglieri
Good to have you here =)
Do you guys have any documentation for setting up a cold start using the reference architectures?
Current docs for that are in flux. Big things are happening but it won’t be ready for a couple months probably.
Any suggestion for spinning up a new account. I really liked the setup
we have revamped our strategy around a something that works well with terraform cloud and a cli for operations
We have departed from the one repo per account strategy because it’s not gitops friendly and heavy to setup
Ok, so you wouldnt suggest going down the route of setting up the reference architectures?
Not that refarch, but we have a new one - just not public yet. We still use geodesic, and our terraform modules. We don’t use the terraform-root-modules repo for new engagements and we have tons of stuff we still need to open source which we use in our engagements.
Ok ill hold off and just setup a interim architecture for now
Apologies for digging this topic back…. are there any updates on when the new bootstrap system would be open-sourced?
I’m also interested in knowing if there’s any updates on this, @Erik Osterman (Cloud Posse).
Also, is there a good place to stay informed on said updates?
See #geodesic
but we will announce this in a very big way once we have more published
stay tuned
I used it a year or so ago but cant seem to find the documentation around setting up an account from scratch
I can show you how to get started
Thanks so much
@Erik Osterman (Cloud Posse), gave variant2 a try today. Pretty cool, though definitely a huge difference from what I’m used to with powershell. I see the biggest draw is all the the functions and effort in terraform are avialable. This means it already is starting with a huge library of well tested functions.
I think i’ll blog on this soon as i test further. I had to write up a bash bootstrap script to download into my docker “codespaces”, so beyond that and lack of formatter right now it started to make sense.
job "deploy" {
description = "Runs serverless command to deploy to AWS"
option "bin_path" {
type = string
description = "The location of the build output for go build"
default = "bin"
}
option "project_dir" {
type = string
description = "build folder for the project"
default = "functions/tacos"
}
exec {
command = "serverless"
dir = opt.project_dir
env = {
AWS_SDK_LOAD_CONFIG=1
AWS_DEFAULT_REGION="eu-west-1"
AWS_PROFILE="nonprod"
}
args = ["deploy"]
}
variant run build
produced
vscode@codespaces_012633:~/workspace/devops-serverless$ variant run deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service tacos.zip file to S3 (10.25 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.........
Serverless: Stack update finished...
Service Information
service: tacos
stage: dev
region: eu-west-1
stack: tacos-dev
resources: 6
api keys:
None
endpoints:
None
functions:
tacoops: dev-tacocops
layers:
None
Also remember you can now compile this and discard variant :-)
So you have a single binary executable artifact
Here’s my challenge in reviewing this.. I know you haven’t worked with PowerShell, so we are coming from different backgrounds. i’m new to go, and here’s what I observed so far.
Variant2 requires much much more work to get basic build/task automation up compared to PowerShell (which is cross platform, I’m running in Linux (docker) and macOS). Variant2 while awesome with hashicorp’s functions, is way less featured than PowerShell, which provides a rich ecosystem and is much much better for argument parsing and arg strings.
Note: I love trying new things. I really love the concept and am trying to give it a shot, but coming from a framework that is “make on steroids” with rich objects and all, it’s hard for me to fully grasp the value of this yet.
If you are ever bored, maybe you’d be willing to do a 30min comparison of my current “runner” vs your experience with variant2 for some basic operations and we could compare? I could write up a blog post on InvokeBuild to show how it works in comparison if that would be of interest as well.
So variant2 is not a replacement for your powershell scripts, makefiles, python clis, go tools, jenkin clis, etc.
the point is more about how to combine a dozen cli UIs together. it’s definitely a niche problem, and maybe not a problem you need to solve.
but the way you call your InvokeBuild scripts might be different than the interface for helm
, and helmfile
and terraform
and kubectl
and make
and …. so if you want to present to your users a single cli that feels like it was written from the ground up to provide a consistent interface, but don’t actually want to write a cli from the ground up, then variant is the right tool for that.
Interesting. I was taking it as have a cross platform “make” replacement using a robust hashicorp library etc. The examples of running args for examples with exec is a replacement for a make file.
I’ll be eager to see more. I’m a new Gopher so anything to help gradually get me more immersed in Go is great.
Got you. So the use case would be something perfect for you then, as you might have helm, terraform, terraform-docs and more all in y our tooling, and it would give you a single cli.
So I’ve always defined this as “functions/libraries” and “runners”. The runners are the interface to tested cli/libraries.
yea, so variant is a runner
If I can chain together a simple build like
build.ps1 -task clean, build, deploy
And it is running cleanup commands, remove commands, go build, terraform deploy/serverless etc…. I’m confused on what variant would offer over that i guess.
it’s opinionated as the rest and it deliberately does not allow many things you would find if you were writing it natively in Go/Python/C/etc. It’s doing that to force business logic into the cli tools, rather than into the cli. Keep the cli as minimal as possible.
build.ps1 -task clean, build, deploy
the user needs to know that they should pass clean, build deploy
and not build, deploy, clean
when they type build.ps1 --help
do they see the full menu of options
how do i pas arguments to build
but not deploy
?
how do i pas different arguments to build
than to deploy
?
how do i know what arguments I can pass to clean?
can I run [build.ps](http://build.ps) -task clean --help
?
This is how I instruct others to normally run. It populates prebuilt build tasks in vscode automatically (not required)
And it generates a full asciidoc help file from a few commands or can give you a full task /job list with build.ps1 ? for example.
The arguments is a good point. I drive all this from parameters or environment variables (powershell is very robust on arg parsing)
This is the raw output without any nice formatting, though I have a task that generates a formatted asciidoc readme too.
so make
is as well very environment driven. and (by my opinion) a very weak cli interface when compared to a modern cli. However, this is why variant is nice. So variant handles all the arg parsing. You can pass that to you invoke build. You can pass it to helm, terraform, bash scripts, etc. The user has a standard way of using the cli that doens’t depend on wiki ops.
Got you. Ok for input standardization.
This is useful. PowerShell argument parsing is object based so it can be extremely rich, that’s why I’ve struggled to see some of these differences. Do you have a variant2 example from the repo they provide that demonstrates your favorite use case for this?
Ok for input standardization.
PowerShell is object based vs text based like most cli tooling, so that’s where probably my paradigm is different (not better, just different).
bingo. input / cli standardization / modernization without rewriting it.
For instance I can provide a full object like, perform terraform fmt on files for example
$files = Get-ChildItem -Recursive -Filter *.tf
Isn’t powershell more comprable to something like bash
or fish
than to a task runner? It’s a full fledged language.
This can be passed into a build like build.ps1 -task format -Files $files
but the difference is powershell easily works with this object, it’s not just a file list. Each $file
in the loop would give you access to a full range of properties that are actual dotnet objects, not parsed text.
And PowerShell is more comparable to Python, but with a better “REPL” you can use just like bash/fish at the same time.
The difference is you are working with objects, not text. Not saying it’s better, but for complex things like argument parsing it means you pass actual bool
values, not text 1/0 that have to be converted to boolean in your tool.
yea, exactly - so this is a total apple to oranges comparison. variant in no way strives to replace python/perl/ruby/bash/go/golang/invokebuild./etc
it’s more of a DSL to express workflows
you can definitely write workflows in python/perl/ruby/bash/go/golang/invokebuild
gotta go
office hours
Got you. Maybe we can chat again sometime if you are up for it. Paradigm is different for my background ( i blogged on this monday) and I admit there are major benefits to everything as text/files, but there are also incredibly rich “task runner” things like this that make me question cli tooling anyway
yep! for sure.
thank you
2020-09-05
Hey everyone, give a warm welcome to our newest members!
- @Obinna Ezeakachi
- @Mohammed Nazib
Good to have you here =)
2020-09-07
Guys? Did You try to integrate https://github.com/siemens/sentry-auth-oidc to sentry? ( https://github.com/cloudposse/helmfiles/blob/master/releases/sentry.yaml ) Thanks
OpenID Connect SSO provider for Sentry. Contribute to siemens/sentry-auth-oidc development by creating an account on GitHub.
No, in our case we have it behind gatekeeper/keycloak
OpenID Connect SSO provider for Sentry. Contribute to siemens/sentry-auth-oidc development by creating an account on GitHub.
also, sentry has a massive/painful update that we don’t support yet.
v9 → v10
Thanks.
Is possible to configu sentry agains claimed roles from jwt/oidc token, or is it working only as single user application ?
Hey everyone, give a warm welcome to our newest members!
- @Honza Novak
- @Manuele Fanelli
- @Chris Lavery
Good to have you here =)
2020-09-08
Hey everyone, give a warm welcome to our newest members!
- @rahul521
- @Simon Emms
- @niekb
- @Alex Jurkiewicz
Good to have you here =)
2020-09-09
Could someone help me work out what’s wrong with this nginx rule? ..
If I hardcode values in place of the $subdomain
variable, it works.. But it doesn’t seem to like the interpolation that’s done here. It throws a 502 error… The header X-Subdomain-Proxy
has the correct subdomain value in it.
server {
server_name ~^(?<subdomain>.+)\.example\.com$;
add_header X-Subdomain-Proxy $subdomain;
location / {
proxy_set_header Host $subdomain.newdomain.com;
proxy_pass https://$subdomain.newdomain.com;
}
}
Have you tried wrapping the $subdomain
variable in curly braces? Eg.
proxy_set_header Host "${subdomain}.newdomain.com";
proxy_pass "https://${subdomain}.newdomain.com";
Yep … Although just figured it out, I needed a resolver apparently! Seems to work if I add
resolver 1.1.1.1;
above the server block
Hey everyone, give a warm welcome to our newest members!
- @gugaiz
- @Sebastian Grateful
- @Aaron Miller
- @lokosubayo
- @Corey Thomas
Good to have you here =)
2020-09-10
Hey everyone, give a warm welcome to our newest members!
- @Chris Warren
- @Andrew Red
- @Kelvin Tan
- @David Lundgren
Good to have you here =)
2020-09-11
Hey everyone, give a warm welcome to our newest members!
- @Rhenusone Rosalia
- @Ken Foreman
- @James Roller
- @Gabriel Tiossi
- @feixiaohuijava
- @Yash
- @Chien Huey
- @sadbarkhattak
Good to have you here =)
2020-09-12
I’m here! What’d I miss?
Hey everyone, give a warm welcome to our newest members!
- @Angelo Pace
- @Steve P
Good to have you here =)
2020-09-13
Hey everyone, give a warm welcome to our newest members!
- @Vugar
- @Maciek Ciach
Good to have you here =)
2020-09-14
Thank you guys! Glad to be here!
Hello everyone, just joined. I’m looking forward to migrate my workloads to using cloudposse’s way.
Hey everyone, give a warm welcome to our newest members!
- @Marcin Cierpicki
- @Patrick Sodré
- @Matt Ryan
Good to have you here =)
2020-09-15
Hey everyone, give a warm welcome to our newest members!
- @Yen Kuo
- @ninja
- @ashu.angiras11
- @varunreddypenna
- @gusse
- @Vamshi Siddarth
- @Ben
- @Martin Canovas
- @João Ferreira
- @mlondeen
- @Vitalii Sikora
- @Sergey Pokatov
Good to have you here =)
2020-09-16
Hey everyone, give a warm welcome to our newest members!
- @Joaquin Menchaca
- @Iouns
- @Alex Taylor
- @Justin Seiser
- @jones.mabea
- @James Villa
Good to have you here =)
2020-09-17
Hey everyone, give a warm welcome to our newest members!
- @Nesti
Good to have you here =)
Hello
2020-09-18
Moin Moin
Moin ^^^
Hey everyone, give a warm welcome to our newest members!
- @Vedant Mhatre
- @srimarakani
- @Jurgen
- @Majan Paul
- @Daniele P
- @Nitin Prabhu
- @rei
- @Jimmie Butler
Good to have you here =)
2020-09-19
Hey everyone, give a warm welcome to our newest members!
- @tomaszjdul
- @Alban
- @Dave Cobb
Good to have you here =)
2020-09-20
Hi folks ! thanks and happy to be here
Hey everyone, give a warm welcome to our newest members!
- @Stephen Bennett
- @Viacheslav Romanov
Good to have you here =)
2020-09-21
https://github.com/docker/cli/issues/267#issuecomment-695149477 that’s all you need to know about docker development speed
Allow the following usage: docker push org/image-a:some_tag org/image-b:other_tag This would make docker push simpler to use in systems that need to build and push multiple images to dockerhub by o…
Hello. Just looking to use https://github.com/cloudposse/terraform-aws-elasticache-redis. Part of the task is to create users on the redis server that are essentially read only users. Is this possible with this module, or terraform in general? We already have a bastion SSH tunnel in place that only allows tunnelling to specific destinations, so no issue with connecting to the redis instances.
My guess is that unless there’s a specific resource to monitor, terraform isn’t going to be involved.
But any suggestions would be appreciated.
Terraform module to provision an ElastiCache Redis Cluster - cloudposse/terraform-aws-elasticache-redis
better to use #terraform
Terraform module to provision an ElastiCache Redis Cluster - cloudposse/terraform-aws-elasticache-redis
Hey everyone, give a warm welcome to our newest members!
- @Kristoffer
- @Richard Quadling
Good to have you here =)
2020-09-22
Hey everyone, give a warm welcome to our newest members!
- @vixus0
- @Viktor Farcic
- @Vasilios Tzanoudakis
Good to have you here =)
2020-09-23
question for today: what is proper way of ensuring that kubectl command called in terraform (via local_exec) will succeed? I often (not all the time) find the command runs before the EKS cluster API server is ready so terraform aborts. If I re-run it again, that 10-20 seconds is sufficient for the server to be ready so terraform then completes the apply. I tried a few things, without success. Any docs on this would be awesome.
Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.
Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.
Thanks Andrew, I thought I had tried this but I just noticed that there are a couple of places in our module where we do the local exec directly, not within the null resource. I’ll have a closer look.
Hey everyone, give a warm welcome to our newest members!
- @Alex
- @Jay Zalowitz
- @Bob Strubel
- @Tomek
- @Charles Kim
- @Fernando Castillo
Good to have you here =)
Hi everyone
Hey guys
2020-09-24
Hey everyone, give a warm welcome to our newest members!
- @Adin Hodovic
- @Smailli Moraes
- @Abhinav Khanna
- @rani
- @Denis Boulas
Good to have you here =)
Hello everyone
Hello!
2020-09-25
Hey everyone, give a warm welcome to our newest members!
- @Lucio Zolezzi
- @Robbie Gladmon
Good to have you here =)
anyone use buildkite here ?
ran into this issue today and i finally got to cut my teeth (and my wrists!) with cloudformation
beware, if you’re using buildkite and their cloudformation stack to launch templates. your stuff may be out of date.
2020-09-26
Hey everyone, give a warm welcome to our newest members!
- @Mohammed Yahya
- @Eli Karchov
- @Peter Oates
Good to have you here =)
2020-09-27
Hey everyone, give a warm welcome to our newest members!
- @aconchillo
- @1efty
Good to have you here =)
2020-09-28
Hey everyone, give a warm welcome to our newest members!
- @Venkata Pavan Kumar Varri
- @t.hiroya
- @Solomon Tekle
- @Jihed Mselmi
- @aob-henry
Good to have you here =)
2020-09-29
Hey everyone, give a warm welcome to our newest members!
- @Didar Rashid
- @VJ
Good to have you here =)
2020-09-30
Hey everyone, give a warm welcome to our newest members!
- @Yasser Isa Manzur
- @jonjitsu
- @Tom Vaughan
Good to have you here =)
Thank you for terraform-aws-efs! I’m wondering what “namespace” does? I’m actually trying to create an EFS volume that can be accessed by pods in several namespaces. In “main.tf” I don’t see a reference to the namespace — is it there just for future expansion? … and will I be able to connect from mulitple namespaces? The internet is a little hazy on this. Thanks!
namespace
in the terraform modules is not Kubernetes namespace
it’s used for uniquely and consistency naming AWS resources
it’s coming from https://github.com/cloudposse/terraform-null-label
Terraform Module to define a consistent naming convention by (namespace, stage, name, [attributes]) - cloudposse/terraform-null-label
we use naming convention like namespace-environment-stage-name-attributes
any of those could be omitted
in the end, it looks like cp-ue2-prod-efs
, where the namespace cp
identifies our company
ok — got it! Yes — as I my purpose is to use in k8, I got confused, but I guess your module has nothing k8-specific in it. Thank you!