#office-hours (2022-09)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2022-09-02

https://github.blog/2022-09-01-github-actions-introducing-the-new-larger-github-hosted-runners-beta/ Finally it’s here. But still arm64 for linux still missing

Now your team can spend less time managing infrastructure and more time writing code.
2022-09-05

Question for next time: anybody using Spacelift with CloudFormation? If yes, what was your experience?

Hi @Vlad Ionescu (he/him) We have a few customers actively using it. I can see if a recent one will share their experience with you. They recently signed up as they have terraform, SAM, and ansible use cases.
https://docs.spacelift.io/vendors/cloudformation/integrating-with-sam.html
Do you have any questions or feedback? Our Eng team is also available to discuss.
Full disclosure: I lead sales at Spacelift.
Collaborative Infrastructure For Modern Software Teams

@Ryan thank you for asking! I don’t have any concrete questions unfortunately — Spacelift with CloudFormation is on my to-try list and I haven’t yet had the time, so I was curious if anybody else tried it.
2022-09-06

Question: You (Eric) mentioned AWS IC (SSO) and federation as an approach to fine grained, cross-account permission/access management. I did not quite get it
Like with IAM in the past or present, I know of users, groups, roles.
• Users belong to groups,
• Groups have permissions enforcing MFA and granting cross-account access (assume role), and the
• Other accounts have appropriate roles which users can assume based on the group they are in. This also worked nicely on the CLI. You configure .aws/credentials and .aws/config with creds and profiles once and then the CLI daily flow is:
• MFA once within the CLI and without the browser (8h or 12h duration)
• Switching the value of AWS_PROFILE (allowing for different terminal windows having different roles/accounts) How does your flow on the CLI look (also scripting)? What is the approach behind it?


Thanks for all the interesting tips

thanks @Adnan! hit us up next week if you want some more concrete examples

Thanks will probably do

BTW, I deleted everything, reinstalled/updated and now it is working without browser interaction except initially.
Starting/generating additional sessions happens without additional browser interaction. So I something was messed up with my config.
2022-09-07

Discussion topic for today: Anyone using ArgoCD-Autopilot? We were evaluating it at Masterpoint and I believe our intuition is to go in another direction to bootstrap clusters with Argo, but I would still be interested to hear others experience and if they’d recommend for or against.

@here office hours is starting in 30 minutes! Remember to post your questions here.

Ben Smith (Cloud Posse) has joined Public “Office Hours”

Allan Swanepoel has joined Public “Office Hours”

Vlad Ionescu has joined Public “Office Hours”

Linda Pham (Cloud Posse) has joined Public “Office Hours”

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Matt Calhoun has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

Antonio Rodriguez has joined Public “Office Hours”

Michael Jenkins has joined Public “Office Hours”

venkata mutyala has joined Public “Office Hours”

Mauricio Wyler has joined Public “Office Hours”

Charles Smith has joined Public “Office Hours”

Patrick McDonald has joined Public “Office Hours”

Roy Sprague has joined Public “Office Hours”

Ralf Pieper has joined Public “Office Hours”

Matt Gowie has joined Public “Office Hours”

Ayobami Bamigboye has joined Public “Office Hours”

Isa Aguilar has joined Public “Office Hours”

dag viggo lokoeen has joined Public “Office Hours”

Mike Martin has joined Public “Office Hours”

Alexandr Vorona has joined Public “Office Hours”

Allen Lyons has joined Public “Office Hours”

Devendra Yadav has joined Public “Office Hours”

Jonathan Poczatek has joined Public “Office Hours”

Gabriel has joined Public “Office Hours”

Hakan Kaya has joined Public “Office Hours”

Bhushan Uparkar has joined Public “Office Hours”

Oliver Schoenborn has joined Public “Office Hours”

Jim Park has joined Public “Office Hours”

Marc Tamsky has joined Public “Office Hours”

Bridget Royer has joined Public “Office Hours”

Antonio Rodriguez has joined Public “Office Hours”

Sam Caneer has joined Public “Office Hours”

Steven Vargas has joined Public “Office Hours”

Eric Berg has joined Public “Office Hours”

Vincent Werner has joined Public “Office Hours”

Vincent Werner has joined Public “Office Hours”

Question: has anyone tried zeet.co? Another multi-cloud build tool new kid on the block.

Fun note for MFA. Now I’d think twice to see if this even makes sense to do, but I’ve seen someone use oauthtool in github actions to answer a MFA.


Manage your Cloud credentials locally and improve your workflow with the only open-source desktop app you’ll ever need.

Links from today’s call: https://hashiconf.com/global/ https://github.com/cloudposse/atmos/releases/tag/v1.5.0 https://aws.amazon.com/about-aws/whats-new/2022/09/automate-aws-control-tower-guardrail-management-apis/ https://aws.amazon.com/about-aws/whats-new/2022/08/workload-consolidation-karpenter/

Join the HashiCorp community in LA and online for two full days of infrastructure and security automation, modern application delivery, and the future of cloud.


@Erik Osterman (Cloud Posse) upload to youtube failed again

Heads up! we received an erroneous DMCA take down notice and are looking into it.
2022-09-08
2022-09-09
2022-09-13


Drift Detection for Terraform Cloud continuously checks infrastructure state to detect and notify operators of any changes, minimizing risk, downtime, and costs.

SMH — They’re so late to the game that it’s actually astounding.

Drift Detection for Terraform Cloud continuously checks infrastructure state to detect and notify operators of any changes, minimizing risk, downtime, and costs.
2022-09-14

Question for OH: Anybody upgraded EKS to 1.23? How did it go? Any issues?

I been missing recent Office Hours but I listen to the YT Videos. You chatted about managing AWS Config files last week, I been using https://granted.dev and It is very gooood.
Main highlights:
- Auto Populated SSO Roles
- Open AWS Console in Separate Chrome Profiles
Granted - the easiest way to access your cloud.

Hm — This looks pretty great. I know the Leapp folks are building some similar functionality into their tool.
Has anyone used this with Leapp? Does it play nicely?
Granted - the easiest way to access your cloud.

Unfortunately I did not try leapp

FYI: I won’t make it to the call today Sorry!

@here office hours is starting in 30 minutes! Remember to post your questions here.

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

Linda Pham (Cloud Posse) has joined Public “Office Hours”

dag viggo lokoeen has joined Public “Office Hours”

Allan Mohr has joined Public “Office Hours”

Patrick McDonald has joined Public “Office Hours”

venkata mutyala has joined Public “Office Hours”

Gabriel has joined Public “Office Hours”

Antonio Rodriguez has joined Public “Office Hours”

Matt Calhoun has joined Public “Office Hours”

Lucas Andrade has joined Public “Office Hours”

Charles Smith has joined Public “Office Hours”

Hm… I’m getting the following when I hit the meeting link today —

I had to open the Zoom app on my machine and join from there. Tried hard refresh and multiple browsers and yet I was continuing to hit that page.
FYI — @Erik Osterman (Cloud Posse) @Linda Pham (Cloud Posse)

I guess I’m the only one who ran into this since no one else commented on this…
Feel free to ignore me on this one I guess — Maybe some weird hiccup on my end with Zoom?

Thanks for reporting this! We had to extend the meeting occurrences. It’s fixed for next week.

Matt Gowie has joined Public “Office Hours”

Isa Aguilar has joined Public “Office Hours”

Amaan Khan has joined Public “Office Hours”

question i have if time: secrets are a pain to delete and recreate (say you delete a stack that has AWS secrets, and then recreate it and the secrets have same names), any tricks?

Are you referring to SSM Parameter Store secrets or AWS Secrets Manager that you manage manually?
Maybe a better solution would be to use sops to check in your secrets and seed them into AWS via terraform?
Something like what I built and am upstreaming to Cloud Posse here: https://github.com/cloudposse/terraform-aws-components/pull/480
what
• Adds a new component: ssm-parameters
why
• We use this component on Masterpoint projects to manage SSM Parameter Store resources in a GitOps-ish fashion. • It enables managing secrets via sops and seeding them into SSM Params – This be useful to reduce manual operations around secrets. • It also supports managing normal SSM Params that are not secrets in the case that is useful for non-secret config.
references
• None

AWS Secrets Manager

if you’re using terraform you can append a random suffix to the secret name using
resource "random_string" "secret_name_suffix" {
length = 4
special = false
}

Yeah I’m keeping that as last resort because services running in the stack (but not managed by it) need to know the secret path as they can get info stored there.
So this approach would now require that secret path be shipped to the application somehow, eg an SSM parameter that is not random.
There is a resource argument for secrets called force_overwrite_replica_secret
but even when set to true, I still get the “resource already exists” error when I re-create the stack. This would be the best solution if it worked since it can be used upon re-creation not deletion.
So I guess I will just have to remember to set the secret destruction time to 0 days before I run tf destroy.
Or I’ll run a script that uses AWS CLI to destroy the secrets after tf destroy has been run, since it is easy to forget to do the apply with 0 day setting.

I know why that force flag does not work as I expected: it is for multi-region replication. https://github.com/hashicorp/terraform-provider-aws/issues/17943. The tf docs for the resource just don’t make that obvious (although the word “replica” in the flag is hint, in hindsight).

Ben Smith (Cloud Posse) has joined Public “Office Hours”

Ralf Pieper has joined Public “Office Hours”

venkata mutyala has joined Public “Office Hours”

Mohammed Yahya has joined Public “Office Hours”

Oliver Schoenborn has joined Public “Office Hours”

Michael Jenkins has joined Public “Office Hours”

Andrew Vitko has joined Public “Office Hours”

Bridget Royer has joined Public “Office Hours”

Ralf Pieper has joined Public “Office Hours”

Roy Sprague has joined Public “Office Hours”

A cli tool to help discover deprecated apiVersions in Kubernetes

Easily check your clusters for use of deprecated APIs

How do you handle updating EKS clusters? We use terraform. Is it safer to recreate everything inside the cluster in the updated version and use some kind of DNS switching? Or just update the version number in the same cluster in terraform. What do people generally do?

Maged Abdelmoeti has joined Public “Office Hours”

Life Lofranco has joined Public “Office Hours”

Ralf Pieper has joined Public “Office Hours”

Links from today’s call: https://hashiconf.com/global/ https://www.legitsecurity.com/blog/github-privilege-escalation-vulnerability-0 https://github.com/contentful-labs/terraform-diff www.github.com/mineiros-io/terramate https://github.com/cloudposse/atmos https://github.com/cloudposse/atmos/pull/200 https://www.hashicorp.com/blog/terraform-partner-integration-program-changes https://granted.dev/ www.github.com/mineiros-io/terramate https://www.hashicorp.com/blog/drift-detection-for-terraform-cloud-is-now-generally-available registry.terraform.io/providers/hashicorp/asws/latest/docs/resources/secretsmanager_secret github.com/cloudposse/terraform-provider-awsutils aws.amazon.com/blogs/containers/aws-controllers-for-kubernetes-ack aws-controllers-k8s.github.io/community/docs/community/services/#amazon-iam https://doc.crds.dev/github.com/crossplane-contrib/provider-jet-aws

Thanks for reporting this! We had to extend the meeting occurrences. It’s fixed for next week.

@Erik Osterman (Cloud Posse) - I couldnt make the office hours today - but wanted to ask if youhave, ot thought of doing demos on some of your tools, such as https://github.com/cloudposse/geodesic/ and https://github.com/cloudposse/atmos ?

@RB or @matt are you free next week to do an geodesic/atmos demo?

2022-09-16

Yo!
Where’s this week’s Office Hour recording

@nian


Hey @Erik Osterman (Cloud Posse) and @nian. I don’t think the recording for the 28th of September is posted to YouTube yet.

Thanks for the heads up! It should be visible now.
2022-09-19

2022-09-21

@here office hours is starting in 30 minutes! Remember to post your questions here.

Can I ask architecture questions? I want to deploy a dotnet 6 application that is backed by PostgreSQL. The application exposes a REST API and also has an internally scheduled process that runs batch processing. I’m torn between splitting up the batch processing from the REST API, using Lambda+API Gateway for the API and a simple ECS container for the batch processing. OR, having containers for both things. I’m thinking about provisioned Aurora for PostgreSQL (serverless v2 seems really pricey for now)
I’m also torn between ECS and EKS, I feel that EKS might be overkill for now.
Any other options I’m missing?

heads up! we answered your question today on the call.
Can I ask architecture questions? I want to deploy a dotnet 6 application that is backed by PostgreSQL. The application exposes a REST API and also has an internally scheduled process that runs batch processing. I’m torn between splitting up the batch processing from the REST API, using Lambda+API Gateway for the API and a simple ECS container for the batch processing. OR, having containers for both things. I’m thinking about provisioned Aurora for PostgreSQL (serverless v2 seems really pricey for now)
I’m also torn between ECS and EKS, I feel that EKS might be overkill for now.
Any other options I’m missing?

Check it out and let us know if that answers it.

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Vlad Ionescu has joined Public “Office Hours”

Gabriel has joined Public “Office Hours”

Brian Choate has joined Public “Office Hours”

dag viggo lokoeen has joined Public “Office Hours”

Jonathan Poczatek has joined Public “Office Hours”

Linda Pham (Cloud Posse) has joined Public “Office Hours”

Yusuf has joined Public “Office Hours”

Charles Smith has joined Public “Office Hours”

Vinko Vrsalovic has joined Public “Office Hours”

David Subar has joined Public “Office Hours”

Matt Calhoun has joined Public “Office Hours”

Andy Roth has joined Public “Office Hours”

Henry Ngo has joined Public “Office Hours”

Josh has joined Public “Office Hours”

Veronika Gnilitska has joined Public “Office Hours”

Steven Vargas has joined Public “Office Hours”

Chris Dobbyn has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

Paul Bullock has joined Public “Office Hours”

Oskar Maria Grande has joined Public “Office Hours”

Harold Sphinx has joined Public “Office Hours”

Mohammed Yahya has joined Public “Office Hours”

Dave Gregory has joined Public “Office Hours”

Guilherme Borges has joined Public “Office Hours”

Matt Gowie has joined Public “Office Hours”

Alexandr Vorona has joined Public “Office Hours”

Alex Siegman has joined Public “Office Hours”

Allan Swanepoel has joined Public “Office Hours”

Oliver Schoenborn has joined Public “Office Hours”

Harold Sphinx has joined Public “Office Hours”

Marc Tamsky has joined Public “Office Hours”

Harold Sphinx has joined Public “Office Hours”

Harold Sphinx has joined Public “Office Hours”

Ralf Pieper has joined Public “Office Hours”


Andy Roth has joined Public “Office Hours”


Dave Gregory has joined Public “Office Hours”

PePe Amengual has joined Public “Office Hours”

Dave Gregory has joined Public “Office Hours”

Amaan Khan has joined Public “Office Hours”

Dave Gregory has joined Public “Office Hours”

Links from today’s call: https://hashiconf.com/global/ https://github.com/hashicorp/terraform/releases/tag/v1.3.0 https://github.com/ahmetb/kubectl-tree https://www.macchaffee.com/blog/2022/k8s-under-documented-security-tips/ https://github.com/spacelift-io/prometheus-exporter https://github.com/hashicorp/terraform/issues?q=is%3Aissue+is%3Aclosed+archived%3Afalse+reason%3A%22not+planned%22 https://github.com/padok-team/tfautomv https://github.com/verifa/terraplate https://github.com/dineshba/tf-summarize https://docs.aws.amazon.com/securityhub/latest/userguide/standards-fsbp-config-resources.html https://aws.amazon.com/about-aws/whats-new/2022/09/aws-fargate-increases-compute-memory-resource-configurations-4x/

@matt @Vlad Ionescu (he/him) @muhaha @Alanis Swanepoel @Ralf Pieper thanks for all your great insights today!

2022-09-22
2022-09-28

@here office hours is starting in 30 minutes! Remember to post your questions here.

was up for last week - but could we get a demo of geodesic and atmos at some point perhaps?

We’re going to do this demo after hashiconf. Skipping this week and next week is hashiconf.

i understand theres a lot to cover from the normal news

also - aws is having a bad day - https://health.aws.amazon.com/health/status
View the overall status and health of AWS services using the AWS Health Dashboard.

Erik Osterman (Cloud Posse) has joined Public “Office Hours”

Johnmary Odenigbo has joined Public “Office Hours”

Jared Richards has joined Public “Office Hours”

Alex Atkinson has joined Public “Office Hours”

Assistant Cloud Posse has joined Public “Office Hours”

Alex Atkinson has joined Public “Office Hours”

Alex Atkinson has joined Public “Office Hours”

Alex Atkinson has joined Public “Office Hours”

Yusuf has joined Public “Office Hours”

Max Lobur has joined Public “Office Hours”

Gurvinder S. has joined Public “Office Hours”

Isaac M has joined Public “Office Hours”

dag viggo lokoeen has joined Public “Office Hours”

Vincent Werner has joined Public “Office Hours”

Alex Atkinson has joined Public “Office Hours”

Max Lobur has joined Public “Office Hours”

Madhusudan Satapathy has joined Public “Office Hours”

Andrew Vitko has joined Public “Office Hours”

Jose Figueredo has joined Public “Office Hours”

Rahul has joined Public “Office Hours”

Jonathan Poczatek has joined Public “Office Hours”

venkata mutyala has joined Public “Office Hours”

Isa Aguilar has joined Public “Office Hours”

Ray Botha has joined Public “Office Hours”

Ben Smith (Cloud Posse) has joined Public “Office Hours”

Matt Gowie has joined Public “Office Hours”

Tim Gourley has joined Public “Office Hours”

Dave Gregory has joined Public “Office Hours”

Charles Smith has joined Public “Office Hours”

Allan Swanepoel has joined Public “Office Hours”

Alex Atkinson has joined Public “Office Hours”

Steven Vargas has joined Public “Office Hours”

Vicken Simonian has joined Public “Office Hours”

Max Lobur has joined Public “Office Hours”

@Erik Osterman (Cloud Posse) - https://www.complianceforge.com/secure-controls-framework-scf-download/
Secure Controls Framework (SCF)

Links from today’s call: https://hashiconf.com/global/ https://health.aws.amazon.com/health/status https://github.com/vulhub/vulhub https://github.blog/changelog/2022-09-26-github-actions-dynamic-names-for-workflow-runs/ https://blog.cloudflare.com/workerd-open-source-workers-runtime/ https://aws.amazon.com/about-aws/whats-new/2022/09/amazon-route-53-support-dns-resource-record-set-permissions/ https://www.hashicorp.com/blog/terraform-enterprise-adds-run-metrics-versioned-docs-and-more https://gist.github.com/AlexAtkinson/fb113e9001058b459d8a69bb52b7a18d
Pre-Built Vulnerable Environments Based on Docker-Compose

Join the HashiCorp community in LA and online for two full days of infrastructure and security automation, modern application delivery, and the future of cloud.
View the overall status and health of AWS services using the AWS Health Dashboard.

GitHub Actions: Dynamic names for workflow runs
2022-09-29

Hi folks, if it interests you I would love to hear some discussion on this question I just posted in #prometheus https://sweetops.slack.com/archives/CE2JD8N1M/p1664446180231159
Hi all, wondering if anyone has a solution they could share for running analytics over historical alerts from alertmanager? For example, “alert X fired 10 times in July and 30 times in August”. This would allow the monitoring team in company Y to investigate the biggest drain for the on-call shift over time. I understand it’s a design decision of AM to not persist any state but in medium/large environments I think it’s an important area that often gets overlooked. Thanks.

DORA 2022 is out
Blog post is up at https://cloud.google.com/blog/products/devops-sre/dora-2022-accelerate-state-of-devops-report-now-out and I attached the PDF cause I find the “let us spam you” form annoying.

Security-enhancing DevOps practices are broadly adopted, this year’s DORA Accelerate State of DevOps Report found, but that’s not the whole story.