#office-hours (2023-03)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2023-03-01
@here office hours is starting in 30 minutes! Remember to post your questions here.
Linda Pham (Cloud Posse) has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
tyler has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Allan Mohr has joined Public “Office Hours”
Joe Caulfield has joined Public “Office Hours”
Joe Perez has joined Public “Office Hours”
Fireflies.ai Notetaker has joined Public “Office Hours”
Charles Smith has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Andy Wortman has joined Public “Office Hours”
Constantin Popa has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Michael Martin has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Adedapo Ajuwon has joined Public “Office Hours”
Antarr Byrd has joined Public “Office Hours”
13155203466 has joined Public “Office Hours”
Radha Ramadoss has joined Public “Office Hours”
emem u has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Paul Marcelin has joined Public “Office Hours”
Jared Richards has joined Public “Office Hours”
Aaron Cutchin has joined Public “Office Hours”
Michael Vasilenko has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Jeremy White has joined Public “Office Hours”
Brad Curfman has joined Public “Office Hours”
Yusuf has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Darren Pham has joined Public “Office Hours”
Michael Vasilenko has joined Public “Office Hours”
Related to the ChatGPT topic: https://twitter.com/multikev/status/1616784555788075009
Matt Ouellette has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Links from today’s office hours:
https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-rds-postgresql-major-version-15/ https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-cloudwatch-internet-monitor-generally-available/ https://blog.lastpass.com/2023/03/security-incident-update-recommended-actions/ https://www.hashicorp.com/blog/writing-terraform-for-unsupported-resources https://github.blog/changelog/2023-02-24-new-forks-page-view/ https://aws.amazon.com/about-aws/whats-new/2023/02/amazon-ecs-deletion-inactive-task-definition-revisions/ https://pages.awscloud.com/GLOBAL-ln-GC-TrainCert-Professional-Certification-Challenge-Registration-2023.html https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-kubernetes-version-1-25/ https://github.com/aws/containers-roadmap/issues/1967 https://github.com/hashicorp/terraform-provider-aws/issues/29682 https://kubernetes.io/docs/concepts/security/pod-security-admission/ https://marketplace.visualstudio.com/items?itemName=DanielSanMedium.dscodegpt https://twitter.com/multikev/status/1616784555788075009 https://typefully.com/iamvlaaaaaaad/codewhisperer-vs-copilot-8HjruKe https://arxiv.org/abs/2211.03622 https://commonfate.io/blog/granted-approvals-release https://github.com/zurawiki/gptcommit https://twitter.com/iamvlaaaaaaad/status/1630327934715064322 https://www.vladionescu.me/posts/scaling-containers-on-aws-in-2022 Introducing MRSK
2023-03-05
I might not use this to deploy/create resources…. but i might just use it as a way to back up the state of an AWS account vs using something like terraformer
https://iasql.com/
Automatically import existing infrastructure
Connect an AWS account to IaSQL to provision a PostgreSQL db and automatically backfill the database with your existing cloud resources. No need to redefine or reconcile existing infrastructure.
2023-03-08
@here office hours is starting in 30 minutes! Remember to post your questions here.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Jeremy Bouse has joined Public “Office Hours”
Brian Pauley has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Nate Garcia has joined Public “Office Hours”
dave lundgren has joined Public “Office Hours”
Fireflies.ai Notetaker has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Paul Marcelin has joined Public “Office Hours”
Evan Pitstick has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
Cannon Palms has joined Public “Office Hours”
John Mitchell has joined Public “Office Hours”
Devendra Yadav has joined Public “Office Hours”
Yonatan Koren has joined Public “Office Hours”
AJ Junior has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Fabian Berisha has joined Public “Office Hours”
Jim C has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Zachary Loeber has joined Public “Office Hours”
Arjun Dandagi has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Madhusudan Satapathy has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Michael Vasilenko has joined Public “Office Hours”
Kris Musard has joined Public “Office Hours”
Harry Moreno has joined Public “Office Hours”
Links from today’s office hours:
https://iasql.com/ https://blog.cloudflare.com/big-pineapple-intro/ https://opensource.googleblog.com/2023/03/introducing-service-weaver-framework-for-writing-distributed-applications.html https://aws.amazon.com/blogs/aws/subscribe-to-aws-daily-feature-updates-via-amazon-sns/ https://github.com/bregman-arie/devops-exercises/tree/master/topics/terraform https://digger.dev/ https://aws.amazon.com/blogs/aws/aws-application-composer-now-generally-available-visually-build-serverless-applications-quickly/ https://github.com/bregman-arie/devops-exercises/tree/master/topics/terraform https://github.com/cresta/atlantis-drift-detection https://www.atlassian.com/software/compass https://backstage.io/ https://twitter.com/iamvlaaaaaaad/status/1617510607946661891 https://www.reddit.com/r/devops/comments/1171it7/backstage_is_not_userfriendly_i_want_something/ https://developer.atlassian.com/cloud/compass/components/create-view-update-and-delete-components/ https://roadie.io/ https://github.com/klothoplatform/klotho https://klo.dev/docs/why-klotho https://aws-new-features.s3.us-east-1.amazonaws.com/update/2023-03-08.json https://www.chainguard.dev/unchained/what-the-fork-imposter-commits-in-github-actions-and-ci-cd https://techcrunch.com/2022/12/15/spotifys-plan-to-monetize-its-open-source-backstage-developer-project/ http://frontside.com/ https://www.sobyte.net/post/2022-01/linus-play-a-trick-of-github-vulnerability/
2023-03-15
maybe interesting for OH https://github.com/awslabs/amazon-s3-tar-tool https://aws.amazon.com/about-aws/whats-new/2023/03/mountpoint-amazon-s3/
@here office hours is starting in 30 minutes! Remember to post your questions here.
I’m not going to make it today, sorry!
I expected this months ago and moved my workflows over to GitHub Packages and AWS Public ECR. I’m not surprised.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
Darren Pham has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
Fireflies.ai Notetaker has joined Public “Office Hours”
Mitchell Gaddy has joined Public “Office Hours”
Jeremy White has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Yonatan Koren has joined Public “Office Hours”
Jeremy Bouse has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Maura Rowell has joined Public “Office Hours”
Gabriel has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Ivan Polchenko has joined Public “Office Hours”
Ashwin Jacob has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Igor B has joined Public “Office Hours”
Josh has joined Public “Office Hours”
Kris Musard has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Sigh. Turning on ICMP is the first thing I do.
Eric Berg has joined Public “Office Hours”
Josh has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Johnmary Odenigbo has joined Public “Office Hours”
I have a TIL I’m excited to share : https://sweetops.slack.com/archives/CCT1E7JJY/p1678906460455109
Antarr Byrd has joined Public “Office Hours”
Peter Dada has joined Public “Office Hours”
Johnmary Odenigbo has joined Public “Office Hours”
Paul Marcelin has joined Public “Office Hours”
Mike Martin has joined Public “Office Hours”
I have a DNS management question; we currently use terraform to manage route53, but now we manage 400+ records in one hosted zone and it takes minutes to plan and on top of that we are beginning to hit rate limits from AWS. Does anyone manage their route53 with anything else?
Terraform isn’t so great at managing many resources at a time. I co-locate route53 resources with the apps they support to spread out the burden. The downside to this is that there is no single pane of DNS glass from a terraform perspective. Other tools, like the Web-UI, have to be used for the whole view.
Yeah - that is the pain we are facing now. We have all of our records from one hosted zone in one terraform directory. We prefer the single view, but as you mentioned Terraform is struggling to keep up. I’m looking for a better way to have speed and a good view of things.
You might benefit from two things:
You can try tweaking parallelism to a value greater than 10. You might also try to collocate the terraform binary executing the apply inside AWS to minimize latency and rate-limiting effects. (normally I don’t advocate for this unless there’s a need, but there might be here!)
Re: Darren’s buildx + ECR problem in office hours today (watch near the end of the recording for context)
It looks like the problem has to do with -o type=registry, which translates to -o type=image,push=true
Basically the random snippet in my gist which was pushing proper OCI compliant (multiarch) images to ECR, probably can be done with -o type=oci,push=true, but I haven’t checked
EDIT:
Nvm, there is no such thing as type=oci,push=true. According to the docs, type=image should be OCI compliant.
But @Darren Pham I am back at the the authorization error in ECR when I do --push=true (even though my IAM policies are correct).
So I have reverted to the export to tar.gz, then importing (and overriding CMD and ENTRYPOINT because it is stripped on import), and then pushing to ECR…
So anyways… hope this somehow helps you.
Even though this is ugly https://gist.github.com/korenyoni/a5bd32da16a428477fce960060f192e9/63fca3862185cae7d33b885164e6c8949e0631ff
It’s the only thing that seems to work for me (and maybe for you)
Would have been timely for today but sharing FYI anyway. Might be fun to see what comes out of it next week.
Links from today’s office hours:
https://github.com/awslabs/mountpoint-s3 https://chanzuckerberg.github.io/fogg/ https://msrc.microsoft.com/update-guide/vulnerability/CVE-2023-23415 https://blog.alexellis.io/docker-is-deleting-open-source-images/ https://aws.amazon.com/blogs/security/how-to-use-policies-to-restrict-where-ec2-instance-credentials-can-be-used-from/ https://github.com/awslabs/amazon-s3-tar-tool https://github.com/slimtoolkit/slim https://github.com/rancher/rancher https://github.com/abiosoft/colima https://github.com/moby/buildkit/issues/1512 https://github.com/docker/setup-buildx-action/issues/122 https://github.com/moby/buildkit/issues/3579#issuecomment-1416438660 https://github.com/actions/runner-images/issues/5631 https://github.com/actions/runner-images/issues/2552 https://azure.microsoft.com/en-us/blog/now-in-preview-azure-virtual-machines-with-ampere-altra-armbased-processors/ https://github.com/nektos/act https://github.com/estesp/manifest-tool https://gist.github.com/korenyoni/a5bd32da16a428477fce960060f192e9 https://github.com/moby/moby/pull/44598
2023-03-16
2023-03-21
2023-03-22
@here office hours is starting in 30 minutes! Remember to post your questions here.
Q: @Erik Osterman (Cloud Posse) could https://github.com/cloudposse/bastion get some love? If you guys have moved on and no longer maintaining it, could I volunteer as a maintainer?
Definitely open to volunteers. We’re not actively using it as we predominantly help customers use Teleport.
awesome, i’d love to volunteer to help with that and keep the project up to date on the latest versions of the libraries and such
Got a use case for this also
I note that https://github.com/cloudposse/bastion/pull/70 just got merged
I’ve pulled latest master, built locally and trying running with
docker run -p 1234:22 -e MFA_PROVIDER=google-authenticator -v ~/.ssh/authorized_keys:/root/.ssh/authorized_keys docker.io/cloudposse/bastion:dev
PAM: Module is unknown for root from 192.168.200.0
Connection closed by authenticating user root 192.168.200.0 port 50332 [preauth]
❯ docker run -p 1234:22 -e MFA_PROVIDER=google-authenticator -v ~/.ssh/authorized_keys:/root/.ssh/authorized_keys cloudposse/bastion
Running the upstream works.
@joshmyers I’ve been still working on it. Can you please test the latest master? I think everything is finally all updated and we fixed all the bugs from upgrading.
Hey @Jonathan - thanks so much for this. Yup tested latest master and seems to be working as I’d expect now.
On a related topic - https://github.com/cloudposse/github-authorized-keys/pull/37
Awesome, thanks for validating @joshmyers!
@joshmyers make sure to chase @Erik Osterman (Cloud Posse) down, he has lots of projects to keep track and manage, things easily fall through the cracks.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Q for Office Hours:
How do you folks track package versions as well as software licenses across repos/languages? We are looking at creating a repo and doing this manually right now but I have a feeling there is a better way of doing this. We are trying to ensure we keep our packages updated (CVEs/new features) and we also want to ensure we remain compliant with all the software licenses we are using.
I watched the office hours recording
I used to work at Sonatype , creators of Nexus IQ, not free
but one of the things is does really well is inventory of software dependencies, licenses and CSVs
one of the powerful features of Nexus
Thanks!
Dima Nelen has joined Public “Office Hours”
Jeremy Bouse has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
dave lundgren has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Fireflies.ai Notetaker has joined Public “Office Hours”
Andy Wortman has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Darren Pham has joined Public “Office Hours”
ashkan jafari has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Kris Musard has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Maura Rowell has joined Public “Office Hours”
Hugo Samayoa III has joined Public “Office Hours”
Taylor Turner has joined Public “Office Hours”
Johnmary Odenigbo has joined Public “Office Hours”
Joe Caulfield has joined Public “Office Hours”
Jeremy White has joined Public “Office Hours”
Jonathan Chan has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Andrew Elkins has joined Public “Office Hours”
emem u has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Gajanand Singh has joined Public “Office Hours”
Igor Bronovskyi has joined Public “Office Hours”
2023-03-23
Links from office hours:
https://aws.amazon.com/blogs/developer/announcing-the-end-of-support-for-node-js-12-x-in-the aws-sdk-for-javascript-v3/ https://aws.amazon.com/about-aws/whats-new/2023/03/s3-object-lambda-amazon-cloudfront-tailor-content-end-users/https://atmos.tools/core-concepts/components/inheritance/ https://docs.spacelift.io/integrations/observability/datadog https://github.com/cloudposse/geodesic/releases/tag/2.0.0 https://github.com/aws-samples/eks-cluster-upgrade https://github.com/ansible/terraform-provider-ansible https://steampipe.io/ https://hub.steampipe.io/plugins/turbot/trivy https://www.sonatype.com/products/repository-oss https://wazuh.com/ https://docs.snyk.io/manage-issues/snyk-reports/reports-overview https://fossa.com/solutions/software-bill-materials-management https://github.com/anchore/syft https://anchore.com/sbom/ https://github.com/poc-hello-world/greeter-service/pull/70 https://snyk.io/ https://github.com/renovatebot/renovate https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file https://codeql.github.com/ https://github.blog/2023-03-22-github-copilot-x-the-ai-powered-developer-experience/ https://github.com/features/preview/copilot-x https://githubnext.com/projects/copilot-for-pull-requests https://githubnext.com/projects/copilot-voice/ https://githubnext.com/projects/copilot-for-docs/ https://githubnext.com/projects/copilot-for-pull-requests#gentest
2023-03-24
https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/
If you run argocd you may have to do a couple of things.
Hah, yes we ran into this
@Igor Rodionov
2023-03-26
It seems to be bullshit and MinIO seems to be the bad actor actually: https://blocksandfiles.com/2023/03/26/we-object-minio-says-no-more-open-license-for-you-weka/
2023-03-27
Data regionality and sovereignty is a developing area of regulation, and fairly adjacent to GDPR or compliance framework like SOC2/ISO27001. I think CCPA has a line item on it? So what should startups that want to be able to have global user registrations do? Generally for compliance, it’s simply operating in shadow mode until they achieve good compliance reports, but data sovereignty as a new point of concern is a bigger challenge. Anyone have any experience with this discussion? I saw somewhere a recommendation to “simply” leave a copy of data in each region where it’s generated, and export to your data store/processing location.
Can we have something related to terraform restrictions while running it based on users
Depending on where you’re running it from… Like anything, if a user runs a module from their local and they have God tier access…
If you’re setting up tf apply out of some CI, like GH actions, you could do some cheap access control like this. Or that enterprise tier approvals feature…
Or you can use a platform like env0, which gets you all the RBAC you could want. https://www.env0.com/blog/custom-rbac-roles https://docs.env0.com/docs/sub-projects
Thanks for input but I’m completely relying on on prem tools and trying to find open source solution for it. And also I’m using bitbucket as repository and jenkins as CI. Env0 seems like paid product is there any open source solution for onprem fit
Often you can pay a little $ for SAAS, or spend \(\) developing an in-house solutions. Sometimes compliance kills that idea though.
Anyone with the code can do tf commands. Their access to affect those changes is something else. If they shouldn’t be able to do something with TF, they shouldn’t have the access necessary to affect those changes to begin with. Look to your access management strategy for internal users.
Not sure but as we knew if lot of team members are involving to contribute IAC. we need user based access and logging and particular user should some restriction on user resources
2023-03-28
2023-03-29
@Erik Osterman (Cloud Posse) you are probably across this but https://youtu.be/G9_DnebevJg
The feature I am excited about and I think you may also like is: https://docs.github.com/en/actions/using-workflows/required-workflows
Given how many repos you folks manage I imagine this could be useful.
They mention the workflows feature towards the end
Yes, was very excited about it, except for they don’t work on pull requests from forks
So for open source it’s still useless.
@here office hours is starting in 30 minutes! Remember to post your questions here.
Question for today: we are looking to implement healthchecks.io for our ecs scheduled tasks. 1. Anyone have experiences to share? From what I’ve seen so far, the internet seems to think this is a good product. 2. We are trying to implement it and looking for ideas on how to create healthchecks that share the same lifecycle as the scheulded task itself. My first though is to use this resource provider https://github.com/kristofferahl/terraform-provider-healthchecksio/tree/master as I can’t think of a better way to tie the creation of a healthcheck to AWS events that seemingly do not exist; ie. ecs only has deployment, container instance and another event that don’t really have anything to do with scheduled events.
Sorry for my long ramble and I will explain this better with my voice
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Dima Nelen has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Andrew Thompson has joined Public “Office Hours”
Nenna Salinas has joined Public “Office Hours”
Mike Martin has joined Public “Office Hours”
Eduardo Wohlers has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Ketan Patel has joined Public “Office Hours”
Fireflies.ai Notetaker has joined Public “Office Hours”
Michael Pursifull has joined Public “Office Hours”
tyler has joined Public “Office Hours”
Luis Masaya has joined Public “Office Hours”
Amer Zec has joined Public “Office Hours”
Paul Bullock has joined Public “Office Hours”
Maura Rowell has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Brian Choate has joined Public “Office Hours”
Alejandro Calbazana has joined Public “Office Hours”
Isa Aguilar has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Jeremy White has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
emem u has joined Public “Office Hours”
Devante Williams has joined Public “Office Hours”
Isa Aguilar has joined Public “Office Hours”
Dariusz Panasiuk has joined Public “Office Hours”
2023-03-30
Links from office hours:
https://www.theregister.com/2023/03/24/github_changes_its_ssh_host/ https://news.ycombinator.com/item?id=35295216 https://github.com/dlvhdr/gh-dash https://registry.terraform.io/providers/pseudo-dynamic/value/latest/docs https://github.com/apps/settings https://github.com/hashicorp/terraform-provider-aws/issues/29842 https://github.com/rogerwelin/cfnctl https://ergomake.dev/blog/docker-compose-as-a-universal-interface/ https://www.docker.com/blog/no-longer-sunsetting-the-free-team-plan/ https://aws.amazon.com/about-aws/whats-new/2023/03/amazon-guardduty-enforcement-threat-detection-organization/ https://registry.terraform.io/providers/hashicorp/awscc/latest https://www.ergomaker.com/ https://aws.amazon.com/about-aws/whats-new/2023/03/application-load-balancer-tls-1-3/ https://aws.amazon.com/about-aws/whats-new/2023/03/aws-copilot-customization-cdk-yaml-overrides/ https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-tls-listener.html https://github.com/qunash/chatgpt-advanced https://stackoverflow.com/questions/68767674/does-aws-application-load-balancer-support-tls-1-3 https://docs.github.com/en/actions/using-jobs/assigning-permissions-to-jobs https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html https://docs.aws.amazon.com/step-functions/latest/dg/connect-ecs.html https://aws.amazon.com/blogs/compute/orchestrating-aws-glue-crawlers-using-aws-step-functions/ https://docs.aws.amazon.com/AmazonECS/latest/userguide/lifecycle-metrics.html