#office-hours (2022-08)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2022-08-02
I’m just embarking on ‘decomposing’ a small number of shared/’monolithic’ AWS accounts into a larger number of focussed AWS accounts. Naming is hard. Feels like every account/workload/stage needs an expressive string to embed in its resource names, so when you’re referencing those resources across accounts, it’s easy to see what’s what without needing to parse account IDs. That string needs to be short (because of the many resource name length restrictions), expressive (so it actually helps) and unique (for facepalm avoidance). Am I thinking about this right? Any advice?
Do you mean something like this? https://registry.terraform.io/modules/cloudposse/label/null/latest
That certainly looks useful, thanks!
I’ll discuss on our call today
2022-08-03
I would like to discuss this today https://github.com/cloudposse/terraform-aws-efs/issues/114
@Jeremy G (Cloud Posse) will you be on the call today?
@Erik Osterman (Cloud Posse) I wasn’t planning to be on the call, but I can join if you want. LMK.
@Mohammed Yahya this question is probably best sorted directly with @Jeremy G (Cloud Posse)
and also there is a nice number of replication supported with Terraform now, which will help set up a DR easily:
• KMS
• ECR
• SecretManager
• EFS
• RDS
Regarding GitOps …
Without going into too much detail, I use Github actions and a structure of application and env repositories. Applications publish charts and terraform modules. Env repositories use those in addition to a env specific configuration. Everything is automated, the infrastructure and Kubernetes resources state is located as code in the env repositories. There is a lot of reusability with GH actions, there are a few cool custom features like automatic conversion of terraform output to helm values, automatic blue/green deployments for k8s version upgrades and others. Because of GH actions I don’t have to deploy/manage/troubleshoot/fix anything myself, I don’t have to worry about scalability. Multi-tenancy is achieved by creating additional env/tenant repositories and updating configuration.
Questions are:
Are solutions like Flux/Argo worth the self managing/scaling/troubleshooting/migration efforts? Can those solutions handle terraform? Or would I need another service like Spacelift in addition to Flux/Argo? Anybody using Flux/Argo? Pros/Cons? Anybody migrated from GH to Flux/Argo?
Sorry for the general open ended questions, I know that in the end I will have to decide for myself but I was curious about your thoughts and experience. Maybe this is something for the Office Hours. I will also go through older episodes try to find GitOps content.
I think a portion of your question was actually answered in the previous oh.
just as an additional point, GCP are going on all in on the Kubernetes Resource Model (KRM) and have comprehensive coverage to let you manage your resources in yaml with the config managed by GCP (e.g. no need to centrally manage/backup the config in a management cluster).
https://cloud.google.com/config-connector/docs/reference/overview
Redhat is investing some resources in this KRM direction also with kcp (https://github.com/kcp-dev/kcp).
AWS is hedging its bets but defaulting to cloud formation it seems. Azure is going with flux and bicep. On AWS the two approaches that support the KRM model are Pulumi and Crossplane but to get the managed config cluster you need to get the hosted version and they may not have full coverage/good components yet.
Great - lets discuss today
Listened to the episode, great input and points regarding flux/argo/etc. Will definitely help with decision making.
Thanks a lot
Thanks @Gabriel
@Eamon Keane thx a bunch for sharing the KRM, wasn’t aware of GCP path; looking at the service coverage it sounds like they put great effort into it.
@here office hours is starting in 30 minutes! Remember to post your questions here.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Chris Barnes has joined Public “Office Hours”
Dave Gregory has joined Public “Office Hours”
Linda Pham (Cloud Posse) has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Christopher Pieper has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Zadkiel has joined Public “Office Hours”
Ian Bartholomew has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Bradley Peterson has joined Public “Office Hours”
Isa Aguilar has joined Public “Office Hours”
Martin Palastanga has joined Public “Office Hours”
Adam Buggia has joined Public “Office Hours”
Josh has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Charles Smith has joined Public “Office Hours”
Zadkiel has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Ashwin Jacob has joined Public “Office Hours”
Antarr Byrd has joined Public “Office Hours”
Ben Smith (Cloud Posse) has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
Ozzy has joined Public “Office Hours”
Vincent Werner has joined Public “Office Hours”
tamsky has joined Public “Office Hours”
Michael Williams has joined Public “Office Hours”
Yusuf has joined Public “Office Hours”
Naiman Daniels has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Azar AKB has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Paul Bullock has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
13153275398 has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
13153275398 has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Links from today’s session: https://medium.com/tinder/exploiting-github-actions-on-open-source-projects-5d93936d189f https://github.com/TinderSec/gh-workflow-auditor https://beta.sayhello.so/ https://github.blog/changelog/2022-08-03-github-actions-remove-offline-self-hosted-runners/ https://github.com/runatlantis/atlantis/pull/997 https://aws.amazon.com/blogs/security/how-to-eliminate-the-need-for-hardcoded-aws-credentials-in-devices-by-using-the-aws-iot-credentials-provider/ https://aws.amazon.com/blogs/iot/securely-ingesting-large-sized-payloads-from-iot-devices-to-the-aws-cloud/ https://docs.aws.amazon.com/iot/latest/developerguide/iot-provision.html https://github.com/cloudposse/terraform-aws-efs/issues/114
2022-08-04
2022-08-05
2022-08-07
2022-08-09
Discussion topic : I have the privilege of designing the VPC’s for my org and i’m looking for insights. What would you do differently if you had this luxury (e.g use IPAM e.t.c)? Should I go multi-account with multi-vpc or embrace the touted simplicity of Shared VPC?
2022-08-10
AWS Storage Day is happening: https://aws.amazon.com/blogs/aws/welcome-to-aws-storage-day-2022/ and we got a couple (small) announcements
@here office hours is starting in 30 minutes! Remember to post your questions here.
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
Andrew Hall has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
Jonathan Poczatek has joined Public “Office Hours”
Linda Pham (Cloud Posse) has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Robert Jordan has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Oscar Jara has joined Public “Office Hours”
Barak Griffis has joined Public “Office Hours”
Alexandr Vorona has joined Public “Office Hours”
Johnmary Odenigbo has joined Public “Office Hours”
Gerard Ceraso has joined Public “Office Hours”
Jan-Arve Nygård has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Ben Smith (Cloud Posse) has joined Public “Office Hours”
Shaun Wang has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Ruslan Butdayev has joined Public “Office Hours”
Mazin Ahmed has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Charles Smith has joined Public “Office Hours”
Amaan Khan has joined Public “Office Hours”
Bridget Royer has joined Public “Office Hours”
Waqar Ahmed has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Antarr Byrd has joined Public “Office Hours”
Disclaimer: I haven’t used this, but only thought it was novel that they are essentially fighting AWS on margin for their NAT Gateways. https://www.cohesive.net/vns3/cloud-nat/
Links from today’s call: https://github.blog/2022-07-27-planning-next-to-your-code-github-projects-is-now-generally-available/ https://github.blog/changelog/2022-08-09-github-actions-self-hosted-runners-now-support-apple-m1-hardware https://bridgecrew.io/blog/checkov-enables-ci-cd-security-with-new-supply-chain-security-policies/ https://saveslack.com/ https://home.robusta.dev/blog/stop-using-cpu-limits/ https://www.hashicorp.com/blog/cdk-for-terraform-now-generally-available https://aws.amazon.com/about-aws/whats-new/2022/08/amazon-ebs-crash-consistent-snapshots-subset-ebs-volumes-attached-amazon-ec2-instance/
2022-08-11
2022-08-12
hey is anyone around?
1
Yep!
yep, call starting in 6min
2022-08-15
2022-08-17
@here office hours is starting in 30 minutes! Remember to post your questions here.
I won;’t make it to today’s call
summer break?
Nope, worky-worky me with a bunch of client meetings that of course are all scheduled for today
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Robert Jordan has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Linda Pham (Cloud Posse) has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Jonathan Poczatek has joined Public “Office Hours”
Thomas Poetke has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Ronnie Coleman has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Johnmary Odenigbo has joined Public “Office Hours”
Jesus Martinez has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Oskar Maria Grande has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
emem u has joined Public “Office Hours”
Matthew Pickens has joined Public “Office Hours”
Allen Lyons has joined Public “Office Hours”
Steven Kalt has joined Public “Office Hours”
Adam Buggia has joined Public “Office Hours”
Alex Atkinson has joined Public “Office Hours”
I recently heard there are cheaper alternatives to AWS Support provided by their third party partners. I’m unclear on which partners offer this but does anyone here use an alternative to AWS Support? If so, who do you pay and is it just as “good”?
Sam Caneer has joined Public “Office Hours”
Adedapo Ajuwon has joined Public “Office Hours”
Linda Pham (Cloud Posse) has joined Public “Office Hours”
emem u has joined Public “Office Hours”
Links from today’s call: https://github.com/cloudposse/terraform-aws-ecr-public https://github.com/cloudposse/terraform-aws-lakeformation https://github.com/cloudposse/terraform-aws-glue https://github.com/cloudposse/terraform-aws-athena https://github.com/cloudposse/terraform-aws-kinesis-stream https://github.blog/changelog/2022-08-10-github-pages-builds-with-github-actions-ga/ https://www.theregister.com/2022/08/12/atlassian_cloud_6999_2023/ https://github.com/jckuester/terradozer https://github.com/DontShaveTheYak/cf2tf https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html
2022-08-19
A dev asked me today whether he should store the rds-ca root cert in his Lambda’s repo (feels icky, would need a code change to update), add it at build time (redeploy to update, not loads better), or get it at run time (potentially adding an HTTP roundtrip to the front of every exec). Wondered if we could store in SSM and inject straight into the context as you can with ECS tasks, but couldn’t see how. Wondered whether Lambda’s temp storage persisted predictably enough that you could ‘memoize’/’cache’ the runtime get. Ran out of ideas. Posted in #office-hours. Any wisdom?
Just to clarify, you’re talking about downloading this file, right? https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html
We seem to be able to get away with just using https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem (linked from here) but essentially yes
For our lambda infra we’ve stored that same cert (which also gets a slight ick from me “as-a-pattern”) alongside DB credentials in secrets manager. But we do have a backlog ticket to have a cron-task/script verify the public .pem matches the cert in our json-based secret.
And you grab it at the start of each invocation?
And you grab it at the start of each invocation?
Yes — and essentially, this is free. We’re already mandated to make a call to SecretsManager to retrieve the RDS credentials (we don’t store the secret in the same account as the Lambda).
What type of RDS creds does your Lambda use? Where are those stored (if not using IAM)
Yes — and essentially, this is free. We’re already mandated to make a call to SecretsManager to retrieve the RDS credentials
Possibly being slow but not completely following this. It’s free because you’re already making a call to SecretsManager and you can ‘bulk retrieve’ the cert from there along with the RDS creds in a single HTTP transaction? Or is it free because it’s just really fast? (or both?)
What type of RDS creds does your Lambda use? Where are those stored (if not using IAM)
Generally we use SSM, but I believe this component recently switched to Secrets Manager as a trial.
Thanks, this is interesting!
2022-08-22
2022-08-24
@here office hours is starting in 30 minutes! Remember to post your questions here.
Q: Is there a deep-dive video covering the Cloudposse Way? Things like how contexts work, incl. the fixtures files, as well as overall phlosophy, etc.?
I’ll start a list
Don’t know yet when we will host it
Bit late to the party but I’m definitely up for an Activation Day, with one or two of my dudes. We’re on UK time, but I’m hopeful we could work something out.
Also, I’m building out TF for Aurora and would appreciate any input on the process.
Interested. [email protected]
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Robert Jordan has joined Public “Office Hours”
Adnan M. has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Allan Swanepoel has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Vincent Werner has joined Public “Office Hours”
Linda Pham (Cloud Posse) has joined Public “Office Hours”
Luis Masaya has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Bridget Royer has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Marcos Soutullo has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Matthew Pickens has joined Public “Office Hours”
Alexandr Vorona has joined Public “Office Hours”
Igor M has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Adedapo Ajuwon has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Ori Yhezkel Mualem has joined Public “Office Hours”
Q: Wondering if anyone knows of operational tools (other than k8s) that manage (config-as-code) and/or sync Consul’s service resolver configs:
• https://www.consul.io/docs/connect/config-entries/service-resolver
dag viggo lokoeen has joined Public “Office Hours”
Brian Choate has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Brian Choate has joined Public “Office Hours”
Mazin Ahmed has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Vincent Werner has joined Public “Office Hours”
Antonio Rodriguez has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Ozzy has joined Public “Office Hours”
Antarr Byrd has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Links from today’s call: https://github.com/cloudposse/terraform-aws-dms/tree/main/examples/complete https://github.blog/changelog/2022-08-23-github-actions-enhancements-to-openid-connect-support-to-enable-secure-cloud-deployments-at-scale https://github.com/multycloud/terraform-provider-multy https://docs.multy.dev/vm_sizes https://github.com/apache/libcloud https://registry.terraform.io/providers/devops-rob/terracurl/latest/docs/resources/request https://github.com/SvenHamers/terraform-provider-oauth https://doordash.engineering/2022/08/09/how-to-handle-kubernetes-health-checks/ https://aws.amazon.com/about-aws/whats-new/2022/08/aws-support-launches-managing-cases-slack/ https://github.com/apache/libcloud https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.Replication.html https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless-v2.upgrade.html
There was a question regarding RDS / Database certificates today - here is a short example of how I run it on my dev machine with docker-compose
2022-08-29
Q: Anyone use AWS code artifact? love it? hate it? We are considering it for pip. CI would push (already uses OIDC with circle CI), and devs would pull for docker container based environments. Are there any weird edges? At my last gig we were using nexus artifactory and nexus would not respect a package being yanked from pip. Nexus would continue to serve that yanked version (until a new version was pushed out) as the latest which was a hassle.
are we talking for ECR for Images?? or In previous life used code artifact for helm packages. before that was using s3 as source package manager.. artifact made it little easier with roles isolate the specific permissions only for CD roles
https://github.com/hypnoglow/helm-s3
This is the plugin i was referring for s3
No it’s for python and node packages ect
I think the biggest limitation from our POV is it’s limited in the supported artifacts
As in how only pypi and npm and maven ect. are supported?
2022-08-30
https://aws.amazon.com/blogs/aws/new-aws-support-app-in-slack-to-manage-support-cases/
this was discussed in the previous office hours, highlights on one of the points discussed.. looks like 100 account per slack channel
After you authorize your Slack workspace, you can add your Slack channels by choosing Add channel. You can add up to 20 channels for a single account. A single Slack channel can have up to 100 AWS accounts.
Thanks @Azar!
2022-08-31
Q: I’m trying to import google_project_iam_policy that looks like this:
data "google_iam_policy" "iam_policy_data_mazin_test_roles_containeranalysis_ServiceAgent" {
binding {
role = "roles/containeranalysis.ServiceAgent"
members = [
"serviceAccount:[email protected]"
]
}
}
resource "google_project_iam_policy" "iam_policy_mazin_test_roles_containeranalysis_ServiceAgent" {
project = "mazin-test"
policy_data = data.google_iam_policy.iam_policy_data_mazin_test_delete_roles_containeranalysis_ServiceAgent.policy_data
}
Based on documentation, it should be imported as:
$ terraform import google_project_iam_policy.my_project your-project-id
However when running this one followed by terraform plan, it shows that the role is being deleted instead. I’m not sure if it’s related to importing data sources, or if there is a better way to import GCP IAM roles. Any ideas?
https://www.terraform.io/internals/debugging - I will try debugging it with this feature Thanks @matt!
@here office hours is starting in 30 minutes! Remember to post your questions here.
Linda Pham (Cloud Posse) has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Mike Martin has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Andy Miguel (Cloud Posse) has joined Public “Office Hours”
Robert Jordan has joined Public “Office Hours”
Luis Masaya has joined Public “Office Hours”
Antonio Rodriguez has joined Public “Office Hours”
Allan Mohr has joined Public “Office Hours”
Andrew Vitko has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Ashwin Jacob has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
Isa Aguilar has joined Public “Office Hours”
Omer Sen has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Mike Drummond has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Life Lofranco has joined Public “Office Hours”
SYED HUSSAIN has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Brian Pauley has joined Public “Office Hours”
Roy Sprague has joined Public “Office Hours”
Adnan M. has joined Public “Office Hours”
Ben Smith (Cloud Posse) has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
dag viggo lokoeen has joined Public “Office Hours”
Dana Carney has joined Public “Office Hours”
Mazin Ahmed has joined Public “Office Hours”
Vincent Werner has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Links from today’s call: https://stackoverflow.com/tags/cloudposse/info https://aws.amazon.com/about-aws/whats-new/2022/08/amazon-cloudfront-origin-access-control/ https://sweetops.slack.com/archives/CB6GHNLG0/p1661965412589009 https://typefully.com/iamvlaaaaaaad/august-24-fargate-outage-N4rzykR https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_example-tag-policies.html
Anyone here tried this yet? https://metacontroller.github.io/metacontroller/