#office-hours (2022-01)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2022-01-05
@here office hours is starting in 30 minutes! Remember to post your questions here.
@Erik Osterman (Cloud Posse) is helmfile
a project you guys manage or is it its own thing?
that’s for @Erik Osterman (Cloud Posse)
do office hours include him as a resource as well?
oh geez
I see what I did. I apologize.
helmfile is a separate project - a tool we’ve used a lot in our engagements
I’ll be on office hours today
oh perfect…..thank you
Emile Fugulin has joined Public “Office Hours”
wasim k has joined Public “Office Hours”
wasim k has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Ralf-Eric Pieper has joined Public “Office Hours”
Yuri Lima has joined Public “Office Hours”
Jim Conner has joined Public “Office Hours”
@Erik Osterman (Cloud Posse) (for office hours) Any insights on provisioning cdns that are optimized to minimize http 2 response delays? We’re not always getting the benefits of removal of the 6 connection limit from http1. There are 2 chrome threads about this issue in case you want to do a bit more reading. https://bugs.chromium.org/p/chromium/issues/detail?id=1074705 https://bugs.chromium.org/p/chromium/issues/detail?id=723748
Brandon vh has joined Public “Office Hours”
Andy Miguel (Cloud Posse) has joined Public “Office Hours”
wasim k has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Florain Drescher has joined Public “Office Hours”
Benjamin Smith has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Zadkiel has joined Public “Office Hours”
Mike Martin has joined Public “Office Hours”
Kevin Edwards has joined Public “Office Hours”
links from today’s session:
• https://aws.amazon.com/about-aws/whats-new/2022/01/acm-kubernetes-cert-manager-plugin-production/
• https://github.com/aws/containers-roadmap/issues/904
• https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/pull/203
• https://registry.terraform.io/providers/paultyng/sql/latest/docs/resources/migrate
Tony Scott has joined Public “Office Hours”
Mohammed Yahya has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Jeremy Bouse has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
wasim k has joined Public “Office Hours”
Gerard Ceraso has joined Public “Office Hours”
Naija ninja has joined Public “Office Hours”
Amazon CloudFront is a content delivery network that can help you increase the performance of your web applications and significantly lower the latency of delivering content to your customers. For CloudFront to access an origin (the source of the content behind CloudFront), the origin has to be publicly available and reachable. Anyone with the origin […]
Andrew Thompson has joined Public “Office Hours”
And WAF might be an option: https://docs.aws.amazon.com/waf/latest/developerguide/what-is-aws-waf.html
• Allow all requests except the ones that you specify – This is useful when you want Amazon CloudFront, Amazon API Gateway, Application Load Balancer, or AWS AppSync to serve content for a public website, but you also want to block requests from attackers.
Use AWS WAF to monitor requests that are forwarded to an Amazon CloudFront distribution, an Amazon API Gateway REST API, an Application Load Balancer, or an AWS AppSync GraphQL API and to control access to your content. Use AWS Shield to help protect against DDoS attacks. Use AWS Firewall Manager to set up your firewall rules and apply the rules automatically across accounts and resources, even as new resources are added.
Paul Tomkinson has joined Public “Office Hours”
Michael Holt has joined Public “Office Hours”
Devendra has joined Public “Office Hours”
Mohammed Yahya has joined Public “Office Hours”
Naiman Daniels has joined Public “Office Hours”
Yuri Lima has joined Public “Office Hours”
Oscar Blanco has joined Public “Office Hours”
Just announced: https://aws.amazon.com/blogs/mt/announcing-aws-cloudtrail-lake-a-managed-audit-and-security-lake/
Organizations managing cloud infrastructure in AWS need effective mechanisms to audit operations in their AWS accounts for security and compliance. In November 2013, we announced AWS CloudTrail as the auditing platform for AWS. Since then, millions of customers have adopted this service. We believe CloudTrail is so important to AWS customers’ success that every new […]
Community Note Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request Please do not leave "+1" or other comme…
Sam Caneer has joined Public “Office Hours”
Thanks as always, y’all!
This has been super interesting. I have another meeting to attend… How long does office hours last? I can reconnect after my meeting if they are still going on.
A V has joined Public “Office Hours”
Naiman Daniels has joined Public “Office Hours”
Mauricio Wyler has joined Public “Office Hours”
Yuri Lima has joined Public “Office Hours”
Manage Tag Policies: https://console.aws.amazon.com/organizations/v2/home/policies/tag-policy#
Here’s how to evaluate compliance: https://docs.aws.amazon.com/ARG/latest/userguide/tag-policies-orgs-finding-noncompliant-tags.html
Learn how to find and correct noncompliant tags on an account’s resources.
As of Late 2021, which AWS service should I use to run my new containerized app in production?
Fargate now support Spot (when this video was recorded that was not available)
@Erik Osterman (Cloud Posse) any chance you have a master doc with the agenda of all the office hours? Sometimes when i’m short on time is easier to skim through the agenda to stay in contact instead of the video. thx
I think @Andy Miguel adds the agenda somewhere?
Our YouTube Office Hours videos have timestamped show notes in the description:
right, much thanks folks !
Jim Conner has joined Public “Office Hours”
2022-01-06
HI folks as a follow up to this issue that we discussed on the call yesterday https://sweetops.slack.com/archives/CHDR1EWNA/p1641410961200500
I wasn’t able to get any traces using x-ray from my lambda. X-Ray is activated for active tracing on the lambda but for some reason there are no traces being shown in the X-Ray dashboard. The lambda is Lambda@Edge as I mentioned on the call yesterday. Also there are other services in api gateway that I enabled X-Ray tracing on, and those services do have traces in the X-Ray dashboard. I also tried watching the network traffic on the device using charles proxy (alternative to wireshark) and it seems there’s a decent amount of latency (1-2s) even outside of downloading the images. I thought it was related to the speed of the request but as the attached session shows, there wasn’t a clear association. I attached the same session info as .trace
and .chls
. So all told I haven’t found the cause of the issue yet, I’m open to any ideas. Please let me know what other useful info I can provide. Thanks!
@Erik Osterman (Cloud Posse) (for office hours) Any insights on provisioning cdns that are optimized to minimize http 2 response delays? We’re not always getting the benefits of removal of the 6 connection limit from http1. There are 2 chrome threads about this issue in case you want to do a bit more reading. https://bugs.chromium.org/p/chromium/issues/detail?id=1074705 https://bugs.chromium.org/p/chromium/issues/detail?id=723748
cc @matt
@Erik Osterman (Cloud Posse) (for office hours) Any insights on provisioning cdns that are optimized to minimize http 2 response delays? We’re not always getting the benefits of removal of the 6 connection limit from http1. There are 2 chrome threads about this issue in case you want to do a bit more reading. https://bugs.chromium.org/p/chromium/issues/detail?id=1074705 https://bugs.chromium.org/p/chromium/issues/detail?id=723748
2022-01-11
2022-01-12
I have a talking point for today’s Q/A
How to Automate granting IAM Permissions to different Teams in growing company ?
• We use Terraform to define our IAM Roles and SSO Roles/PermissionSets.
• Recently my company has been growing fast, with many services created and many teams are being formed. We have new extra security requirements that require least privileged access to everything. And we got overwhelmed with requests whenever any new service is created.
• Although having IAC helps, it’s still slow and require Terraform knowledge if someone wanted to contribute and open a PR.
• We also can’t give our CI/CD IAM full access yet, so Terraform for IAM is currently applied from our Computers.
• I was thinking if there is a tool that manages IAM in way that users/team request IAM permissions, and admins or security engineers can Approve this change and it get applied automatically.
Potential talking point that I’d love to get others insight on if they’ve implemented similar:
Integrations with Github Actions, Vault and Terraform Cloud
• Github Actions acquiring GITHUB_TOKEN creds via Githup App
• Hashicorp Vault Github OIDC authenticated
• Terraform Cloud workspaces, multiple per repo or 1:1
@here office hours is starting in 30 minutes! Remember to post your questions here.
Eric Berg has joined Public “Office Hours”
Andrey Nazarov has joined Public “Office Hours”
Ben Dollinger has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
tony Scott has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Ray Myers has joined Public “Office Hours”
Zachary Loeber has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Madhusudan Satapathy has joined Public “Office Hours”
Dave Lundgren has joined Public “Office Hours”
Florain Drescher has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Jeremy Bouse has joined Public “Office Hours”
Mazin Ahmed has joined Public “Office Hours”
Jim Conner has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
Ralf-Eric Pieper has joined Public “Office Hours”
Thayne Trevenen has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Paul Scarrone has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Oscar Blanco has joined Public “Office Hours”
Kareem Shahin has joined Public “Office Hours”
Another potential talking point: https://www.theverge.com/2022/1/9/22874949/developer-corrupts-open-source-libraries-projects-affected
The sabotaged versions produce an endless string of illegible text.
Matt Gowie has joined Public “Office Hours”
Ben Dollinger has joined Public “Office Hours”
Neil Gealy has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Ralf-Eric Pieper has joined Public “Office Hours”
Validate and define text-based and dynamic configuration
Vicken Simonian has joined Public “Office Hours”
Uwaila Adams has joined Public “Office Hours”
Mauricio Wyler has joined Public “Office Hours”
Hoping we talk about the #helmfile discussion https://sweetops.slack.com/archives/CE5NGCB9Q/p1641494827015300 https://sweetops.slack.com/archives/CE5NGCB9Q/p1641495251016200 https://sweetops.slack.com/archives/CE5NGCB9Q/p1641833841070300?thread_ts=1641546392.029800&cid=CE5NGCB9Q
our infra guys decided to use terraform to deploy helm charts using the terraform helm provider (which I think Mumoshu maintains/wrote actually) but imo, deploying software with terraform is a really really really bad idea blending infrastructure with software, which is poor practice. You really should keep those two things separate. The infra guys and we have been recognizing a great number of problems with this make up so I’m separating all of that out into helmfile. That’s proving to be a chore because of what I’ve identified problematic given the size of our environment and what helmfile documentation speaks to.
This is a good #office-hours topic. Let’s discuss next week
In most typical cluster setups, there are generally and pretty consistently, in my experience, at least three components for environments….
- the cluster/infrastructure a. when using a cloud provider b. if using bare-metal then this could be split into two completely disparate components
- the substrate e.g. components running on infrastructure to support the main application/micro-service(s)
- the application Practices which could be discussed:
- always separate infrastructure from software deployments to eliminate accidental actions from occurring. a. terraform was meant for infrastructure. Some people believe it’s a good idea to use terraform for infra and software but I personally think this is a really really bad idea (we’ve been bit in the collective butts by this). b. day-1 deployments typically work fine…but c. extra precautions, code, software must be taken or used to keep unintended issues from occurring on day-2 deployments. which could just add to tech debt…moreover, d. when using terraform to deploy substrate, day-2 operations require terraform to run which means terraform must check the state and possibly apply to your infrastructure before getting to substrate/software deployment, which is usually not desired and unless expected is definitely not suggested. e. in our experience, if something in the substrate needs attention, terraform helm provider doesn’t always catch the correct state of helm charts deployed and extra work needs to be done to “fix” that which for us usually means side-deleting a chart and then re-applying terraform; see #a
- substrate software should be deployed using helm and better if deployed using some kind of package release software like helmfile, helmwave (maybe, I don’t know much about this) or home-grown wrapper…to ease release – a. It’s up to engineers how best to perform this task. We’re migrating out of terraform and trying out helmfile.
- for deploying your software stack, something like argocd is likely requisite but this is also up to engineering teams to decide. The idea behind tooling is to:
- ease toil and burden on operations teams and devs
- effectively perform the tasks necessary for proper operations and uptime of application(s)
Mauricio Wyler has joined Public “Office Hours”
Yusuf has joined Public “Office Hours”
Yuri Lima has joined Public “Office Hours”
Least privilege AWS IAM Terraformer. Contribute to bridgecrewio/AirIAM development by creating an account on GitHub.
Yuri Lima has joined Public “Office Hours”
Abraham Olu has joined Public “Office Hours”
Hey @Erik Osterman (Cloud Posse) it seems you are one the few I’ve seen who uses ADR (or something similar) successfully:) (https://adr.github.io/)
Homepage of the ADR GitHub organization
Yes, we use a modified version of this for our purposes
Homepage of the ADR GitHub organization
It’s been critical to helping customers understand the justifications and tradeoffs for all decisions
wedneyyuri has joined Public “Office Hours”
following today’s office-hours call I was taking a look through the geodesic tutorial… Not sure if this is a known issue or an artifact of my workstation. I’m running Ubuntu 20.04 LTS under WSL2 on Windows 10 Pro. When I start geodesic
I am greeted with a notice about it running on an Apple M1 CPU and that it isn’t supported. My system isn’t MacOS and it’s actually running on a AMD Ryzen 7
2022-01-13
2022-01-19
@here office hours is starting in 30 minutes! Remember to post your questions here.
Question for today: In cicd, what is a good way to ensure that a terraform plan
on one git branch does not block other branches from doing terraform plan
, if the tf state is stored in AWS S3 and plan crashes without releasing the backend lock? I was thinking of checking directly with aws dynamodb
command to see if lock is there but I’m not seeing an obvious way to determine whether the lock is for the plan that just crashed.
Isa Aguilar has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Paul Scarrone has joined Public “Office Hours”
Isa Aguilar has joined Public “Office Hours”
Mohammed Almusaddar has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Loren Gordon has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
Florain Drescher has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Stan Miroshnikov has joined Public “Office Hours”
Ben Azoulay has joined Public “Office Hours”
Ben Smith (Cloud Posse) has joined Public “Office Hours”
Josh B has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Amer Zec has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Mikey Carr has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Chocks Subramanian has joined Public “Office Hours”
Chocks Subramanian has joined Public “Office Hours”
Andrew Bost has joined Public “Office Hours”
Chocks Subramanian has joined Public “Office Hours”
Andrey Nazarov has joined Public “Office Hours”
Jim Conner has joined Public “Office Hours”
Uwaila Adams has joined Public “Office Hours”
PePe Amengual has joined Public “Office Hours”
Mauricio Wyler has joined Public “Office Hours”
Andrew Thompson has joined Public “Office Hours”
Marc Slayton has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
TACOS to the rescue for sure)
Is it really a common practice to run terraform plan continuously to detect drifts? I don’t see it that often.
Isaac M has joined Public “Office Hours”
2022-01-21
This one might be a fun one to discuss/review for office hours: https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/
Starting October 28th and fully resolving on October 31st, Roblox experienced a 73-hour outage. We’re sharing these technical details to give our community an understanding of the root cause of the problem, how we addressed it, and what we are doing to prevent similar issues from happening in the future.
2022-01-26
@here office hours is starting in 30 minutes! Remember to post your questions here.
I’m working on a commandline tool to support automated refactoring of Terraform code, similar to what we have in IDEs for refactoring Java etc…
I’m curious, what tools are people already using to help refactor Terraform and what types of operations do they wish were more automated?
What are your thoughts on having terraform code in the application repository for resources specific to the application and more static resources like databases and VPCs in their own repo?
Oops, I missed these messages.
We’ll address this week.
Does it make sense to run terraform/CDK deployments in the same pipeline as your app’s ci/cd? We use CDK and it’s inside our monorepo, but wanted…
Deirdre Rodgers has joined Public “Office Hours”
Marcelo Santoro has joined Public “Office Hours”
Bhavik Patel has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Marcelo Santoro has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Jim Conner has joined Public “Office Hours”
Ralf Pieper has joined Public “Office Hours”
Paul Bullock has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Houman Jafarnia has joined Public “Office Hours”
Ray Myers has joined Public “Office Hours”
Marcos Soutullo has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Isaac M has joined Public “Office Hours”
Michael Manganiello has joined Public “Office Hours”
Steven Vargas has joined Public “Office Hours”
Zachary Loeber has joined Public “Office Hours”
David Lozano has joined Public “Office Hours”
Mikey Carr has joined Public “Office Hours”
Yonatan Koren (Cloud Posse) has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
Luis Masaya has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Container runtimes on macOS (and Linux) with minimal setup - GitHub - abiosoft/colima: Container runtimes on macOS (and Linux) with minimal setup
Jeremy Bouse has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
emem peace has joined Public “Office Hours”
Gerard Ceraso has joined Public “Office Hours”
Emem has joined Public “Office Hours”
Naren arjun has joined Public “Office Hours”
Florain Drescher has joined Public “Office Hours”
Stelios Latsinoglou has joined Public “Office Hours”
wasim k has joined Public “Office Hours”
Mauricio Wyler has joined Public “Office Hours”
Andrew Thompson has joined Public “Office Hours”
Otilio Reyes has joined Public “Office Hours”
Kevin Huang has joined Public “Office Hours”
Andy Roth has joined Public “Office Hours”
Stelios Latsinoglou has joined Public “Office Hours”
Oliver Schoenborn has joined Public “Office Hours”
Amer Zec has joined Public “Office Hours”