#office-hours (2021-10)
“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours
Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!
https://cpco.io/slack-office-hours
Meeting password: sweetops
2021-10-06
@here office hours is starting in 30 minutes! Remember to post your questions here.
Sahil Kamboj has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Andy Miguel (Cloud Posse) has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Alexandr Vorona has joined Public “Office Hours”
Benjamin Smith has joined Public “Office Hours”
Florain Drescher has joined Public “Office Hours”
Ross Rfd has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Michael Manganiello has joined Public “Office Hours”
Ryan Ryke has joined Public “Office Hours”
Justin Davis has joined Public “Office Hours”
Erin Angerer has joined Public “Office Hours”
Yusuf has joined Public “Office Hours”
Adam Blackwell has joined Public “Office Hours”
Mukesh Sharma has joined Public “Office Hours”
Nate Faerber has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Mike Crowe has joined Public “Office Hours”
Antarr Byrd has joined Public “Office Hours”
Arjun Venkatesh has joined Public “Office Hours”
Mike Crowe has joined Public “Office Hours”
Andrew Thompson has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
links from today’s session:
• https://twitter.com/lukehoban/status/1443658655467900936?s=21
• https://github.com/hashicorp/terraform-provider-awscc
• https://www.hashicorp.com/blog/announcing-terraform-aws-cloud-control-provider-tech-preview
• https://www.pulumi.com/blog/announcing-aws-native/
• https://blog.cloudflare.com/introducing-r2-object-storage/
• https://github.com/cloudposse/terraform-aws-config/tree/master/modules/conformance-pack
Erin Angerer has joined Public “Office Hours”
Tony Scott has joined Public “Office Hours”
Geoff Weinhold has joined Public “Office Hours”
Patrick Joyce has joined Public “Office Hours”
Chocks Subramanian has joined Public “Office Hours”
So excited about this!!
Oscar Blanco has joined Public “Office Hours”
Zadkiel AHARONIAN has joined Public “Office Hours”
Mohammed Yahya has joined Public “Office Hours”
Patrick Joyce has joined Public “Office Hours”
Oscar Blanco has joined Public “Office Hours”
@matt, I feel like I hit a related issue regarding encrypted AMI (or at least EBS volume) in EKS using eksctl. I ended up falling back to some defaults that got me by so I can’t say this would have solved my problem but maybe it’s a data point: https://github.com/weaveworks/eksctl/issues/952#issuecomment-507219406
Is the principal listed here one you have tried?
Must be since this links back to official AWS docs which I assume you have checked. I’ve had issues with ASGs using encrypted AMI cross-account that I never solved so your story was bringing back bad memories. I’d like to know the solution when you figure it out.
Thanks, I’ll take a look and make sure I didn’t miss anything.
For this particular customer, they have HIPAA requirements and have an org policy that doesn’t allow any EBS volumes that are unecrypted, so I don’t have any choice but to use the encrypted AMIs for packer
yeah. Makes sense.
2021-10-12
Agnello Dsouza has joined Public “Office Hours”
2021-10-13
@here office hours is starting in 30 minutes! Remember to post your questions here.
Oscar Blanco has joined Public “Office Hours”
Matteo Migliaccio has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Mukesh Sharma has joined Public “Office Hours”
Phillip Hocking has joined Public “Office Hours”
Oscar Blanco has joined Public “Office Hours”
Carlos Ybere Rodrigues has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Ian Bartholomew has joined Public “Office Hours”
Andy Miguel (Cloud Posse) has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Benjamin Smith has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Daniel Miller (CloudPosse) has joined Public “Office Hours”
Justin Davis has joined Public “Office Hours”
emem peace has joined Public “Office Hours”
Michael Manganiello has joined Public “Office Hours”
Sarah Donehower has joined Public “Office Hours”
Jason Skidmore has joined Public “Office Hours”
Agnello dsouza has joined Public “Office Hours”
James Haughey has joined Public “Office Hours”
Patrick Joyce has joined Public “Office Hours”
Oskar Maria Grande has joined Public “Office Hours”
Mauricio Wyler has joined Public “Office Hours”
Othman Musleh has joined Public “Office Hours”
Florain Drescher has joined Public “Office Hours”
Marc Slayton has joined Public “Office Hours”
yong chen has joined Public “Office Hours”
Andrew Thompson has joined Public “Office Hours”
yong chen has joined Public “Office Hours”
Ian Bartholomew has joined Public “Office Hours”
I feel like Theo Schlossnagle’s Art and Science of the Service Level Objective is a solid reference: https://www.circonus.com/whitepapers/art-and-science-of-the-service-level-objective.pdf
I also like the SLO chapter from the Google SRE book: https://sre.google/sre-book/service-level-objectives/
Ian Bartholomew has joined Public “Office Hours”
2021-10-20
Does anyone have a demo app framework to recommend? Something along the lines of https://codebase.show/projects/realworld but with an emphasis on infrastructure?
via @Vlad Ionescu (he/him) https://github.com/poc-hello-world/greeter-service
KHAI Anis has joined Public “Office Hours”
@here office hours is starting in 30 minutes! Remember to post your questions here.
Hi, about the layered infrastructure, the first layer (fundamental infrastructure) and EKS (which we consider platform, right?) are into different tf dirs. Would terragrunt be a valid option to maintain these two, or that would be overkilling and a terraform_remote_state could suffice to get the resources from the first terraform (into the EKS) one.. what’s the best approach for you?
Oliver Schoenborn has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Charles Sperbeck has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
venkata mutyala has joined Public “Office Hours”
Mike Cox has joined Public “Office Hours”
Mauricio Wyler has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
Jason Skidmore has joined Public “Office Hours”
Ian Bartholomew has joined Public “Office Hours”
Nate Faerber has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
Aleksandr Fofanov has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Sarah Donehower has joined Public “Office Hours”
Andy Miguel (Cloud Posse) has joined Public “Office Hours”
Daniel Miller (CloudPosse) has joined Public “Office Hours”
Matteo has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Matt Gowie has joined Public “Office Hours”
Justin Davis has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Antarr Byrd has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Yusuf has joined Public “Office Hours”
links from today’s session:
• https://github.com/mitchellh/terraform-provider-multispace
• https://www.pulumi.com/blog/introducing-pulumi-registry/
• https://github.com/FairwindsOps/saffire
• https://registry.terraform.io/providers/hashicorp/time/latest
• https://blog.cloudflare.com/automatic-remediation-of-kubernetes-nodes/
Mukesh Sharma has joined Public “Office Hours”
Obi One has joined Public “Office Hours”
Eric Berg has joined Public “Office Hours”
Florain Drescher has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
i say this with humility because we are building a framework
… even more humility since it reads like it was written this morning and not 1~5~6 years ago….
That was very informative. Thank to everybody involved! I’m missing being there live though))
2021-10-27
Yo, Sherif here 
I am looking forward to join today’s office hours, I just discovered cloudposse community (although I’ve been using the modules for quite sometime )
I work as a SRE/DevOps/Infa/Platform ( whatever the name is it’s confusing )
I apologies for the wall of text
My current work is moving our Infrastructure to a more modern setup, this include using Terraform, Kubernetes, and all the things in between. We’ve built something very cool and I am proud of it.
We do have a Production, Staging and QA Environments ( 1 Per QA Member, about ~30 in total), and we have about 30 Service.
Ironically I modeled our Infrastructure in a way very similar to SweetOps 4 Layer model. Aside from the Foundational, Platform, and Shared Services part, my main point of struggle that I am very eager to improve is the Application Layer.
Especially the Application Infrastructure (call it AppInfra) which are the Application 1-to-1 Infrastructure components such as S3 Buckets, RDS, CloudFront. and Let me explain what I want to achieve.
I just made a deal with my workplace to give me a AWS Account to build a Greenfield virtual company that I’ll try to explore good practices using. And I am trying to solve some problems we’re already having (but without caring so much about disturbing business, it will be a greenfield project)
So
Our current way
- Applications are Deployed using Push Model, thus mean, CI (jenkins sadly ) will Apply the Terraform part, then proceed with Applying the Helm Chart.
- We use Terraform to release the helmchart, which is a very intuitive combo in theory especially when passing Terraform outputs to the Helm Values. My Goals:
- GitOps GitOps GitOps… I want to implement GitOps ( the reconciliation part is what is missing atm)
- Implement Feature Branch Preview Environments
- I tried to shift any thing possible to Kubernetes, using the service’s helm chart (it’s just soo intuitive imo), this includes the: a. All application components like Webservers, Consumers, Jobs, and CronJobs are in the Chart. b. Route53 Records using External DNS c. Monitoring and Alerting Using Prometheus Operator d. Grafana Dashboard using Grafana Operator e. And of course Kubernetes stuff like Autoscaling, KEDA, etc are all part of the Helm Chart.
- However… I can’t use Helm Chart for AppInfra yet… a. I looked into Crossplane.io, and AWS Operator, but they’re not there yet imo. Infrastructure is different, The CRDs are fine for the optimistic situations, but letting my Kubernetes Operator do RDS Restore, Scaling, or other tricky Infrastructure tweaks ( The DB Migration analogy comes to mind), I don’t feel Kubernetes Operator pattern is the best method here. b. Using Terraform to deploy changes in the PRs (before merging and before Argo does its sync) seems fine. We won’t deploy our App to Kubernetes unless Terraform is in sync. c. However… This breaks when dealing with highly dynamic environments, when I have like ~5 interdependent services, each one has their own AppInfra, I can’t use GitOps methods to just point Argo to sync them. I’ll need to deploy them 1 by 1, as well as Cleaning Them Up. d. And now I will use my CI to handle this deployments and ask it to trigger 5 services, then ask it later to go destroy them too. and I am drifting away from Kubernetes, and more coupled to my CI to do all the operations. I am asking, is there is a better way to do it ? What do you think ?
Questions ( for OfficeHours)
- How do I deploy on-demand environments (say ~5 interdependent services) using GitOps & Terraform together ? a. Using: i. Terraform to Apply the App Infrastructure (1-to1 Infra) ii. ArgoCD for syncing the Chart itself. b. And avoiding maintaining the services dependency tree and triggering my CI in specific sequence to provision an environment ?
- How to pass information between Terraform and the Helm Chart (deployed via Argo) ? a. For Testing/QA we have DB Snapshot Volumes that we will use Terraform to clone, and pass it for the HelmChart b. The HelmChart then uses to create a PV & PVC for the DB Pod.
You take a look at env0 yet? That product squarely targets your ephemeral environment needs I believe
Huge thanks for the shoutout @Zachary Loeber. @Sherif there are a few of us from env0 here in this community. We would be glad to setup a time to talk with you about your use case and how could help you solve the issues, if you’d like.
For example, we have a blog about how to get the Feature Branch Previews you mentioned: https://www.env0.com/blog/from-feature-branches-to-feature-environments
Thanks @Zachary Loeber & @tim.davis.instinct I don’t think my company is ready yet to introduce yet again more components, they’re already still digesting the Kubernetes move and we’ve only migrated 40% of our stack.
So as much as I’d love to have a talk ( Let’s set this up! ), I don’t think we can get an approval yet.
Also, the enthusiastic engineer in me also wanna have a discussion about implementing this setup using purely OSS tools, that’s why I got an approval under the learning budget to have a separate AWS Account for a greenfield project that I can use for the “”radical”” changes
That’s totally fair. A lot of folks can get overloaded with all the new stuff. But, if you are trying to get to GitOps or other methodology, the tools can always help you get there faster, vs trying to get there then implementing the tools. We’re here to help in any way we can, even if it’s just having a call and helping you answer some high level questions that aren’t env0 product related. We’re all about helping out the community and sharing our knowledge any way we can :)
Hah, fair call on the OSS tools. Honestly, a lot of customers that end up looking at our platform, attempted to use all OSS or build their own, and then realized that it’s easier to let someone else do all the hard work. Re-inventing the wheel can be…less fun, sometimes
Yea, Practically if we’re going this path as a company, I’ll push to use a managed specialized service myself
We’re more than happy to answer any questions you have anytime. It can be a long and complicated path sometimes, but always for the best. You seem to have a great handle on where you want to be.
I’ll definitely reach out and we can have a talk, I might try to introduce env0 to my team, we can definitely use it after we’ve migrated more of our stack to Kubernetes.
@Sherif Cool introduction! I’m wondering how you create and delete these QA envs. Do you have a possibility to trigger the creation/deletion of the environment from CI by clicking buttons or your folks trigger it via CLI? Or they are permanent and created all together?
Right now, each app and its Infra can be deployed to any QA Environment, even if it didn’t exist before. ( will dynamically create everything ). And this is done through CI/CD jobs.
However dependencies between services is completely manually. Teams would go to each service and deploy it themselves until they “complete” their dependencies.
We tried building the dependency tree before and automate this part, however we didn’t find a neat solution for that yet.
Thanx for sharing!
@here office hours is starting in 30 minutes! Remember to post your questions here.
Emile Fugulin has joined Public “Office Hours”
Emile Fugulin has joined Public “Office Hours”
Yusuf has joined Public “Office Hours”
Erik Osterman (Cloud Posse) has joined Public “Office Hours”
Vlad Ionescu has joined Public “Office Hours”
Zachary Loeber has joined Public “Office Hours”
Andy Miguel (Cloud Posse) has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Benjamin Smith has joined Public “Office Hours”
Guilherme Borges has joined Public “Office Hours”
Jim Park has joined Public “Office Hours”
James Haughey has joined Public “Office Hours”
Oskar Maria Grande has joined Public “Office Hours”
David Hawthorne has joined Public “Office Hours”
Justin Davis has joined Public “Office Hours”
Matt Calhoun has joined Public “Office Hours”
JB Beck has joined Public “Office Hours”
Kyle Korth has joined Public “Office Hours”
Yusuf has joined Public “Office Hours”
Daniel Miller (CloudPosse) has joined Public “Office Hours”
Antarr Byrd has joined Public “Office Hours”
Matthew Zeemann has joined Public “Office Hours”
Tim Gourley has joined Public “Office Hours”
Jim Antoniou has joined Public “Office Hours”
Mohammed Almusaddar has joined Public “Office Hours”
dario erregue has joined Public “Office Hours”
Isa Aguilar has joined Public “Office Hours”
Marc Tamsky has joined Public “Office Hours”
Hao Wang has joined Public “Office Hours”
Sam C has joined Public “Office Hours”
Tony Scott has joined Public “Office Hours”
Taylor M has joined Public “Office Hours”
Github actions telemetry => Datadog by Scribd (https://github.com/scribd/github-action-datadog-reporting)
As mentioned on the office hours call, Charity Majors, the founder of Honeycomb.io is prolific and a really good read on observability
1
Patrick Joyce has joined Public “Office Hours”
Sherif Abdel-Naby has joined Public “Office Hours”
Michael Jenkins has joined Public “Office Hours”
Anere Faithful has joined Public “Office Hours”
Vicken Simonian has joined Public “Office Hours”
Blaise pabon has joined Public “Office Hours”
Just (officially) announced: https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/
Eric Berg has joined Public “Office Hours”
@Erik Osterman (Cloud Posse) you might like that
it is new
^ @Erik Osterman (Cloud Posse) I think the video is corrupted
yeah, automation bit us hehe