#office-hours (2024-04)

“Office Hours” are every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers! https://cloudposse.com/office-hours

Public “Office Hours” are held every Wednesday at 11:30 PST via Zoom. It’s open to everyone. Ask questions related to DevOps & Cloud and get answers!

https://cpco.io/slack-office-hours

Meeting password: sweetops

2024-04-03

rohit avatar

office hours:

we have a goal to deliver a working helm project to a customer, so customer BYOI (brings your own infra), and we ask them to helm install our project. this normally works fine for simpler projects, but we’re facing difficulties in ordering some of the k8s resources we want to create before we deploy our services AND ensuring the dependency helm charts (alb + fluent-bit) are setup PRIOR to our helm chart.

our flow right now looks like this:

  1. install external helm chart (aws-load-balancer-controller)
  2. install external helm chart (fluent-bit)
  3. we install service-account.yaml before secret.yaml
  4. run this vault script to ensure service account has access to vault to fetch secrets
  5. helm install (2 jobs, service, deployment, and ingress.yaml)
  6. we wait for restarts until “things eventually work out” (app restarts multiple times because jobs are not done) this doesn’t seem feasible managing all of this in helm. am i wrong? if not, what are my options here?

I looked into helmfile and splitting up our helm chart into 2 helm charts, where our main app will have dependencies in Chart.yaml to ensure deps are met, but AFAIK we can’t “order” the dependency installs. or even writing a shell script that does this step by step.

thank you!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

cc @Yonatan Koren

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s important to know what the assumptions are as well • What flavor of Kubernetes (e.g. EKS, GKE, AKS, etc) • Are they running a current version of Kubernetes • Is deploying an ALB even viable? E.g. they might already have ALB controller deployed • Is using only helm a requirement? Maybe helmfile will simplify things

Yonatan Koren avatar
Yonatan Koren

I used to work at Codefresh and we had to solve similar issues for delivering on on-prem installation to customers.

You can take a look at the documentation for the on-prem codefresh helm chart: https://artifacthub.io/packages/helm/codefresh-onprem/codefresh

My devil’s advocate response to wanting to package all of your low-level app dependencies for customers is, what happens when you have two customers with different underlying cluster requirements?

For example, when I worked at Codefresh we had to support both aws-load-balancer-controller and nginx-ingress-controller. So we saved these as open ended for the customer to choose, and these served as pre-requisites for our app. Yes it’s less automated and we needed support engineers to help new customers, but it was a complicated architecture and the expectation was that setup needs some manual preparation anyways. So we focused more on ensuring compatibility with different ingress controllers, different public cloud providers, even added support for openshift customers. This was more important to us than to automate everything from end to end only for one type of customer.

codefresh 2.3.0 · codefresh-io/codefresh-onpremattachment image

Helm Chart for Codefresh On-Prem

Yonatan Koren avatar
Yonatan Koren

So in my opinion, take #1 and #2 out of the equation. This needs to be handled by the customer. This way you can onboard more types of customers who have different k8s environments.

If you can automate #3 and #4 using a Kubernetes resource instead of a script, you definitely should do that. Have you looked into https://external-secrets.io/latest/provider/hashicorp-vault/ ? If External Secrets Operator can sync that vault secret you need for the service account, you don’t need to manually intervene with a script. It will eventually sync and your application will come up. You should also give your customers different options to get this secret into their cluster. Maybe they have it in AWS Secrets Manager. You can instruct them how to set up ESO to sync that secret as well. This ESO resource probably won’t be inside your helm templates, but you will give an option to change the name of the secret inside the SA template. Then they deploy it either in templates/ with your chart as a sub-chart, or just deploy it manually beforehand.

Yonatan Koren avatar
Yonatan Koren

We can discuss this on the office hours call today if you’d like

elvis lim avatar
elvis lim
aws-ia/terraform-aws-eks-blueprints-addon

Terraform AWS module which creates an EKS addon (helm release + IRSA)

rohit avatar

Sorry and thank you for the replies. Our first customer is running kubernetes on VMware TKGI (kind cluster behinds the scenes). So I cannot take advantage of any cloud managed k8s

rohit avatar

@Yonatan Koren i am reading your replies. thank you!

np1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Our first customer is running kubernetes on VMware TKGI
Hah! Exactly the challenge

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So many ways to run k8s. Other considerations that we ran into were exposing web interfaces over TLS without self-signed certs, which required Let’s Encrypt and public DNS names.

Yonatan Koren avatar
Yonatan Koren

As another example, take a look at the Teleport project and how they support their helm installation for different scenarios.

https://goteleport.com/docs/deploy-a-cluster/helm-deployments/

They have a single helm chart but support EKS, GKE, AKS, and others. There are semi-manual steps involved, such as setting up IAM, setting up TLS, setting up DynamoDB, etc. Their helm chart only installs their app, but they walk you through how to install it based on your scenario.

IMO it’s much better to keep new potential customers happy by giving and documenting different options for them to use. One of the worse things that can happen is if a customer wants to install your product on their cluster and you say that they have to use cert-manager instead of AWS Certificate Manager. Or that they have to use aws-load-balancer-controller instead of nginx-ingress-controller. In my experience this leaves them with a bad taste in their mouth, because it makes them feel like they need to do a lot of changes to install your app, and that your app isn’t engineered in a way that had compatibility in mind.

Guides for running Teleport using Helm | Teleport Docsattachment image

How to install and configure Teleport in Kubernetes using Helm

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:26 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Adnan avatar

Am I doing it wrong or did office hours not start yet?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Make sure. you click the link from the invite

Adnan avatar

I clicked the one in my gmail/calendar. Still says waiting for host to start the meeting. I guess no OH for me today.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Others mentioned that you might have the old invite link

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In september 2023, we changed meeting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha, Public "Office Hours" is definitely the old invite

2024-04-04

managedkaos avatar
managedkaos
OpenTofu may be showing us the wrong way to forkattachment image

Disagree with a license? Fork the project, but don’t lift the code and say it was always publicly available. Compare HashiCorp code and license to OpenTofu’s version.

jose.amengual avatar
jose.amengual

That post should be deleted, is actually not accurate

OpenTofu may be showing us the wrong way to forkattachment image

Disagree with a license? Fork the project, but don’t lift the code and say it was always publicly available. Compare HashiCorp code and license to OpenTofu’s version.

jose.amengual avatar
jose.amengual
John Rowley (@johnrowleyster1) on X

@adamhjk @ibuildthecloud @InfoWorld @mjasay The author probably isn’t aware that the hashicorp MPL license is being applied in bulk in opentofu. The “deceitfully misappropriated” code is licensed this way not out of guilt, but out of an abundance of caution.

https://t.co/vtsqQFODEx

https://t.co/hc52draILF

managedkaos avatar
managedkaos


That post should be deleted, is actually not accurate
the InfoWorld post or this Slack post?

1
jose.amengual avatar
jose.amengual

not yours, the infoworld post

1
jose.amengual avatar
jose.amengual

or it should be corrected

jose.amengual avatar
jose.amengual

we should discuss it in office hours

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Will do!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
OpenTofu on LinkedIn: OpenTofu Project was recently made aware of a letter by HashiCorp’s… | 45 commentsattachment image
OpenTofu Project was recently made aware of a letter by HashiCorp’s lawyers, alleging that OpenTofu was not respecting the terms of its BSL license governing…45 comments on LinkedIn
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hey everybody, OpenTofu core team member here,

On April 3rd, OpenTofu received a Cease and Desist letter from Hashicorp claiming copyright infringement on the part of one of our core developers.

The OpenTofu team vehemently disagrees with any suggestion that it misappropriated, mis-sourced, or otherwise misused HashiCorp’s BSL code. All such statements have zero basis in facts.

You can find our response, along with the cease & desist letter, our response letter, as well as the source code origin document resulting from our investigation in this blog post: https://opentofu.org/blog/our-response-to-hashicorps-cease-and-desist/

Despite these events, we have managed to carry out significant development on OpenTofu 1.7, and we will be releasing a new pre-release version next week, including provider-defined functions!

Michael avatar
Michael

This quote is from OpenTofu: “In the future, if you should have any concerns or questions about how source code in OpenTofu is developed, we would ask that you contact us first. Immediately issuing DMCA takedown notices and igniting salacious negative press articles is not the most helpful path to resolving concerns like this.”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s such a reasonable request by OpenTofu and how reasonable parties should operate on both sides.

2024-04-06

SlackBot avatar
SlackBot
04:48:19 PM
SlackBot avatar
SlackBot
04:48:19 PM

2024-04-10

managedkaos avatar
managedkaos

Randomly, another whiteboarding/diagramming app. this one’s open source and embeddable…. https://excalidraw.com/ https://github.com/excalidraw/excalidraw

Excalidraw — Collaborative whiteboarding made easyattachment image

Excalidraw is a virtual collaborative whiteboard tool that lets you easily sketch diagrams that have a hand-drawn feel to them.

excalidraw/excalidraw

Virtual whiteboard for sketching hand-drawn like diagrams

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:30 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Allan Swanepoel avatar
Allan Swanepoel
We discovered an AWS access vulnerability | Stedi - Modern EDIattachment image

Last year, we extended our use of IAM to build and enforce role-based access control (RBAC) for our customers using AWS Security Token Service (STS), an IAM service you can use to provide temporary access to AWS resources. Along the way, we discovered a vulnerability in STS that caused role trust policy statements to be evaluated incorrectly. Learn how we discovered this vulnerability, our disclosure process with AWS, and what we learned from the experience.

1
Allan Swanepoel avatar
Allan Swanepoel

AWS Ref architecture for wordpress - https://github.com/aws-samples/aws-refarch-wordpress

aws-samples/aws-refarch-wordpress

This reference architecture provides best practices and a set of YAML CloudFormation templates for deploying WordPress on AWS.

Allan Swanepoel avatar
Allan Swanepoel
aws-samples/aws-serverless-airline-booking

Airline Booking is a sample web application that provides Flight Search, Flight Payment, Flight Booking and Loyalty points including end-to-end testing, GraphQL and CI/CD. This web application was the theme of Build on Serverless Season 2 on AWS Twitch running from April 24th until end of August in 2019.

managedkaos avatar
managedkaos

Yes this is exactly along the lines of what I’m looking for. Thanks for sharing!!

aws-samples/aws-serverless-airline-booking

Airline Booking is a sample web application that provides Flight Search, Flight Payment, Flight Booking and Loyalty points including end-to-end testing, GraphQL and CI/CD. This web application was the theme of Build on Serverless Season 2 on AWS Twitch running from April 24th until end of August in 2019.

2024-04-11

2024-04-12

2024-04-16

SlackBot avatar
SlackBot
04:28:17 AM
SlackBot avatar
SlackBot
04:28:17 AM

2024-04-17

Ranjith avatar
Ranjith

Question for #office-hours

We are trying to automate the service deployment process in AWS using terraform/terragrunt. Currently the Services are deployed in ECS. We are starting to use Atlantis for Pull Request automation.

Once a new Service version is deployed in the integration environment and tested, we want the same version to be automatically deployed in different regions of the production environment.

We want the automated tests for each Service to be run against the new deployed version to validate the deployment.

There should be a bake time in each region/environment. Only after validating the health of the deployment in that region/environment, it should be propagated to the next region/environment.

The Service should be rolled back to the earlier version in case of any test failures or increase in error rates during the bake period.

We are trying to understand how different teams have implemented something similar for AWS.

Is AWS Step Functions a good mechanism to orchestrate these steps for automating service deployment? Are there other options that folks have successfully implemented?

Please let us know…

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You might also enjoy this other recent thread. It’s not entirely related, but talks about some of the challenges we’ve encountered managed ECS tasks purely with Terraform. https://sweetops.slack.com/archives/CCT1E7JJY/p1712686050689659

Does anyone here know of any open-source (or paid even) connector Lambda project to take items off of an SQS queue and put them into a bucket on S3?

Ranjith avatar
Ranjith

Thanks for taking up my question yesterday in the office hours….Unfortunately i could not join. But I went through the video recording and got the pointers. Will investigate further on those ….

Thanks again…

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:27 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

managedkaos avatar
managedkaos
LeakyCLI: New Vulnerability Exposes Credentials In AWS, Azure & Google Cloudattachment image

Multiple Cloud Service providers like Google Cloud, AWS and Azure have been discovered with a new vulnerability that has been termed as

1
1

2024-04-19

SlackBot avatar
SlackBot
12:23:26 AM
SlackBot avatar
SlackBot
12:23:26 AM

2024-04-24

omerfsen avatar
omerfsen

Hi, for helm templating which tool are you using ? still helmfile? (i used helmsman before but i didn’t like it) .. I used ArgoCD before but this case i need to use something like helm.. cli

1
venkata.mutyala avatar
venkata.mutyala

This probably isn’t going to be very helpful, but on my team we use vanilla helm to install argocd and then do the rest within argocd. Were there any issues you had with argocd? If so, please share and i’ll try to help.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Will try to discuss today

omerfsen avatar
omerfsen

Hey I missed this stuff. Have you discussed it ? If so I will rewatch session

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hah, I can’t recall actually

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can you make it today?

1
omerfsen avatar
omerfsen

Would be nice

omerfsen avatar
omerfsen

I checked last 4 videos no they are not mentioned

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Discussed today

omerfsen avatar
omerfsen

Sorry could not able to attend but will Watch recording

omerfsen avatar
omerfsen

Sorry guys i feel embarrassed not joining to the questions i asked but thank you i watched https://youtu.be/7LQJtWxDT1w?si=ScXEIgwJWUhyqN2g

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Someone yesterday at the office-hours asked about other approaches to managing deployments than using helm : have a look at nelm by werf. It is on github.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:00:22 PM

@here office hours is starting in 30 minutes! Remember to post your questions here.

Eric D. Berg avatar
Eric D. Berg

Not sure if this is just me, but the calendar invitation doesn’t have the zoom link anymore.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmmmmm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That would be bad!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Matt Gowie @venkata.mutyala notice anything?

Matt Gowie avatar
Matt Gowie

I’ve still got a link…

1
Matt Gowie avatar
Matt Gowie

On upcoming events as well.

1
Eric D. Berg avatar
Eric D. Berg

Hmm…i’ll resubscribe and see if that fixes it. could be from switching email to stavvy.com after recently being acquired.

Eric D. Berg avatar
Eric D. Berg

Thanks, guys.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let me know if you don’t get the invite sorted. I will manually jigger it.

1

2024-04-25

2024-04-27

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

OpenTofu reaches 20K stars, more than Pulumi.

1
2
2
Adnan avatar

Streisand effect

this1

2024-04-28

2024-04-29

venkata.mutyala avatar
venkata.mutyala
S3 Bucket Encryption Doesn't Work The Way You Think It Worksattachment image

Let’s try all the different S3 encryption options, see why it’s more like access control than encryption, and why that matters.

1

2024-04-30

managedkaos avatar
managedkaos
The Arc browser arrives on Windows to take on Chrome and Edgeattachment image

Arc for Windows is going to get better on a weekly basis.

2
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Coincidence?

The Arc browser arrives on Windows to take on Chrome and Edgeattachment image

Arc for Windows is going to get better on a weekly basis.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Is Google anti-competitive against Arc?

managedkaos avatar
managedkaos

ha! i wonder!

    keyboard_arrow_up