#helm (2020-09)

Archive: https://archive.sweetops.com/helm/

2020-09-18

Joaquin Menchaca avatar
Joaquin Menchaca

helm has broken idempotence  in versions 3.3.2 and 3.3.3. Thus you cannot use helmfile with those versions.

2020-09-07

2020-09-05

2020-09-03

Gowiem avatar
Gowiem
Any clever ways that folks get around the fact that [Helm’s >lookup<https://helm.sh/docs/chart_template_guide/functions_and_pipelines/#using-the-lookup-function function](https://helm.sh/docs/chart_template_guide/functions_and_pipelines/#using-the-lookup-function) won’t access the cluster during an install / upgrade –dry-run?

Keep in mind that Helm is not supposed to contact the Kubernetes API Server during a helm template or a helm install|update|delete|rollback --dry-run, so the lookup function will return nil in such a case. This makes it so trying to dry-run my templates locally is a no go and even breaks helm lint which sucks. Anything I can do there or do I just have to live with that limitation?

Template Functions and Pipelines

Using functions in templates.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I don’t believe we’ve been bit by this so why do you need to use this lookup function?

Template Functions and Pipelines

Using functions in templates.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

xyproblem :-)

Gowiem avatar
Gowiem

@Erik Osterman (Cloud Posse) I believe you use Chamber + Helmfile heavily so you may be getting around this issue via Env Vars. But here is the issue I was trying to suss out:

{{- define "oc-lib.ingressAnnotations" -}}
[kubernetes.io/ingress.class](http://kubernetes\.io/ingress\.class): alb
[alb.ingress.kubernetes.io/scheme](http://alb\.ingress\.kubernetes\.io/scheme): internet-facing
[alb.ingress.kubernetes.io/tags](http://alb\.ingress\.kubernetes\.io/tags): {{ printf "Environment=%s,Customer=%s,Namespace=%s,ManagedByTerraform=false,ManagedByK8s=true" .Values.environment .Values.customer "oc" }}
[alb.ingress.kubernetes.io/listen-ports](http://alb\.ingress\.kubernetes\.io/listen\-ports): '[{"HTTPS":443}]'
[alb.ingress.kubernetes.io/certificate-arn](http://alb\.ingress\.kubernetes\.io/certificate\-arn): {{ (lookup "v1" "ConfigMap" "oc-app" "oc-config").data.certificate_arn }}
{{ toYaml .Values.ingress.annotations }}
{{- end -}}

I’m populating each environment’s certificate ARN into a config map via Terraform. Then my goal is to use it in Helm/K8s to allow ALB-ingress to associate the correct Cert.

We use this pattern in a few spots so that Terraform drives more of the cluster’s configuration and Helm / bootstrap templates are agnostic and we don’t need to hardcode these types of values.

Gowiem avatar
Gowiem

What I ended up doing to get around this limitation was to provide a default value for local –dry-run / helm template usage:

{{- define "oc-lib.ingressAnnotations" -}}
{{- $certArnDefault := (dict "data" (dict "certificate_arn" "DEFAULT_VALUE")) -}}
[kubernetes.io/ingress.class](http://kubernetes\.io/ingress\.class): alb
[alb.ingress.kubernetes.io/scheme](http://alb\.ingress\.kubernetes\.io/scheme): internet-facing
[alb.ingress.kubernetes.io/tags](http://alb\.ingress\.kubernetes\.io/tags): {{ printf "Environment=%s,Customer=%s,Stage=%s,Namespace=%s,ManagedByTerraform=false,ManagedByK8s=true" .Values.environment .Values.customer .Values.customer "oc" }}
[alb.ingress.kubernetes.io/listen-ports](http://alb\.ingress\.kubernetes\.io/listen\-ports): '[{"HTTPS":443}]'
[alb.ingress.kubernetes.io/certificate-arn](http://alb\.ingress\.kubernetes\.io/certificate\-arn): {{ (coalesce (lookup "v1" "ConfigMap" "oc-app" "oc-config") $certArnDefault ).data.certificate_arn }}
{{ toYaml .Values.ingress.annotations }}
{{- end -}}

That did the trick.

Gowiem avatar
Gowiem

If I understand CP’s view and usage of Helm, you’d store this type of upstream configuration in SSM PStore, use chamber to pull it out at time of invoking Helmfile, and then include the certificate_arn env var to create that Annotation’s value. Which I would probably prefer if building entirely from scratch since the Chamber pattern can be used elsewhere extensively.

Sadly, my client already decided to use Flux / Helm Operator which does the trick but is not as flexible to roll in Chamber.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(so we’re currently using chamber less and less because helmfile natively supports SSM)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. in helmfile, we can now do:

{{- $pgpassword := urlquery (fetchSecretValue "<ref+awsssm://aurora_postgres/primary_aurora_postgres_master_password>") }}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or for an arn, they are very deterministic, so we do:

    serviceAccount:
      annotations:
        [eks.amazonaws.com/role-arn](http://eks\.amazonaws\.com/role\-arn): {{ printf "arn:aws:iam::%v:role/eg-gbl-%v-eks-myapp" .Values.account_number .Values.stage | quote }}
Gowiem avatar
Gowiem

Aha well that is sweet. Surprised that Helmfile would natively support that, but very cool.

Gowiem avatar
Gowiem

I could go the deterministic route, but AFAIR Cert Arns and similar do have unique IDs in them.. Will have to check that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya, @mumoshu integrated #helmfile with vals which is his swissarmy knife for pulling data out of all the most common types of data stores

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
variantdev/vals

Helm-like configuration values loader with support for various sources - variantdev/vals

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
02:28:25 AM
Gowiem avatar
Gowiem

Hahah now that’s sweet. I like it.

Gowiem avatar
Gowiem

And it seems that is very similar to what I was looking for over here: https://sweetops.slack.com/archives/CB6GHNLG0/p1599088270240500

I remembered the SSM + Chamber pattern after someone brought it up, and I’ve used that in the past, but if I wanted to be more cloud agnostic maybe I would want to go the vals route with the Terraform State backend.

Does anyone know a good tool for pulling values from Terraform state outside of terraform itself?

As in, I have a CD process that is running simple bash commands to build and deploy a static site. I’d like to get my CloudFront CDN Distribution ID and the bucket that the static site’s assets should be shipped to from my Terraform state file in S3. I could pull the state file, parse out the outputs I need, and then go about it that way but I am figuring that there must be a tool written around this.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya, you were on the right track

Gowiem avatar
Gowiem

Though… I like the PStore approach. Supports a more consistent mental model I think. “These configs are for downstream systems, they’re for this environment, and this service”. Instead of just the potentially ugly resource name path that gets created by Terraform. Using that path as a lookup in a CI / CD tool I’m sure will cause some others to scratch their heads.

    keyboard_arrow_up