#helm (2020-09)

I don’t believe we’ve been bit by this so why do you need to use this lookup function?

xyproblem :-)

@Erik Osterman (Cloud Posse) I believe you use Chamber + Helmfile heavily so you may be getting around this issue via Env Vars. But here is the issue I was trying to suss out:

{{- define "oc-lib.ingressAnnotations" -}}
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/tags: {{ printf "Environment=%s,Customer=%s,Namespace=%s,ManagedByTerraform=false,ManagedByK8s=true" .Values.environment .Values.customer "oc" }}
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: {{ (lookup "v1" "ConfigMap" "oc-app" "oc-config").data.certificate_arn }}
{{ toYaml .Values.ingress.annotations }}
{{- end -}}

I’m populating each environment’s certificate ARN into a config map via Terraform. Then my goal is to use it in Helm/K8s to allow ALB-ingress to associate the correct Cert.

We use this pattern in a few spots so that Terraform drives more of the cluster’s configuration and Helm / bootstrap templates are agnostic and we don’t need to hardcode these types of values.

What I ended up doing to get around this limitation was to provide a default value for local –dry-run / helm template usage:

{{- define "oc-lib.ingressAnnotations" -}}
{{- $certArnDefault := (dict "data" (dict "certificate_arn" "DEFAULT_VALUE")) -}}
kubernetes.io/ingress.class: alb
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/tags: {{ printf "Environment=%s,Customer=%s,Stage=%s,Namespace=%s,ManagedByTerraform=false,ManagedByK8s=true" .Values.environment .Values.customer .Values.customer "oc" }}
alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
alb.ingress.kubernetes.io/certificate-arn: {{ (coalesce (lookup "v1" "ConfigMap" "oc-app" "oc-config") $certArnDefault ).data.certificate_arn }}
{{ toYaml .Values.ingress.annotations }}
{{- end -}}

That did the trick.

If I understand CP’s view and usage of Helm, you’d store this type of upstream configuration in SSM PStore, use chamber to pull it out at time of invoking Helmfile, and then include the certificate_arn env var to create that Annotation’s value. Which I would probably prefer if building entirely from scratch since the Chamber pattern can be used elsewhere extensively.

Sadly, my client already decided to use Flux / Helm Operator which does the trick but is not as flexible to roll in Chamber.

(so we’re currently using chamber less and less because helmfile natively supports SSM)

e.g. in helmfile, we can now do:

{{- $pgpassword := urlquery (fetchSecretValue "<ref+awsssm://aurora_postgres/primary_aurora_postgres_master_password>") }}

or for an arn, they are very deterministic, so we do:

    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: {{ printf "arn:aws:iam::%v:role/eg-gbl-%v-eks-myapp" .Values.account_number .Values.stage | quote }}

Aha well that is sweet. Surprised that Helmfile would natively support that, but very cool.

I could go the deterministic route, but AFAIR Cert Arns and similar do have unique IDs in them.. Will have to check that.

ya, @mumoshu integrated #helmfile with vals which is his swissarmy knife for pulling data out of all the most common types of data stores

https://github.com/variantdev/vals/

Hahah now that’s sweet. I like it.

And it seems that is very similar to what I was looking for over here: https://sweetops.slack.com/archives/CB6GHNLG0/p1599088270240500

I remembered the SSM + Chamber pattern after someone brought it up, and I’ve used that in the past, but if I wanted to be more cloud agnostic maybe I would want to go the vals route with the Terraform State backend.

ya, you were on the right track

Though… I like the PStore approach. Supports a more consistent mental model I think. “These configs are for downstream systems, they’re for this environment, and this service”. Instead of just the potentially ugly resource name path that gets created by Terraform. Using that path as a lookup in a CI / CD tool I’m sure will cause some others to scratch their heads.