#helmfile (2018-12)

Have you encountered issues with a chart that creates jobs? When running sync a 2nd time k8s errors with field is immutable

@Daren This would not be a issue with helmfile or helm. Certain fields are not mutable in kubernetes. What field are you trying to modify?

Im changing a value which is used as an ENV by the job

so we’re deploying jobs regularly with our charts - mostly to run db migrations

we’ve not run into that

fwiw, our envs are referring to values in configmaps and secrets

hmm maybe thats the issue

Im doing it directly from values

monochart monochart monochart

@Daren If you are creating a job directly via helm you likely should have it as part of a lifecycle hook.

Jobs are not mutable and once it’s create it should run and then complete

Hm even when using a configmap it still fails on immutable

I thought of doing init-container originally, but its tricky for this situation. Im deploying an official chart which requires that a couple scripts be run against the DB first. I wanted to avoid having to build a container just for that

So I tried using a chart that runs a couple jobs using the official image.

Here’s a job being deployed several times a day that runs migrations

Heres mine, its simple:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ template "harbor-init.fullname" . }}-registry
  labels:
    app.kubernetes.io/name: {{ include "harbor-init.name" . }}
    helm.sh/chart: {{ include "harbor-init.chart" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- if .Values.podAnnotations }}
  annotations:
{{ toYaml .Values.podAnnotations | indent 4 }}
{{- end }}
spec:
  template:
    metadata:
      labels:
        app: {{ template "harbor-init.fullname" . }}-registry
        release: "{{ .Release.Name }}"
    spec:
      restartPolicy: OnFailure
      containers:
        - name: registry
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          envFrom:
            - configMapRef:
               name: {{ include "harbor-init.fullname" . }}
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "harbor-init.fullname" . }}
                  key: password
          command: ["psql"]
          args:
            - "-f"
            - "/docker-entrypoint-initdb.d/initial-registry.sql"

Im not doing the checksum annotations

Does it matter?

Nothing in a job’s spec an change. If you want to do this on install I would recommend using a helm lifecycle hook for post-install

with your current job if the image or tag changes it will cause helm to fail

ohhh a helm hook, nice

nginx uses one: https://github.com/helm/helm/blob/master/docs/examples/nginx/templates/post-install-job.yaml#L17-L19

thanks @Shane - i didn’t know why @Igor Rodionov was doing this

but i guess that’s why

@Igor Rodionov has joined the channel

The hooks allow you to delete the job so when it runs again it can run the job again. if you don’t delete the job it will fail if the job ever changes

Due to the immutability

someone get this man a

I always accept beer as payment

If you only want the job to run once, do you need the delete hook?

do not run on any subsequent sync

then i thikn you want

or basically, something to indicate that it shouldn’t get triggered on upgrades

i don’t know the hooks off the top of my head

https://docs.helm.sh/developing_charts/#the-available-hooks

@Daren you can find more info here https://docs.helm.sh/developing_charts/#hooks

oh.. you found already

Is there a way to bundle some resources into an official chart?

what chart?

monochart ?

In this case its https://github.com/goharbor/harbor-helm

I mean does helmfile allow you to attach another resource to a chart you are installing

helmfile is basically a Makefile for helm

so if you can do it with helm, you can do it with helmfile

now, i know i’m going to sound like a broken record - but…

we’re doing something similar. let me explain.

so we install granfa from official chart repos

but then we need some additional configmaps

in our helmfile we have one release for grafana and another release for the configmaps which get installed using our monochart

you can not inject jobs into existing chart. We have workaround of that with helmfile and monorchat

we have an example - will show you

Sounds like my use case. Im installing the Harbor chart. But I need to run a script against the DB it will use beforehand

https://github.com/cloudposse/helmfiles/blob/master/helmfile.d/0400.kube-prometheus.yaml#L262-L310

This is example how we “extend” official nginx chart with additional resources https://github.com/cloudposse/helmfiles/blob/master/helmfile.d/0320.nginx-ingress.yaml#L156

With both your examples, you are ordering the helm releases and using wait: true?

yea, concurrency 1

wait true

and release order playes role

cool, we have the same approach

also I sugget to deploy it with 2 bash commands

like

helmfile –selector component=job helmfile –selector component=harbor

well, or have a helmfile.yaml which includes the helmfiles in the right order

My ordering is less sensitive

If the pre-job runs at the same time or after its fine. Harbor will eventually recover once it does run

then helmfile fits perfect

Agreed

@Erik Osterman (Cloud Posse) check out harbor sometime, its working out very well

that’s great to hear

i want to add more security/compliance features to our solution so it sounds like something good to have there.

did you go enterprise?

Its open source…

Docker registry & Chart museum: https://goharbor.io/

backed by CNCF

yea, my understanding was though to get access to a larger library of vulns, you needed to go ent

though I met those guys ~2 years ago at a meetup and a lot might have changed

I was not aware of an enterprise version

It uses clair (https://github.com/coreos/clair) for vuln.

hah, my fault

https://anchore.com/

all these nautical product names mixed me up

this is an open core + enterprise

don’t know how it compares to harbor vs clair

isn’t harbor providing redundant capabilities to twistlock which you’re already using?

https://github.com/future-architect/vuls is pretty nice for a lightweight vuln scanner

Ive added a chart repository that requires auth + mfa to helm. However every helm operation involving the repo (install, repo update, push) trigger an MFA verification. Is there a way for helm to cache a session?

helm-s3

?

or MFA with harbor?

In this harbor

Im wondering if its because harbor uses LDAP for auth

oh, yea, in that case no clue.

@jdolitsky probably could answer it though

@jdolitsky has joined the channel

@Shane what do you think about this? https://github.com/roboll/helmfile/issues/347

our use-case is we write a lot of helmfiles for our clients, but i’d like to reuse them while version pinning

https://github.com/cloudposse/helmfiles/tree/master/helmfile.d

got ya, one issue I would see with it is helmfile is not really self contained.

hrmm

everything would have to be in the helmfile without any external file references.

yes - it would be limited in that regard

but those external file references in our case are usually the non-portable settings

things like URLs, configuration files, keys, etc

got ya

https://github.com/cloudposse/statup/pull/2/files#diff-c9e5b83e3877738c2e3854641c0144c5

this is the most common use-case though for us

everytime we deploy an app for some staging environment we copy and paste this over

I’ll check it out I got to jump off the wife wants to eat.

haha, no rush.

basically i don’t want to copy all these snippets everywhere which add a lot of technical debt.

(heading out myself! ttyl)

Anyone know how to remove a lifecycle hook from a resource and not get the helm state out of sync? This chart https://github.com/helm/charts/blob/master/stable/sumologic-fluentd/templates/secrets.yaml#L11 has a pre hook on the secret, which is not needed and causes issues. Removing the hook and performing a sync causes helm to error out thinking the secret does not exist.

hrm… can you create a stubsecret deployed as a standalone release using helmfile?

…. not sure if i full comprehend the problem