#helmfile (2018-12)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2018-12-18

Daren avatar
Daren

Anyone know how to remove a lifecycle hook from a resource and not get the helm state out of sync? This chart https://github.com/helm/charts/blob/master/stable/sumologic-fluentd/templates/secrets.yaml#L11 has a pre hook on the secret, which is not needed and causes issues. Removing the hook and performing a sync causes helm to error out thinking the secret does not exist.

helm/charts

Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm… can you create a stubsecret deployed as a standalone release using helmfile?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…. not sure if i full comprehend the problem

2018-12-13

2018-12-12

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

our use-case is we write a lot of helmfiles for our clients, but i’d like to reuse them while version pinning

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Shane avatar
Shane

got ya, one issue I would see with it is helmfile is not really self contained.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmm

Shane avatar
Shane

everything would have to be in the helmfile without any external file references.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes - it would be limited in that regard

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but those external file references in our case are usually the non-portable settings

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

things like URLs, configuration files, keys, etc

Shane avatar
Shane

got ya

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add helmfile and codefresh by osterman · Pull Request #2 · cloudposse/statup

what Add helmfile for deployment with monochart Add codefresh build manifest why Easy deployment to kubernetes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is the most common use-case though for us

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

everytime we deploy an app for some staging environment we copy and paste this over

Shane avatar
Shane

I’ll check it out I got to jump off the wife wants to eat.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, no rush.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically i don’t want to copy all these snippets everywhere which add a lot of technical debt.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(heading out myself! ttyl)

2018-12-10

Daren avatar
Daren

Ive added a chart repository that requires auth + mfa to helm. However every helm operation involving the repo (install, repo update, push) trigger an MFA verification. Is there a way for helm to cache a session?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

helm-s3

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or MFA with harbor?

Daren avatar
Daren

In this harbor

Daren avatar
Daren

Im wondering if its because harbor uses LDAP for auth

Daren avatar
Daren
LDAP with Okta and MFA · Issue #6524 · goharbor/harbor

summary When configuring LDAP against Okta where an MFA policy is enforced, every docker and helm operation performed by users triggers an MFA challenge. We would like to allow users to have a sess…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh, yea, in that case no clue.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@jdolitsky probably could answer it though

jdolitsky avatar
jdolitsky
08:07:03 PM

@jdolitsky has joined the channel

2018-12-06

Daren avatar
Daren

Have you encountered issues with a chart that creates jobs? When running sync a 2nd time k8s errors with field is immutable

Shane avatar
Shane

@Daren This would not be a issue with helmfile or helm. Certain fields are not mutable in kubernetes. What field are you trying to modify?

Daren avatar
Daren

Im changing a value which is used as an ENV by the job

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we’re deploying jobs regularly with our charts - mostly to run db migrations

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ve not run into that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

fwiw, our envs are referring to values in configmaps and secrets

Daren avatar
Daren

hmm maybe thats the issue

Daren avatar
Daren

Im doing it directly from values

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

party_parrot monochart party_parrot monochart party_parrot monochart

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/charts

The “Cloud Posse” Distribution of Kubernetes Applications - cloudposse/charts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Shane avatar
Shane

@Daren If you are creating a job directly via helm you likely should have it as part of a lifecycle hook.

Shane avatar
Shane

Jobs are not mutable and once it’s create it should run and then complete

Daren avatar
Daren

Hm even when using a configmap it still fails on immutable

Daren avatar
Daren

I thought of doing init-container originally, but its tricky for this situation. Im deploying an official chart which requires that a couple scripts be run against the DB first. I wanted to avoid having to build a container just for that

Daren avatar
Daren

So I tried using a chart that runs a couple jobs using the official image.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
08:49:34 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s a job being deployed several times a day that runs migrations

Daren avatar
Daren

Heres mine, its simple:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ template "harbor-init.fullname" . }}-registry
  labels:
    [app.kubernetes.io/name](http://app.kubernetes.io/name): {{ include "harbor-init.name" . }}
    [helm.sh/chart](http://helm.sh/chart): {{ include "harbor-init.chart" . }}
    [app.kubernetes.io/instance](http://app.kubernetes.io/instance): {{ .Release.Name }}
    [app.kubernetes.io/managed-by](http://app.kubernetes.io/managed-by): {{ .Release.Service }}
{{- if .Values.podAnnotations }}
  annotations:
{{ toYaml .Values.podAnnotations | indent 4 }}
{{- end }}
spec:
  template:
    metadata:
      labels:
        app: {{ template "harbor-init.fullname" . }}-registry
        release: "{{ .Release.Name }}"
    spec:
      restartPolicy: OnFailure
      containers:
        - name: registry
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          envFrom:
            - configMapRef:
               name: {{ include "harbor-init.fullname" . }}
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "harbor-init.fullname" . }}
                  key: password
          command: ["psql"]
          args:
            - "-f"
            - "/docker-entrypoint-initdb.d/initial-registry.sql"
Daren avatar
Daren

Im not doing the checksum annotations

Daren avatar
Daren

Does it matter?

Shane avatar
Shane

Nothing in a job’s spec an change. If you want to do this on install I would recommend using a helm lifecycle hook for post-install

Shane avatar
Shane

with your current job if the image or tag changes it will cause helm to fail

Daren avatar
Daren

ohhh a helm hook, nice

Daren avatar
Daren
helm/helm

The Kubernetes Package Manager. Contribute to helm/helm development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
08:54:58 PM
Shane avatar
Shane
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks @Shane - i didn’t know why @Igor Rodionov was doing this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but i guess that’s why

Igor Rodionov avatar
Igor Rodionov
08:55:51 PM

@Igor Rodionov has joined the channel

Shane avatar
Shane

The hooks allow you to delete the job so when it runs again it can run the job again. if you don’t delete the job it will fail if the job ever changes

:--1:1
Shane avatar
Shane

Due to the immutability

Daren avatar
Daren

someone get this man a

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Shane avatar
Shane

I always accept beer as payment

Daren avatar
Daren

If you only want the job to run once, do you need the delete hook?

Daren avatar
Daren

do not run on any subsequent sync

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then i thikn you want

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
"[helm.sh/hook](http://helm.sh/hook)": pre-install
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or basically, something to indicate that it shouldn’t get triggered on upgrades

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i don’t know the hooks off the top of my head

Daren avatar
Daren

Igor Rodionov avatar
Igor Rodionov

@Daren you can find more info here https://docs.helm.sh/developing_charts/#hooks

Helm - The Package Manager for Kubernetes.

Documentation for Helm - The Kubernetes Package Manager.

Igor Rodionov avatar
Igor Rodionov

oh.. you found already

Daren avatar
Daren

Is there a way to bundle some resources into an official chart?

Igor Rodionov avatar
Igor Rodionov

what chart?

Igor Rodionov avatar
Igor Rodionov

monochart ?

Daren avatar
Daren
goharbor/harbor-helm

The helm chart to deploy Harbor. Contribute to goharbor/harbor-helm development by creating an account on GitHub.

Daren avatar
Daren

I mean does helmfile allow you to attach another resource to a chart you are installing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

helmfile is basically a Makefile for helm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so if you can do it with helm, you can do it with helmfile

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

now, i know i’m going to sound like a broken record - but…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re doing something similar. let me explain.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we install granfa from official chart repos

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but then we need some additional configmaps

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in our helmfile we have one release for grafana and another release for the configmaps which get installed using our monochart

Igor Rodionov avatar
Igor Rodionov

you can not inject jobs into existing chart. We have workaround of that with helmfile and monorchat

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we have an example - will show you

Daren avatar
Daren

Sounds like my use case. Im installing the Harbor chart. But I need to run a script against the DB it will use beforehand

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Igor Rodionov avatar
Igor Rodionov

This is example how we “extend” official nginx chart with additional resources https://github.com/cloudposse/helmfiles/blob/master/helmfile.d/0320.nginx-ingress.yaml#L156

cloudposse/helmfiles

Comprehensive Distribution of Helmfiles. Works with helmfile.d - cloudposse/helmfiles

Daren avatar
Daren

With both your examples, you are ordering the helm releases and using wait: true?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, concurrency 1

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wait true

Igor Rodionov avatar
Igor Rodionov

and release order playes role

Daren avatar
Daren

cool, we have the same approach

Igor Rodionov avatar
Igor Rodionov

also I sugget to deploy it with 2 bash commands

Igor Rodionov avatar
Igor Rodionov

like

Igor Rodionov avatar
Igor Rodionov

helmfile –selector component=job helmfile –selector component=harbor

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

well, or have a helmfile.yaml which includes the helmfiles in the right order

Daren avatar
Daren

My ordering is less sensitive

Daren avatar
Daren

If the pre-job runs at the same time or after its fine. Harbor will eventually recover once it does run

Igor Rodionov avatar
Igor Rodionov

then helmfile fits perfect

Daren avatar
Daren

Agreed

Daren avatar
Daren

@Erik Osterman (Cloud Posse) check out harbor sometime, its working out very well

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s great to hear

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i want to add more security/compliance features to our solution so it sounds like something good to have there.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

did you go enterprise?

Daren avatar
Daren

Its open source…

Daren avatar
Daren

Docker registry & Chart museum: https://goharbor.io/

Daren avatar
Daren

backed by CNCF

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, my understanding was though to get access to a larger library of vulns, you needed to go ent

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

though I met those guys ~2 years ago at a meetup and a lot might have changed

Daren avatar
Daren

I was not aware of an enterprise version

Daren avatar
Daren

It uses clair (https://github.com/coreos/clair) for vuln.

coreos/clair

Vulnerability Static Analysis for Containers. Contribute to coreos/clair development by creating an account on GitHub.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hah, my fault

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Anchore - The Open Platform for Container Security and Compliance - Anchore

The Open Platform for Container Security and Compliance

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

all these nautical product names mixed me up

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is an open core + enterprise

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

don’t know how it compares to harbor vs clair

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

isn’t harbor providing redundant capabilities to twistlock which you’re already using?

Daren avatar
Daren

a bit of overlap

Daren avatar
Daren

Twistlock does provide scanning of image like Clair. However Twistlock also provides real time reports of your infrastructure along with runtime analysis

Daren avatar
Daren

Twistlock has very high value in that regard. Clair is limited to telling me during build time what my exposure is

Daren avatar
Daren

Or by clicking through every image+tag in the registry UI

joshmyers avatar
joshmyers

https://github.com/future-architect/vuls is pretty nice for a lightweight vuln scanner

future-architect/vuls

Vulnerability scanner for Linux/FreeBSD, agentless, written in Go - future-architect/vuls

    keyboard_arrow_up