#helmfile (2020-2)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2020-02-16

Roderik van der Veer avatar
Roderik van der Veer

Is there a way to have the u/p for a repository to come from a file? Trying to do the following where the ‘docker’ key is part of an environments values file. I tried making a base repositories file and make it a gotmpl but no joy there.

  - name: settlemint
    url: <https://harbor.settlemint.com/chartrepo/launchpad>
    username: {{ .Values.docker.username }}
    password: {{ .Values.docker.password }}
Erik Osterman avatar
Erik Osterman

It’s probably related to the double rendering bug

Graeme Gillies avatar
Graeme Gillies

is helmfiles: the only section of a helmfile that can take git repo refs/paths. E.g. can I reference another file in bases: or values: ?

2020-02-14

Alex Siegman avatar
Alex Siegman

Wish there was a cloudposse group I could @ mention, but I have a very small PR to make a helmfile a bit more friendly: https://github.com/cloudposse/helmfiles/pull/224

[fluentd-elasticsearch-aws] configurable resource requesets/limits by asiegman · Pull Request #224 · cloudposse/helmfiles

what [fluentd-elasticsearch-aws] Allow configurable resource requests/limits from environment why Lots of CPU throttling happening, would like to provide a higher limit.

creature avatar
creature

they are very active on here…I bet you won’t wait too long

Alex Siegman avatar
Alex Siegman

Usually don’t

Erik Osterman avatar
Erik Osterman

@maxim is working full time on our PR backlog

maxim avatar
maxim

@Alex Siegman will put it on top of my queue

:--1:1
Alex Siegman avatar
Alex Siegman

It’s a 5 minute PR tops unless I missed some naming convention, I matched a similar one I found in Sentry helmfile

maxim avatar
maxim

@Alex Siegman here it is: <https://github.com/cloudposse/helmfiles/releases/tag/0.92.0>

:--1:1

2020-02-12

2020-02-11

Bart M. avatar
Bart M.

hmm doesn’t seem to be possible to define in a helmfile which helm version should be used?

Bart M. avatar
Bart M.

it only seems to be possible on the cli

Bart M. avatar
Bart M.

this is a bit messy since I currently have multiple environments using helm2 and helm3… now I have to specify the --helm-binary / -b every time

Nelson Jeppesen avatar
Nelson Jeppesen

From within helmfile.yaml, is there anyway I can detect if helmfile is being run with helm2 or helm 3?

Nelson Jeppesen avatar
Nelson Jeppesen

We have a shared repo that we’ll be migrating namespace by namespace and I’d like to add logic that says “this helmfile.yaml is only deloyed with helm 2 or 3”

Erik Osterman avatar
Erik Osterman

I believe the right way is to set helm_binary inside each helmfile, then install both versions of helm.

Erik Osterman avatar
Erik Osterman

looks like this parameter isn’t supported

Erik Osterman avatar
Erik Osterman
Set helm binary globally in helmfile.yaml · Issue #1083 · roboll/helmfile

In my team we use both helm 2 and 3 on different clusters and we ship a docker image containing all the binaries such as helm (named helm2 and helm3), helm plugins, helmfile, etc. What happens is t…

Nelson Jeppesen avatar
Nelson Jeppesen

Thanks for the help, maybe I can find out another way

Erik Osterman avatar
Erik Osterman

If you’re using alpine linux, we distribute helm2 and helm3 packages

Erik Osterman avatar
Erik Osterman
cloudposse/packages

Cloud Posse installer and distribution of native apps, binaries and alpine packages - cloudposse/packages

2020-02-10

DanB avatar

Hmm, so is there a consensus on the differences between apply and sync?

roth.andy avatar
roth.andy

apply is diff then sync if diff shows a change. Sync skips the diff step.

DanB avatar

right, but if there aren’t any deltas with the resources in k8s, nothing happens either in the case of sync. It’s not clear what apply is trying to achieve.

roth.andy avatar
roth.andy

sync still runs helm upgrade --install even if there is nothing to change. apply won’t do that if nothing has changed

roth.andy avatar
roth.andy

apply is the intended command to use

DanB avatar

is there some penalty for running helm upgrade --install if nothing has changed?

roth.andy avatar
roth.andy

probably not, other than whatever computer/memory the command uses

roth.andy avatar
roth.andy

then again, depending on your setup you might have your pods restarted for instance

2020-02-08

Roderik van der Veer avatar
Roderik van der Veer

I’m having some issues with helmfile + helm3 + dynamic clusterIP’s. It is this issue: https://github.com/helm/helm/issues/7082#issuecomment-575514155 but when i put force to false in my helmfile i’m still hit with Error: UPGRADE FAILED: an error occurred while cleaning up resources. original upgrade error: failed to replace object: Service "violet-reindeer-mint-webserver" is invalid: spec.clusterIP: Invalid value: "": field is immutable: unable to cleanup resources: object not found, skipping delete

proposal: `helm upgrade --recreate` · Issue #7082 · helm/helm

Steps to reproduce the issue: helm create tmp echo &#39; apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: {{ include &quot;tmp.fullname&quot; $ }} roleRef: apiGroup: rbac….

Andrey Nazarov avatar
Andrey Nazarov

For us, setting force: to false helped. Before that we had to delete the resource manually and then to apply again.

proposal: `helm upgrade --recreate` · Issue #7082 · helm/helm

Steps to reproduce the issue: helm create tmp echo &#39; apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: {{ include &quot;tmp.fullname&quot; $ }} roleRef: apiGroup: rbac….

Roderik van der Veer avatar
Roderik van der Veer

it also puts my release in a failed state

Roderik van der Veer avatar
Roderik van der Veer

and i have cleanupOnFail set to true, but all the pods and services appear to be there

Roderik van der Veer avatar
Roderik van der Veer

and i do not have clusterIP in my templates in case you were wondering

Zachary Loeber avatar
Zachary Loeber

You must have clusterIP: “” somewhere in your source template or this would not occur right?

Zachary Loeber avatar
Zachary Loeber

in either case, I’ve found I’ve had to blow away the entire deployment to get around this three-way merge madness/issue myself :(*

2020-02-07

David Hubbell avatar
David Hubbell

Is there a recommendation for storing the helm definition alongside the application code in the same git repo vs storing it in a separate infrastructure/IaC repo?

grv avatar

Good question. We keep terraform things separate so I would like people to give their opinions here, cz I am soon switching to Helm structure as well. Currently we have a legacy setup of bunch of yaml files (deploy,service,ingress etc) in a separate repo. CI-CD systems read from the repo and control of that repo is with Devops/SRE people

David Hubbell avatar
David Hubbell

yeah, our terraform is separate, too, which is why I could see it going either way….

Zachary Loeber avatar
Zachary Loeber

I’m on the fence with this one. I personally keep a rather large omnichart (or archtype chart or whatever one would call the thing) in its own repo then use it in downstream per-app charts that can reside in the application repose themselves

Zachary Loeber avatar
Zachary Loeber

the idea of keeping these massive standardized charts in each app repo just doesn’t appeal to me with any more than a handful of apps at play

DanB avatar

I’m reading the README and it isn’t clear what the difference between apply and sync is. sync does upgrade –install, apply runs a diff before, but doesn’t that mean the same thing?

James Huffman avatar
James Huffman

just speaking for where i work, we do the following:

  1. git repository for each microservice app
  2. src folder contains the app source code
  3. charts folder contains the helm chart source
  4. build process generates container image and helm chart, pushes both to container/chart registry
:--1:1
David Hubbell avatar
David Hubbell

@James Huffman for #4, when you say the process generates the chart, you’re referring to a “helm package” call, right?

James Huffman avatar
James Huffman

correct

James Huffman avatar
James Huffman

as i understand it, apply is being deprecated and sync is the way forward

Andrey Nazarov avatar
Andrey Nazarov

Where did you get this info?

James Huffman avatar
James Huffman

i must have misread something. there were certain features that only worked on a sync and not apply and it seemed as if apply had reduced capabilities

James Huffman avatar
James Huffman

apologies for any misinformation!

James Huffman avatar
James Huffman

we recently migrated all our helmfile usage from apply to sync

grv avatar

@James Huffman so you keep the source chart and a backup in some registry as well. What tooling you use for Helm chart for registry? S3 or something?

James Huffman avatar
James Huffman

Azure Container Registry and Google Container Registry

James Huffman avatar
James Huffman

you can store both container images and helm charts in those

Zachary Loeber avatar
Zachary Loeber

@James Huffman for #4, do you keep the whole chart there or do you use subcharts at all?

James Huffman avatar
James Huffman

because we have dozens of microservices, we’ve made a “common” chart for which every app’s chart is a thin subchart

:100:1
Josh Hudson avatar
Josh Hudson

I’m in the process of implementing this pattern myself

Andrey Nazarov avatar
Andrey Nazarov

We are doing it the same way.

James Huffman avatar
James Huffman

that way we can add features to every app’s chart just by changing the common one

grv avatar

Does ECR support helm charts? We use ECR for dockerized images of microservices

Zachary Loeber avatar
Zachary Loeber

@James Huffman That sounds sane to me

James Huffman avatar
James Huffman

yeah, keeps us from having 30 different ways of building charts

:100:1
Erik Osterman avatar
Erik Osterman

We use this pattern too

1
Zachary Loeber avatar
Zachary Loeber

I totally used that pattern today to migrate ingress on a cluster. It made me feel like a wizard.

Zachary Loeber avatar
Zachary Loeber

because, you know, my feelings are pertinent to this conversation…

1

2020-02-06

Josh Hudson avatar
Josh Hudson

Where in helmfiles is .Release in scope? As far as I can tell the only place I can access it is in Release Templates with double interpolation. I’m finding it very painful to get access to anything from .Release in my values templates. Anyone have any tips?

Josh Hudson avatar
Josh Hudson

For example I have several values I’d like to do something like someUrl: {{ printf "https://%[s.mydomain.co](http://s.mydomain.co)" .Release.Namespace }} in my values.yaml.gotmpl.

Josh Hudson avatar
Josh Hudson

Luckily I realized .Namespace also exists and is in scope for the values templates, but I think the point about .Release remains for things like .Release.Name.

2020-02-05

DanB avatar

How can I reference a Value in my environments.yaml.gotmpl? This produces an error indicating clusterName is not set. This is odd because in my releases I use .gotmpl for values files and the same Value can be found in that context. I run this via helmfile -e developer apply with the CLUSTER_NAME and AWS_REGION env vars exported correctly.

DanB avatar

current solution is to duplicate the '{{ coalesce (env "CLUSTER_NAME") (env "CLUSTER") }}' in the environments.yaml.gotmpl

DanB avatar

this is the error err: error during environments.yaml.gotmpl.part.0 parsing: template: stringTemplate:5:19: executing “stringTemplate” at <.Values.clusterName>: map has no entry for key “clusterName” in ./helmfile.yaml: error during environments.yaml.gotmpl.part.0 parsing: template: stringTemplate:5:19: executing “stringTemplate” at <.Values.clusterName>: map has no entry for key “clusterName”

Josh Hudson avatar
Josh Hudson

Is a top-level values map supported in a helmfile? I thought values had to be defined within an environment, release, or release template

DanB avatar

Hmm, I’m not sure if it is supported but it does work. I can reference the values in the top level helmfile in my release values files

DanB avatar

This makes sense for me since I want to single source several different values from env vars, I prefer to do it in one place.

Josh Hudson avatar
Josh Hudson
Josh Hudson avatar
Josh Hudson


I want to single source several different values from env vars, I prefer to do it in one place.
Maybe readFile could be a good option?

{{ readFile "common.yaml" }}
DanB avatar

so it’d be 1) set env vars 2) generate common.yaml programmatically from env vars 3) run helmfile?

Roderik van der Veer avatar
Roderik van der Veer

Has anyone done work to get helmfile (and diff etc) output in a computer readable format? Mostly looking for a JSON format of the info displayed after a release is completed.

Erik Osterman avatar
Erik Osterman

@Roderik van der Veer that’s a cool idea! then we could use OPA to set up some policies.

Erik Osterman avatar
Erik Osterman

@mumoshu have you seen anyone do that?

Erik Osterman avatar
Erik Osterman

@Roderik van der Veer what’s your use-case for the JSON data?

Roderik van der Veer avatar
Roderik van der Veer

We are using helmfile to orchestrate dedicated k8s clusters + a lot of services from a web platform. The only feedback that i can five to the user waiting is “start helmfile run” and “helmfile run complete”. Having each “step” output a json log line, i can read those and show “deployed x” “deployed y” but also filter out failures when they happen.

Erik Osterman avatar
Erik Osterman

aha, ok - so a bit different use-case

Roderik van der Veer avatar
Roderik van der Veer

it might be as easy as supporting the json output from helm directly, we use that in some other cases (list releases etc)

mumoshu avatar
mumoshu

sounds more like a ndjson log format rather than json output, but interesting use-case!

mumoshu avatar
mumoshu

https://github.com/roboll/helmfile/issues/913 could be about adding --output json to helmfile.

this ndjson one could be about logging and possibly addressed by adding --log-format json where the default is text to helmfile.

JSON output · Issue #913 · roboll/helmfile

We recently had a need to parse the output of helmfile list. It would be ideal if we could utilize an –output json flag (like helm allows) to return structured json output.

2020-02-04

erik-stephens avatar
erik-stephens

Anyone able to get --helm-binary declared in their helmfiles? I’d prefer user not have to know which version of helm should be used.

Erik Osterman avatar
Erik Osterman

@Jeremy Grodberg you are doing this right?

Jeremy Grodberg avatar
Jeremy Grodberg

No, I am using export HELM_BINARY=helm3 to set which binary helmfile should use. I don’t think you can select from within the helmfile itself.

:--1:1
erik-stephens avatar
erik-stephens

@Bart M. Here is another option that might prove less onerous.

Bart M. avatar
Bart M.

hmm couldn’t find anything about this in the sourcecode?

Bart M. avatar
Bart M.

I would expect this to be present somewhere, but that env var is never referenced… unless that cli flag package does some magic?

Zachary Loeber avatar
Zachary Loeber

Yeah, where is this thing referenced? I’m keen to know if it gets around some of the helm plugin issues I’ve seen where they ignore the users $PATH variable when looking for and using the helm binary

2020-02-03

Jonathan avatar
Jonathan

Has anyone set up keycloak using helmfile? We used to have to set everything up manually, and had a startup script to add the custom realms as well as a role-mapping, but would like to move away from that. I’ve managed to add the realms properly, but I’m not sure how to add the role-mappings without using some sort of script.

Erik Osterman avatar
Erik Osterman

We deploy keycloak with helmfile, but only to the point the software is running.

Erik Osterman avatar
Erik Osterman

The configuration therein is manual

Erik Osterman avatar
Erik Osterman
cloudposse/helmfiles

Comprehensive Distribution of Helmfiles for Kubernetes - cloudposse/helmfiles

Bart M. avatar
Bart M.

I’ve done something like that a long time ago - but with Ansible

Bart M. avatar
Bart M.

was a pain in the ass

Graeme Gillies avatar
Graeme Gillies

I have a command that will call an external source and return json output, which I am currently using inside an {{ exec }} block in a helmfile template file for values, I don’t suppose there is a way in template I can unmarshal that json into a struct or map so I can access inner attributes easily?

Josh Hudson avatar
Josh Hudson

Has anyone had any luck getting inline values working with release templating? I cannot get this example from the best practices doc to work: inline values map:


\# ...
  valuesTemplate:
  - image:
      tag: `{{ .Release.Labels.tag }}`

\# ...
Josh Hudson avatar
Josh Hudson

In my case I’m trying to have access to .Release.Name in a values.yaml.gotmpl.

Josh Hudson avatar
Josh Hudson

It seems like the issue is Environment.Values completely clobbers my values: and valuesTemplates: definitions from my template as I get:

executing "stringTemplate" at <.Values.releaseName>: map has no entry for key "releaseName"
error in first-pass rendering

When I strip out the environment section, the templates render without error, but I get: err: no releases found that matches specified selector() and environment(staging), in any helmfile.

bradym avatar
bradym

In my helmfile when I reference Release.Name I use:

{{`{{ .Release.Name }}`}}
bradym avatar
bradym

have you tried that?

Josh Hudson avatar
Josh Hudson

I feel like I’ve tried every variation of the double bracket syntax. I can use it for the path to a values file but not inline values despite the example in the best practices doc. This works:

  valuesTemplate:
  - config/{{`{{ .Release.Name }}`}}/values.yaml

This doesn’t seem to:

  valuesTemplate:
  - releaseName: `{{ .Release.Name }}

`

bradym avatar
bradym

I think you need to wrap it in quotes when it’s used at the start of the line:

    labels:
      app: "{{`{{ .Release.Name }}`}}"
bradym avatar
bradym

^^ that works in my helmfile

Josh Hudson avatar
Josh Hudson

You are able to reference {{ [Values.labels.app](http://Values.labels.app) }} in a template?

Josh Hudson avatar
Josh Hudson

Trying your syntax above I still get: executing "stringTemplate" at <.Values.releaseName>: map has no entry for key "releaseName" when I try to render it in a template.

bradym avatar
bradym

Oh, I’m using that in the helpfile, not in a template.

bradym avatar
bradym

May have mis-read what you posted.

Josh Hudson avatar
Josh Hudson

Well I would love to call .Release.Name directly in a template, but because it’s only in scope of the helmfile, the best practices suggests to use an inline value, but I can’t get it to work.

Josh Hudson avatar
Josh Hudson

I see something similar in a go test as your usage, but I don’t think it tries it with a template: https://github.com/roboll/helmfile/blob/fc75f25293055003d8159a841940313e56a164c6/pkg/app/app_test.go#L3701-L3702

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Josh Hudson avatar
Josh Hudson

I would imagine using .Release.Name in a values template would be a common use case, so I feel like I’m missing something obvious.

bradym avatar
bradym

I’m doing a similar thing with .Environment.Name not sure if it would work with .Release.Name. Here’s what I’ve got:

templates:
  default: &default
    chart: stable/{{`{{ .Release.Name }}`}}
    labels:
      app: "{{`{{ .Release.Name }}`}}"
    namespace: infra
    missingFileHandler: Error
    values:
      - environment: {{ .Environment.Name }}
      - config/{{`{{ .Release.Name }}`}}.values.yaml.gotmpl

I don’t really understand the valuesTemplate thing, maybe try putting it in values and it’ll work?

Josh Hudson avatar
Josh Hudson

I’ve also tried it in values and valuesTemplate, but happy to try again.

bradym avatar
bradym

I’m new to helmfile, so I’m honestly just making some guesses and hoping they’ll work.

Josh Hudson avatar
Josh Hudson

I am as well. I appreciate the help in any case

Josh Hudson avatar
Josh Hudson

I ended up using helmfiles overrides as a workaround to hardcode my release names: https://github.com/roboll/helmfile/issues/387#issuecomment-513737164

Make values available as .Values in termplates and helmfile.yaml · Issue #387 · roboll/helmfile

It would really help if values from different places were available as .Values (like in standard Helm), to be referred and used. Not alle values are Environment values, example: We have set up Open…

    keyboard_arrow_up