#helmfile (2020-02)

Has anyone set up keycloak using helmfile? We used to have to set everything up manually, and had a startup script to add the custom realms as well as a role-mapping, but would like to move away from that. I’ve managed to add the realms properly, but I’m not sure how to add the role-mappings without using some sort of script.

I’ve done something like that a long time ago - but with Ansible

was a pain in the ass

I have a command that will call an external source and return json output, which I am currently using inside an {{ exec }} block in a helmfile template file for values, I don’t suppose there is a way in template I can unmarshal that json into a struct or map so I can access inner attributes easily?

Has anyone had any luck getting inline values working with release templating? I cannot get this example from the best practices doc to work: inline values map:

# ...
  valuesTemplate:
  - image:
      tag: `{{ .Release.Labels.tag }}`
# ...

In my case I’m trying to have access to .Release.Name in a values.yaml.gotmpl.

It seems like the issue is Environment.Values completely clobbers my values: and valuesTemplates: definitions from my template as I get:

executing "stringTemplate" at <.Values.releaseName>: map has no entry for key "releaseName"
error in first-pass rendering

When I strip out the environment section, the templates render without error, but I get: err: no releases found that matches specified selector() and environment(staging), in any helmfile.

In my helmfile when I reference Release.Name I use:

{{`{{ .Release.Name }}`}}

have you tried that?

I feel like I’ve tried every variation of the double bracket syntax. I can use it for the path to a values file but not inline values despite the example in the best practices doc. This works:

  valuesTemplate:
  - config/{{`{{ .Release.Name }}`}}/values.yaml

This doesn’t seem to:

  valuesTemplate:
  - releaseName: `{{ .Release.Name }}

`

I think you need to wrap it in quotes when it’s used at the start of the line:

    labels:
      app: "{{`{{ .Release.Name }}`}}"

^^ that works in my helmfile

You are able to reference {{ [Values.labels.app](http://Values.labels.app) }} in a template?

Trying your syntax above I still get: executing "stringTemplate" at <.Values.releaseName>: map has no entry for key "releaseName" when I try to render it in a template.

Oh, I’m using that in the helpfile, not in a template.

May have mis-read what you posted.

Well I would love to call .Release.Name directly in a template, but because it’s only in scope of the helmfile, the best practices suggests to use an inline value, but I can’t get it to work.

I see something similar in a go test as your usage, but I don’t think it tries it with a template: https://github.com/roboll/helmfile/blob/fc75f25293055003d8159a841940313e56a164c6/pkg/app/app_test.go#L3701-L3702

I would imagine using .Release.Name in a values template would be a common use case, so I feel like I’m missing something obvious.

I’m doing a similar thing with .Environment.Name not sure if it would work with .Release.Name. Here’s what I’ve got:

templates:
  default: &default
    chart: stable/{{`{{ .Release.Name }}`}}
    labels:
      app: "{{`{{ .Release.Name }}`}}"
    namespace: infra
    missingFileHandler: Error
    values:
      - environment: {{ .Environment.Name }}
      - config/{{`{{ .Release.Name }}`}}.values.yaml.gotmpl

I don’t really understand the valuesTemplate thing, maybe try putting it in values and it’ll work?

I’ve also tried it in values and valuesTemplate, but happy to try again.

I’m new to helmfile, so I’m honestly just making some guesses and hoping they’ll work.

I am as well. I appreciate the help in any case

I ended up using helmfiles overrides as a workaround to hardcode my release names: https://github.com/roboll/helmfile/issues/387#issuecomment-513737164

Anyone able to get --helm-binary declared in their helmfiles? I’d prefer user not have to know which version of helm should be used.

@Jeremy G (Cloud Posse) you are doing this right?

No, I am using export HELM_BINARY=helm3 to set which binary helmfile should use. I don’t think you can select from within the helmfile itself.

How can I reference a Value in my environments.yaml.gotmpl? This produces an error indicating clusterName is not set. This is odd because in my releases I use .gotmpl for values files and the same Value can be found in that context. I run this via helmfile -e developer apply with the CLUSTER_NAME and AWS_REGION env vars exported correctly.

Has anyone done work to get helmfile (and diff etc) output in a computer readable format? Mostly looking for a JSON format of the info displayed after a release is completed.

@Roderik van der Veer that’s a cool idea! then we could use OPA to set up some policies.

@mumoshu have you seen anyone do that?

@Roderik van der Veer what’s your use-case for the JSON data?

We are using helmfile to orchestrate dedicated k8s clusters + a lot of services from a web platform. The only feedback that i can five to the user waiting is “start helmfile run” and “helmfile run complete”. Having each “step” output a json log line, i can read those and show “deployed x” “deployed y” but also filter out failures when they happen.

aha, ok - so a bit different use-case

it might be as easy as supporting the json output from helm directly, we use that in some other cases (list releases etc)

sounds more like a ndjson log format rather than json output, but interesting use-case!

https://github.com/roboll/helmfile/issues/913 could be about adding --output json to helmfile.

this ndjson one could be about logging and possibly addressed by adding --log-format json where the default is text to helmfile.

Where in helmfiles is .Release in scope? As far as I can tell the only place I can access it is in Release Templates with double interpolation. I’m finding it very painful to get access to anything from .Release in my values templates. Anyone have any tips?

For example I have several values I’d like to do something like someUrl: {{ printf "https://%[s.mydomain.co](http://s.mydomain.co)" .Release.Namespace }} in my values.yaml.gotmpl.

Is there a recommendation for storing the helm definition alongside the application code in the same git repo vs storing it in a separate infrastructure/IaC repo?

Good question. We keep terraform things separate so I would like people to give their opinions here, cz I am soon switching to Helm structure as well. Currently we have a legacy setup of bunch of yaml files (deploy,service,ingress etc) in a separate repo. CI-CD systems read from the repo and control of that repo is with Devops/SRE people

I’m reading the README and it isn’t clear what the difference between apply and sync is. sync does upgrade –install, apply runs a diff before, but doesn’t that mean the same thing?

just speaking for where i work, we do the following:

1. git repository for each microservice app
2. src folder contains the app source code
3. charts folder contains the helm chart source
4. build process generates container image and helm chart, pushes both to container/chart registry

as i understand it, apply is being deprecated and sync is the way forward

we recently migrated all our helmfile usage from apply to sync

@James Huffman so you keep the source chart and a backup in some registry as well. What tooling you use for Helm chart for registry? S3 or something?

Azure Container Registry and Google Container Registry

you can store both container images and helm charts in those

@James Huffman for #4, do you keep the whole chart there or do you use subcharts at all?

because we have dozens of microservices, we’ve made a “common” chart for which every app’s chart is a thin subchart

that way we can add features to every app’s chart just by changing the common one

Does ECR support helm charts? We use ECR for dockerized images of microservices

@James Huffman That sounds sane to me

yeah, keeps us from having 30 different ways of building charts

We use this pattern too

I totally used that pattern today to migrate ingress on a cluster. It made me feel like a wizard.

because, you know, my feelings are pertinent to this conversation…

I’m having some issues with helmfile + helm3 + dynamic clusterIP’s. It is this issue: https://github.com/helm/helm/issues/7082#issuecomment-575514155 but when i put force to false in my helmfile i’m still hit with Error: UPGRADE FAILED: an error occurred while cleaning up resources. original upgrade error: failed to replace object: Service "violet-reindeer-mint-webserver" is invalid: spec.clusterIP: Invalid value: "": field is immutable: unable to cleanup resources: object not found, skipping delete

it also puts my release in a failed state

and i have cleanupOnFail set to true, but all the pods and services appear to be there

and i do not have clusterIP in my templates in case you were wondering

You must have clusterIP: “” somewhere in your source template or this would not occur right?

in either case, I’ve found I’ve had to blow away the entire deployment to get around this three-way merge madness/issue myself :(*

Hmm, so is there a consensus on the differences between apply and sync?

hmm doesn’t seem to be possible to define in a helmfile which helm version should be used?

it only seems to be possible on the cli

this is a bit messy since I currently have multiple environments using helm2 and helm3… now I have to specify the --helm-binary / -b every time

From within helmfile.yaml, is there anyway I can detect if helmfile is being run with helm2 or helm 3?

We have a shared repo that we’ll be migrating namespace by namespace and I’d like to add logic that says “this helmfile.yaml is only deloyed with helm 2 or 3”

~I believe the right way is to set helm_binary inside each helmfile, then install both versions of helm.~

If you’re using alpine linux, we distribute helm2 and helm3 packages

https://github.com/cloudposse/packages

Wish there was a cloudposse group I could @ mention, but I have a very small PR to make a helmfile a bit more friendly: https://github.com/cloudposse/helmfiles/pull/224

they are very active on here…I bet you won’t wait too long

Usually don’t

@Maxim Mironenko (Cloud Posse) is working full time on our PR backlog

@Alex Siegman will put it on top of my queue

It’s a 5 minute PR tops unless I missed some naming convention, I matched a similar one I found in Sentry helmfile

@Alex Siegman here it is: <https://github.com/cloudposse/helmfiles/releases/tag/0.92.0>

Is there a way to have the u/p for a repository to come from a file? Trying to do the following where the ‘docker’ key is part of an environments values file. I tried making a base repositories file and make it a gotmpl but no joy there.

  - name: settlemint
    url: <https://harbor.settlemint.com/chartrepo/launchpad>
    username: {{ .Values.docker.username }}
    password: {{ .Values.docker.password }}

It’s probably related to the double rendering bug

is helmfiles: the only section of a helmfile that can take git repo refs/paths. E.g. can I reference another file in bases: or values: ?

hey all. has anyone deployed the elk stack with metricbeat on aws with helm? I’m going with the defaults and it sort of works, but i get an error on one of the monitoring services

Hi all, is helmfile usable with helm3 ?

Is there a way of pre render a single release, i dont see any options in helmfile template --help

helmfile -e your-environment -l name=name-of-release template

Hi all, how to make helmfile to delete chart, when I remove it from helmfile.yaml ? Neither helm apply nor helm sync seems working for me

try to use installed option. set it to false and do apply (here is documentation: <https://github.com/roboll/helmfile/blame/master/README.md#L160>)

Hi all, I’m doing helmfile -n myns apply and then helmfile -n myns status from my CI \ CD jenkins pipeline Although helmfile apply executes successfully , helmfile status fails with following error

23:34:14  err: release "eshepelyuk-fastapi" failed: helm exited with status 1:
23:34:14    Error: release: not found
23:34:14  err: release "hrzn-docs" failed: helm exited with status 1:
23:34:14    Error: release: not found
23:34:14  in ./helmfile.yaml: 2 errors:
23:34:14  err 0: release "eshepelyuk-fastapi" failed: helm exited with status 1:
23:34:14    Error: release: not found
23:34:14  err 1: release "hrzn-docs" failed: helm exited with status 1:
23:34:14    Error: release: not found

Locally, both commands work, Can anyone give an advice ?

Anyone plz, at least some clues where to start from ?

maybe supply the namespace with helmfile status command?

oh, seems like you are.

Well after checking debug log, i see that -n is not passed to helm status command from helmfile status

23:20:47  Getting status eshepelyuk-fastapi
23:20:47  exec: helm status eshepelyuk-fastapi
23:20:47  worker 1/2 started
23:20:47  Getting status hrzn-docs
23:20:47  exec: helm status hrzn-docs

Is it a bug ?

I found this PR https://github.com/roboll/helmfile/pull/1098 that should be fixed in 0.99.1 I am using 0.100.0, and bug is still here

Interesting observation when trying to workaround as suggested in that PR

1. when running helmfile status -n myns --args "--namespace myns" - I receive the same error
2. when running helmfile status --args "--namespace myns" - all works fine, So, apparently there’s a bug with -n flg in helmfile status command Could someone confirm ?

Another question is about helmfile.lock In case I’m not gonna use version ranges (ie 1.x or ~2.0) is there any reason to care about helmfile.lock Can I put it to gitignore ?

Anybody here ?

You can safely ignore the helmfile.lock if you’re not going to use the functionality.

@Ievgenii Shepeliuk Hey! Are you sure you’re using the same version of Helm(3 or 2) on your machine and Jenkins?

@Ievgenii Shepeliuk Thanks. One more thing - Does helm list -n myns show you eshepelyuk-fastapi on both your machine and Jekins?

I need to configure replica sets with different configurations for mongodb - https://github.com/helm/charts/tree/master/stable/mongodb-replicaset/templates. could anyone point me in the right direction?

I’ve got a question about helmfile netsted helmfiles and the values key in nested helmfiles. What exactly is that meant to override and what format is expected?

Guys, i don’t know who are the maintainers, but please accept a great appreciation from our team to all of them. we’ve been able to adopt helmfile in a day (sure with some issues, i’ve asked in this chat) and this tool really enriched our CD pipelines, so many problems solved and so easy to adopt it’s really a great tool for k8s ecosystem pity, we’re not golang team and can’t contribute freely as we do to JVM OSS projects

@scorebot let’s keep tabs!

@scorebot has joined the channel

Thanks for adding me emojis used in this channel are now worth points.

Wondering what I can do? try @scorebot help

Hello. this is my first post here. I would like to provision thousands of namespaces, one for each customer, or even several, namespaces per customer. Every day more and more customers would join and more and more namespaces would be created. Would helmfile accomodate that? What would be the best practices in this case, please? Thanks a lot

sure, but it may be more efficient to just script out the creation of namespaces based on your naming convention and flat yaml/json at those numbers.

Here is a great write up on kube thresholds and scalability: https://github.com/kubernetes/community/blob/master/sig-scalability/configs-and-limits/thresholds.md#kubernetes-thresholds

@Zachary Loeber: Thanks a lot

@Richard Gomes thousands would be the point where I’d consider writing a dedicated thing in Go if all these things are very similar… you can for example use helm3 as a go library, and accessing the API is pretty easy…

{{- $namespace := requiredEnv "GITLAB_KUBE_NAMESPACE" }}

How can I make that fail if the string is longer than 23 characters?

Well this is kinda clever -> https://github.com/bitsofinfo/helmfile-deploy

That’s interesting

Feels quite similar to what we are doing with Helmfile and monochart, but his more opinionated approach is appealing

@Igor Rodionov

maybe i’m doing it wrong, but i did not expect that with 2 releases with the same name, one in a cluster-specific helmfile and one later in a common helmfile, helmfile will take the later one?

anyone else thats deploying prometheus-operator getting this issue? It was supposedly fixed in helm 2.14 but even testing with more recent versions of helm2 (2.16.3) or helm3, I still get the error:

failed processing release prometheus-operator: helm exited with status 1:
  Error: validation failed: unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"

https://github.com/coreos/prometheus-operator/issues/1866

I know its a race condition that I can easily script around, but it reads to me that it should be fixed?