#helmfile (2021-02)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2021-02-25

jason800 avatar
jason800

Hey has anyone made use of jsonPatches in helmfile? I am trying a very simple use-case for them when running a release against a directory of manifest files and I cannot seem to get it to work. A strategic merge patch works just fine but I wanted the ability to target and apply specifically

jason800 avatar
jason800

I have this manifest:

---
  2 apiVersion: rbac.authorization.k8s.io/v1
  3 kind: ClusterRoleBinding
  4 metadata:
  5   name: cluster-readonly-all
  6   annotations:
  7     meta.helm.sh/release-name: kubernetes-manifests
  8     meta.helm.sh/release-namespace: default
  9   labels:
 10     app.kubernetes.io/managed-by: Helm
 11     app.kubernetes.io/name: kubernetes-manifests
 12 roleRef:
 13   apiGroup: rbac.authorization.k8s.io
 14   kind: ClusterRole
 15   name: cluster-readonly-all
 16 subjects:
 17 - kind: Group
 18   name: system:authenticated
 19   apiGroup: rbac.authorization.k8s.io
jason800 avatar
jason800

and then this in my helmfile

jason800 avatar
jason800
    jsonPatches:
      - target:
          version: rbac.authorization.k8s.io/v1
          kind: ClusterRoleBinding
          name: cluster-readonly-all
        patch:
          - op: add
            path: "/metadata/labels/cluster"
            value: "{{ $cluster }}"
          - op: add
            path: "/metadata/labels/realm"
            value: "{{ $realm }}"
          - op: add
            path: "/metadata/labels/region"
            value: "{{ $region }}"
jason800 avatar
jason800

I see the patches getting generated in what looks to be a correct way from the output of --debug

jason800 avatar
jason800
generated and using kustomization.yaml:
kind: ""
apiversion: ""
resources:
- templates/ClusterRole-ReadOnly.yaml
- templates/ClusterRoleBinding-ReadOnly.yaml
patchesJson6902:
- target:
    kind: ClusterRole
    name: cluster-readonly-all
    namespace: default
    version: rbac.authorization.k8s.io~1v1

  path: jsonpatches/patch.0.yaml
- target:
    kind: ClusterRoleBinding
    name: cluster-readonly-all
    version: rbac.authorization.k8s.io/v1

  path: jsonpatches/patch.1.yaml
jason800 avatar
jason800

This is incredibly frustrating. I cannot get jsonPatches to pick up the target or properly modify the manifest being ingested. strategicMergePatch works, but it fails under my go-templating which iterates over many regions, the metadata/labels only get the region of the first iterated release

2021-02-24

Evan Pitstick avatar
Evan Pitstick

is there a standard way of “injecting” a template into helm charts before it’s parsed with helmfile? i started playing with hooks looking for the right way but the working dir doesn’t seem to be current dir. I can think of a number of cases where I might want to add a CRD or something to a chart but not want to have to keep up our own custom downstream version of the chart just to keep that one file in there. I see that there is a way to get kustomize to run with the chart but it would be nice to use all the _helpers in the chart i’m working on.

Evan Pitstick avatar
Evan Pitstick

also, I see a lot about tiller in the docs. i didn’t start using helm until 3 was around. is there a reason to use tiller with helmfile besides still being on helm 2?

voron avatar
voron

no reason to use tiller now

Igor Urazov avatar
Igor Urazov

@ your question is covered at https://github.com/roboll/helmfile/issues/1157. personally i’m using incubator/raw chart for this

How to deal with simple kubernetes definitions that don't have a chart · Issue #1157 · roboll/helmfile

This is more of a questions. There are more and more CRDs in the wild nowdays and they all come with their own CRs that of course don't come with a Chart. If I want to keep my helm/helmfile wor…

rmk avatar

is anyone successfully using the remote environment values from 1296? i’m on helmfile v0.138.4 and it just treats the URLs as a local path

Add the ability to load a remote environment values file by kevinjqiu · Pull Request #1296 · roboll/helmfile

This PR enables the user to specify a remote path for an environment values file, e.g., environments: cluster-azure-us-west: values: - git://git.company.org/helmfiles/global/azur>…

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

Hmm useful feature, thnx for pointing it out Should it only fetch from git or from other sources as well ?

Add the ability to load a remote environment values file by kevinjqiu · Pull Request #1296 · roboll/helmfile

This PR enables the user to specify a remote path for an environment values file, e.g., environments: cluster-azure-us-west: values: - git://git.company.org/helmfiles/global/azur>…

rmk avatar

theoretically it should work with anything supported by go-getter, but I haven’t got it to work at all. :confused: haven’t had time to do any digging yet. have to deliver this today so I just used an exec template call to pull the git repo for now

hashicorp/go-getter

Package for downloading things from a string URL using a variety of protocols. - hashicorp/go-getter

2021-02-23

Roderik van der Veer avatar
Roderik van der Veer

I was wondering how a “selector” and “needs” works together. I have one “mongodb” chart and 2 “apps” that depend on it, Apps are labeled aith the selector “app=app1/2”. The apps have “need” setting to the mongo chart. When I use a selector to install app1, does it also install mongodb?

bradym avatar
bradym

Yep! I’m doing something similar and it’s working well for me.

Roderik van der Veer avatar
Roderik van der Veer

awesome!

Roderik van der Veer avatar
Roderik van der Veer

Well, this is so not working for me do you have an example? the releases ther are “needed” are never installed

Roderik van der Veer avatar
Roderik van der Veer
Not installing dependancies when using selector · Issue #1692 · roboll/helmfile

i have some thing like releases: - name: cert-manager namespace: cert-manager chart: cert-manager/cert-manager version: v1.2.0 missingFileHandler: Warn - name: prometheus needs: - cert-manager/cert…

2021-02-22

joey avatar

perhaps this is not a helmfile question, but i think it might be? i know there’s https://github.com/hashicorp/terraform-provider-aws/issues/16542#issuecomment-782606500 which is making nice progress but looking at the pr there’s something that’s missing for me.

i want to run coredns using a nodeSelector and coredns comes as a preinstalled Deployment with eks and/or an addon. i really don’t want to overwrite the coredns deployment with my own deployment.. i’d rather just patch the existing deployment with a nodeSelector. i don’t know any simple way to do this with helmfile?

Support Managing EKS Cluster Add-Ons · Issue #16542 · hashicorp/terraform-provider-aws

Community Note Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request Please do not leave "+1" or other comme…

joey avatar

any feedback on how people solve this sort of scenario?

Support Managing EKS Cluster Add-Ons · Issue #16542 · hashicorp/terraform-provider-aws

Community Note Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request Please do not leave "+1" or other comme…

mumoshu avatar
mumoshu

@joey Hey! I tend to just “import” those existing resource beforehand and use helmfile for management afterwards

https://stackoverflow.com/a/62528643 https://github.com/roboll/helmfile/issues/1281

Use Helm 3 for existing resources deployed with kubectl

We used to deploy kubernetes resources using the normal kubectl command for services, deployments, configmap…etc. Now we need to start using Helm 3 and integrate it to our pipelines, but when I t…

Importing existing k8s resources into a release · Issue #1281 · roboll/helmfile

I would like to import existing resources into my release. This new helm feature allows you to adopt existing resources by annotating them: helm/helm#7649 I was thinking of implementing this using …

mumoshu avatar
mumoshu

Does it help if Helmfile had ability to auto-annotate/label the existing resource so that it automagically imports to helmfile?

It should look like built-in version of: https://github.com/roboll/helmfile/issues/1281#issuecomment-636494016

Importing existing k8s resources into a release · Issue #1281 · roboll/helmfile

I would like to import existing resources into my release. This new helm feature allows you to adopt existing resources by annotating them: helm/helm#7649 I was thinking of implementing this using …

joey avatar

i had no idea about helmfile import. that definitely solved my use case.

1

2021-02-16

jason800 avatar
jason800

Hey All, have a weird error. I’m providing a values file via a helmfile release to a helm chart. I’m providing it via a variable containing the full file path (the file is generated at run-time). The path I’m providing should be absolutely but is erroring because it doesn’t exist in the relative .

values file matching "/tmp/manage-secrets-IhNjd4/combined_secrets.yaml " does not exist in "."

Has anyone ever seen anything like this?

jason800 avatar
jason800

removing quotes from the variable worked

jason800 avatar
jason800

well, it actually didn’t. it got rid of the above error, but helmfile simply does not process the file and the values are not available to the helm chart

2021-02-15

2021-02-14

2021-02-13

2021-02-12

Balazs Varga avatar
Balazs Varga
10:45:54 AM

hello all, I would like to add the Environment.Name as label to all alerts so alter I could setup a rule based on that. If I do like this helmfile does not catches it and in configmap I see without any changes.

Vadim Bauer avatar
Vadim Bauer

I am seeking Helmfile advice:

We are deploying some Helm charts that are infra related. At the moment we have only one cloud provider and use environments called stage and prod A new cloud provider will be added next and we want to install the same set of helm charts there as well. The new cloud provider has some values that are only relevant to him.

How ca we best support that with helm? I see this two options:

  1. Add more envs eg. stage-aws, stage-azr, prod-aws, prod-azr and so on.
  2. Continue using stage/prod but move to cloud provider specific stuff to selectors. And then do if selector aws do aws specific and so on Do you see any addition option how to accomplish this?
Andrey Bronin avatar
Andrey Bronin

Environments should be a better choice

  1. You can add cloud-specific values like production: true or cloud: azure and conditionally set installed option to releases to prevent them to be installed on the wrong stage/cloud.
  2. You can define different engines/config to cloud specific charts like external-dns or external-secrets
Vadim Bauer avatar
Vadim Bauer

Hello Andrey, should I set key value cloud: azure and then test it? Its not clear to me, can you maybe give an example?

Andrey Bronin avatar
Andrey Bronin

For example, if you’ll add production: true or production:false you can do this trick with some of your releases:

  - name: datadog
    <<: *default
    installed: {{ eq .Environment.Values.production true | toYaml }}
    chart: datadog/datadog
    version: 2.8.1
Andrey Bronin avatar
Andrey Bronin

As you can see it here, we’re installing datadog to production environments only.

Christian avatar
Christian

You should take a look at how adobe does it with helmfile: https://medium.com/adobetech/how-to-set-up-a-kubernetes-clusters-with-helmfile-4ab7866fc2fc Would give you some ideas.

How to Set Up a Kubernetes Clusters with Helmfile attachment image

Code examples of how Adobe Experience Platform uses helmfile in Kubernetes to streamline large-scale application management.

1
Andrey Bronin avatar
Andrey Bronin

@ are you using templates in your helmfiles? Were you able to use bases like environments.yaml and then use Environment.Values.something with templates?

Christian avatar
Christian

Yup. I generally have this format for almost all my helmfiles.

bases:
  - environments.yaml
---
repositories:
  - name: autoscaler
    url: <https://kubernetes.github.io/autoscaler>

releases:
  - name: cluster-autoscaler
    namespace: kube-system
    chart: autoscaler/cluster-autoscaler
    labels:
      id: "cluster-autoscaler"
      chart: "cluster-autoscaler"
      namespace: "kube-system"
      tier: "base"
    version: {{ .Values.chartVersion | quote }}
    values:
      ...

2021-02-11

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

Hi all Is it a possible to share files that later could be used as bases in helmfile.yaml For instance put them in Git or S3 or HTTP And then fetch them I’ve tried to use go-getter style path, but obtained an error

roth.andy avatar
roth.andy

not sure what you mean by bases

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk
04:23:01 PM
roth.andy avatar
roth.andy

Ah. I haven’t used bases before, so I’m sorry I can’t help there.

What specific problem are you trying to solve? the helmfiles: section lets you pull from external sources, we use that a lot to compose larger patterns out of smaller modular helmfiles

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

Hmm, let me read about helmfiles. Thnx for the suggestion.

1
Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk

For instance, I want to share repoaitories betwenn dozens of helmfiles in different git repositories. Now, we just copy paste this block?

roth.andy avatar
roth.andy

I agree that it makes sense that bases: should support the same path: functionality as is found in helmfiles:

Ievgenii Shepeliuk avatar
Ievgenii Shepeliuk
Add support for referencing base helmfiles from a URL · Issue #1193 · roboll/helmfile

Use Case If you have multiple repositories with their own helmfiles being deployed it would be helpful to syncronise certain aspects, for example the environments section. This would make managemen…

1
jason800 avatar
jason800

Is it possible to set helmDefaults in environments.yaml?

jason800 avatar
jason800

I’m looking to apply a global --wait=true , but i execute individual helmfiles in my environment and don’t want to add a line to every file i’d rather add it in the exec of helmfile

jason800 avatar
jason800

is it maybe possible using the --args flag in helmfile ?

jason800 avatar
jason800
jason800 avatar
jason800

@mumoshu hey if you have a moment could you give a review to https://github.com/roboll/helmfile/pull/1678 ?

Add ability to specify / override helmDefaults.wait via cli by jwitko · Pull Request #1678 · roboll/helmfile

Fixes #64 This PR gives the ability to append –wait to either helmfile sync or helmfile apply. The setting will append –wait to every helm upgrade … command and it will also override helmDefau…

jason800 avatar
jason800

I also left a comment on the above issue giving some context

mumoshu avatar
mumoshu

I’ve added it to my todo list for this weekend thanks for pr!

jason800 avatar
jason800

Np! There is a test failing because the Wait method isn’t found inapp_test. I’m not that familiar with Go so any help you could provide on that would be greatly appreciated

mumoshu avatar
mumoshu
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

jason800 avatar
jason800

ty, had to add that and the function below

jason800 avatar
jason800

tests are now passing!

jason800 avatar
jason800

Hey @mumoshu did you ever get a chance to check it out? no problem if not, just happy to answer questions if so

jason800 avatar
jason800

thinking since its so similar I’ll probably handle the issue for helm 3.5 new --wait-for-jobs feature

jason800 avatar
jason800

but want to make sure this pattern is OK before doing that

jason800 avatar
jason800

thats a bummer. just tried using --args "--wait", and it throws errors because it applies to every helm command like helm list

2021-02-09

2021-02-05

Justin Seiser avatar
Justin Seiser

Is it possible, to mark a release so that it only runs when a --selector name=whatever is applied? wanting to do https://github.com/roboll/helmfile/tree/master/examples#managing-oneshot-jobs-with-helmfile this, but our CI runs helmfile apply on changes, and I really only want those jobs to run when explicitly told to do so.

roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Balazs Varga avatar
Balazs Varga

we use this if we would like to install something only on dev:

installed: {{ eq .Environment.Name "default" | toYaml }}
roboll/helmfile

Deploy Kubernetes Helm Charts. Contribute to roboll/helmfile development by creating an account on GitHub.

Justin Seiser avatar
Justin Seiser

I dont think that would prevent a sync/apply from running the deployment?

Balazs Varga avatar
Balazs Varga

ah. yeah sorry. I misread your message

Justin Seiser avatar
Justin Seiser

NP.

Justin Seiser avatar
Justin Seiser

condition: {{ env "ENABLE_MIGRATIONS" | default false }}

Justin Seiser avatar
Justin Seiser

would something like that work? Where it is false, unless ran with ENABLE_MIGRATIONS=true ?

Justin Seiser avatar
Justin Seiser


Environment variables can be used in most places for templating the helmfile. Currently this is supported for namenamespacevalue (in set), values and url (in repositories).

Justin Seiser avatar
Justin Seiser

i assume no

Balazs Varga avatar
Balazs Varga

can you share the error message ?

Justin Seiser avatar
Justin Seiser

Im basing my assumption off what the documentation says.

Justin Seiser avatar
Justin Seiser
Labels should allow excluding chart by default · Issue #168 · roboll/helmfile

This is probably a feature request. Labels in current form allow a nice separation of charts, but they do not change the default helmfile sync behavior, when all charts are executed. For example, i…

Justin Seiser avatar
Justin Seiser

I assume this means, we cant, not run these jobs when someone does a sync

2021-02-04

Igor Urazov avatar
Igor Urazov

Chart incubator/raw was marked as deprecated, part of helm charts repository deprecation (https://github.com/helm/charts/tree/master/incubator/raw). Is there a new home/repo for it or anyone step up for maintaining it? Or is there a better way define resources via values? I’m using this chart in helmfile to define CRs, so they nicely coupled with other releases, for example cert-manager deployment and ClusterIssuer defined in single helmfile.

helm/charts

(OBSOLETE) Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

Andrew Nazarov avatar
Andrew Nazarov

We created our our chart for cluster issuer and made its release dependent on a cert-manager release using needs:

helm/charts

(OBSOLETE) Curated applications for Kubernetes. Contribute to helm/charts development by creating an account on GitHub.

Igor Urazov avatar
Igor Urazov

Yeah, that should work. It’s just a bit more convenient to be able to define CRs via values inline in helmfile, instead of writing whole chart for this (which will be stored separately, requires versioning and other stuff). Also ClusterIssuer was just an example, there more CRs I’m defining with incubator/raw.

voron avatar
voron

AFAIR mumoshu just uses some another helm repo mirror with same deprecation warning. IDK about dedicated helm repo w/o warning. There is nothing wrong with chart itself, it’s just the whole repo deprecation.

Balazs Varga avatar
Balazs Varga

Helmfile w/o environments ? I have a task to simplify our helmfile, and let only run with a simple helmfile apply. Currently we provide cluster related data (fqdn, cert arn etcc) using env. I have few ideas:

• using vals with secret manager and grab all data from there wit ref. Can get all data from here

• using hashicorp vault inside cluster and dynamically use secrets. get cluster related info with exec…

• using randAlpha to generate missing secrets and use exists if there is any. get cluster related info with exec. Any advice what else I should check and try ?

2021-02-03

2021-02-02

Balazs Varga avatar
Balazs Varga

Can somebody provide an example for helmfile with vals and aws secret store ? Thanks

Andrew Nazarov avatar
Andrew Nazarov
variantdev/vals

Helm-like configuration values loader with support for various sources - variantdev/vals

Balazs Varga avatar
Balazs Varga

thanks

    keyboard_arrow_up