#helmfile (2021-02)
Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/
2021-02-02
Can somebody provide an example for helmfile with vals and aws secret store ? Thanks
You might want to consult this https://github.com/variantdev/vals#aws-secrets-manager
thanks
2021-02-03
2021-02-04
Chart incubator/raw was marked as deprecated, part of helm charts repository deprecation (https://github.com/helm/charts/tree/master/incubator/raw). Is there a new home/repo for it or anyone step up for maintaining it? Or is there a better way define resources via values? I’m using this chart in helmfile to define CRs, so they nicely coupled with other releases, for example cert-manager deployment and ClusterIssuer defined in single helmfile.
We created our our chart for cluster issuer and made its release dependent on a cert-manager release using needs:
Yeah, that should work. It’s just a bit more convenient to be able to define CRs via values inline in helmfile, instead of writing whole chart for this (which will be stored separately, requires versioning and other stuff). Also ClusterIssuer was just an example, there more CRs I’m defining with incubator/raw.
AFAIR mumoshu just uses some another helm repo mirror with same deprecation warning. IDK about dedicated helm repo w/o warning. There is nothing wrong with chart itself, it’s just the whole repo deprecation.
Helmfile w/o environments ? I have a task to simplify our helmfile, and let only run with a simple helmfile apply. Currently we provide cluster related data (fqdn, cert arn etcc) using env. I have few ideas:
• using vals with secret manager and grab all data from there wit ref. Can get all data from here
• using hashicorp vault inside cluster and dynamically use secrets. get cluster related info with exec…
• using randAlpha to generate missing secrets and use exists if there is any. get cluster related info with exec. Any advice what else I should check and try ?
2021-02-05
Is it possible, to mark a release so that it only runs when a --selector name=whatever is applied? wanting to do https://github.com/roboll/helmfile/tree/master/examples#managing-oneshot-jobs-with-helmfile this, but our CI runs helmfile apply on changes, and I really only want those jobs to run when explicitly told to do so.
we use this if we would like to install something only on dev:
installed: {{ eq .Environment.Name "default" | toYaml }}
I dont think that would prevent a sync/apply from running the deployment?
ah. yeah sorry. I misread your message
NP.
condition: {{ env "ENABLE_MIGRATIONS" | default false }}
would something like that work? Where it is false, unless ran with ENABLE_MIGRATIONS=true ?
Environment variables can be used in most places for templating the helmfile. Currently this is supported for name, namespace, value (in set), values and url (in repositories).
i assume no
can you share the error message ?
Im basing my assumption off what the documentation says.
I assume this means, we cant, not run these jobs when someone does a sync
2021-02-09
2021-02-11
Hi all
Is it a possible to share files that later could be used as bases in helmfile.yaml
For instance put them in Git or S3 or HTTP
And then fetch them
I’ve tried to use go-getter style path, but obtained an error
not sure what you mean by bases
Ah. I haven’t used bases before, so I’m sorry I can’t help there.
What specific problem are you trying to solve? the helmfiles: section lets you pull from external sources, we use that a lot to compose larger patterns out of smaller modular helmfiles
For instance, I want to share repoaitories betwenn dozens of helmfiles in different git repositories.
Now, we just copy paste this block?
I agree that it makes sense that bases: should support the same path: functionality as is found in helmfiles:
Is it possible to set helmDefaults in environments.yaml?
I’m looking to apply a global --wait=true , but i execute individual helmfiles in my environment and don’t want to add a line to every file i’d rather add it in the exec of helmfile
is it maybe possible using the --args flag in helmfile ?
I see https://github.com/roboll/helmfile/issues/64 is still open
@mumoshu hey if you have a moment could you give a review to https://github.com/roboll/helmfile/pull/1678 ?
I also left a comment on the above issue giving some context
I’ve added it to my todo list for this weekend thanks for pr!
Np! There is a test failing because the Wait method isn’t found inapp_test. I’m not that familiar with Go so any help you could provide on that would be greatly appreciated
i believe you need to add the Wait func to this struct https://github.com/roboll/helmfile/blob/2a716400959dbbeac49470be5988cf81d44b9512/pkg/app/app_test.go#L2300
ty, had to add that and the function below
tests are now passing!
Hey @mumoshu did you ever get a chance to check it out? no problem if not, just happy to answer questions if so
thinking since its so similar I’ll probably handle the issue for helm 3.5 new --wait-for-jobs feature
but want to make sure this pattern is OK before doing that
thats a bummer. just tried using --args "--wait", and it throws errors because it applies to every helm command like helm list
2021-02-12
hello all, I would like to add the Environment.Name as label to all alerts so alter I could setup a rule based on that. If I do like this helmfile does not catches it and in configmap I see without any changes.
I am seeking Helmfile advice:
We are deploying some Helm charts that are infra related. At the moment we have only one cloud provider and use environments called stage and prod A new cloud provider will be added next and we want to install the same set of helm charts there as well. The new cloud provider has some values that are only relevant to him.
How ca we best support that with helm? I see this two options:
- Add more envs eg. stage-aws, stage-azr, prod-aws, prod-azr and so on.
- Continue using stage/prod but move to cloud provider specific stuff to selectors. And then do if selector aws do aws specific and so on Do you see any addition option how to accomplish this?
Environments should be a better choice
- You can add cloud-specific values like
production: trueorcloud: azureand conditionally setinstalledoption to releases to prevent them to be installed on the wrong stage/cloud. - You can define different engines/config to cloud specific charts like
external-dnsorexternal-secrets
Hello Andrey,
should I set key value cloud: azure and then test it?
Its not clear to me, can you maybe give an example?
For example, if you’ll add production: true or production:false you can do this trick with some of your releases:
- name: datadog
<<: *default
installed: {{ eq .Environment.Values.production true | toYaml }}
chart: datadog/datadog
version: 2.8.1
As you can see it here, we’re installing datadog to production environments only.
You should take a look at how adobe does it with helmfile: https://medium.com/adobetech/how-to-set-up-a-kubernetes-clusters-with-helmfile-4ab7866fc2fc Would give you some ideas.
@Christian are you using templates in your helmfiles? Were you able to use bases like environments.yaml and then use Environment.Values.something with templates?
Yup. I generally have this format for almost all my helmfiles.
bases:
- environments.yaml
---
repositories:
- name: autoscaler
url: <https://kubernetes.github.io/autoscaler>
releases:
- name: cluster-autoscaler
namespace: kube-system
chart: autoscaler/cluster-autoscaler
labels:
id: "cluster-autoscaler"
chart: "cluster-autoscaler"
namespace: "kube-system"
tier: "base"
version: {{ .Values.chartVersion | quote }}
values:
...
2021-02-13
2021-02-14
2021-02-15
2021-02-16
Hey All, have a weird error. I’m providing a values file via a helmfile release to a helm chart. I’m providing it via a variable containing the full file path (the file is generated at run-time). The path I’m providing should be absolutely but is erroring because it doesn’t exist in the relative .
values file matching "/tmp/manage-secrets-IhNjd4/combined_secrets.yaml " does not exist in "."
Has anyone ever seen anything like this?
removing quotes from the variable worked
well, it actually didn’t. it got rid of the above error, but helmfile simply does not process the file and the values are not available to the helm chart
2021-02-22
perhaps this is not a helmfile question, but i think it might be? i know there’s https://github.com/hashicorp/terraform-provider-aws/issues/16542#issuecomment-782606500 which is making nice progress but looking at the pr there’s something that’s missing for me.
i want to run coredns using a nodeSelector and coredns comes as a preinstalled Deployment with eks and/or an addon. i really don’t want to overwrite the coredns deployment with my own deployment.. i’d rather just patch the existing deployment with a nodeSelector. i don’t know any simple way to do this with helmfile?
any feedback on how people solve this sort of scenario?
@joey Hey! I tend to just “import” those existing resource beforehand and use helmfile for management afterwards
https://stackoverflow.com/a/62528643 https://github.com/roboll/helmfile/issues/1281
Does it help if Helmfile had ability to auto-annotate/label the existing resource so that it automagically imports to helmfile?
It should look like built-in version of: https://github.com/roboll/helmfile/issues/1281#issuecomment-636494016
2021-02-23
I was wondering how a “selector” and “needs” works together. I have one “mongodb” chart and 2 “apps” that depend on it, Apps are labeled aith the selector “app=app1/2”. The apps have “need” setting to the mongo chart. When I use a selector to install app1, does it also install mongodb?
Yep! I’m doing something similar and it’s working well for me.
awesome!
Well, this is so not working for me do you have an example? the releases ther are “needed” are never installed
Guess i’m not the only one https://github.com/roboll/helmfile/issues/1692
2021-02-24
is there a standard way of “injecting” a template into helm charts before it’s parsed with helmfile? i started playing with hooks looking for the right way but the working dir doesn’t seem to be current dir. I can think of a number of cases where I might want to add a CRD or something to a chart but not want to have to keep up our own custom downstream version of the chart just to keep that one file in there. I see that there is a way to get kustomize to run with the chart but it would be nice to use all the _helpers in the chart i’m working on.
also, I see a lot about tiller in the docs. i didn’t start using helm until 3 was around. is there a reason to use tiller with helmfile besides still being on helm 2?
no reason to use tiller now
@Evan Pitstick your question is covered at https://github.com/roboll/helmfile/issues/1157. personally i’m using incubator/raw chart for this
is anyone successfully using the remote environment values from 1296? i’m on helmfile v0.138.4 and it just treats the URLs as a local path
Hmm useful feature, thnx for pointing it out Should it only fetch from git or from other sources as well ?
theoretically it should work with anything supported by go-getter, but I haven’t got it to work at all. :confused: haven’t had time to do any digging yet. have to deliver this today so I just used an exec template call to pull the git repo for now
2021-02-25
Hey has anyone made use of jsonPatches in helmfile? I am trying a very simple use-case for them when running a release against a directory of manifest files and I cannot seem to get it to work. A strategic merge patch works just fine but I wanted the ability to target and apply specifically
I have this manifest:
---
2 apiVersion: rbac.authorization.k8s.io/v1
3 kind: ClusterRoleBinding
4 metadata:
5 name: cluster-readonly-all
6 annotations:
7 meta.helm.sh/release-name: kubernetes-manifests
8 meta.helm.sh/release-namespace: default
9 labels:
10 app.kubernetes.io/managed-by: Helm
11 app.kubernetes.io/name: kubernetes-manifests
12 roleRef:
13 apiGroup: rbac.authorization.k8s.io
14 kind: ClusterRole
15 name: cluster-readonly-all
16 subjects:
17 - kind: Group
18 name: system:authenticated
19 apiGroup: rbac.authorization.k8s.io
and then this in my helmfile
jsonPatches:
- target:
version: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
name: cluster-readonly-all
patch:
- op: add
path: "/metadata/labels/cluster"
value: "{{ $cluster }}"
- op: add
path: "/metadata/labels/realm"
value: "{{ $realm }}"
- op: add
path: "/metadata/labels/region"
value: "{{ $region }}"
I see the patches getting generated in what looks to be a correct way from the output of --debug
generated and using kustomization.yaml:
kind: ""
apiversion: ""
resources:
- templates/ClusterRole-ReadOnly.yaml
- templates/ClusterRoleBinding-ReadOnly.yaml
patchesJson6902:
- target:
kind: ClusterRole
name: cluster-readonly-all
namespace: default
version: rbac.authorization.k8s.io~1v1
path: jsonpatches/patch.0.yaml
- target:
kind: ClusterRoleBinding
name: cluster-readonly-all
version: rbac.authorization.k8s.io/v1
path: jsonpatches/patch.1.yaml
This is incredibly frustrating. I cannot get jsonPatches to pick up the target or properly modify the manifest being ingested. strategicMergePatch works, but it fails under my go-templating which iterates over many regions, the metadata/labels only get the region of the first iterated release