#helmfile (2019-08)
Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/
2019-08-01
are you only able to pull .Environment.Values into a values.yaml.gotmpl?
like something like this doesnt work
"service.beta.kubernetes.io/aws-load-balancer-ssl-cert": {{ .Values.cert.arn }}
if my values: is setup like
values:
- ../../values/kube2iam/values.yaml.gotmpl
- ../../values/kube2iam/{{ .Environment.Name }}.values.yaml
values.yaml.gotmpl contains the .Values.cert.arn and environment.values.yaml has an entry like
cert.arn: "fdjklfsj"
there is no way to reference that value from another values template
or would i need to do some {{ readFile }} magic
@ag Hey! Yeah, you need readFile magic, or leverage sub-helmfiles like https://github.com/roboll/helmfile/issues/756#issuecomment-514013382
Helmfile values and chart(release) values are completely different things
Let’s gather all the tips and tricks worth included in the documentation, by linking from various questions answered in this repo.
@ag We’re discussing about setting values from a template rendered in a custom context(=.Values)
https://github.com/roboll/helmfile/issues/745#issuecomment-510737741
releases:
- name: myapp`
chart: my-chart
valuesFrom:
- templateFile:
path: values/values-file.yaml.gotmpl
valuesFrom:
# Instead of implicitly passing helmfile's Values to the values-file.yaml.gotmpl
# Overrides it with `releaseName: myapp`
- inlineTemplate:
releaseName: `{{.Release.Name}}` #<- you'll need {{` `}} to defer rendering until the evaluation time of inlineTemplate. otherwise this fails due to that .Release.Name isn't available at loading-time of your helmfile.yaml :)
I'd love to have the functionality of #662 but I wanted to throw my use-case in here while it's topical as what i'm suggesting is probably closer to #705. We're deploying a bunch of…
2019-08-02
Hi guys, I’m having troubles with installing chart via helmfile. helfile
repositories:
- name: stable
url: <https://kubernetes-charts.storage.googleapis.com>
- name: helm
url: <utl>
username: <username>
password: <password>
helmDefaults:
tillerNamespace: kube-system
verify: true
wait: true
timeout: 600
recreatePods: true
force: true
tls: false
# The desired states of Helm releases.
#
# Helmfile runs various helm commands to converge the current state in the live cluster to the desired state defined here.
releases:
- name: consul
namespace: default
chart: helm/consul
version: 0.8.1
values:
- charts/chart-consul/values.yaml
I’m getting an error Error: failed to download "helm/consul" (hint: running helm repo update may help)
If I try to install via helm, everything works fine.
When tracing, I found out that helm tries to download tgz.pov chart file, while in index.yaml there is only tgz one.
Any ideas?
Interesting. How a successful helm command look like?
Can you compare it with the helm command being run by Helmfile if you run helmfile --log-level=debug
?
successful via helm - https://gist.github.com/djackyn/725ac0638c33c89a3cb903d03707f43c failure via helmfile - https://gist.github.com/djackyn/725ac0638c33c89a3cb903d03707f43c
will try to run same command as helmfile now
Okay, have the same problems. Seems like a bug in the helm, will check. Thanks for the direction
2019-08-05
If I do the following:
- deploy a chart (deployment) with
helmfile
- edit that deployment with
kubectl
should a helmfile diff
show a change?
Because when I run helmfile diff
with a manual edited deployment, it shows no changes. This is with helmfile
0.80.2 and tested with helm
2.13.1, 2.14.1 and 2.14.3
yes, it should, however helmfile
just calls out to helm-diff
https://github.com/databus23/helm-diff
A helm plugin that shows a diff explaing what a helm upgrade would change - databus23/helm-diff
as far as i understand, helm diff
doesn’t show diff for actual k8s resources
helm persists the desired state of k8s manifests per each helm release in a release configmap/secret in tiller namespace
helm-diff
shows diffs from that, not from the current state of your cluster
aha, wow, didn’t know that.
so the downside of helm-diff
is that you cant use it to detect “skews” due to manual changes
A helm plugin that shows a diff explaing what a helm upgrade would change - databus23/helm-diff
i wonder if kubectl diff
is useful in that case though
maybe it can be used like this:
helm get $YOUR_RELEASE_NAME --template {{.Release.Manifest}} | kubectl diff -f -
Yes, I recently encountered this behavior. helm diff
diffs the new desired state against the old desired state. It is blind to manual changes in the state. I didn’t even know kubectl diff
existed.
Thank ya’ll, that’s very helpful. I was not aware of kubectl diff
either.
Guess I need to keep some hands out of the kubectl
pot in the future
darn, can’t use kubectl diff with k8s 1.11 it seems
2019-08-06
Hi….Is ordering possible in helmfile for single helmfile.yaml….say i have 10 helm charts in a single helmfile and i want to run run them in the order they are defined is it possible?
Unforunately, no.
But we have a feature request to add a declarative way to manage ordering https://github.com/roboll/helmfile/issues/715 Pls feel free to +1 if you think it helps your use-case!
I know Helmfile can already do release dependencies by ordering releases manually via naming conventions for helmfiles or limiting concurrency. Ability to explicitly declare dependencies would be a…
Without labels it is working with –concurrency=1 but with labels it is not
use helmfiles:
is how im doing it
what you are specifying in helmfile
can you give example
helmfiles:
- path: releases/external-dns/helmfile.yaml.gotmpl
values:
- versions.yaml
- path: releases/prometheus-operator/helmfile.yaml.gotmpl
values:
- versions.yaml
- path: releases/prometheus-pushgateway/helmfile.yaml.gotmpl
values:
- versions.yaml
will deploy them in that order
but i want to use only one helmfile i dont want to use multiple helmfiles….ok if this i do is it possible to have helmfile specific to envvironment
like i have three environments i want each corresponding helmfile for them
not sure with a single helmfile
i want that the helmfile with –environment set should work
can i have helmfile per environment
2019-08-07
Thanks….Hi is it possible to read the values in gotmpl file from the configmap in k8s?
interesting idea! but not possible now.
it would worth a dedicated feature request. would you mind writing one?
maybe we can enhance the syntax being discussed/proposed in https://github.com/roboll/helmfile/issues/745#issuecomment-510737741 to also accept k8s secrets as sources.
I'd love to have the functionality of #662 but I wanted to throw my use-case in here while it's topical as what i'm suggesting is probably closer to #705. We're deploying a bunch of…
in the meantime, use {{ exec "sh" (list "-c", "kubectl get secret .....") }}
as a workaround
@mumoshu so i have https://github.com/roboll/helmfile/pull/790 open but looking at the environmentSecrets, i’m thinking a package scoped cache might be better so that exec contexts don’t need to be threaded everywhere
Related to #782 and #444 Allows concurrent decryption of different secrets files Caches decrypted secrets by original file path and returns decrypted results from memory Secrets being run through …
package scoped cache sounds like a good starting point! i do think that we’ll eventually need per-HelmState(or something nearby) cache but it isn’t needed as long as we don’t use helmfile as a go library
Related to #782 and #444 Allows concurrent decryption of different secrets files Caches decrypted secrets by original file path and returns decrypted results from memory Secrets being run through …
and thanks for the pr i’ll take a look soon
hm yeah. i’ll see if moving the cache up the stack makes more sense
2019-08-10
When running helmfile deps
, how to scope it to a single release?
I tried helmfile --selector name=my-app-1 deps
, but that removes everything except my-app-1
from helmfile.lock
Hey! There seems no way to scope deps to a single release without breaking helmfile.lock currently
Just curious, why you need to do so?
Pretty simple, I have an observe namespace with grafana, elasticsearch, prometheus etc etc etc. Maybe 20 or so charts. Upgrading everything at once is considerable change, particularly with complex ones like jenkins
or prometheus
I think it worth a feature request to add helmfile an optional argument for which chart to receive updates
like helmfile deps one_of_your_chart_name_here
That’d be lovely
I can put that in tomorrow
thanks!
no, thank you!
2019-08-14
I have a chart, namely prometheus-msteams that reads in a file via Files.Get. I’d like to override the contents of that file, i.e what’s in card.tmp. Is there some way of doing so when installing the chart via helmfile?
I can of course modify the chart so that it has the option to specify what configmap to use, but if I don’t have to because helmfile already can do this for me, I will be eternally happy
https://github.com/bzon/prometheus-msteams/tree/master/chart/prometheus-msteams
A lightweight Go Server that sends Prometheus Alert Manager notifications to Microsoft Teams - bzon/prometheus-msteams
You have an interesting use-case! Unfortunately Helmfile doesn’t have a feature that helps it today, but worth a feature request to helmfile and helm-x
A lightweight Go Server that sends Prometheus Alert Manager notifications to Microsoft Teams - bzon/prometheus-msteams
one of featuers of helm-x is to add adhoc changes to the chart before installing it(like patches resources
so enhancing it to replace chart files (perhaps like helm x upgrade --install --replace-chart-file card.tmpl=yourown.card.tmpl
?
and exposing the helm-x option to helmfile.yaml makes sense(perhaps with replaceFiles: {"card.tmpl": "yourown.card.tmpl
}
?
@mumoshu do you know when you’re anticipating a new helmfile release?
(I believe a new release is cut pretty much for every merge to master)
…or it used to be
I see 0.81.0 is composed of multiple PRs
basically yes
i thought it would be better to cut a single version containing 3 prs from @tgroth this time
Yea, I think the main intent is just to keep releases flowing regularly so features (even if unintentionally broken and not fully baked) are readily available for testing.
definitely
there was a bit of a joke at work last week that @mumoshu might be dead since it had been a week since the last release
haha, thank god not! @mumoshu get some time off finally?
not really i was just busy handling reviews, questions and writing a few complementary go libraries for helmfile!
hopefully i can take some time off after the upcoming helm summit
@mumoshu are you going to attend Helm Summit, btw?
@Andrew Nazarov yes will you?
I would like to go, but don’t know yet. It contradicts my other plans and commitments. ~It’s too bad, there will be no talk about helmfile. At least I haven’t seen anything related to it in the schedule.~
and i see an another helmfile user is giving a talk, too https://helmsummit2019.sched.com/event/S8tf/helm-for-anybody-combining-chatops-and-declarative-configurations-jonathan-striebel-scalable-minds-gmbh
View more about this event at Helm Summit 2019
2019-08-15
2019-08-16
#helmfile I am thinking of the following structure
*modules* readme.md sites
./modules:
postgresql prometheus
./modules/postgresql:
postgresql.yaml
./modules/prometheus:
prometheus.yaml
./sites:
minikube
./sites/minikube/local:
helmfile.yaml postgresql_values.yaml prometheus_values.yaml
./sites/infra/dev:
helmfile.yaml
./sites/infra/staging:
Where the sites folder would have the components that needs to be installed in an environment, and the modules holding the common modules. I saw the helmfiles.d/ and values/ separation, here it would give us better view on what is going on per site. What are the problems with the above structure? Since nothing much has been actually done, it would be very easy to change at this point. Thanks
@Shiva Gopal I don’t see any use of “environments”
just want to make sure that’s on your radar
@Erik Osterman (Cloud Posse) - the structure is ./sites/customer/{env,namespace} - At least that’s the idea
2019-08-20
#helmfile - I want to run https://github.com/helm/charts/tree/master/stable/grafana repeatedly, with the adminPassword set in values.yaml file
I have tried different combinations, the secret is getting generated each time. What am I doing wrong? Thanks @mumoshu
@Shiva Gopal Hey! You need it included in the grafana.yaml
:
releases:
- name: grafana
namespace: grafana
chart: stable/grafana
values:
- rbac:
create: false
- ./grafana_values.yaml
helmfiles:
- path: "../../modules/grafana/grafana.yaml"
values:
- "./grafana_values.yaml"
this isn’t valid as helmfiles[].values
is for setting helmfile values, rather than release values.
@mumoshu - got it. if I want env specific values, is there a way to put in say sites/infra/dev/grafana_values.yaml ?
Yeah that’s one of the good ways! You may also use {{.Environment.Name}}
one more thing… is there any way to the use the values file only if present? meaning if there is no additional config required, the users need not have the values file.
you can set the missing file handler to be warning or even debug
missingFileHandler parameter under releases. there’s a comment for it in the main README.md
Also is there a way to say use the directory from where the program was launched and use that as a parameter? It will save me from setting one ENV var
referencing PWD might do it
i haven’t tried it but it sounds like it might work
{{ requiredEnv "PLATFORM_ENV" }}
syntax
no it’s a standard env var that already has your current directory
requiredEnv should get it (i think that’s a sprig or gotemplate function)
@tgroth - thanks it was user error Used PWD directly instead of “PWD”
2019-08-22
It’s too bad we cannot see the full history here. I wanted to recall something by checking out old messages, but got hit by this limitation.
@Andrew Nazarov you can use https://archive.sweetops.com/helmfile/
SweetOps is a collaborative DevOps community. We welcome engineers from around the world of all skill levels, backgrounds, and experience to join us! This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build sweet infrastructure.
One of the most important principles to being an engineer, is being able to admit when you are wrong. Well folks. I was wrong. Some of you…
by our very own @stobiewankenobi
2019-08-23
Hey ya’ll, I feel like I’m misunderstanding something about environments feature in helmfile and was hoping someone could point me in the right direction. Trying to do this but my environment values aren’t getting passed down to my sub helmfiles
environments:
beta:
values:
- externaldns:
installed: false
namespaces:
installed: false
helmfiles:
- path: releases/external-dns
values:
- installed: {{ Environment.Values.externaldns.installed }}
- path: releases/namespaces
values:
- installed: {{ Environments.Values.namespaces.installed }}
@sung kang Hey!
Could you try running it with debug logs like helmfile --log-level debug sync
so that you can see how it is rendering your helmfile.yaml?
Anyways, I think {{ Environment.Values.externaldns.installed }}
should be {{ .Environment.Values.externaldns.installed }}
. It needs .
before Environment
I can try to reproduce it if you could share the actual files you’re having the trouble
At least for now, I don’t see any issue in the example you’ve shared. It should work
Turns out the issue I was having had to do with boolean values
When I use the default function it doesn’t handle it correctly, switching to {{ get “” “”}} function addressed my issue
Am I missing something here entirely?
I know environments aren’t shared down to sub-helmfiles but I’d thought I could expose them through the state this way
2019-08-24
2019-08-26
Hello, anyone around to answer a question?
@Erik Osterman (Cloud Posse) / @mumoshu / others with terraform experience. Has anyone studied the pros/cons of using terraform helm provider vs helmfile? I am obviously biased towards helmfile (don’t have much experience with the tf helm provider though) and would like to present any known items of one vs the other. Thanks
First, helmfile
was born long before the terraform-helm-provider
existed. Before that time, terraform wasn’t really an option.
Since then, the terraform-helm-provider
came out. It was tempting at first, until we considered all that we would lose. In terraform 0.11, the templating was just not robust enough. It just handled simple replacement of variables. When working with values.yaml
, it’s important to be able to deal with more complicated maps/lists, etc. Sometimes an entire section should be conditionally disabled.
Then terraform 0.12 came out. The gap closed. Now robust templating is possible in templates. It’s an appealing alternative, especially when everything else is in terraform. For example, many of our helm releases have AWS backing services. It would be nice to deploy the RDS database with terraform and then the helm chart immediately there after.
This is possible today.
However, doing so would give up the helm diff
functionality. That’s a deal breaker in my book.
Basically, without helm diff
, you have zero clue what’s going to happen.
There’s an option issue for this on terraform-helm-provider
Unless I'm wrong, this provider will not detect changes if neither the chart version, nor the values changed. However, one can have made manual changes using the k8s api (update or delete objec…
(someone even suggested making it work more like helmfile
! Props to @ausov
.)
The issue has been open for over 1 year now with no movement. Not holding my breath.
Also, while I have the stage, want to call out a project @mumoshu is working on
Kubernetes operator that continuously syncs any set of Chart/Kustomize/Manifest fetched from S3/Git/GCS to your cluster - mumoshu/helmfile-operator
Also, there’s https://github.com/Kubedex/terraform-helmfile
Run Helmfile from Terraform. Contribute to Kubedex/terraform-helmfile development by creating an account on GitHub.
Not sure how I feel about that implementation though. We run terraform
in a container, and this expects to run helmfile
in a container, which causes a catch-22
@Shiva Gopal was this helpful? anything else you’re curious about…
@Erik Osterman (Cloud Posse) - this was super helpful. Let me digest the above and come back with any questions that I have.
(I jumping on a call but will get back to you today)
Thanks @Erik Osterman (Cloud Posse)
@devoperandi - welcome , loved your blog and still using git2consul
@Shiva Gopal Thanks man. I have utterly failed to blog over the last year. Need to get back to it.
2019-08-27
2019-08-28
Hi, I have a set of helmfiles in helmfiles.d and a values.gotmpl created by other people. I get an error complaining about .Values.clusterRegion not being set, even though it looks to me that it should be set. Is there a way to render the .gotmpl-file to see that that is ok and/or see what helmfile thinks should be in the .Values variable ?
@Daniel Johansson Hey! Unfortunately helmfile doesn’t have support for that today.
I can help debuggging tho
even though it looks to me that it should be set.
Would you mind sharing the snippet of your values.gotmpl or relevant files so that I can explain why it should/not work?
I’ve tried with: helmfile –log-level debug template and also adding a dummy parameter with {{ .Values | toYaml }} But nothing showed presumed contents of the .gotmpl
2019-08-29
@Robert has joined the channel