#helmfile (2018-11)

@Erik Osterman (Cloud Posse) has joined the channel

set the channel description: Questions and discussion around helmfile https://github.com/roboll/helmfile

@Andriy Knysh (Cloud Posse) has joined the channel

@Max Moon has joined the channel

@Daren has joined the channel

@michal.matyjek has joined the channel

@mumoshu has joined the channel

@Shane has joined the channel

@alebabai has joined the channel

@nian has joined the channel

set the channel description: Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles

@mallen has joined the channel

@davidvasandani has joined the channel

hi @Erik Osterman (Cloud Posse) @Andriy Knysh (Cloud Posse) @Shane happy to chat with you here!

@Shane I think I may have same feeling as yours. One thing is the sub-helmfiles feature. atlantis arrived in the landscape, making me think I have no specific use-cases for sub-helmfiles. It seems like a matter of splitting a helmfile.yaml into multiple independent ones, atlantis could sync everything or only part of helmfiles.
My main concern is all of the ways that helmfile works… so many usecases have been added that supporting it all proves painful.

and sub-helmfiles introduced much confusion to users, like “how to use global variables” and “how to share variables across helmfiles”, “how to pass arguments only to a subset of sub-helmfiles” and so on

hi @mumoshu

i agree that’s not an easy thing to do

what we used, instead of having sub-helmfiles, we targeted a particular chart helmfile --file helmfile.yaml --selector namespace=kube-system,chart=prometheus-operator sync, or a group of charts by their labels

maybe that will work (with atlantis)?

@Andriy Knysh (Cloud Posse) i believe that would almost work

atlantis doesnt allow specifying arbitrary args at runtime. so how to handle helmfile args like --file helmfile.yaml --selector namespace=kube-system,chart=prometheus-operator seems the issue for me

we can change atlantis we already forked it here https://github.com/cloudposse/atlantis and added a few new features https://github.com/cloudposse/atlantis/releases

but sure, this is not a long term solution

atlantis does support changing directory before running plan or apply via -d DIR flag. so i was in an impression that splitting your helmfile.yaml by what you use in selectors, and giving each helmfile an independent directory works nicely with atlantis.

so that altlantis could detect and run helmfile sync against the namespace=kube-system,chart=prometheus-operator part of your former helmfile.yaml, if that was the only changed helmfile.yaml in your project.

ah yes, change detection would be a challenge with one file, agree

ya! that’s definitely a way to make it work. think im just trying to align as much as possible with current atlantis’ design, so that we don’t diverge from the upstream too far

change detection would be a challenge with one file

yeah, this!

at the time of implementing the most of helmfile’s feature, there was no atlantis. i wanted helmfile to detect changes and only update necessary helm releases. that is helmfile diff and helmfile apply.

but i wonder, in a normal circumstance, if it would be nicer if we could detect changes more efficiently, without running helmfile diff.

atlantis basically seems to do a kind of git-diff to detect changes, with would indeed faster than helmfile diff … :slightly_smiling_face:

helmfile diff is superior though, because it allows us to detect configuration drifts.

im not yet sure how we should set up our overall workflow. atlantis plan would run helmfile diff against only changed helmfile.yamls. atlantis apply would run helmfile apply against only changed helmfile.yamls. what for configuration drift detetion?

what atlantis does is it calls the API (GitHub for example) and the API returns a list of changed files

so if helmfile supported sub-files, there would be no difference in the detection flow

from this point of view, it would be the easiest solution. Any changes in a sub-helmfile would trigger plan/apply

@mumoshu what configuration drifts are you thinking of?

thx for the info! never realized github api allowed that

i was thinking of e.g. my colleague manually upgraded one of helm releases by hand

ah that one

it’s not easy i guess

yes i think so too

b/c there would be no changed files

helmfile doesn’t know about helm releases not defined in helmfile.yaml, in the first place. so configuration drift detection by helmfile isn’t perfect

(don’t allow your colleagues to do anything manually )

yes, that

true!

one idea is that running an external tool to detect all the helm releases that aren’t defined in helmfile.yamls while beating its authors. then helmfile could do the rest with your suggested setup of helmfile sync being run whenever one of subhelmfiles changed

in helmfile, do you use something like state-file as Terraform does?

nope. but i had an idea like that

yes, TF uses it and just compares what you have in code with the state from the state file

i thought i have discussed about the state thing here https://github.com/roboll/helmfile/issues/194#issuecomment-406458671

basically we wanted not to introduce an external state store at that time

ah, in that sense, helm releases are already stored in k8s as secrets or configmaps

so you do have the state

understandable… it’s a project in itself and not easy

helmfile diff renders releases locally and compare with the remote state(=helm releases stored by helm in the cluster)

yeah i think so..

so if I change a helmfile on GitHub, atlantis will detect it, then helmfile diff would do the trick?

yep. it will detect any changes as long as the helm releases are listed under helmfile.yaml’s releases:

but if your change was removing a single release from releases:, helmfile diff detects nothing

so the only thing that’s missing is to support sub-helmfiles? I saw something about it in the repo. How far are you from implementing it?

but if your change was removing a single release from releases:, helmfile diff detects nothing

yea, that’s why state file like TF’s would help here

agreed

ah i remember that i have once proposed helm to allow tagging helm releases

so that we can tag helm releases with something like managed-by=helmfile, which allows helmfile to detect releases removed from helmfile.yml

yea, that would work too

https://github.com/helm/helm/issues/4639

and easier than maintaining external state with all its syncing and locking

btw the `sub-helmfiles feature is already implemented. it is just that (i think) it has many edge-cases to use properly > https://sweetops.slack.com/archives/CE5NGCB9Q/p1542436482007600

hmm, i think we never saw any of those issues

can you show an example of any of the edge-cases?

just dumping options i have in my mind.

option 1: helmfile.yaml + sub-helmfiles + atlantis + external unmanaged helm release detector => almost works, but sub-helmfiles seems to be hard to use

option 2: independent helmfile.yaml + atlantis + external detector => how can we detect config drifts? use find . -name helmfile.yaml | xargs helmfile apply -f {} \; for atlantis apply?

option 3: basically option 1 with all the sub-helmfiles merged into the top-level one => have not thought much about this yet

sure, hold on..

https://github.com/roboll/helmfile/issues/388 https://github.com/roboll/helmfile/issues/361 (should they propagate to sub-helmfiles?) https://github.com/roboll/helmfile/issues/344 (use a specific selector for sub-helmfiles only)

i’ll add once i find more

thanks, I’ll take a look

@tolstikov has joined the channel

One issue I see is with atlantis if I’m not mistaken it expects all configuration for a thing to be in the directory. In the helm file situation changes in a directory above helmfile could effect releases in a different location. I also see “globals” as part of a problem that I currently have as changing a global effects all environments which is overall a bad design, but I’m part of the problem

With terraform it’s easy to create modules that contain those defaults, but with helm creating a new chart just to wrap a existing chart is a unpleasant undertaking.

@monsoon.anmol.nagpal has joined the channel

@Shane If I read correctly, your concern is about “how to trigger helmfile sync against dir/helmfile.yaml if affecting changes are made outside of the dir/”, right?

For that, I’m about to give up elegantly handling it…

Validate or just encourage people, so that all the deps to dir/helmfile.yaml are collocated inside dir/. No sub-helmfiles or values files in nested or sibling directories.

This is basically the option 3 shown in my above comment.

Btw, here’s my initial attempt to add the nice-to-have feature to atlantis, that would enable atlantis to accept custom commands(or comment-command or stage in atlantis terminology)

https://github.com/cloudposse/atlantis/pull/20

apprecite any feedback. im still unfamiliar with atlantis

That would still require you syncing everything in the helmfile or at least doing apply.

Although I guess that is pretty much the same thing that is happening in terraform

@mumoshu thanks, reviewed your PR, LGTM

if you think it’s OK, we can merge to cloudposse/atlantis

That would still require you syncing everything in the helmfile or at least doing apply.

@Shane thanks for the res. yep. so in other words, you shouldn’t use helmfile [sync|apply] -l foo=bar, right?

other than that, i’d also like we could run a kind of helmfile apply against the whole cluster, so that we can detect any release that aren’t defined by any helmfile.yaml.

rel: https://sweetops.slack.com/archives/CE5NGCB9Q/p1542438629036000 https://sweetops.slack.com/archives/CE5NGCB9Q/p1542439119045900

@Andriy Knysh (Cloud Posse) thx for the review! i’ll double-check and proceed once done

@joshmyers has joined the channel