#helmfile (2018-11)

https://github.com/roboll/helmfile

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles Archive: https://archive.sweetops.com/helmfile/

2018-11-27

joshmyers avatar
joshmyers
10:32:16 AM

@joshmyers has joined the channel

2018-11-19

mumoshu avatar
mumoshu

@Shane If I read correctly, your concern is about “how to trigger helmfile sync against dir/helmfile.yaml if affecting changes are made outside of the dir/”, right?

mumoshu avatar
mumoshu

For that, I’m about to give up elegantly handling it…

Validate or just encourage people, so that all the deps to dir/helmfile.yaml are collocated inside dir/. No sub-helmfiles or values files in nested or sibling directories.

This is basically the option 3 shown in my above comment.

mumoshu avatar
mumoshu

Btw, here’s my initial attempt to add the nice-to-have feature to atlantis, that would enable atlantis to accept custom commands(or comment-command or stage in atlantis terminology)

https://github.com/cloudposse/atlantis/pull/20

apprecite any feedback. im still unfamiliar with atlantis

feat/wip: Custom stages by mumoshu · Pull Request #20 · cloudposse/atlantis

This is currently an alpha-level work of what the subject states. I have not tried to think throughout all the edge-cases, but it should work in normal cases. I want to run arbitrary helmfile comma…

Shane avatar
Shane

That would still require you syncing everything in the helmfile or at least doing apply.

Shane avatar
Shane

Although I guess that is pretty much the same thing that is happening in terraform

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@mumoshu thanks, reviewed your PR, LGTM

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if you think it’s OK, we can merge to cloudposse/atlantis

mumoshu avatar
mumoshu


That would still require you syncing everything in the helmfile or at least doing apply.

@Shane thanks for the res. yep. so in other words, you shouldn’t use helmfile [sync|apply] -l foo=bar, right?

mumoshu avatar
mumoshu

other than that, i’d also like we could run a kind of helmfile apply against the whole cluster, so that we can detect any release that aren’t defined by any helmfile.yaml.

rel: https://sweetops.slack.com/archives/CE5NGCB9Q/p1542438629036000 https://sweetops.slack.com/archives/CE5NGCB9Q/p1542439119045900

i thought i have discussed about the state thing here https://github.com/roboll/helmfile/issues/194#issuecomment-406458671

basically we wanted not to introduce an external state store at that time

Feature Request: List Helm releases filtered by labels · Issue #4639 · helm/helm

I'd want to label "helm releases" to efficiently retrieve only part of helm releases. To achieve that, I'm imagining at least one enhancement to helm list. That is to accept –sel…

mumoshu avatar
mumoshu

@Andriy Knysh (Cloud Posse) thx for the review! i’ll double-check and proceed once done

2018-11-18

monsoon.anmol.nagpal avatar
monsoon.anmol.nagpal
04:24:48 PM

@monsoon.anmol.nagpal has joined the channel

2018-11-17

tolstikov avatar
tolstikov
11:33:16 AM

@tolstikov has joined the channel

Shane avatar
Shane

One issue I see is with atlantis if I’m not mistaken it expects all configuration for a thing to be in the directory. In the helm file situation changes in a directory above helmfile could effect releases in a different location. I also see “globals” as part of a problem that I currently have as changing a global effects all environments which is overall a bad design, but I’m part of the problem

Shane avatar
Shane

With terraform it’s easy to create modules that contain those defaults, but with helm creating a new chart just to wrap a existing chart is a unpleasant undertaking.

2018-11-16

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
04:57:46 PM

@Erik Osterman (Cloud Posse) has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
04:57:46 PM

@Erik Osterman (Cloud Posse) set the channel purpose: Questions and discussion around helmfile https://github.com/roboll/helmfile

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
04:57:46 PM

@Andriy Knysh (Cloud Posse) has joined the channel

Max Moon avatar
Max Moon
04:57:46 PM

@Max Moon has joined the channel

Daren avatar
Daren
04:57:46 PM

@Daren has joined the channel

michal.matyjek avatar
michal.matyjek
04:57:47 PM

@michal.matyjek has joined the channel

mumoshu avatar
mumoshu
04:57:47 PM

@mumoshu has joined the channel

Shane avatar
Shane
04:57:47 PM

@Shane has joined the channel

alebabai avatar
alebabai
04:57:47 PM

@alebabai has joined the channel

nian avatar
nian
05:21:18 PM

@nian has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:56:29 PM

@Erik Osterman (Cloud Posse) set the channel purpose: Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles

mallen avatar
mallen
07:36:07 PM

@mallen has joined the channel

davidvasandani avatar
davidvasandani
08:58:07 PM

@davidvasandani has joined the channel

mumoshu avatar
mumoshu

hi @Erik Osterman (Cloud Posse) @Andriy Knysh (Cloud Posse) @Shane happy to chat with you here!

mumoshu avatar
mumoshu

@Shane I think I may have same feeling as yours. One thing is the sub-helmfiles feature. atlantis arrived in the landscape, making me think I have no specific use-cases for sub-helmfiles. It seems like a matter of splitting a helmfile.yaml into multiple independent ones, atlantis could sync everything or only part of helmfiles.
My main concern is all of the ways that helmfile works… so many usecases have been added that supporting it all proves painful.

mumoshu avatar
mumoshu

and sub-helmfiles introduced much confusion to users, like “how to use global variables” and “how to share variables across helmfiles”, “how to pass arguments only to a subset of sub-helmfiles” and so on

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

hi @mumoshu

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i agree that’s not an easy thing to do

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what we used, instead of having sub-helmfiles, we targeted a particular chart helmfile --file helmfile.yaml --selector namespace=kube-system,chart=prometheus-operator sync, or a group of charts by their labels

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

maybe that will work (with atlantis)?

mumoshu avatar
mumoshu

@Andriy Knysh (Cloud Posse) i believe that would almost work

mumoshu avatar
mumoshu

atlantis doesnt allow specifying arbitrary args at runtime. so how to handle helmfile args like --file helmfile.yaml --selector namespace=kube-system,chart=prometheus-operator seems the issue for me

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we can change atlantis we already forked it here https://github.com/cloudposse/atlantis and added a few new features https://github.com/cloudposse/atlantis/releases

cloudposse/atlantis

GitOps for Teams (experimental hard fork of atlantis) - cloudposse/atlantis

cloudposse/atlantis

GitOps for Teams (experimental hard fork of atlantis) - cloudposse/atlantis

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but sure, this is not a long term solution

mumoshu avatar
mumoshu

atlantis does support changing directory before running plan or apply via -d DIR flag. so i was in an impression that splitting your helmfile.yaml by what you use in selectors, and giving each helmfile an independent directory works nicely with atlantis.

mumoshu avatar
mumoshu

so that altlantis could detect and run helmfile sync against the namespace=kube-system,chart=prometheus-operator part of your former helmfile.yaml, if that was the only changed helmfile.yaml in your project.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

ah yes, change detection would be a challenge with one file, agree

mumoshu avatar
mumoshu

ya! that’s definitely a way to make it work. think im just trying to align as much as possible with current atlantis’ design, so that we don’t diverge from the upstream too far

mumoshu avatar
mumoshu


change detection would be a challenge with one file

yeah, this!

mumoshu avatar
mumoshu

at the time of implementing the most of helmfile’s feature, there was no atlantis. i wanted helmfile to detect changes and only update necessary helm releases. that is helmfile diff and helmfile apply.

mumoshu avatar
mumoshu

but i wonder, in a normal circumstance, if it would be nicer if we could detect changes more efficiently, without running helmfile diff.

atlantis basically seems to do a kind of git-diff to detect changes, with would indeed faster than helmfile diff … :slightly_smiling_face:

helmfile diff is superior though, because it allows us to detect configuration drifts.

mumoshu avatar
mumoshu

im not yet sure how we should set up our overall workflow. atlantis plan would run helmfile diff against only changed helmfile.yamls. atlantis apply would run helmfile apply against only changed helmfile.yamls. what for configuration drift detetion?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what atlantis does is it calls the API (GitHub for example) and the API returns a list of changed files

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so if helmfile supported sub-files, there would be no difference in the detection flow

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

from this point of view, it would be the easiest solution. Any changes in a sub-helmfile would trigger plan/apply

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@mumoshu what configuration drifts are you thinking of?

mumoshu avatar
mumoshu

thx for the info! never realized github api allowed that

mumoshu avatar
mumoshu

i was thinking of e.g. my colleague manually upgraded one of helm releases by hand

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

ah that one

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s not easy i guess

mumoshu avatar
mumoshu

yes i think so too

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

b/c there would be no changed files

mumoshu avatar
mumoshu

helmfile doesn’t know about helm releases not defined in helmfile.yaml, in the first place. so configuration drift detection by helmfile isn’t perfect

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(don’t allow your colleagues to do anything manually )

mumoshu avatar
mumoshu

yes, that

mumoshu avatar
mumoshu

true!

mumoshu avatar
mumoshu

one idea is that running an external tool to detect all the helm releases that aren’t defined in helmfile.yamls while beating its authors. then helmfile could do the rest with your suggested setup of helmfile sync being run whenever one of subhelmfiles changed

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

in helmfile, do you use something like state-file as Terraform does?

mumoshu avatar
mumoshu

nope. but i had an idea like that

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes, TF uses it and just compares what you have in code with the state from the state file

mumoshu avatar
mumoshu

i thought i have discussed about the state thing here https://github.com/roboll/helmfile/issues/194#issuecomment-406458671

basically we wanted not to introduce an external state store at that time

Proposal: `helmfile sync --prune` removes any charts not declared · Issue #194 · roboll/helmfile

This idea comes from the kubectl apply -f –prune where kubectl deletes any resources that aren't referenced. Helmfile should have an option for the sync command that checks what helm charts ar…

mumoshu avatar
mumoshu

ah, in that sense, helm releases are already stored in k8s as secrets or configmaps

mumoshu avatar
mumoshu

so you do have the state

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

understandable… it’s a project in itself and not easy

mumoshu avatar
mumoshu

helmfile diff renders releases locally and compare with the remote state(=helm releases stored by helm in the cluster)

mumoshu avatar
mumoshu

yeah i think so..

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so if I change a helmfile on GitHub, atlantis will detect it, then helmfile diff would do the trick?

mumoshu avatar
mumoshu

yep. it will detect any changes as long as the helm releases are listed under helmfile.yaml’s releases:

mumoshu avatar
mumoshu

but if your change was removing a single release from releases:, helmfile diff detects nothing

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so the only thing that’s missing is to support sub-helmfiles? I saw something about it in the repo. How far are you from implementing it?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


but if your change was removing a single release from releases:, helmfile diff detects nothing

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea, that’s why state file like TF’s would help here

mumoshu avatar
mumoshu

agreed

mumoshu avatar
mumoshu

ah i remember that i have once proposed helm to allow tagging helm releases

mumoshu avatar
mumoshu

so that we can tag helm releases with something like managed-by=helmfile, which allows helmfile to detect releases removed from helmfile.yml

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea, that would work too

mumoshu avatar
mumoshu
Feature Request: List Helm releases filtered by labels · Issue #4639 · helm/helm

I'd want to label "helm releases" to efficiently retrieve only part of helm releases. To achieve that, I'm imagining at least one enhancement to helm list. That is to accept –sel…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and easier than maintaining external state with all its syncing and locking

:--1:1
mumoshu avatar
mumoshu

btw the `sub-helmfiles feature is already implemented. it is just that (i think) it has many edge-cases to use properly > https://sweetops.slack.com/archives/CE5NGCB9Q/p1542436482007600

and sub-helmfiles introduced much confusion to users, like “how to use global variables” and “how to share variables across helmfiles”, “how to pass arguments only to a subset of sub-helmfiles” and so on

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

hmm, i think we never saw any of those issues

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

can you show an example of any of the edge-cases?

mumoshu avatar
mumoshu

just dumping options i have in my mind.

option 1: helmfile.yaml + sub-helmfiles + atlantis + external unmanaged helm release detector => almost works, but sub-helmfiles seems to be hard to use

option 2: independent helmfile.yaml + atlantis + external detector => how can we detect config drifts? use find . -name helmfile.yaml | xargs helmfile apply -f {} \; for atlantis apply?

option 3: basically option 1 with all the sub-helmfiles merged into the top-level one => have not thought much about this yet

mumoshu avatar
mumoshu

sure, hold on..

mumoshu avatar
mumoshu

https://github.com/roboll/helmfile/issues/388 https://github.com/roboll/helmfile/issues/361 (should they propagate to sub-helmfiles?) https://github.com/roboll/helmfile/issues/344 (use a specific selector for sub-helmfiles only)

Using helmfiles, but unable to pass Environment down to helmfile.yaml · Issue #388 · roboll/helmfile

When using {{ readFile "environments.yaml }}} — helmfiles: - ./*/helmfile.yaml Environment.Values are not passed to the globbed helmfiles. I would expect that to happen/work, in order to kee…

feat: Allow environment values to be specified on the CLI · Issue #361 · roboll/helmfile

Allow environment values to be specified on the CLI: helmfile –environment default –set foo.bar=123 Not sure what the CLI switch should be named and whether it should allow at the global level or…

Possible to set selector in parent helmfile? · Issue #344 · roboll/helmfile

I have a helmfile that pulls in another helmfile cluster1-helmfile.yaml — context: cluster1 helmfiles: - apps-helmfile.yaml Inside the second helmfile there are selectors apps-helmfile.yaml relea…

mumoshu avatar
mumoshu

i’ll add once i find more

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks, I’ll take a look

:--1:1
    keyboard_arrow_up