#helmfile (2019-10)

Seems adding --skip-deps seems to resolve this

@Martin Devlin Hey!

You seem to have found the answer but yeah, i’d recommend --skip-deps or just not manage the repo from helmfile(just omit it from repositories:)

Now add your Azure Container Registry Helm chart repository to your Helm client using the az acr helm repo add command. This command gets an authentication token for your Azure container registry that is used by the Helm client

After reading this I think there’s no static “password” that can be set in helmfile.yaml’s repositories[].password in this scenario.

This command gets an authentication token for your Azure container registry sounds like it is generating a temporary, short life token that should be regenerated each time you run helm/helmfile

@Martin Devlin If you got some time, I’d greatly appreciate it if you could submit a PR to add some guidnce to acr as a helm repo in the context of helmfile(perhaps a few lines in README.md would suffice

@mumoshu Thanks for the reply. “sounds like it is generating a temporary, short life token that should be regenerated each time you run helm/helmfile”. The az acr helm repo add command adds a JWT token to ~/.helm/repository/repositories.yaml

This works fine with helm commands, but not helmfile, which is a little frustrating as it’s a neat way to avoid managing secrets. As I say, --skip-deps makes the problem go away so we’re using that for now.

I can make a note of that in README.md as you requested

https://github.com/roboll/helmfile/pull/889

theoretically, your approach should work. might you try running helmfile --log-level debug apply just to identify root-cause

Thanks for the tip.

Seems like the helmfile apply diff step ignores the args in my helmfile:

exec: helm diff upgrade --reset-values --allow-unreleased account-service xxx/account-service --tiller-namespace acc --kube-context acc --values /var/folders/9h/1pr987354xj3n7y3tbcy5f2d606v15/T/values267183981 --detailed-exitcode

Whereas helmfile diff does not:

exec: helm diff upgrade --reset-values --allow-unreleased account-service xxx/account-service --tiller-namespace acc --kube-context acc --values /var/folders/9h/1pr987354xj3n7y3tbcy5f2d606v15/T/values801725413 --devel

Using helmfile sync instead of helmfile apply ended up solving my problem.

seems to be a bug. Worth creating an issue on github, imo.

@mbilliet @starets Hey! Have you tried setting devel: true in your releases in helmfile.ymal like this?

releases:
- name: myapp
  devel: true

--args is very hard to reason about and use - i’d recommend declaring everything in your helmfile.yaml

haven’t used it, but judging by https://github.com/roboll/helmfile/blob/master/main.go#L64 and https://github.com/roboll/helmfile/blob/master/main.go#L479-L489

it’s exactly what’s stated in Usage:

(can specify multiple or separate values with commas: key1=val1,key2=val2)

well - but the “key=value” doesn’t seem to work

I have an ‘image.tag’ in my helm chart, and when I define “image.tag=foo” - it still outputs the wrong image tag in my deployment if I try this out with helmfile template

I tried prefixing with the chartname, release name, Values, bare, …

also, if I try with --log-level debug - and I nowhere see the value I try to override

hmm ok, these only seem to be available in the helmfile files themselves, if I want to propagate them to the charts I have to add them to the values loaded by helmfile…

--state-values-set should be used like --state-values-set mykey=myvalue so that it becomes available in your helmfile.yaml like:

releases:
- name: myapp
  values:
  - foo: {{ .Values.mykey }}

Or

helmfiles:
- path: path/to/subhelmfile.yaml
  values:
  - foo: {{ .Values.mykey }}

The important point is that any state values are not implicitly propagated to any releases or any sub-helmfiles.

yeah I figured that out by now

thanks anyway!

@Vlad Ionescu (he/him) Hey! I’ve tried calling helmfile from atlantis with my own “custom stages” feature a year ago

https://github.com/cloudposse/atlantis/pull/20

Does atlantis officially have such feature today?

Would you make it possible to set a default image.tag per environment, too?

Assuming so, I’d guess it should be

releases:
  - name: myrel-{{ env "BRANCH_NAME" | default "master" }}
    namespace: dataproduct-{{ env "BRANCH_NAME" | default "master" }}
    chart: ../../../../charts/data-product-service-chart
    force: true
    atomic: true
    values:
      - image:
          repository: myrepo.com/rel/myrel
          tag: major-1.1.123
      - ../../env/{{ .Environment.Name }}.yaml
      - {{ tag := get "image.tag" "" .Values }}{{ if $tag }}{"image": {"tag": {{ $tag | quote }} } } {{ end }}

The important point here is that state values are not automatically propagated to releases as state values and chart values are completely different things

We don’t really want the image.tag to vary between environments so setting a default for each environment introduces redundant environment files (essentially boilerplate). Regardless, I tried adding image.tag to env/minikube.yaml and adding your suggested code above but I got the following error;

template: stringTemplate:17: function "tag" not defined

ah, perhaps it should be {{ $tag := get not {{ tag := get

and as you aren’t going to define environmental defaults, values can just be

    values:
      - image:
          repository: myrepo.com/rel/myrel
          tag: {{ .Values | get "image.tag" "major-1.1.123" }}
      - ../../env/{{ .Environment.Name }}.yaml

So should the value of image.tag there come from --state-values-set image.tag=mytag?

Doesn’t seem to work for me

Yes

Hmm, what would you see if you run helmfile build with log-level=debug?

helmfile --log-level=debug -f helmfile.yaml --state-values-set image.tag=foo build

when no --set-values-set are provided, it does use default_tag

$ helmfile --log-level=debug -f helmfile.yaml build
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=&{default map[] map[]}, overrode=<nil>
first-pass uses: &{default map[] map[]}
first-pass produced: &{default map[] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[] map[]}
vals:
map[]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: releases:
 1:   - name: myapp
 2:     chart: stable/nginx
 3:     values:
 4:     - image:
 5:         tag: default_tag
 6:

merged environment: &{default map[] map[]}
---
#  Source: helmfile.yaml

filepath: helmfile.yaml
releases:
- chart: stable/nginx
  name: myapp
  values:
  - image:
      tag: default_tag
templates: {}

That is pretty much exactly what I’m doing but it always uses the default

and with --state-values-set

$ helmfile --log-level=debug -f helmfile.yaml --state-values-set image.tag=foo build
processing file "helmfile.yaml" in directory "."
first-pass rendering starting for "helmfile.yaml.part.0": inherited=<nil>, overrode=&{default map[image:map[tag:foo]] map[]}
first-pass uses: &{default map[image:map[tag:foo]] map[]}
first-pass produced: &{default map[image:map[tag:foo]] map[]}
first-pass rendering result of "helmfile.yaml.part.0": {default map[image:map[tag:foo]] map[]}
vals:
map[image:map[tag:foo]]
defaultVals:[]
second-pass rendering result of "helmfile.yaml.part.0":
 0: releases:
 1:   - name: myapp
 2:     chart: stable/nginx
 3:     values:
 4:     - image:
 5:         tag: foo
 6:

merged environment: &{default map[image:map[tag:foo]] map[]}
---
#  Source: helmfile.yaml

filepath: helmfile.yaml
releases:
- chart: stable/nginx
  name: myapp
  values:
  - image:
      tag: foo
templates: {}

@Ben would you mind sharing your logs with --log-level=debug? (but please beware not to leak creds/secrets in it though!

Seems to be related to environments. This is my exact helmfile

bases:
  - ../../env/helmfile-environments.yaml

releases:
  - name: rdc-{{ env "BRANCH_NAME" | default "master" }}
    namespace: dataproduct-{{ env "BRANCH_NAME" | default "master" }}
    chart: ../../../../charts/data-product-service-chart
    force: true
    atomic: true
    values:
      - image:
          repository: registry.encompasshost.com/encompass/cdp/rdc-cdp
          tag: {{ .Values | get "image.tag" "major-humpback-1.1.5888" }}
      - service:
          internalPort: 8081
      - ../../env/{{ .Environment.Name }}.yaml

if I comment out the bases section and the last line then image.tag is overriden as expected

Does the inclusion of environments section clear the state passed from CLI maybe?

Sounds like so - and it might be a bug! I’ll take a deeper look soon

@Gourav Hey! Are you talking about how you’d use stable/kiam chart, right?

https://github.com/helm/charts/blob/master/stable/kiam/templates/agent-serviceaccount.yaml#L12

https://github.com/helm/charts/blob/ce946802c980eac7e741931c92e0305df65b10ec/stable/kiam/templates/_helpers.tpl#L75

it seems like you should use serviceAccounts, not serviceAccount

try

serviceAccounts:
  agent:
    create: true
    name: dev-ops-kiam-agent
  server:
    create: true
    name: dev-ops-kiam-server

ok… I will try.. thank you @mumoshu

Thanks.. it worked…

so probably you want a central place to affect every aspect of every release in every (sub-)helmfile, right?

maybe it isn’t possible today -

if i read correctly, what you might need is

releases:
# render myapp-us-west-1
{{ include "the-central-place/release-template.tpl" (dict "Values" .Values "name" "myapp" "region" "us-west-1" "opts" (dict "key1" "val1")) }}
# render myapp-us-west-2
{{ include "the-central-place/release-template.tpl" (dict "Values" .Values "name" "myapp" "region" "us-west-2" "opts" (dict "key1" "val1")) }}
# ....

and in release-template.tpl, you write a nice go template that generates a release as you like

where include includes the given tpl from the file system(as we can do in helm

if this sounds good to you, would you mind opening a issue to add include to helmfile?

hmm that might be a solution, not sure though

still trying to understand. could you provide me a small example to reproduce what you’ve tried?

well - I have a

templates:
  default: &default
    values:
      - /central/path/to/values.yaml

releases:
  - release-v1:
    <<*: *default
    values:
    - somevalue: "to override"

if I do this, the values from the template are ignored

if I could somehow merge them, I could declare the template in a central base that’s included everywhere

and add additional custom values per release

but that’s all due to ugly YAML trickery I’m afraid

not sure if helmfile could do that in some way

@pjbecotte Hey! Assuming you’re talking about remote ghelmfiles stored under .helmfile/cache/something - yeah you might need to manually run git-pull there if you’ve specified a changing git branch

helmfile doesn’t try to re-fetch already cached git branches and tags.

my expected usage for the remote helmfile feature was you’d tag your git repo with semantic versions. that is, you change the git tag included in the remote helmfile url(go-getter-style url) from v1.0.0 to v2.0.0 when you need any update.

does it make sense?

Sure, that is a workflow that will work. Of course, it was a pain while I was iterating trying to get things working. Was just thinking I probably missed something.

@pjbecotte i hear you.

perhaps it would be nicer if helmfile git-pulled every cached remote helmfile repo by default?

and ability to optionally skip git-pulling cached remote helmfiles?

JFYI: Issued opened https://github.com/roboll/helmfile/issues/901

Oh, neat, thanks!

@Naseem Hey! Replied to you in the k8s slack also but anyway

It depends, but I guess you should use sub-helmfile per region.

Thati s, there could be a production environment in a “global” helmfile. the global helmfile would delegate per-region deployment to respective sub-helmfile

Thanks @mumoshu ! I will try this approach!

if you have releases that are almost identical across all regions, with maybe 1 value thats different per region, is this still the approach you would suggest?

Yep.

Try injecting state values for regions like:

helmfiles:
- path: regional.yaml
   values:
   - region: us-west-1
- path: regional.yaml
   values:
   - region: us-west-2

Regarding globally distributed clusters again:

Currently in a non-globally-distributed setup, I know if a release should be installed based on which environment it is: installed: {{ eq .Environment.Name "staging" }} <— only installs in staging.

How does one achieve the flexibility of: “installed if env is staging AND region is us-west-1” OR “installed if env is staging AND cluster name is Bob” … either of these would be great

Can we somehow extract the cluster name from kube context or something?

re the gotmpl expression, it would look like {{ and (eq .Environment.Name "staging") (eq $clusterName "cluster1") }}

Can we somehow extract the cluster name from kube context or something? I think you should think conversely. You’d provide the cluster name via values, and select appropriate kubeconfig based on that

you should split your kubeconfig per context beforehand with kubectl config view --minify

Thanks @mumoshu!

hm… maybe helm-tiller timedout/failed in the middle of the installation process and left the cluster in a half-baked state?

i suspect it can potentially a fundamental issue in helm-tiller then(helm3 would help

I bumped the K8s version in my cluster and it started working ok. Probably it was a coincidence, but everything is ok so far.

Ah maybe we’re missing the implementation for rendering release template expressions within the inlined values

Issue opened: https://github.com/roboll/helmfile/issues/900

thanks

Not yet. but it would be great to have one!

Ok, I might take a shot at that in the coming days.

(Currently using helmfile as ‘templating engine’ and seems not efficient at this point to invest in Helm2 and all the tiller shenanigans)

it’s ok if you use the tillerless plugin

(which is what we do)

Considering that as well. But afaik there is no clear upgrade path yet to migrate helm2 to helm3 deployments. So you’d have to remove and reinstall which is not ideal. So that’s my main reason for looking directly at helm3.

Do you use a single namespace for all helm2 release data (kube-system and apps)?

That sounds like a good strategy!

Anyways, we will be able to use https://github.com/helm/helm-2to3 for upgrading without reinstalling

Great to see helm3 images are built now! Work (other work) got in the way, so hadn’t found time for that (and need to familiarize myself a bit with the ins/outs of Helmfile ci setup).

Ran into some CRD-related issues on a first attempt using Helm3 on prometheus-operator. Then again: Not the simplest chart. And stuff moves so fast, might have been fixed already. (Also: Not a helmfile issue).

@Tiago Meireles hey! you’d also need to update this in the yaml:

bases:
  - envs/environments.yaml

to

bases:
- ../envs/environments.yaml

every refs from a helmfile.yaml should be relative to itself

That is what I expected. It didn’t work when i tried it.

thx! seems like you also need to fix environments.yaml as well.

hey! could you share your configs(or maybe smaler versions of them for reproduction?

all i can say form the sole example is that your config does miss the default anchor undefined in your specific case.

@mumoshu, sorry for the late response: sub-helmfile:

---
environments:
  {{ .Environment.Name }}:
    values:
    - ../envs/{{ .Environment.Name }}/defaults.yaml
    - ../envs/{{ .Environment.Name }}/charts.yaml
---
{{ readFile "../templates.yaml" }}

releases:
- name: x-creds-{{ .Environment.Name }}
  chart: x-stg/docker-registry-creds-chart
  version: {{ .Environment.Values.charts | getOrNil "x.chartVersion" | default "0.0.1" }}
  installed: {{ .Environment.Values.charts | getOrNil "x.enabled" | default false }}
  <<: *default
  values:
  - imageCredentials:
      username: {{ requiredEnv "X_USER" }}
      password: {{ requiredEnv "X_PASS" }}
  - fullnameOverride: x-creds-{{ .Environment.Name }}

templates.yaml:

---
bases:
- ../repos.yaml
- ../helmdefaults.yaml
---
templates:
  default: &default
    namespace: "{{ .Environment.Values.namespace }}"
    wait: false
    missingFileHandler: Error

thx! ah, so it won’t work at all.

&default needs *default written in the same yaml document. and --- nor bases doesn’t result in concatenating files as texts. they all read and render files independently.

maybe this works?

{{ readFile "../repos.yaml" }}
{{ readFile "../helmdefaults.yaml" }}

templates:
  default: &default
    namespace: "{{ .Environment.Values.namespace }}"
    wait: false
    missingFileHandler: Error

@mumoshu actually my original config is working, it just prints this error before going forward

ah gotcha! just curious, but you see that warning only with --log-level=debug?

nope, both with and w/o debugging

interesting.

ah okay you already have this in your original helmfile

{{ readFile "../templates.yaml" }}

maybe helmfile should turn off that warning by default

but anyway… it’s there due to how helmfile’s feature called double rendering work

when changing my templates yaml to :

{{ readFile "../repos.yaml" }}
{{ readFile "../helmdefaults.yaml" }}

instead of using bases im getting this:

could not deduce `environment:` block, configuring only .Environment.Name. error: failed to read x-creds.yaml.part.1: reading document at index 1: yaml: unknown anchor 'default' referenced
in ./x-creds.yaml: failed to read x-creds.yaml: reading document at index 1: yaml: unmarshal errors:
  line 2: cannot unmarshal !!map into string

there’s a chicken-and-egg problem while loading a helmfile.yaml. that is, you need environment values loaded before rendering any go template in hlemfile.yaml. but to load env values it must be a plain yaml without go template.

you mean this part:

---
environments:
  {{ .Environment.Name }}:
    values:
    - ../envs/{{ .Environment.Name }}/defaults.yaml
    - ../envs/{{ .Environment.Name }}/charts.yaml

in subhelmfile?

to workaround that, helmfile renders your sub-helmfile twice

partially yes. double rendering occurs on the whole file

that is, helmfile firstly renders

{ readFile "../templates.yaml" }}

releases:
- name: x-creds-{{ .Environment.Name }}
  chart: x-stg/docker-registry-creds-chart
  version: {{ .Environment.Values.charts | getOrNil "x.chartVersion" | default "0.0.1" }}
  installed: {{ .Environment.Values.charts | getOrNil "x.enabled" | default false }}
  <<: *default
  values:
  - imageCredentials:
      username: {{ requiredEnv "X_USER" }}
      password: {{ requiredEnv "X_PASS" }}
  - fullnameOverride: x-creds-{{ .Environment.Name }}

with readFile replaced with a noop func, and with an empty values set

which results in

releases:
- name: x-creds-default
  chart: x-stg/docker-registry-creds-chart
  version: "0.0.1"
  installed: false
  <<: *default
  values:
  - imageCredentials:
      username: VALUE_OF_X_USER
      password: VALUE_OF_X_PASS
  - fullnameOverride: x-creds-default

which indeed miss &default and results in the error on *default

helmfile uses this to load env values used to render your helmfile.yaml. this time readFile is not a noop func and as your helmfile.yaml is correct, it works…

helmfile has no way to know your helmfile.yaml contains environments or not before rendering it

thats why this double rendering thing always happen regardless of there’s environments section defined or not in your helmfile.yaml.

i get it, but with a simple change in the release i dont see this warning/error, thats what surprises me

using the same templates.yaml but changing the release to this:

---
environments:
  {{ .Environment.Name }}:
    values:
    - ../envs/{{ .Environment.Name }}/defaults.yaml
    - ../envs/{{ .Environment.Name }}/charts.yaml

---
{{ readFile "../templates.yaml" }}

releases:
- name: my-app
  chart: x-stg/my-app-v2-chart
  version: {{ .Values.charts.myApp.chartVersion }}
  installed: {{ .Environment.Values.charts | getOrNil "myApp.enabled" | default false }}
  <<: *default
  values:
   - ../envs/{{ .Environment.Name }}/apps/{{ `{{ .Release.Name }}` }}/values.yaml
   - ../envs/common/ms-affinity-rule.yaml
   - fullnameOverride: my-app-{{ .Environment.Name }}

the only noticeable change here is the version

wow, really?!

yes

i can share privately debug logs if you wish

what do you see as the “first rendering result” when you add --log-level=debug like helmfile --log-level=debug build ?

im on 0.87 btw

wait will share the result

sorry it takes time, some sensitive names that i need to remove

if you wish i can share a debug log after i change the version to use getOrNil and then i get the deduce error

yes, please!

at glance i see the expected thing here:

first-pass rendering input of "my-app.yaml.part.1":
 0: {{ readFile "../templates.yaml" }}
 1:
 2: releases:
 3: - name: myproject-my-app
 4:   chart: bams-stg/myproject-my-app-v2-chart
 5:   version: {{ .Values.charts.myprojectLatiTagger.chartVersion }}
 6:   installed: {{ .Environment.Values.charts | getOrNil "myprojectLatiTagger.enabled" | default false }}
 7:   <<: *default
 8:   values:
 9:    - ../envs/{{ .Environment.Name }}/apps/{{ `{{ .Release.Name }}` }}/values.yaml
10:    - ../envs/common/ms-affinity-rule.yaml
11:    - fullnameOverride: myproject-my-app-{{ .Environment.Name }}

i thought this would emit the warning(which didnt

i wondering if its because in one case im using .Values.charts.somevalue and the other case is: .Environment.Values.charts.somevalue

I read the log but am not yet sure what’s going on here. There’s indeed a few switches that relates to whether you use .Environment.Values or .Values. I’ll take a deeper look in coming days!

Thanks for your cooperation

Thank you for the support!

@mumoshu hi did u had the chance to look at this? should i try to use env bases to avoid this warning? i still can’t figure out what causes this behavior

@yuri Yes I have some progress. The main problem was that it was missing some error log that occurred in the first-pass render.

Improving it, I get this:

irst-pass rendering input of "helmfile.3.yaml.part.0":
 0: releases:
 1: - name: myproject-my-app
 2:   chart: bams-stg/myproject-my-app-v2-chart
 3:   version: {{ .Values.charts.myprojectLatiTagger.chartVersion }}
 4:   installed: {{ .Environment.Values.charts | getOrNil "myprojectLatiTagger.enabled" | default false }}
 5:   <<: *default
 6:   values:
 7:    - ../envs/{{ .Environment.Name }}/apps/{{ `{{ .Release.Name }}` }}/values.yaml
 8:    - ../envs/common/ms-affinity-rule.yaml
 9:    - fullnameOverride: myproject-my-app-{{ .Environment.Name }}
10:

template syntax error: template: stringTemplate:4:21: executing "stringTemplate" at <.Values.charts.myprojectLatiTagger.chartVersion>: nil pointer evaluating interface {}.myprojectLatiTagger
first-pass rendering output of "helmfile.3.yaml.part.0":
 0: releases:
 1: - name: myproject-my-app
 2:   chart: bams-stg/myproject-my-app-v2-chart
 3:   version:

pls see template syntax error: template: stringTemplate:4:21: executing "stringTemplate" at <.Values.charts.myprojectLatiTagger.chartVersion>: nil pointer evaluating interface

hmmm

this means that, the first-pass render “stops” at any nil pointer access(it has no way to tolerate it…), which results in the incomplete yaml not containing anything after the installed: ... line

that’s why it doesn’t emit the reading document at index 1: yaml: unknown anchor 'default' referenced warning

anyway, regardless of the warning is emitted or not, the second-pass should produce same same result

ah ok so if i understand since i have 2 “values” of getOrNil it throws this warning

so probably you don’t need to worry about the warning at all..?(i agree it’s red herring though..

yes the sync works as expected, it just drives me crazy in the ci process to see some extra messages that i dont wish to see

yeah. maybe we should enhance the debug logs for the first-pass render?

maybe my “pattern” of templating this is incorrect the idea was to hold some yaml file with

chartName:
  installed: true/false
  vesrion: x.y.z

generally speaking any error in the first-pass is tolerable

hmm maybe. your idea makes sense

what works today would be to use getOrNil whenever you dig Values or Environment.Values

we have the same application that we want to install with different versions… for example qa/ppe/…

hmm but i do use getOrNil

ah sry i’m a bit confused

from helmfile’s perspective, emitting reading document at index 1: yaml: unknown anchor 'default' referenced is rather the correct behaviour

ok so it a matter of log verbosity and maybe should appear in debug ?

so avoiding the warning by exploiting the fact that the first-pass render “stops” at the first template error seems wrong..

yes

i think so

ok i hope just want to make sure i dont break any logic/patterns in helmfile the can hurt me later

ah and back to your original problem

you should avoid using anchors in your pattern

if

hmm so no templates with our current release definition?

if you see errors like this one https://sweetops.slack.com/archives/CE5NGCB9Q/p1571818484076100 anywhere other than the first-render

getting reading document at index 1: yaml: unknown anchor 'default' referenced in the first-pass is ok

ah ok got you

for now its seems like only the first render

ok great!

then there’s nothing wrong on your side

thank u again for the support!

the only remaining todo would be - i’d prefix warning from the first-pass render nicely so that it won’t confuse you anymore

thanks!

my pleasure. thanks for your support and using helmfile!

Not yet

Haven’t used it

absolutely. just write a helmfile like this:

releases:
- name: opa
  chart: stable/opa
  values:
  - values.yaml

assuming you use https://github.com/helm/charts/tree/master/stable/opa

@mumoshu Thank you

This is not really an answer to what you are asking, but we have the same problem all the time

We use our monochart very frequently to get around the perceived shortcomings of a lot of charts out there

See our usage here: https://github.com/cloudposse/helmfiles/tree/master/releases

Basically the monochart makes implementing most services extremely easy, so we use that combined with Helmfile as our escape hatch

Yeah, that is basically how we deploy our services :)

@pjbecotte Hey! I haven’t tested it extensively but helmfile has a secret feature that allows you to jsonpatch/strategicmergepatch manifests before installing a chart:

https://github.com/roboll/helmfile/pull/673

Would it make it unnecessary to fork charts if you use helmfile template to generate the patches dynamically?

in k8s secrets

?

try adding --suppress-secrets like helmfile apply --suppress-secrets

like i have multiple helm charts in the helmfile and i want only there status which deployments have been deployed etc but not the complete deployment

providing the above flag stops printing sensitive info

i want only there status which deployments have been deployed etc but not the complete deployment

this sounds like a different issue than protecting sensitive info! do you actually need it?(and why?

i only need helm info like

`RESOURCES: ==> v1/Deployment NAME READY UP-TO-DATE AVAILABLE AGE help 0/1 1 0 33d

==> v1/Pod(related) NAME READY STATUS RESTARTS AGE help-699b97d548-jd9zg 0/1 Terminating 0 2m58s help-74688d5f45-jm6sx 0/1 ContainerCreating 0 76s

==> v1/Service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE help ClusterIP 172.30.115.158 8080/TCP 33d

==> v1beta1/Ingress NAME HOSTS ADDRESS PORTS AGE help help.dev.onprem.dmsuitecloud.com 80 33d

NOTES: Helm Chart installed : help in namespace dmp-system Your release is named : help.`

no the full ` # Source: help/templates/help-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: com.fico.dmp/instanceId: help com.fico.dmp/name: help reloader.stakater.com/auto: “true” labels: com.fico.dmp/instanceId: help com.fico.dmp/name: help name: help`

got it. that’s not possible today. why would you like that?

Actually i made a docker image for installation using helmfile.I want only the helm charts info to be printed in the logs not the complete deployments

btw i’m asking because i’ve seen that everyone has different opinions and requirements for which output they want

i see. how would you debug the installation when it faiiled?

We have implemented the rollback feature for client we wont be giving him the complete info about deplyments.If the deployment fails then our team will look into it locally

but if you like the default output from that helmfile container to include only k8s resources installed/upgraded by helm(https://sweetops.slack.com/archives/CE5NGCB9Q/p1572505407002900?thread_ts=1572505148.001300&cid=CE5NGCB9Q)

how would your team debug it?

i.e. if rollback happens the the logs will notify the user and user will contact us..so that client dont have to do anything and we will fix the issue and will make a new image and push it

so perhaps you want an ability to configure multiple log output channels? like stdout contains k8s resources affected by helm only, and something like debug.log contains all the logs?

actually for this we will run locally on our cluster the same without disabling output and try..the cluster with us is exact copy of the clients cluster and with the help of helm diff we can see the changes and try to work on them…

mostly there will be image change only in the helm chart so we can recify that easily

ah, it would work out ok!

so is it possible to disable diff output?

no, as i said above

but i’m eager to add a feature to configure what’s included in the helmfile log. does that sound good to you?

yup…i think it will be great if we can add feature to add logs incrementally like if ye want to add diff logs or not and etc

maybe something like helmfile --log-filter helm,exec,helmfile,...

Yup like this only…so can we disable diff output with this currently?

or even helmfile --info-log-filter helm,exec,helmfile,...…

nope

all you can do today would be pipe it to tee

and grep only what you want

so try that way if you want something that works today

Yup thanks….but the logs are clattered like we dont know how long will be notes and other thing so it wont work fine…but thanks for your help..

yeah, but i think helmfile apply | grep -v '^\(+|-\) would mostly work

as the diff output mostly begins with any of +, -

Yup thanks…i will check it..thanks alot for your help!..

or even better helmfile apply | grep -v '^\(+|-|Comparing |Release was not present in Helm|\*\)'

my pleasure! good luck

and thanks for using hlemfile

hey!

unfortunately the kiam chart doesn’t seem to support annotations at the daemonset level

see https://github.com/helm/charts/blob/6e94f8f328eaee60d21d80e6fbd8a81f6bc0c78e/stable/kiam/templates/agent-daemonset.yaml#L4-L11

you’ll see there’s no templates set up for the daemonset annotations

maybe worth a feature request to the kiam chart

Thanks for your inputs @mumoshu .

I was working to add the annotations in helmfile.. I think we can do something like this in helmfile for kiam

    hooks:
      - events: ["postsync"]
        command: "/bin/sh"
        args: ["-c", "kubectl annotate --overwrite --namespace={{`{{ .Release.Namespace }}`}} DaemonSet/{{`{{ .Release.Name }}`}}-agent secret.reloader.stakater.com/reload=kiam-agent-certificate-secret,kiam-ca-cert"]
      - events: ["postsync"]
        command: "/bin/sh"
        args: ["-c", "kubectl annotate --overwrite --namespace={{`{{ .Release.Namespace }}`}} DaemonSet/{{`{{ .Release.Name }}`}}-server secret.reloader.stakater.com/reload=kiam-server-certificate-secret,kiam-ca-cert"]

yeah maybe..

did it work?

yes.. it worked

awesome!

ideally helmfile should provide a way to patch some resources included in the release, without forking the original chart

i’d love to but we don’t have one today

hey! currently, no, helmfile doesn’t have such feature. probably worth a feature request?

but why you need that?

i think your releases are usually backward-compatible and therefore you don’t need to rollback successful releases rolled out before the failed release

or perhaps you want helmfile to rollback only the failed release?

Ideally each release would be backwards compatible yes, but we’re maturing to that point as we break apart a single runtime/release into multiples

And as it sits currently, we’d like to ensure the same version of code is deployed at a given time with rollback if it is not.

I’ll open an request. Appreciate the response @mumoshu

that makes sense. thx for clarifying! im looking forward to the feature request

just to be sure, what you want helmfile to do is basically running helm rollback $RELEASE_NAME $(helm history --output json $RELEASE_NAME | jq -r .[].revision | tail -n 2 | head -n 1) for all the affected releases in a failed Helmfile run?

@Cameron Boulton

I think so? Ideally the logic that’s already used when atomic:true for a given release fails

But instead for ALL releases if ANY release in the helmfile release array/list fails

If that makes sense?

jq -r .[].revision | tail -n 2 | head -n 1 is for obtaining the second latest release of the release, assuming the latest release is one created by the failed helmfile run

But instead for ALL releases if ANY release in the helmfile release array/list fails

im still trying to understand. how is it different from for all the affected releases in a failed Helmfile run?

It is not different from for all the affected releases in a failed Helmfile run

But for all the affected releases in a failed Helmfile run is different from current helmfile behavior today correct?

Ideally the logic that’s already used when atomic:true for a given release fails

yeah that makes sense. implementation-wise, we can’t use the exact logic used by --atomic as it’s just a flag provided by helm itself

Ah

But for all the affected releases in a failed Helmfile run is different from current helmfile behavior today correct?

ah, so you’re talking about the current behavior ouf helmfile when a release has atomic: true set?

if so, yes, it rollbacks the failed release only

Yes, we’re using that now

gotcha

then what we need might be

a new flag like helmfile apply --rollback-on-failure instructs helmfile to (1) rollback the failed release if the release didn’t have atomic: true set and (2) all the successful releases rolled out before the failed one with https://sweetops.slack.com/archives/CE5NGCB9Q/p1572565299026800?thread_ts=1572563989.024200&cid=CE5NGCB9Q

i.e. helmfile doesn’t need to explicitly rollback the failed release if the release had atomic: true set

as atomic: true results in helm upgrade --atomic that would rollback the failed release automatically for you

does this make sense?

Yes, the actual failed release(s) themselves would already be rolled back if using atomic: true

So helmfile really only needs to rollback any successful releases if any one failed

absolutely!

i thought we’d better name it helmfile apply --atomic but …

How do you decide between helmfile argument instead of option in helmfile YAML?

Are the latter for helm options only?

generally any helmfile-run-wide operational option is available via flags only

Gotcha

helmfile apply --atomic seems great to me

yeah but i think i have a few questions if we do so

like

should it imply atomic: true in all the releases?

That seems more problematic to me, but still thinking

or maybe we can just deprecate atomic: true in favor of helmfile apply --atomic?

Mmm but I think there are cases such as today’s behavior where people want PER release atomicity, but not across ALL releases (whole helmfile)

Does that make sense?

or evangelize to use atomic: true and not helmfile apply --atomic when you can ensure all the releases are backward compatible and you do need a automated rollback?

Exactly

Backwards compatible or maybe entirely unrelated

Such as kafka and redis

One might not care if kafka failed but redis succeeded

makes sense

Which is the behavior we have today that probably should not change

apply --atomic would be a superset

and helmfile apply --atomic makes atomic: true irrelevant, right?

Seems like it does logically

As in, if you used --atomic but omitted atomic: true and > 0 release failed any successful would be rolled back

as helmfile would rollback the failed release regardless of it had atomic: true or not anyway

Right

But I guess what it lets you do:

exactly

Have atomicity PER release (like today)

And then optionally you COULD use --atomic on demand if/when it was needed

And then not use --atomic but still keep the per release atomic:true behavior in that case

Does that make sense?

yes i believe so

so you basically use helmfile apply --atomic only when one of your new releases has known backward-incompatibility

Exactly

Which for some users might be always

But at least it doesn’t force others with a changed behavior of what we have today

that’s great

Really appreciate the conversation @mumoshu

Would you still like me to open a GitHub issue/feature request @mumoshu?

so am i! it was very inspiring. i’m awaiting your feature request(probably including a link to this slack thread would help to provide ctx to other uses)

yes, i’d appreciate it if you could do so!

it’s a bit annoying task given we already have a great conversation here and settled on something

No problem. I understand the benefits of the formality for tracking, updates and visibility to other users of the project.

i just want to ensure that helmfile looks like being developed openly

Exactly

exactly!

thx for the conversation and your understanding!

Welcome and thank you