#variant (2020-04)

https://github.com/mumoshu/variant

Discuss variant (the “Universal CLI”) https://github.com/mumoshu/variant

Archive: https://archive.sweetops.com/variant/

2020-04-30

2020-04-29

nian avatar

For variant … this tool appears to be running locally, as opposed to something in a cluster from the local terminal. Is that correct?

nian avatar

For example, would we use this in a distributed system? Is there an agent mode?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There are 2 modes of invocation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

one is as a slackbot

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the other is as a cli

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, as a cli you could incorporate it with any sort of CI/CD pipelines you have

nian avatar

:–1:

mumoshu avatar
mumoshu

@nian Erik’s correct.

Would it be nice for you if it worked in a cluster? (K8s?

mumoshu avatar
mumoshu

Usually it would be a matter of building a docker image containing the binary built by running variant export binary and run it via an AWS ECS task or K8s Job/Pod/etc

mumoshu avatar
mumoshu

But I have considered about if I could add a client mode to Variant.

It would probably look like variant client run --config someconnectioninfo.yaml CMD ARGS which creates e.g. K8s pod running the CMD in the K8s cluster as configured in the someconnectioninfo.yaml

mumoshu avatar
mumoshu

But I stopped there as I had no specific use-case at the time. If you have one, i’d appreciate it if you could share!

nian avatar

Yes … distributed in k8s cluster.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think I’d prefer deploying it in k8s more along the lines of how we deploy other things so as not to introduce a new mechanism

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That said, a UI would make this more appealing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
feat: Multi-tenancy of Web UI · Issue #17 · mumoshu/variant2

Importing from mumoshu/variant#33 Add a Login page and a project-selection page in front of #32. It should be deployed along with multiple instances of variant server #31 (+ perhaps #32). This may …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but at what point are we reinventing jenkins

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

0.28 seems to be pretty solid. tests are passing with no changes.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu I’m getting weird results from simply reading my file.

contents:

triggers:
  smoke-test:
    description: Smoke testing the CLI
    order:
      - job: terraform plan eks
        args:
          - -detailed-exitcode

error:

panic: inconsistent map element types (cty.List(cty.String) then cty.String)

goroutine 1 [running]:
[github.com/zclconf/go-cty/cty.MapVal(0xc0009ea880](http://github\.com/zclconf/go\-cty/cty\.MapVal\(0xc0009ea880), 0xc0009ea880, 0xc0009d20f8, 0x3, 0xc0009ea9b8)
        /home/runner/go/pkg/mod/github.com/zclconf/[email protected]/cty/value_init.go:207 +0x4b3
[github.com/mumoshu/variant2/pkg/app.goToCty(0x1767100](http://github\.com/mumoshu/variant2/pkg/app\.goToCty\(0x1767100), 0xc0004d38f0, 0x0, 0x0, 0x1, 0xc0009bb8e0, 0x0, 0x1)
        /home/runner/work/variant2/variant2/pkg/app/go_to_cty.go:28 +0x60e
[github.com/mumoshu/variant2/pkg/app.(*App).execMultiRun(0xc0000e3140](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execMultiRun\(0xc0000e3140), 0xc000239580, 0xc000819280, 0xc0009eadf8, 0x16, 0xc0009d1160, 0x16)
        /home/runner/work/variant2/variant2/pkg/app/app.go:1214 +0x13f
[github.com/mumoshu/variant2/pkg/app.(*App).execJob(0xc0000e3140](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execJob\(0xc0000e3140), 0xc000239580, 0xc0000bd997, 0x7, 0x0, 0x205fbc0, 0xc0006fc540, 0xc0000c7610, 0xc000577ea0, 0x2, ...)
        /home/runner/work/variant2/variant2/pkg/app/app.go:816 +0x246
[github.com/mumoshu/variant2/pkg/app.(*App).Job.func1(0x0](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.Job\.func1\(0x0), 0x0, 0x0)
        /home/runner/work/variant2/variant2/pkg/app/app.go:644 +0x939
[github.com/mumoshu/variant2/pkg/app.(*App).Run(0xc0000e3140](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.Run\(0xc0000e3140), 0xc0000bd997, 0x7, 0xc00080adb0, 0xc00080ade0, 0xc00020b9e0, 0x1, 0x1, 0x0, 0x0, ...)
        /home/runner/work/variant2/variant2/pkg/app/app.go:513 +0xc3
[github.com/mumoshu/variant2.(*Runner).Cobra.func1(0xc00037d180](http://github\.com/mumoshu/variant2\.\(\*Runner\)\.Cobra\.func1\(0xc00037d180), 0xc00080acf0, 0x2, 0x3, 0x0, 0x0)
        /home/runner/work/variant2/variant2/variant.go:661 +0x114
[github.com/spf13/cobra.(*Command).execute(0xc00037d180](http://github\.com/spf13/cobra\.\(\*Command\)\.execute\(0xc00037d180), 0xc00080ac00, 0x3, 0x3, 0xc00037d180, 0xc00080ac00)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
[github.com/spf13/cobra.(*Command).ExecuteC(0xc0000e9b80](http://github\.com/spf13/cobra\.\(\*Command\)\.ExecuteC\(0xc0000e9b80), 0x2034ba0, 0xc0000be010, 0x2034ba0)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
[github.com/spf13/cobra.(*Command).Execute(...)](http://github\.com/spf13/cobra\.\(\*Command\)\.Execute\(\.\.\.\))
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
[github.com/mumoshu/variant2.(*Runner).Run(0xc0007c3040](http://github\.com/mumoshu/variant2\.\(\*Runner\)\.Run\(0xc0007c3040), 0xc0000f8c90, 0x4, 0x4, 0xc00020bd78, 0x1, 0x1, 0x0, 0x0)
        /home/runner/work/variant2/variant2/variant.go:791 +0x2cc
[github.com/mumoshu/variant2.Main.Run(0x7ffe211094bb](http://github\.com/mumoshu/variant2\.Main\.Run\(0x7ffe211094bb), 0x3, 0x0, 0x0, 0x0, 0x7ffe211094b4, 0xa, 0x2034ba0, 0xc0000be008, 0x2034ba0, ...)
        /home/runner/work/variant2/variant2/variant.go:396 +0x12d
main.main()
        /home/runner/work/variant2/variant2/pkg/cmd/main.go:13 +0xb8
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

another interesting one, @mumoshu.

Parsing this YAML works fine:

    order:
      - job: terraform plan eks
        args: ""
      - job: helmfile diff teleport
        args: --selector chart=teleport-node

Parsing this YAML seems to have a problem with:

    order:
      - job: terraform plan eks
        # args: ""
      - job: helmfile diff teleport
        args: --selector chart=teleport-node

It throws the following error, but the issue is a missing args not a missing job:

Error: Missing map element

  on ../cli/trigger.variant line 36:
  (source code not available)

This map does not have an element with the key "job".

Error: ../cli/trigger.variant:36,29-33: Missing map element; This map does not have an element with the key "job".
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so it seems mixing a job with args and without is some weird issue

mumoshu avatar
mumoshu

Interesting. I thought it doesn’t have any specific logic to handle keys named “args” and “jobs” in a map

mumoshu avatar
mumoshu

Maybe it depends on the context?

mumoshu avatar
mumoshu

Could you share your trigger.variant, so that I can see the code around L36 and L29-33

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

the line numbers are likely off due to recent changes

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

the issue was with an array of objects without the same keys in each

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

weird…looks like the code didn’t come through.

it was referring to:

  depends_on "trigger switch" {
    items = conf.file.triggers[var.trigger-name].order

    args = {
      item            = item.job
      item-args       = try(item.args, "")
      dry-run         = opt.dry-run
      tenant          = param.tenant
      trigger-name    = var.trigger-name
    }
  }

2020-04-27

2020-04-25

2020-04-24

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I think there is a regression in 0.24.2. the tests worked in 0.24.0 and now fail with:

  26:       aws --profile ${var.namespace}-${var.tenant}-helm eks update-kubeconfig --name=${var.namespace}-${var.tenant}-eks-cluster --region=${var.region} --kubeconfig=${var.kubeconfig-path}/kubecfg.${var.tenant}-helm

There is no variable named "var".
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu this is inside a case

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it seems like there is an issue with the case in some scenarios too.

  10:       bash -c terraform workspace select ${case.tenant} || terraform workspace new ${case.tenant}

There is no variable named "case".
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
  case "ok" {
    project = "account"
    tenant = "client"

    err = ""
    exitstatus = 0
    stdout = trimspace(<<-EOS
      bash -c terraform init
      bash -c terraform workspace select ${case.tenant} || terraform workspace new ${case.tenant}
      terraform destroy -var defaults_config_file=../../defaults.yaml -var tenant_config_file=../../${case.tenant}.yaml -auto-approve
    EOS
    )
  }
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

went back to 0.24.0 and all tests pass

mumoshu avatar
mumoshu

well are you trying to read case variables from within the case itself?

mumoshu avatar
mumoshu

i’ve never intended to make it work

mumoshu avatar
mumoshu

not sure how it worked before

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i was, yeah.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so my output is based on the same vars my run is using

mumoshu avatar
mumoshu
  26:       aws --profile ${var.namespace}-${var.tenant}-helm eks update-kubeconfig --name=${var.namespace}-${var.tenant}-eks-cluster --region=${var.region} --kubeconfig=${var.kubeconfig-path}/kubecfg.${var.tenant}-helm

i think this one is due to the change made for https://sweetops.slack.com/archives/CFFQ9GFB5/p1587702567304900?thread_ts=1587681605.300200&cid=CFFQ9GFB5

  case "ng1" {
    concurrency = 0
    stdout = ""
    delayone = 0
    delaytwo = 1
  }

  variable "err" {
     value = "delay of ${case.delayone} is less than ${case.delaytwo}"
  }

  run "test" {
    concurrency = case.concurrency
    delayone = case.delayone
    delaytwo = case.delaytwo
  }

  assert "..." {
      condition = var.err == ...
mumoshu avatar
mumoshu

the dependency was case -> variable before

mumoshu avatar
mumoshu

which is now variable -> case

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

use case:

• command takes 3 args

case 1: verify it works with defaults

case 2: verify it works with custom values

case 3: verify it fails with invalid values

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

for that, i want my case to define the values

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i want my case.stdout to reference values from case.*

mumoshu avatar
mumoshu

i think that’s where variable is used

mumoshu avatar
mumoshu

yes that’s okay

mumoshu avatar
mumoshu
aws --profile ${var.namespace}-${var.tenant}-helm eks update-kubeconfig --name=${var.namespace}-${var.tenant}-eks-cluster --region=${var.region} --kubeconfig=${var.kubeconfig-path}/kubecfg.${var.tenant}-helm

i think this is a different beast

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so i have to create a variable for every case property i want to reuse?

mumoshu avatar
mumoshu

this should better be a variable

mumoshu avatar
mumoshu

nope

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it’ll fail when it hits a property for a case without a specific value; say the scenario case 1 above

mumoshu avatar
mumoshu
  case "ok" {
    project = "account"
    tenant = "client"

    err = ""
    exitstatus = 0
    stdout = trimspace(<<-EOS
      bash -c terraform init
      bash -c terraform workspace select ${case.tenant} || terraform workspace new ${case.tenant}
      terraform destroy -var defaults_config_file=../../defaults.yaml -var tenant_config_file=../../${case.tenant}.yaml -auto-approve
    EOS
    )
  }
mumoshu avatar
mumoshu

I think there is a regression in 0.24.2. the tests worked in 0.24.0 and now fail with:

  26:       aws --profile ${var.namespace}-${var.tenant}-helm eks update-kubeconfig --name=${var.namespace}-${var.tenant}-eks-cluster --region=${var.region} --kubeconfig=${var.kubeconfig-path}/kubecfg.${var.tenant}-helm

There is no variable named "var".
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
10:34:48 PM
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

a var is expected to exist in the test for a case or a run

mumoshu avatar
mumoshu

bidirectional dependency between variable <-> case is too hard to be implemented

mumoshu avatar
mumoshu

well i dont understand. i thought you will only need either if you rewrite it?

mumoshu avatar
mumoshu

so if we bring back access to previously defined case fields from later case fields, you can rewrite this

  26:       aws --profile ${var.namespace}-${var.tenant}-helm eks update-kubeconfig --name=${var.namespace}-${var.tenant}-eks-cluster --region=${var.region} --kubeconfig=${var.kubeconfig-path}/kubecfg.${var.tenant}-helm

to

  26:       aws --profile ${case.namespace}-${case.tenant}-helm eks update-kubeconfig --name=${casenamespace}-${case.tenant}-eks-cluster --region=${case.region} --kubeconfig=${case.kubeconfig-path}/kubecfg.${case.tenant}-helm
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

ok…so i’ll leave the best decision to you, but the need is for my case block to have access to all case vars

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

a plus is having a case block have access to all var declarations

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i’d standardize my run on case vars only

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

my case would ref var for any global property that doesn’t change value per case

mumoshu avatar
mumoshu

so you don’t need access to case from vars?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

let me revisit this.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i’m about to write some tests so let me try it out

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yes. that wasn’t working, though

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

but to get that var.value was removed inside of a case

mumoshu avatar
mumoshu

but if we could previously refer to case fields from within later case fields, we wont need variable under case in the first place

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

we couldn’t previously

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

well…wait….could we?

mumoshu avatar
mumoshu

yeah it wasnt working so i added it in v0.24.1, which breaks existing behavior on accessing var from case

mumoshu avatar
mumoshu

i want the dependency to be one direction here. either variable can depend on case, or case can depend on variable

mumoshu avatar
mumoshu

well anyways, give me a minute and i’ll publish new variant2 release for testing

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

ok…maybe case can use var

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

seems more logical

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

mumoshu avatar
mumoshu


maybe case can use var
yeah i agree

mumoshu avatar
mumoshu


my case would ref var for any global property that doesn’t change value per case
this made sense to me

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yeah

mumoshu avatar
mumoshu

okay so this seems to have worked only when you are so lucky to have a specific Go map to have a specific key ordering

https://sweetops.slack.com/archives/CFFQ9GFB5/p1587746383322000?thread_ts=1587745980.321500&cid=CFFQ9GFB5

mumoshu avatar
mumoshu

We are unable to get the case fields in the order of their definitions. So I’d need to add some dependency analysis between the fields

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

gotcha

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so case is largely just a map of values and not some special block

mumoshu avatar
mumoshu

exactly

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

gotcha. i thought it was a special block like an exec or something

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

then in that case i think it makes sense to not overcomplicate it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

maybe make that clear with case = vs case { or maybe it is just me

mumoshu avatar
mumoshu

well it is a special block in that sense. it is just that no variant block has support for self referencing yet

mumoshu avatar
mumoshu

i do think there are 3 types of variables that should be useful within a test:

  1. case-independent variables (variable blocks today)
  2. case-dependent variables (case block fields today)
  3. case-dependent variables that depend on 1 and 2 (does not exist today. adding support for self-referencing case fields from within case fields would be one of possible solutions to this
:--1:1
mumoshu avatar
mumoshu

3 is added in v0.25.0.

variant now builds a DAG of case fields and evaluates it in an order that all the required case fields are known when evaluating dependent fields.

so this just works:

case "ok" {
  bar = case.foo
  foo = "FOO"
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu was talking with @Erik Osterman (Cloud Posse) about a new cli command and it’d be clean if I could use an if statement on an internal depends_on. is that possible?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@johncblandii (Cloud Posse) Can you provide a mockup?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
job "something" {
  run "some job" {
    condition = opt.bool-value
  }

  run "some other job" {
    condition = !opt.bool-value
  }
}
:--1:1
mumoshu avatar
mumoshu

i understand but i’m afraid this would make the call side too complex to be tested

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

k

mumoshu avatar
mumoshu

what’s the exact usecase?

mumoshu avatar
mumoshu

i thought i would rather add condition to exec or run

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @mumoshu - don’t want to make things harder on ya!

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

on exec/run might be enough, tbh

mumoshu avatar
mumoshu

ah no! i meant im afraid making it hard for you guys to use!

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

the use case was simply being able to trigger a specific job or depends_on only if some condition is met

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

this one was specifically:…

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

job:

job 1
  param x

cli: cli job

output: [list available param.x options]

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

basically, dynamic help

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so it would be:

run "some-job" {
  condition = param.x != ""
}

exec {
  command = "echo
  condition = params.x == ""
  args = "docs here"
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

^ steps and stuff like that in there, but that’s the idea

mumoshu avatar
mumoshu

hmm? i think i’d rather do

variable "help_wanted" {
  value = param.x == ""
}

variable "show_help" {
  value = {
    job = "show help"
    with = {
       text = "docs here"
    }
  }
}

variable "help_or_run" {
  value = help_wanted ? var.show_help : var.run_it
}

run {
  job = help_or_run.job
  with = help_or_run.with
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

works if there are two things

mumoshu avatar
mumoshu

also variant2 restricts a single job to have either exec or run, not both, to force the job to be simple

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

if you need to diff for more than 2 jobs, it gets to need extra

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

no worries, though. this isn’t a blocker

mumoshu avatar
mumoshu

this is easy to implement. but what i’m afraid is that the more you do it imperatively, it gets more difficult to debug

mumoshu avatar
mumoshu

please post more examples like that. probably i’d eventually come up with something that helps you

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

well it could be used on source to dynamically load a file

mumoshu avatar
mumoshu


if you need to diff for more than 2 jobs, it gets to need extra
i’d use maps for 3 or more conditional jobs. but not sure it’s applicable to every case.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
  config "file" {
    source file {
      path = "${opt.tenants-dir}/defaults.yaml"
    }

    source file {
      if = params.tenant
      path = "${opt.tenants-dir}/${param.tenant}.yaml"
    }
  }
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

or

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
  config "file" {
    source file {
      path = "${opt.tenants-dir}/defaults.yaml"
    }

    source file {
      if = fileexists("${opt.tenants-dir}/${param.tenant}.yaml")
      path = "${opt.tenants-dir}/${param.tenant}.yaml"
    }
  }
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

could also log when things change:

depends_on "echo" {
  if = params.x
  args = { message = "blah" }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that would log when a condition is met (ex: running a specific log message for a client as opposed to without one)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

take this:

job "terraform plan" {
  step "plan init" {
    run "terraform init" {
    }
  }

  step "plan workspace" {
    run "terraform workspace" {
    }
  }

  step "plan cmd" {
    run "terraform subcommand" {
      command = "plan"
    }
  }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that init and workspace is copied to multiple places

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i could put that only in subcommand with if to toggle it based on some option/arg I pass to it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

something like:

  step "plan cmd" {
    run "terraform subcommand" {
      command = "plan"
      init = true
    }
  }
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

and subcommand could easily have:

  step "plan init" {
    if = param.init
    run "terraform init" {
    }
  }
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

(or either inside of the run)

mumoshu avatar
mumoshu

umm, sorry i dont get it yet. why you can’t run terraform plan directly/why you need terraform subcommand?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that’s just a DRY command

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it handles the dir, etc

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

don’t worry about the commands, though

mumoshu avatar
mumoshu
job "init_and_workspace" {
  step "plan init" {
    run "terraform init" {
    }
  }

  step "plan workspace" {
    run "terraform workspace" {
    }
  }
}

job "terraform" {
  depends_on "init_and_workspace" {
  }

  parameter "subcmd" {
    type = string
  }
  
  run {
    job = "util terraform run-subcommand ${param.subcmd}"
    args = ...
  }
}

job "util terraform run-subcommand plan" {
   step "do more common things" 
     ....
   }

   step "exec terraform plan" 
   }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yes and you still need to copy/paste code to multiple places

  depends_on "init_and_workspace" {
  }
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

OR you end up unnecessarily running init_and_workspace for terraform runs that don’t need it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

you end up in the same place

mumoshu avatar
mumoshu

does adding if resolve the issue of copy-pasting that depends_on?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

exactly

mumoshu avatar
mumoshu

are you saying that you would create a higher kind job that can run any low-level job w/ depends_on only run when necessary?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yes

mumoshu avatar
mumoshu

ok i believe i understand

mumoshu avatar
mumoshu

my point is, we should at least avoid adding control structures to every kind of blocks

mumoshu avatar
mumoshu

cuz that makes things too hard to test/maintain due to that there becomes many ways to achieve one thing

mumoshu avatar
mumoshu

i’d rather add items and condition to any of run, exec and step

mumoshu avatar
mumoshu

maybe run should be the best place

mumoshu avatar
mumoshu

also let’s allow calling multiple sequential runs in a job

mumoshu avatar
mumoshu

with that you could write the ideal command like

job "terraform" {
  depends_on "init_and_workspace_if_needed" {
    ...
  }

  run {
    job = "util terraform run-subcommand ${param.subcmd}"
    with = var.args_for_subcmd
  }
}

job "init_and_workspace_if_needed" {
  parameter "subcmd" {
    type = string
  }

  run {
     condition = contains(["plan", "apply"], param.subcmd)
      job = "init_and_workspace"
  }
}

job "util terraform run-subcommand plan" {
   run "do common things" 
     ....
   }

   run "exec terraform plan" 
   }
}
mumoshu avatar
mumoshu

also, the more we enhance run, it is more likely we can merge depends_on into run

mumoshu avatar
mumoshu
job "terraform" {
  run {
    condition = contains(["plan", "apply"], param.subcmd)
    job = "terraform "init_and_workspace"
  }

  run {
    job = "util terraform run-subcommand ${param.subcmd}"
    with = var.args_for_subcmd
  }
}

job "util terraform run-subcommand plan" {
   run "do common things" 
     ....
   }

   run "exec terraform plan" 
   }
}
mumoshu avatar
mumoshu

just deprecate/remove depends_on in favor of enhanced run? or just make it an alias to run? not sure which is better

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yeah, depends_on and run pretty much are the same thing anyway from a cli dev perspective

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i think the idea of branches making it too complex, variant should support us using this complexity.

we have options. that inherently means we do X or Y or Z at times so won’t be uncommon for us to do extra things at times

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

…and not do them at other times

mumoshu avatar
mumoshu


we have options. that inherently means we do X or Y or Z at times so won’t be uncommon for us to do extra things at times
yeah probably that makes sense now.

i was wondering if everything can be generalized to mapping variant opts/params to exec

mumoshu avatar
mumoshu

which isn’t realistic as it turned out that we wanna do more things “within” variant

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yeah, sometimes. sometimes we need to add a var, use a source to load something, or not based on an opt

mumoshu avatar
mumoshu

like branching, looping, etc

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yup

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

owe my son some fishing time so i’ll bbiab

mumoshu avatar
mumoshu

happy fishing thx for your feedback as always!

:--1:1
mumoshu avatar
mumoshu

Multiple conditional run blocks has been added in v0.26.0

job "terraform" {
  run {
    condition = contains(["plan", "apply"], param.subcmd)
    job = "terraform "init_and_workspace"
  }

  run {
    condition = ...
  # snip
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu along the lines of the previous question, can you do a source dynamically (only pull if param.x exists) or try/catch on a failed source load?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu it seems options are not available to a job when it is used as a source of a config

  config "state" {
    source job {
      name = "state"
      args = {
        tenant = param.tenant
      }
      key = "key"
      format = "text"
    }
  }

that works, but it does not recognize the opts.tenants-dir in the state job and it does recognize the opt when I call it directly

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

and is there a way to suppress the output when we use it with source?

deploy
Deploy entire tenant stack
------------------------------------------------------------------------
[CLIENT] Deploying terraform project: project1
------------------------------------------------------------------------

that first deploy is just reading the state and echo’ing it. I’d rather not output that when using it as a source to a config

:--1:1
mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) regarding the first question, does the state have option "tenants-dir"? if so, what’s the expected value of it in your specific example?

mumoshu avatar
mumoshu

are you expecting something to be automatically propagated/set in the state job as it is called from within a config block?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

tenants-dir is a top-level option

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i expect anything top-level to propagate down every job/run/source/etc no matter the chain

mumoshu avatar
mumoshu

gotcha! seems like i’ve missed adding support for propagating global opts/params for that

:--1:1
mumoshu avatar
mumoshu

should be fixed in v0.25.1

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

you’re ridiculously fast, man. lol

fast_parrot1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i love it

:--1:1
mumoshu avatar
mumoshu

Also, since v0.25.2 config source job output is suppressed

bananadance1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
02:58:48 AM
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu can a variable not reference a config?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
  variable "trigger-name" {
    value = param.name == "state" ? conf.file.state : param.name
  }

error:

  23:     value = param.name == "state" ? conf.file.state : param.name

There is no variable named "conf".
mumoshu avatar
mumoshu

no. it’s opposite

mumoshu avatar
mumoshu

i can reverse the evaluation order. but not sure which is better

mumoshu avatar
mumoshu

I’ve just reversed the order anyway. Please try v0.27.0!

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

will do

2020-04-23

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu I’m working with tests and it seems null should be allowed for err as opposed to requiring an empty string.

  case "ok" {
    tenant = "acme"
    exitstatus = 0
    err = ""
  }

^ this works just fine.

take err out and you get an error: This object does not have an attribute named "err".

set it to err = null and you get:

panic: handler for type dynamic not implemented yet [recovered]
        panic: handler for type dynamic not implemented yet

goroutine 10 [running]:
testing.tRunner.func1(0xc0000cb500)
        /opt/hostedtoolcache/go/1.13.10/x64/src/testing/testing.go:874 +0x3a3
panic(0x21ebee0, 0xc0000968e0)
        /opt/hostedtoolcache/go/1.13.10/x64/src/runtime/panic.go:679 +0x1b2
[github.com/mumoshu/variant2/pkg/app.(*App).execAssert(0xc00020b080](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execAssert\(0xc00020b080), 0xc0005cc980, 0xc0004dda0a, 0x5, 0x2840660, 0xc000598a80, 0x0, 0x0)
        /home/runner/work/variant2/variant2/pkg/app/app.go:965 +0x9d0
[github.com/mumoshu/variant2/pkg/app.(*App).execTestCase(0xc00020b080](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execTestCase\(0xc00020b080), 0xc000035740, 0x12, 0xc0005b5ec0, 0x1, 0x1, 0xc0005ab5e0, 0x1, 0x1, 0xc000035820, ...)
        /home/runner/work/variant2/variant2/pkg/app/app.go:1091 +0x659
[github.com/mumoshu/variant2/pkg/app.(*App).execTest.func1(0xc0000cb500)](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execTest\.func1\(0xc0000cb500\))
        /home/runner/work/variant2/variant2/pkg/app/app.go:1037 +0xc2
testing.tRunner(0xc0000cb500, 0xc0005be790)
        /opt/hostedtoolcache/go/1.13.10/x64/src/testing/testing.go:909 +0xc9
created by testing.(*T).Run
        /opt/hostedtoolcache/go/1.13.10/x64/src/testing/testing.go:960 +0x350
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu I’d like to dynamically run different parameters to run and reference them in my case without manually typing them again. case.* access seems only available in a run or assert and not within the case itself or in variable.

example from https://github.com/mumoshu/variant2/blob/9753895bdea9bd949596159aaf2f3f9f489575dc/examples/concurrency/concurrency_test.variant:

  case "ng1" {
    concurrency = 0
    err = "delay of ${case.delayone} is less than ${case.delaytwo}"
    stdout = ""
    delayone = 0
    delaytwo = 1
  }

  run "test" {
    concurrency = case.concurrency
    delayone = case.delayone
    delaytwo = case.delaytwo
  }

instead of having type type:

err = "delay of 0 is less than 1"

…which then requires me to change 0 in multiple places if I change the value of delayone to anything else.

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

mumoshu avatar
mumoshu

i think this should better be covered by variable

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

mumoshu avatar
mumoshu
  case "ng1" {
    concurrency = 0
    stdout = ""
    delayone = 0
    delaytwo = 1
  }

  variable "err" {
     value = "delay of ${case.delayone} is less than ${case.delaytwo}"
  }

  run "test" {
    concurrency = case.concurrency
    delayone = case.delayone
    delaytwo = case.delaytwo
  }

  assert "..." {
      condition = var.err == ...
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i tried a var from a case and it threw an error

mumoshu avatar
mumoshu

not sure if i’ve already added support for variable under test blocks. let me check and add it if not exist yet

mumoshu avatar
mumoshu

which error was it?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
  19:     value = case.tenant

There is no variable named "case".
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
  variable "tenant" {
    type = string
    value = case.tenant
  }

  case "ok" {
    tenant = "client"
    ...
  }
mumoshu avatar
mumoshu

oo too bad

mumoshu avatar
mumoshu

i think it should just work. pls expect me it fix it today :)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

in typical mumoshu fashion.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

much appreciated. i’ll be around to test

mumoshu avatar
mumoshu

This should be fixed in v0.24.1

:--1:1
mumoshu avatar
mumoshu

since v0.25.0, you should use this instead

  case "ng1" {
    concurrency = 0
    stdout = ""
    delayone = 0
    delaytwo = 1
    err = "delay of ${case.delayone} is less than ${case.delaytwo}"
  }
:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu can we get some output clean-up on test failures? separating the runs + their failures and maybe some color coding would really help with readability.

https://github.com/mumoshu/variant2/issues/20

Improve readability of test output · Issue #20 · mumoshu/variant2

➜ variant test terraform init terraform plan -out=acme.planfile aws –profile company-blah2-helm eks update-kubeconfig –name=company-blah2-eks-cluster –region=us-east-2 –kubeconfig=/path/to/kube…

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu i think something is up with global options again. this bool option I can echo at the top level and it is true. then in jobs run from that job it is false.

deploy (dry-run = true) -> helmfile deploy (dry-run = false)

➜ ./nbo deploy acme --tenants-dir=../tenants --dry-run --kubeconfig-path=~/.kube

INTERNAL: dry? false. kc-path? /dev/shm

TOP-LEVEL: dry? true. kc-path? ~/.kube

the kc opt is a string and dry is a bool. those should both internally be different values.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

deploy uses depends_on in a loop to call deploy switch which calls other methods using run job/with (the dynamic approach)

mumoshu avatar
mumoshu

Thx for reporting! This should be fixed since v0.24.2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Sweet!!

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I can’t pinpoint why this is failing. Any thoughts here, @mumoshu? I tried wrapping the run.res.exitstatus in trimspace since case.exitstatus used it too.

 (string)--- FAIL: deploy (0.06s)
    --- FAIL: deploy/ok (0.05s)
        app.go:1040: case "ok": assertion "out" failed: this expression must be true, but was false: run.res.stdout == case.out
            , where run.res.stdout=------------------------------------------------------------------------
            [CLIENT] Deploying terraform project: project1
            ------------------------------------------------------------------------
            terraform deploy project1 
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying terraform project: project2
            ------------------------------------------------------------------------
            terraform deploy project2 
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying helmfile project: project1
            ------------------------------------------------------------------------
            helmfile apply project1 
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying helmfile project: project2
            ------------------------------------------------------------------------
            helmfile apply project2 
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying terraform project: project3
            ------------------------------------------------------------------------
            terraform deploy project3 
            
             (string) case.out=------------------------------------------------------------------------
            [CLIENT] Deploying terraform project: project1
            ------------------------------------------------------------------------
            terraform deploy project1
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying terraform project: project2
            ------------------------------------------------------------------------
            terraform deploy project2
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying helmfile project: project1
            ------------------------------------------------------------------------
            helmfile apply project1
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying helmfile project: project2
            ------------------------------------------------------------------------
            helmfile apply project2
            
            ------------------------------------------------------------------------
            [CLIENT] Deploying terraform project: project3
            ------------------------------------------------------------------------
            terraform deploy project3
             (string)
FAIL
Error: test exited with code 1
mumoshu avatar
mumoshu

i often see this when i have

assert "out" {
    condition = (run.res.set && run.res.stdout == case.out) || !run.res.set
  }

and

mumoshu avatar
mumoshu
  case "ok1" {
    exitstatus = 0
    err = ""
    out = trimspace(<<EOS
expected output
EOS
    )
mumoshu avatar
mumoshu

perhaps the actual output contains more new lines at the end so avoid using trimspace would fix it

mumoshu avatar
mumoshu
  case "ok1" {
    exitstatus = 0
    err = ""
    out = <<EOS
expected output
EOS
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i tried without trimspace

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
test "deploy" {
  variable "kubeconfig-path" {
    type = string
    value = "/path/to/kube"
  }

  variable "namespace" {
    type = string
    value = "nbo"
  }

  variable "region" {
    type = string
    value = "us-east-2"
  }

  variable "tenant" {
    type = string
    value = "client"
  }

  case "ok" {
    exitstatus = 0
    err = ""
    out = <<EOS
------------------------------------------------------------------------
[CLIENT] Deploying terraform project: project1
------------------------------------------------------------------------
terraform deploy project1

------------------------------------------------------------------------
[CLIENT] Deploying terraform project: project2
------------------------------------------------------------------------
terraform deploy project2

------------------------------------------------------------------------
[CLIENT] Deploying helmfile project: project1
------------------------------------------------------------------------
helmfile apply project1

------------------------------------------------------------------------
[CLIENT] Deploying helmfile project: project2
------------------------------------------------------------------------
helmfile apply project2

------------------------------------------------------------------------
[CLIENT] Deploying terraform project: project3
------------------------------------------------------------------------
terraform deploy project3

    EOS
  }

  run "deploy" {
    dry-run = true # only echo the args
    kubeconfig-path = var.kubeconfig-path
    namespace = var.namespace
    region = var.region
    tenant = var.tenant
    tenants-dir = "./fixtures"
  }

  assert "error" {
    condition = run.err == case.err
  }

  assert "exitstatus" {
    condition = run.res.exitstatus == case.exitstatus
  }

  # TODO: this doesn't work as expected in variant2 0.24.1
  assert "out" {
    condition = run.res.stdout == case.out
  }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

one thing i did notice was ` (string) case.out` is prefixed with a space. i’m unsure if that is variant injecting one or something about the general output

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

hrmm…i copied out the output and it seems to be a newline character or something. trimspace doesn’t seem to be doing the full job

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yup. removed the forced \n and it no longer errored. interesting

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that’s even with condition = trimspace(run.res.stdout) == trimspace(case.out)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so it seems stdout isn’t actually processing properly

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

…newline chars, specifically

mumoshu avatar
mumoshu

so we may have extra space(s) prefixed in case.out AND run.res.stdout having more newlines than it should?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i don’t think the extra spaces is a problem on case.out. it seems stdout newlines when an echo something\n happens

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i removed the \n and it worked

mumoshu avatar
mumoshu

so you mean you get too much newlines(not only one added by \n) when you had echo something\n,

but not when echo something?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yeah, there is an extra character after something that is not cleaned up by trimspace so I’m thinking it isn’t stored w/ the newline or something

mumoshu avatar
mumoshu

ah interesting!

it can be variant2 is doing something nasty after trimspace is applied in case but before it’s processed in assert

mumoshu avatar
mumoshu

thx, i’ll investigate

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

coolio

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

(sorry about the flood today; digging into a new area with variant2)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

btw, @mumoshu, I was able to completely write full CLI coverage with 0 knowledge of the test approach within a day of work for about 15+ commands

mumoshu avatar
mumoshu

awesome!!

party_parrot1
mumoshu avatar
mumoshu

have u also tried mocking/successfully mocked dependent command like terraform in tests?

https://github.com/mumoshu/variant2/blob/master/examples/simple/simple_test.variant#L26

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i saw that, but copying around the path and setting env on it all seemed a bit much

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i stuck in a dry-run check and turned it into echo vs the actual command

mumoshu avatar
mumoshu

yeah i understand

mumoshu avatar
mumoshu

makes sense

mumoshu avatar
mumoshu

should we add a helper that works in 80% of cases

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

mocking as a first-class citizen would be sweet, though

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

definitely

mumoshu avatar
mumoshu

like

add_to_path = "path/to/the/mock/executable"

run "job" {
   ...
}

assert "whatever"
  ...
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

also, testing in general wouldn’t really care about what cli or whether the CLI runs. i just care that exec was called or run or depends_on

mumoshu avatar
mumoshu

i also considered about adding an inline syntax for mock creation. but that seemed to bloat the test code

mumoshu avatar
mumoshu


i just care that exec was called or run or depends_on
agree.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

allow us to not actually execute the command but just check the args passed to exec

mumoshu avatar
mumoshu

interesting. that might work

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
assert "exec args" {
  condition = run.exec.command == "helmfile"
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

also, just checking to see that a run is called and not actually running it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

example:

job "helmfile apply" {
  description = "Apply the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to operate on"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile shell" {
    command = "apply"
    project = param.project
    tenant  = param.tenant
  }
}
mumoshu avatar
mumoshu

one job run can results in multiple execs. probably we’d need a syntax for asserting on a sequence of multiple execs

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I don’t care what helmfile shell does here. I just want to make sure the command , project, and tenant were passed

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

< yeah

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

fair point

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

heading out, but i’ll check back tomorrow

mumoshu avatar
mumoshu

maybe just list expected exec and runs in sequence under a specific block for mocking?

mock {
  # this should match the first invocation on terraform
  exec {
    command = "terraform"
    args = ["plan"]
    dir = "expectedir"
  }


\# this should match the second invocation on terraform
  exec {
    command = "terraform"
    args = ["apply"]
   dir = "expecteddir"
  }
}

run "job to test" {
   ...
}

assert "..." {
   ...
}
party_parrot1
mumoshu avatar
mumoshu

w/ another wording:

expect {
  # this should match the first invocation on terraform
  exec {
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that’d be sweet if we could use case in there

mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) Thanks a lot for all your feedbacks! cu

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

args = [case.command]

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

np

mumoshu avatar
mumoshu

would it be like

case "ok1" {
  expect {
     exec {

?

mumoshu avatar
mumoshu

ah ok

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yeah, something like that with the args passed to it would be great

mumoshu avatar
mumoshu

starting v0.28.0, you can write expectations on execs like:

expect exec {
   command = ...
   args = ...
   dir = ...
}
1

2020-04-22

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Issue created for building with a . as the path: https://github.com/mumoshu/variant2/issues/19

Exporting with a dot for current directory throws a Go error · Issue #19 · mumoshu/variant2

Problem ➜ variant export binary . mycli When exporting a binary, the path must be a directory name or an absolute path. Using the . throw an error. Error go: malformed import path &quot;.&quot;: in…

1
mumoshu avatar
mumoshu

thx!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Zachary Loeber you have a pulse on everything. Question for you: https://github.com/mumoshu/variant2/issues/17#issuecomment-617835218

feat: Multi-tenancy of Web UI · Issue #17 · mumoshu/variant2

Importing from mumoshu/variant#33 Add a Login page and a project-selection page in front of #32. It should be deployed along with multiple instances of variant server #31 (+ perhaps #32). This may …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Where’s/What’s the silver bullet for building an enterprise-grade Web UI today?

Zachary Loeber avatar
Zachary Loeber
03:04:57 PM

@Zachary Loeber has joined the channel

Zachary Loeber avatar
Zachary Loeber

I’ve been seeking such a bullet myself. I was going to look towards some of the fairwinds projects for inspiration (https://github.com/FairwindsOps/polaris for instance) as they seem to use pure Go based solutions but I haven’t gotten that far yet. Most solutions I’ve seen incorporate some java frameworks that instantly turn me off.

FairwindsOps/polaris

Validation of best practices in your Kubernetes clusters - FairwindsOps/polaris

Zachary Loeber avatar
Zachary Loeber

I think they use buffalo framwork behind the scenes but, again, I’m barely scratching at this particular itch of mine yet

Zachary Loeber avatar
Zachary Loeber

sorry

2020-04-21

2020-04-20

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu is there a way to exec a cli and allow the user to enter a response to the cli?

ex: mycli terraform apply will prompt to apply unless I tell it to automatically say “yes” and mycli just exits skipping over the prompt.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes he added support for this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Look for keyword interactive

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Support Interactive Commands · Issue #9 · mumoshu/variant2

what Running variant with interactive commands fails I believe the original variant supported this mode use-case Terraform apply can be run interactively on the console. While this is not the prima…

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I saw that…
Auto-prompting via interactive messages

One of cool features of the bot is that when you missed to specify values for certain options, it will automatically start a interactive session to let you select and input missing values within Slack. You don’t need to remember all the flags nor repeat lengthy commands anymore.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that’s where you can enter a variant job without required options and it’ll ask you about those options

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ya can’t wait to test the slack bot functionality later

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

this request was more about a tf destroy asking for yes after you start the process

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha, though in this case terraform supports that out of the box so we can get around it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yes, if we don’t want to allow a non -auto-approve scenario

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so i figured it’d be fine for our scenario, but i’ve hit it a couple times where i wanted to inspect a destroy before running

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ya so maybe a new feature request for “prompt” - but I see this getting complicated with paths

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yuppers

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i was thinking of just respecting prompts from exec commands

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

the second was support for our own prompts

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

can kinda do it now if we set an option as required but not provide it on the cmd

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Respecting prompts is supported today, right?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

unless i’m missing something, it will fly right through it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

for an exec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Right so see the link above :-)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

no. that’s not it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Need to add “interactive” flag to the job

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that link is interactive for your variant cli

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it is not interactive for an exec command

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrm so see the example in the associated commit?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That is allowing the user to interact with the command in exec and not fly through

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

interactive = true

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

^ that’s it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But also a command line arg to rm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

rm -i makes it interactive

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yes, but i was looking at the fix

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

good to know. thx for the ref. i should’ve read the code in the first place.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So maybe we should dynamically pass auto-approve=false or true based on options to variant

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Could probably be done using a ternary

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

prob can be automated

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

default to false unless in a deploy command where it passes true

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

not going to sweat it now, but should be easy

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu another one; can we control the exit status in the event we’re looping over something?

ex (pseudo code):

loop item as [call1, call2, call3]
  item()

Let’s say call1 passes, call2 fails, I would expect call3 to not run. Right now it continues to try and run 3 even though I need 2 to pass before 3 runs.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think this what what “need” solves

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Call3 needs call2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yeah

mumoshu avatar
mumoshu


Right now it continues to try and run 3 even though I need 2 to pass before 3 runs.
What did you observe this behavior with? steps ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(also, was this using the need keyword @johncblandii (Cloud Posse)?)

mumoshu avatar
mumoshu

FYI: The needs attribute is documented here: https://github.com/mumoshu/variant2#concurrency

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Will that work within a dynamic loop?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

And yes, it was with a step/run

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I’ll bbiab and can provide code

mumoshu avatar
mumoshu

pls! an example code that doesn’t work would always be helpful

mumoshu avatar
mumoshu

well do we have a dynamic loop thing? oh, is it depends_on "JOB" { items = ... }?

mumoshu avatar
mumoshu

depends_on is for multiple independent dependencies of the parent job

mumoshu avatar
mumoshu

so trying to build a dependency graph across the items of a depends_on seems conceptually wrong

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Yes, it using that part for the looping. It isn’t intended for a dependency graph. I just need to exec jobs in a loop and stop when they fail

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Is there a concept of looping outside of depends on?

mumoshu avatar
mumoshu


exec jobs in a loop and stop when they fail
this seems like building a dependency graph that each next job depends on its previous job?

but anyways,
[10:35 AM] Is there a concept of looping outside of depends on?
no

mumoshu avatar
mumoshu

and depends_on items should be executed serially and stop on the first error

mumoshu avatar
mumoshu

are you seeing different behavior?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yes, i saw different behavior. i’ll try to replicate

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)


this seems like building a dependency graph that each next job depends on its previous job?
i guess you could call it one, but that’s not the intent. it isn’t a dependency as in the cli depends on job 1 to run for 2. it is a dependency in how we’re managing our config file for looping

mumoshu avatar
mumoshu


that’s not the intent
true! that makes sense

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Context:

order:
  - terraform.account
  - terraform.iam-tenant-roles
  - terraform.cloudtrail
  - terraform.vpc

deploy job:

  depends_on "deploy switch" {
    items = conf.file.order

    args = {
      item   = item
      tenant = param.tenant
    }
  }

switch is the job that splits the left/right of the . and calls an internal job

job "deploy switch" {
  run {
    job = "${var.subcommand} deploy"

    with = {
      project = var.project
      tenant  = param.tenant
    }
  }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so based on the order above, you end up with:

job terraform deploy account job terraform deploy iam-tenant-roles job terraform deploy cloudtrail job terraform deploy vpc

mumoshu avatar
mumoshu

thx. the job ordering seems correct.

mumoshu avatar
mumoshu
variable "file" {
  value = {
    order = [
      "terraform.account",
      "terraform.iam-tenant-roles",
      "terraform.cloudtrail",
      "terraform.vpc",
    ]
  }
}

job "example" {
  depends_on "deploy switch" {
    items = var.file.order
    args = {
      item   = item
    }
  }
}

job "deploy switch" {
  option "item" {
    type = string
  }

  variable "subcommand" {
    value = split(".", opt.item)[0]
  }

  variable "project" {
    value = split(".", opt.item)[1]
  }

  run {
    job = "${var.subcommand} deploy"
    with = {
      project = var.project
    }
  }
}

job "terraform deploy" {
  option "project" {
    type = string
  }

  exec {
    command = "bash"
    args = ["-c", <<SCRIPT
echo job terraform deploy ${opt.project}; if [ ${opt.project} == "cloudtrail" ]; then echo simulated error 1>&2; exit 1; fi
SCRIPT
    ]
  }
}

stops on the first (simulated) error, as expected, for me:

 VARIANT_DIR=examples/issues/depends_on_stop_on_first_error ./variant run example
job terraform deploy account
job terraform deploy iam-tenant-roles
job terraform deploy cloudtrail
simulated error
Error: command "bash -c echo job terraform deploy cloudtrail; if [ cloudtrail == "cloudtrail" ]; then echo simulated error 1>&2; exit 1; fi
": exit status 1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I’ll revisit, @mumoshu.

:--1:1

2020-04-17

2020-04-16

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu it looks like indirect run works in a step

fails:

  step "deploy run" {
    run {
      job = "${var.subcommand} deploy"

      with = {
        project = var.project
        tenant  = param.tenant
      }
    }
  } 

works:

  # step "deploy run" {
    run {
      job = "${var.subcommand} deploy"

      with = {
        project = var.project
        tenant  = param.tenant
      }
    }
  # }

Use case: I added a previous step to do a simple log message and thought that was the problem, but ran into this.

mumoshu avatar
mumoshu

indirect run in step isn’t implemented yet. will make it work later!

mumoshu avatar
mumoshu

would you be okay if it removed the older syntax run "NAME" { ... } ?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i think it is more verbose for normal runs

mumoshu avatar
mumoshu

true

mumoshu avatar
mumoshu

it’s just that direct/indirect run distinction makes it a bit harder to explain and maintain for me

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it might make since overall since the normal syntax is type name

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

this is run "name of another type"

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

understandable

mumoshu avatar
mumoshu

also to me, any block with label(s) like someblock "LABEL1" "LABEL2" { } makes me think that if it can be referenced from within hcl expressions with someattr = someblock.LABEL1.LABEL2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

right and you can’t with run

mumoshu avatar
mumoshu

exactly.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so that seems like a valid reason to go with job =

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

btw, i have this all automated. the latest changes are indeed working

mumoshu avatar
mumoshu

yes. but on the other hand i do think that the newer syntax is verbose. not sure what i should do

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I like the foo "something" { ... } syntax more than I like foo = "something"with = { ... }

That said, not definitely not worth keeping both if it makes it harder for you to implement things. This is just a stylistic preference.

mumoshu avatar
mumoshu

great!

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

sometimes verbose is a must

mumoshu avatar
mumoshu

that’s very insightful

mumoshu avatar
mumoshu

thx. i’ll try to build a more complex example variant2 command myself and see if the verbosity is acceptable.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, we could do a screenshare if you want to see how we’re using it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is a “terragrunt killer”

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

cool

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu what’s the relationship to a job’s run status code and the next step running?

mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) are you talking about the case that you’ve two or more steps in a job?

job "example" {
  step "one" {
    run "x" {
    }
  }

  step "two" {
    run "y" {
    }
  }
}
mumoshu avatar
mumoshu

the second step “two” runs only when the first step exists with 0

mumoshu avatar
mumoshu

i.e. job example exits on the first step with non-zero exit code

mumoshu avatar
mumoshu

and example inherits the exit status of the step if it returned a non-zero exit status

mumoshu avatar
mumoshu

for example, x exited with 1 results in variant run example exits with 1

mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) Thanks for all your feedbacks!

I think I’ve finished all the important bugs reported/features requested so far. But please feel free to poke me if I’m missing something

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

:boom: :boom: :boom: :boom: :boom:

Just verified the dry-run option now works properly as a bool.

I also confirmed the env works perfectly fine with cli shell env and merges internal env with the system env

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu is there a reason you have to manually pass a global option into tertiary runs?

Ex: run cli b --namespace hi

option "namespace"
  default = "cp"

job "b"
  echo opt.namespace // "hi"

  run "c" // no namespace set here

  run "d"
    namespace = opt.namespace

job "c"
  echo opt.namespace // "cp"

job "d"
  echo opt.namespace // "hi"
mumoshu avatar
mumoshu

just that i thought verbosity is important there

mumoshu avatar
mumoshu

perhaps variant could just try to fill in missing option/parameter arguments from the global parameter/option?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

well for jobs needing calling jobs calling jobs calling jobs is pushing a global down N levels

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

but you don’t have to define the option for the > secondary jobs

mumoshu avatar
mumoshu

yeah probably

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so i either have to define global options on every single job or hope devs understand i can set args on a run that don’t exist on the job

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

devs working on the same cli, i mean

mumoshu avatar
mumoshu

I was mostly concerned about a like

option "namespace" {
   type = string
}

job "deploy" {
  option "dir" {
    type = string
  }
  exec {
    command = "kubectl"
    args = ["-n", opt.namespace, "-f", opt.dir]
  }
}

job "all" {
  option "namespace" {
     type = string
     default = ""
  }

  step "app1" {
    run "deploy" {
      namespace = opt.namespace == "" ? "app1" : opt.namespace
      dir = "app1"
   }

  step "app2" {
    namespace = "app2"
    dir = "app2"
  }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

as a way to override

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i gotcha

mumoshu avatar
mumoshu

not sure we should allow that though

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it seems conflicts internal to a job to a global might confusing

mumoshu avatar
mumoshu

perhaps just forbidding to shadow a global option with a local option, while

mumoshu avatar
mumoshu

automatically filling missing global args as suggested above would be nice?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

duplication-wise, yes. for example:

job "helmfile apply" {
  description = "Apply the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "apply"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}

job "helmfile destroy" {
  description = "Destroy the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "destroy"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}

job "helmfile diff" {
  description = "Diff the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "diff"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}

job "helmfile lint" {
  description = "Lint the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  step "cmd" {
    run "helmfile" {
      command   = "lint"
      namespace = opt.namespace
      region    = opt.region
      project   = param.project
      tenant    = param.tenant
    }
  }
}

job "helmfile sync" {
  description = "Sync the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "sync"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
    namespace = opt.namespace
    region    = opt.region
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

^ those are all duplicated

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

add a new global, add a new line for every job

mumoshu avatar
mumoshu

that’s indeed painful!

mumoshu avatar
mumoshu

okay then, let’s disallow option/parameter shadowing AND do enhance run to fill on missing args from global opt/param

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

sweet

mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) Just released v0.23.0 for this.

2
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
job "deploy switch": shadowing global option "dry-run" with option "dry-run" is not allowedError: job "deploy switch": shadowing global option "dry-run" with option "dry-run" is not allowed

party_parrot

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s great! ya, i think this is much better. if a job wants to redefine a global option seems like a bad inconsistency. instead the job should define a new option. this change you made addresses that! I like the exception msg.

:--1:1
1

2020-04-15

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu does this work?

option "dry-run" {
  default     = false
  description = "simulate an install"
  type        = bool
}

I’m getting this error no matter what I do:

job "deploy switch": option "dry-run": can't convert Go string to boolError: job "deploy switch": option "dry-run": can't convert Go string to bool

it happens because of:

  depends_on "deploy switch" {
    items = conf.file.order

    args = {
      dry-run = opt.dry-run
      item    = item
      tenant  = param.tenant
    }
  }

I tried to run it with tobool(opt.dry-run), but I get that error every time I try to pass the option or just plain ol’ false to the run

mumoshu avatar
mumoshu

Could you share me a fuller example? Is the option dry-run defined in the same job as the depends_on is defined in?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it is a global

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

full job:


\#!/usr/bin/env variant

\# vim: filetype=hcl

option "dry-run" {
  default     = false
  description = "simulate an install"
  type        = bool
}


\# option "tenant" {

\#   description = "Tenant to interact with"

\#   short       = "t"

\#   type        = string

\# }


\# option "project" {

\#   default     = ""

\#   description = "Terraform project to process"

\#   short       = "p"

\#   type        = string

\# }

job "deploy" {
  description = "Init all projects"

  parameter "tenant" {
    type = string
  }

  config "file" {
    source file {
      path = "${param.tenant}.yaml"
    }
  }

  depends_on "deploy switch" {
    items = conf.file.order

    args = {
      item    = item
      tenant  = param.tenant
    }
  }
}

job "deploy switch" {
  private = true

  parameter "item" {
    default     = []
    description = "a config param to deploy in the format: cli.project; e.g. terraform.eks; e.g. helmfile.reloader"
    type        = string
  }

  parameter "tenant" {
    type = string
  }

  # option "dry-run" {
  #   default     = false
  #   description = "simulate an install"
  #   type        = bool
  # }

  variable "item-split" {
    type  = list(string)
    value = split(".", param.item)
  }

  variable "type" {
    type  = string
    value = var.item-split[0]
  }

  variable "project" {
    type  = string
    value = var.item-split[1]
  }

  variable "subcommand" {
    type = string
    # value = opt.dry-run ? "echo ${var.type}" : var.type
    value = "echo ${var.type}"
  }

  run {
    job = "${var.subcommand} deploy"

    with = {
      project = var.project
      tenant  = param.tenant
    }
  }
}

job "echo terraform deploy" {
  parameter "project" {
    description = "args to pass to subcommand"
    type        = string
  }

  parameter "tenant" {
    type = string
  }

  exec {
    command = "echo"
    args    = [opt.dry-run, param.tenant, "terraform deploy", param.project]
  }
}

job "echo helmfile deploy" {
  parameter "project" {
    description = "args to pass to subcommand"
    type        = string
  }

  parameter "tenant" {
    type = string
  }

  exec {
    command = "echo"
    args    = [param.tenant, "helmfile apply", param.project]
  }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

still refining a bit

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so this one runs since i’m not passing it directly to the switch job

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it always outputs false, though

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

but it seems depends_on args are string conversions

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

but it fails

mumoshu avatar
mumoshu

Thanks for the detailed report! Just spotted the cause and fixed locally.

I’ll publish the next patch release a few hours later

mumoshu avatar
mumoshu

I believe your config is correct. Please just wait for the patch release

mumoshu avatar
mumoshu

This one is working fine after the fix

job "deploy switch" {
  option "dry-run" {
    type = bool
  }

  option "item" {
    type = string
  }

  option "tenant" {
    type = string
  }

  exec {
    command = "bash"
    args = ["-c", "echo deploy switch tenant=${opt.tenant} item=${opt.item}"]
  }
}

job "example" {
  config "file" {
    source file {
     path = "${context.sourcedir}/conf.yaml"
   }
  }

  option "dry-run" {
    type = bool
    default = false
  }

  parameter "tenant" {
    type = string
    default = "mytenant"
  }

  depends_on "deploy switch" {
    items = conf.file.order
    args = {
      dry-run = opt.dry-run
      item    = item
      tenant  = param.tenant
    }
  }

  exec {
    command = "bash"
    args = ["-c", "echo Done."]
  }
}
variant run example
go build -o variant ./pkg/cmd
deploy switch tenant=mytenant item=foo
deploy switch tenant=mytenant item=bar
Done.
mumoshu avatar
mumoshu

There was definitely a bug around type conversion. Having two or more types of values in args was causing the issue.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

ahhhh

mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) Just released v0.22.1 with the fix

2020-04-14

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

can we run dynamic depends_on like this? depends_on "${var.cmd} init" {

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

or would this be a place for an exec?

mumoshu avatar
mumoshu

Unfortunately it isn’t supported

mumoshu avatar
mumoshu

Would you mind sharing your exact use-case? I’m considering how much we might need it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i’ll share code in a sec. let me give context

mumoshu avatar
mumoshu

I mean, I’m eager to make the depends_on target dynamic if it’s necessar

mumoshu avatar
mumoshu

THx

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

we have a file with a list like this:

order:
  - terraform.proj1
  - terraform.proj2
  - helmfile.proj3
  - terraform.proj4
mumoshu avatar
mumoshu

Note that HCL2 doesn’t support expressions inside the block label (e.g. NAME in depends_on "NAME" { }

mumoshu avatar
mumoshu

So we need an alternative syntax if we end up implementing it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
  1. read the file
  2. loop over order
  3. run internal job for ${var.command} plan or ${var.command} diff
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

ah, gotcha

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

so this code outputs a simple echo of each project with the command split:

➜ ../cli/nbo deploy all -t acme
terraform account
terraform cloudtrail
terraform vpc
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

config file looks like:

order:
  - terraform.account
  - terraform.cloudtrail
  - terraform.vpc
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

code to output it is like this:


\#!/usr/bin/env variant

\# vim: filetype=hcl

option "tenant" {
  description = "Tenant to interact with"
  short       = "t"
  type        = string
}

option "project" {
  default     = ""
  description = "Terraform project to process"
  short       = "p"
  type        = string
}

config "file" {
  source file {
    path = "${opt.tenant}.yaml"
  }
}

job "deploy all" {
  description = "Init all projects"

  depends_on "e" {
    items = conf.file.order

    args = {
      a = item
      project = opt.project
      tenant = opt.tenant
    }
  }
}

job "e" {
  option "a" {
    default     = coalesce(list(""))
    description = "args to pass to subcommand"
    type        = string
  }

  variable "asplit" {
    type = list(string)
    value = split(".", opt.a)
  }

  # variable "cmd" {
  #   type = string
  #   value = var.asplit[0]
  # }

  # variable "project" {
  #   type = string
  #   value = var.asplit[1]
  # }

  exec {
    command = "echo"
    args    = var.asplit
  }

  # run "${opt.a} init" {

  # }
  # depends_on "${var.cmd} init" {
  #   args = {
  #     project = var.project
  #     tenant  = opt.tenant
  #   }
  # }
}
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

some comments still in there

mumoshu avatar
mumoshu

makes sense

mumoshu avatar
mumoshu

and you wanna make e in depends_on "e" dependent on each item?

mumoshu avatar
mumoshu

so that you can run helmfile apply for helmfile.proj3 and terraform apply for items like terraform.proj1?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i technically want to run internal jobs within the variant cli

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yes

mumoshu avatar
mumoshu

that makes total sense

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i could break it out to individual commands via exec, but plan would need init and workspace

mumoshu avatar
mumoshu

firstly, i don’t think variant2 as of today has a good way to express that

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

ok

mumoshu avatar
mumoshu

but i do want great support for your use-case

mumoshu avatar
mumoshu

actually i was planning to create a command that looks very similar to yours

mumoshu avatar
mumoshu

myself

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

nice!

mumoshu avatar
mumoshu

let me dump my ideas

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

cool

mumoshu avatar
mumoshu
job "deploy all" {
  description = "Init all projects"

  depends_on "deploy" {
    items = conf.file.order

    args = {
      dir = item
    }
  }
}

job "deploy" {
  description = "deploy the app defined in the directory"
  
  option "dir" {
     type = string
  }

  variable "cmd" {
    value = "${ match(opt.dir, "helmfile") ? "helmfile" : "apply" }"
  }

  exec {
    command = var.cmd
    dir = opt.dir
     args = ["apply"]
  }
}
mumoshu avatar
mumoshu

This is the first option. The idea is that you define a deploy job can is able to call helmfile apply or terraform apply depending on the dir it is targeted

mumoshu avatar
mumoshu

so that it can be used from depends_on

mumoshu avatar
mumoshu

The second option is that you make each item of conf.file.order a object which has two atrributes “type” and “dir”

job "deploy all" {
  description = "Init all projects"

  depends_on "deploy" {
    items = conf.file.order

    args = {
      dir = item.dir
      type = item.type
    }
  }
}

job "deploy" {
  description = "deploy the app defined in the directory"
  
  option "dir" {
     type = string
  }

  option "type" {
     type = string
  }

  variable "cmd" {
    value = opt.type
  }

  exec {
    command = var.cmd
    dir = opt.dir
    args = ["apply"]
  }
}
mumoshu avatar
mumoshu

Does either of the two options look good to you? If so we don’t need to enhance variant2 in any way for this. This should just work with today’s variant2.

mumoshu avatar
mumoshu

Otherwise we need something more. Not sure if that’s about making depends_on dynamic. The downside of doing so would be that the deploy all job can be more complex which makes unit-testing it harder

mumoshu avatar
mumoshu

On the other hand, either of the above two options is easily testable. I mean, you can write test for deploy all and deploy separately and independently

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

the all vs the individual is the plan. so this setup is where we have multiple internal jobs we want to call.

ex: tf plan calls tf init followed by tf workspace then the exec for terraform plan

the deploy will need to call apply and that will consist of init -> workspace -> plan -> apply

mumoshu avatar
mumoshu

Makes sense. As we don’t have any notion of mutliple conditional execs within a single job, that’s not possible

mumoshu avatar
mumoshu

perhaps the only way would be exec another variant command

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

mmm…so command is the cli with the values passed there

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that could be a good workaround

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

is there a concept of “self” in this case?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

or would we need to bake in the cli name in the exec ’s command?

mumoshu avatar
mumoshu


would we need to bake in the cli name in the exec ’s command?
yes, that’s the way it works now.

but there’s a potential solution. we already have context that is used for context.sourcedir which basically translates to dirname $($0)

mumoshu avatar
mumoshu

so it would be straight-forward to add context.self or perhaps context.command that represents $0

mumoshu avatar
mumoshu

which one do you prefer? or i’m open to alternatives

mumoshu avatar
mumoshu

The third possible option would be to enhance steps.

Variant2 job can have two or more steps. Each step is able to depend on other step in the same job, run concurrently, and call an another job.

It doesn’t support conditional execution(i.e. if). But if we add support for if, like

job "apply" {
  # params and opts here

  # instead of exec or run, use steps

  step "do terraform apply" {
    if = opt.type == "terraform"
    run "terraform apply" {
      # args here
    }
  }

  step "do helmfile apply" {
    if = opt.type == "helmfile"
    run "helmfile apply" {
      # args here
    }
  }

we neither need self nor dynamic depends_on target, or write/call a lengthy shell script.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that looks spot on, @mumoshu! i definitely love the simplicity of an if statement to allow control of the run as well

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

that’d be SWEET.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

thoughts, @Erik Osterman (Cloud Posse)?

mumoshu avatar
mumoshu

ideally, i’d like each variant2 job to have only a single responsibility, so that we can enforce testability of each job.

that is, a job should only do any of the followings:

  • selectively run a job(new)
  • run a static graph of jobs(steps. note that steps has no conditions or loops)
  • run a job(job with a run. note that you can’t have multiple runs within a single job. use steps if you need to do so)
  • run a command(job with a exec. note that you can’t have multiple execs within a job)

so i got to think i would prefer a dedicated block selectively { } for it, like:

job "apply" {
  # params and opts here

  # instead of exec or run, use steps

  selectively {
    run "helmfile deploy" {
      args = {}
      if = opt.type == "helmfile"
    }

    run "terraform deploy" {
      args = {}
      if = opt.type == "terraform"
    }
  }
:--1:1
mumoshu avatar
mumoshu

or change the run syntax to accept an expression for the job name, so that we can write

job "apply" {
  # params and opts here

  # instead of exec or run, use new run syntax

  variable "runs" {
    value = {
       helmfile = {
         name = "helmfile deploy"
         args = {...}
      }
      terraform = {
        name = "terraform deploy"
        args = {...}
      }
    }
  }

  variable "job" {
    value = var.runs[opt.type]
  }
  
  run {
    job = var.job.name
    with = var.job.args
  }
mumoshu avatar
mumoshu

coincidentally, the last solution requires https://github.com/mumoshu/variant2/issues/15

Add support for variables to use variables in the value · Issue #15 · mumoshu/variant2

This specific use-case is to help DRY up code and make it a bit more readable inside of the job. job &quot;myjob&quot; { option &quot;value&quot; { default = coalesce(list(&quot;&quot;)) descriptio…

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I think I like the latter. Being able to dynamically choose a job in this way could be beneficial.

I do think anyone coming from Terraform (hcl background) would look for a way to use or not use a job (see count on resources) so maybe both?

mumoshu avatar
mumoshu

Got it. run without the job name in a label should now work as documented in https://github.com/mumoshu/variant2/#indirect-run

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

:100:1
mumoshu avatar
mumoshu

Just published v0.22.0. It also includes the solution for the issue 15

party_parrot1
mumoshu avatar
mumoshu

Perhaps we’d better deprecate the old run syntax because having both seems confusing?

mumoshu avatar
mumoshu

For the consistency reason, I guess we’d better change the step syntax as well

BEFORE

  step "STEP_NAME" {
    run "JOB_NAME" {
      dir = "deploy/environments/${opt.env}/manifests"
    }
  }

AFTER

  step "STEP_NAME" {
    job = "JOB_NAME"
    with = {
      dir = "deploy/environments/${opt.env}/manifests"
    }
  }
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Oh, man. That looks solid. Going to dig into this later today and will report back my results

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m continuously baffled by how fast you implement things @mumoshu

1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Seriously. It baffles me.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in the past, you used args instead of with ; is there a subtle difference I’m not picking up?

mumoshu avatar
mumoshu

I’ve been long wanted a clear distinction between exec’s args of list(string) and job run’s args of map(any). I thought that picking a different name for job run’s would achieve that.

But I’m open to suggestions

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

along the previous lines above, I’m passing in cmd.project as a string, splitting it, then using it. can I have a variable depend on a variable?

  variable "asplit" {
    type = list(string)
    value = split(".", opt.a)
  }

  variable "cmd" {
    type = string
    value = var.asplit[0]
  }

  variable "project" {
    type = string
    value = var.asplit[1]
  }

I get an error when I do:

Error: Unknown variable

  on ../cli/main.variant line 51:
  (source code not available)

There is no variable named "var".

Error: Unknown variable

  on ../cli/main.variant line 51:
  (source code not available)

There is no variable named "var".

Error: ../cli/main.variant:51,13-16: Unknown variable; There is no variable named "var".
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

btw, I can create github issues for these things at any point. just let me know.

mumoshu avatar
mumoshu


can I have a variable depend on a variable?
No, but seems valid and feasible to add support for it.

Would u mind creating a github issue for that?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
Add support for variables to use variables in the value · Issue #15 · mumoshu/variant2

This specific use-case is to help DRY up code and make it a bit more readable inside of the job. job &quot;myjob&quot; { option &quot;value&quot; { default = coalesce(list(&quot;&quot;)) descriptio…

mumoshu avatar
mumoshu

thank you so much!

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

will do

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

gonna crash, but i’ll check in tomorrow. variant2 is good stuff, @mumoshu. there is a lot of potential here. thx for all the work

mumoshu avatar
mumoshu

thanks for all your feedback! it’s helpful and encouraging

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

John just showed me the working bones of what we wanted to get working!

party_parrot2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

almost there

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s so sweet.

2

2020-04-13

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)


Error: handler for type object not implemneted yet
@mumoshu happens with in 0.21.1

job "shell" {
  description = "Run a command in a shell"
  private = true

  option "dir" {
    default = ""
    description = "Directory to run the command"
    type = string
  }

  option "env" {
    default = {}
    description = "Directory to run the command"
    type = map(string)
  }

  parameter "commands" {
    description = "List of commands to execute"
    type = list(string)
  }

  exec {
    dir = opt.dir
    env = opt.env
    command = "bash"
    args = ["-c", join("\n", param.commands)]
  }
}
mumoshu avatar
mumoshu

This should be fixed in v0.21.2 as well :–1:

1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

@mumoshu any thoughts on the above exception?

mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) hey! thx - im trying to reproduce it and fixing a few bugs along the way. you can expect me to cut a new release soon :)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i’m around for a bit so ping whenever

mumoshu avatar
mumoshu

maybe you can reproduce it only when you cann the shell job from another job, right? i got different issues when tried to run from variant run shell whatever

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

hrmm…i didn’t try it from run

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

btw, VARIANT_TRACE fails for short options. have you seen that?

mumoshu avatar
mumoshu

I suppose not. What do you mean by “short options”?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
Error: unknown shorthand flag: 'p' in -p
Usage:
  nbo [flags]
Flags:
  -h, --help   help for nbo
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

changed -p to --project and the command would run

mumoshu avatar
mumoshu

Whoa! I have no idea how that could happen!

mumoshu avatar
mumoshu

Just curious but you’re enabling the trace like VARIANT_TRACE=1 ./variant run shell 'echo foo' 'echo bar', right?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

using a shim

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

VARIANT_TRACE=1 ../cli/nbo -p acme

mumoshu avatar
mumoshu

seems ok

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i’ll try to recreate. 1 sec

mumoshu avatar
mumoshu

Are you sure you’ve set short = "p" within option "project" { ... }?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yeah

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I’ll keep an eye on it if i can recreate it

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i had a code issue i was trying to trace when i hit that

mumoshu avatar
mumoshu

okay. anyways, i managed to reproduce the handler for type object not implemneted yet error. i’ll shortly fix it and then try to pass run "shell" on my own env

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
06:14:37 AM
mumoshu avatar
mumoshu

after that there would be much less probability that you’d encounter any issue i hope so

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

awesome

mumoshu avatar
mumoshu

Re: VARIANT_TRACE issue - I couldn’t reproduce it

$ VARIANT_TRACE=1 PATH=$(pwd):$PATH ./test1/test1 example -d
deploy switch tenant=mytenant item=foo
deploy switch tenant=mytenant item=bar
Done.
mumoshu avatar
mumoshu

I’ve created the shim by modifying an example command contained in the variant2 repo examples/issues/cant-convert-go-str-to-bool to have short = "d" for the example’s dry-run option:

job "example" {
  config "file" {
    source file {
     path = "${context.sourcedir}/conf.yaml"
   }
  }

  option "dry-run" {
    type = bool
    default = false
    short = "d"
  }


\# snip

and ran export shim to create the shim:

$ ./variant export shim examples/issues/cant-convert-go-str-to-bool ./test1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I can’t reproduce the issue any longer either, @mumoshu.

mumoshu avatar
mumoshu

on my machine variant run shell 'echo foo' 'echo bar' is working fine

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

it is the map part that’s the problem

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

1 sec

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

here is a run from a job.

  run "shell" {
    commands = param.commands
    dir      = "../projects/helmfiles/${opt.project}"
    project  = opt.project
    tenant   = opt.tenant

    env = {
      AWS_PROFILE = "nbo-${opt.tenant}-helm"
      TENANT      = opt.tenant
      PROJECT     = opt.project
      KUBECONFIG  = "~/.kube/kubecfg.${opt.tenant}-helm"
    }
  }
mumoshu avatar
mumoshu

Much appreciated!

:--1:1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

here’s an interesting one; details to follow:

in ./helmfile.yaml: error during ../environments.yaml.part.0 parsing: template: stringTemplate:22:9: executing "stringTemplate" at <exec "yq" (list "read" (printf "../../../tenants/%v.yaml" (env "TENANT")) (printf "projects.helmfile.%v.values" (env "PROJECT")))>: error calling exec: exec: "yq": executable file not found in $PATH

COMMAND:
  yq read, ../../../tenants/acme.yaml, projects.helmfile.reloader.values

ERROR:
  exec: "yq": executable file not found in $PATH
Error: command "helmfile --environment tenant diff" in "../projects/helmfiles/reloader": exit status 1
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

yq exists in my shell

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

running this formerly with the shell command (bash -C …) works perfectly fine

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

here’s the job:

job "helmfile shell" {
  description = "Run a command in a shell targeted specifically at a projects/helmfiles project for Helmfile commands"
  private     = true

  parameter "args" {
    description = "List of args to execute"
    type        = list(string)
  }

  exec {
    args    = param.args
    command = "helmfile"
    dir     = "../projects/helmfiles/${opt.project}"

    env = {
      AWS_PROFILE = "${opt.tenant}-helm"
      KUBECONFIG  = "~/.kube/kubecfg.${opt.tenant}-helm"
      PROJECT     = opt.project
      TENANT      = opt.tenant
    }
  }
}
mumoshu avatar
mumoshu

I think you need to set PATH in exec { }

mumoshu avatar
mumoshu

Perhaps variant2 should just inherit the process envvars when you use exec { env = ... } but omitted PATH in the env attr?

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

i would expect it to

mumoshu avatar
mumoshu

alright! i’ll see what i can do(probably i’ll just make it so

:--1:1
mumoshu avatar
mumoshu

I couldn’t reproduce the PATH issue with variant2 alone

I ran this successfully:

job "example" {
  exec {
    command = "jq"
    args = ["-h"]
  }
}
mumoshu avatar
mumoshu

Ooops, it does reproduce when env = { whatever } is set

mumoshu avatar
mumoshu

v0.22.2 fixes this.

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

got a sigsegv when using an empty list args = []

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1f9bafe]

goroutine 1 [running]:
[github.com/mumoshu/variant2/pkg/app.ctyToGo(0x283bb40](http://github\.com/mumoshu/variant2/pkg/app\.ctyToGo\(0x283bb40), 0xc0004b3da0, 0x21295e0, 0xc0004b3dc0, 0xc0004b3d80, 0x0, 0x0, 0x0)
        /home/runner/work/variant2/variant2/pkg/app/app.go:1271 +0x2be
[github.com/mumoshu/variant2/pkg/app.(*App).execRunInternal(0xc0000a38c0](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execRunInternal\(0xc0000a38c0), 0xc000031280, 0xc0004b3c80, 0xc00000c3c0, 0x0, 0x0, 0x0)
        /home/runner/work/variant2/variant2/pkg/app/app.go:1048 +0x1bf
[github.com/mumoshu/variant2/pkg/app.(*App).execRun(0xc0000a38c0](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execRun\(0xc0000a38c0), 0xc000031280, 0xc0004b3c80, 0xc00000c3c0, 0x28268c0, 0xc00049dc70, 0x0, 0x0, 0x0)
        /home/runner/work/variant2/variant2/pkg/app/app.go:1187 +0x81
[github.com/mumoshu/variant2/pkg/app.(*App).execJob(0xc0000a38c0](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.execJob\(0xc0000a38c0), 0xc000031280, 0xc0005e6fd0, 0x7, 0x0, 0x283b980, 0xc00045f740, 0x0, 0x0, 0x0, ...)
        /home/runner/work/variant2/variant2/pkg/app/app.go:812 +0x5fa
[github.com/mumoshu/variant2/pkg/app.(*App).Job.func1(0x0](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.Job\.func1\(0x0), 0x0, 0x0)
        /home/runner/work/variant2/variant2/pkg/app/app.go:626 +0x9d2
[github.com/mumoshu/variant2/pkg/app.(*App).Run(0xc0000a38c0](http://github\.com/mumoshu/variant2/pkg/app\.\(\*App\)\.Run\(0xc0000a38c0), 0xc0005e6fd0, 0x7, 0xc00052c5d0, 0xc00052c600, 0xc00058b9e0, 0x1, 0x1, 0x0, 0x0, ...)
        /home/runner/work/variant2/variant2/pkg/app/app.go:497 +0xc3
[github.com/mumoshu/variant2.(*Runner).Cobra.func1(0xc0005e1180](http://github\.com/mumoshu/variant2\.\(\*Runner\)\.Cobra\.func1\(0xc0005e1180), 0xc000520740, 0x0, 0x4, 0x0, 0x0)
        /home/runner/work/variant2/variant2/variant.go:624 +0x114
[github.com/spf13/cobra.(*Command).execute(0xc0005e1180](http://github\.com/spf13/cobra\.\(\*Command\)\.execute\(0xc0005e1180), 0xc000520700, 0x4, 0x4, 0xc0005e1180, 0xc000520700)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:826 +0x460
[github.com/spf13/cobra.(*Command).ExecuteC(0xc000032780](http://github\.com/spf13/cobra\.\(\*Command\)\.ExecuteC\(0xc000032780), 0x2812d00, 0xc000098010, 0x2812d00)
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:914 +0x2fb
[github.com/spf13/cobra.(*Command).Execute(...)](http://github\.com/spf13/cobra\.\(\*Command\)\.Execute\(\.\.\.\))
        /home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:864
[github.com/mumoshu/variant2.(*Runner).Run(0xc000483090](http://github\.com/mumoshu/variant2\.\(\*Runner\)\.Run\(0xc000483090), 0xc00024e0f0, 0x6, 0x6, 0xc00058bd78, 0x1, 0x1, 0x0, 0x0)
        /home/runner/work/variant2/variant2/variant.go:754 +0x2cc
[github.com/mumoshu/variant2.Main.Run(0x7ffeefbff81f](http://github\.com/mumoshu/variant2\.Main\.Run\(0x7ffeefbff81f), 0x3, 0x0, 0x0, 0x0, 0x7ffeefbff818, 0xa, 0x2812d00, 0xc000098008, 0x2812d00, ...)
        /home/runner/work/variant2/variant2/variant.go:359 +0x12d
main.main()
        /home/runner/work/variant2/variant2/pkg/cmd/main.go:13 +0xb8
johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I think @Erik Osterman (Cloud Posse) mentioned this before. Using his workaround works:

arg     = coalesce(list(""))
mumoshu avatar
mumoshu

Thanks! This should be fixed in v0.21.2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

great

2020-04-12

2020-04-11

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

how can i properly type env to allow it to be optional here:

job "shell" {
  description = "Run a command in a shell"
  private = true

  option "dir" {
    default = ""
    description = "Directory to run the command"
    type = string
  }

  option "env" {
    default = {}
    description = "Directory to run the command"
    type = map(string)
  }

  parameter "commands" {
    description = "List of commands to execute"
    type = list(string)
  }

  exec {
    dir = opt.dir
    env = opt.env
    command = "bash"
    args = ["-c", join("\n", param.commands)]
  }
}

for context, this is a simple wrapper around shell commands so we can simply do things like this (note: this uses a similar wrapper used to set a dir specific to tf runs):

job "tf clean" {
  run "tfshell" {
    project  = opt.project
    tenant   = opt.tenant
    commands = ["rm -rf .terraform *.planfile"]
  }
}
mumoshu avatar
mumoshu

please let me recall how I might have designed this to work… anyways, i believe this is due to how HCL differentiates map and objefct.

i thought {} and {k=v} was for the syntax for hcl objects, not maps.

mumoshu avatar
mumoshu
mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I tried that and it didn’t work. This is for a dynamic map of different keys: map(string).

mumoshu avatar
mumoshu

i think i managed to reproduce it. will try to fix it soon! than ks

mumoshu avatar
mumoshu

@johncblandii (Cloud Posse) this should be fixed since v0.21.1.

you should be able to just use {} as seen in https://github.com/mumoshu/variant2/blob/master/examples/defaults/defaults.variant#L16

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

Sweet. Will check it out

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just add a default value for it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s a map I believe

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Vs parameter is a list

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

I tried a default. I’ll post the errors when I get back to things.

2020-04-10

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


I’m not sure if integrations with external services (e.g. Slack) should be inside variant core
I get what you’re saying - but this is so rad. In the end, we all keep saying “I want a to do chatops, just never get around to it.” Well, no more excuses.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


a lot of half-baked integrations inside the core (Docker runner, Github actions, etc), which are not maintained for the obvious reasons
that’s true…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

My guess is this is a peak in to @mumoshu personal laboratory. What I love about it is he doesn’t hold back, inhibited by figuring out where to put something.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


as v1 is not supported anymore

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu is this true? @Igor Rodionov has a massive variant1 script he’s working on right now. It would be really hard to convert it to variant2.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

understand if no new feature requests in v1. but maybe bugfixes, if they come up? (none right now)

2020-04-08

mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) depends_on under job is available since v0.20.0

mumoshu avatar
mumoshu
mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Very clean! I like it. Thanks for doing that so quickly! Going to have a demo probably next weeks #office-hours

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

whoot! I got it working now with depends_on and dynamically getting the list of items by calling keys on my configuration.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Seems like order is not preserved

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu will keys always be returned in the same order as in the configuration file?

mumoshu avatar
mumoshu

i guess no - variant doesn’t do any fancy things to preserve key ordering and i thought go’s map doesn’t preserve the key ordering

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya, fair enough. probably just means I’m using the wrong data type for the job.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
job "terraform init all" {
  description = "Init all projects"
  depends_on "terraform init" {
    items = keys(conf.terraform.projects)
    args = {
      project = item
      env = opt.env
    }
  }
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


The keys are returned in lexicographical order, ensuring that the result will be identical as long as the keys in the map don’t change.

loren avatar
loren

And in variant3 we’ll get named IDs like for_each to avoid annoying issues with order changes…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i realized I should probably be using a yaml list instead.

2020-04-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu is there something like need but for jobs rather than steps?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In a Makefile it’s very easy to have a dynamic list of target dependencies. I want to do the same with job dependencies.

e.g.


\# static deps (like steps in variant)
target: dep1 dep2 dep3

could be expressed


\# dynamically compute deps
DEPS = $(shell echo dep1 dep2 dep3) 
target: $(DEPS)

How to do this is variant2?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

E.g. define a job for “all” that runs a list of jobs (dynamically discovered from the yaml configuration file)

mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) variant2 equivalent of that would be concurrent steps:

job "dep1" {
}

job "dep2" {
}

job "terraform init all" {
  concurrency = 2

  step "run-dep1" {
    run "dep1" {
    ...
    }
  }

  step "run-dep2" {
    run "dep2" {
    ...
    }
  }

  step "do something with dep1 and dep2" {
    needs = ["run-dep1", "run-dep2"]
    run "something" {
    }
  }
}
mumoshu avatar
mumoshu

would you prefer a shorthand syntax for this?

mumoshu avatar
mumoshu

which has indeed limited usage but easier to write

mumoshu avatar
mumoshu
job "terraform init all" {
  needs = ["dep1", "dep2"]
}

hmm but how should we choose which parameters/options to be passed to dep1 and dep2 then?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So what I was trying to do was create a project configuration in yaml

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then use variant to operate on that configuration

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then create an all job (e.g. mycli terraform init all) that would run terraform init $project on each of the projects in the config in order

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

with the step notation, I have to know ahead of time each project

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but i was hoping to create my own “schema” if you will, and then using the same job to operate on each one in an unattended fashion using an all job

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I have it working now being explicit

mumoshu avatar
mumoshu

probably an ideal and better shorthand syntax for your specific example would be

job "terraform init all" {
   need "dep1" {
     param1 = "paramvalue"
     opt1 = "optvalue"
  }
  need "dep2" {
    param1 = "paramvalue"}
  }

   run "foo" {
     ...
   }
}

which will be translated to todays

job "terraform init all" {
  concurrency = 2
   step "run-dep1" {
     run "dep1" {
       param1 = "paramvalue"
       opt1 = "optvalue"
     }
  }
  step "run-dep2" {
    run "dep2" {
      param1 = "paramvalue"}
    }
  }
  step "run-foo" {
    needs = ["run-dep1", "run-dep2"]
    run "foo" {
      ...
    }
  }
}
mumoshu avatar
mumoshu


with the step notation, I have to know ahead of time each project
ah gotcha.

1
mumoshu avatar
mumoshu

this is what’s missing in the variant2 today

mumoshu avatar
mumoshu

so we don’t have a kind of for expression in variant2 dsl

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, and I am not crazy about that convention for terraform.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think it was a necessary evil, but maybe for variant there’s another syntax.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Maybe it doesn’t need to be an iterator.

mumoshu avatar
mumoshu

i hope so.. but have no concrete idea yet

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let me write up how I thought it could work maybe…

mumoshu avatar
mumoshu

how about adding items to the step syntax, so that it will “expanded” into a multiple concurrent steps?

variable "projects" {
  value = [
    map("dir", "web"),
    map("dir", "infra"),
  }
}

step "all-deps" {
  items = var.projects
  run "terraform init" {
    dir = item.dir
  }
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
job "terraform init" {
  parameter "project" {
    ...
  }
  option "extra_args" {
    ... 
  }
  exec {
    ...
    args = opt.extra_args
  }
}

job "terraform init all" {
  # note this list of "depends on" can be calculated dynamically
  depends_on = ["terrraform init eks", "terraform init efs"]
  extra_args = ["-refresh=true"]
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In this example, terraform init eks resolves to the first job and passes eks as the first positional parameter.

mumoshu avatar
mumoshu


# note this list of “depends on” can be calculated dynamically
:–1:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, I haven’t yet figured out how to do that, but haven’t tried. I assume there’s some way I could extract that from the yaml configuration.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Worst case I exec out to jq

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and globals would be passed automatically

mumoshu avatar
mumoshu

the last space-separated item in each depends_on entry gets turned into the project parameter value?

1
mumoshu avatar
mumoshu

how about options?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Updated example.

mumoshu avatar
mumoshu

thanks!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The key is that when you depends_on , you can only pass all the same parameters or options

mumoshu avatar
mumoshu

okay i see how it works

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

depends_on could maybe be steps and it expands to a set of inline steps

mumoshu avatar
mumoshu

how about this then?

job "terraform init all" {
   depends_on "terrraform init" {
     project = "eks"
  }
  depends_on  "terrraform init" {
     project = "efs"
  }
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Right, so that’s what it should expand to.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The thing is eks and efs come from the config.

mumoshu avatar
mumoshu

ah no, i mean this is the short-hand syntax to be expanded.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For reference, here’s a sample config I’m working with

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
projects:
  eks:
    module: "git::<https://github.com/cloudposse/terraform-root-modules.git//aws/eks?ref=tags/0.122.0>"
    min_nodes: 3
    max_nodes: 10
  efs:
    module: "git::<https://github.com/cloudposse/terraform-root-modules.git//aws/efs?ref=tags/0.122.0>"
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Think astro by uber: https://github.com/uber/astro

uber/astro

Astro is a tool for managing multiple Terraform executions as a single command - uber/astro

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but generalized with variant

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so it will work with helm, helmfile, terraform, etc.

mumoshu avatar
mumoshu

what i worry about is this requires me to write a command parser that should behave equivalent to shells
depends_on = [“terrraform init eks”, “terraform init efs”]
extra_args = [“-refresh=true”]

mumoshu avatar
mumoshu

hopefully i could avoid it. that’s the same reason why i’ve added exec { dir = "..." } yesterday

mumoshu avatar
mumoshu

but we definitely need a way to dynamically generate dependent job runs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


write a command parser that should behave equivalent to shells
So, it’s not unheard of in make to have a target call make ; not crazy about it, but I guess that’s always a possibility to have variant job call $0

mumoshu avatar
mumoshu

yeah true. but i won’t add a syntax sugar to just call variant run <whatever> from within variant

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

mumoshu avatar
mumoshu

so a few possible options in my mind now:

job "terraform init all" {
  exec {
    # exec will be expanded to multiple concurrent execs by replacing `item` in `args` to each item in this tuple:
    items = ["efs", "eks"]
    cmd = "variant"
    args = ["run", "terraform", "init", item, "--refresh=true"]
  }
}
mumoshu avatar
mumoshu
job "terraform init all" {
  need {
    # this step will be expanded to multiple concurrent steps by replacing `item` in the run body to each item in this tuple:
    items = ["efs", "eks"]

    run "terraform init" {
      project = item
      refresh = true
    }
  }
}
mumoshu avatar
mumoshu
job "terraform init all" {
  depends_on {
    # this step will be expanded to multiple concurrent steps by replacing `item` in the run body to each item in this tuple:
    items = ["efs", "eks"]

    run "terraform init" {
      project = item
      refresh = true
    }
  }
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Should items maybe be parameters since that’s how it’s used?

mumoshu avatar
mumoshu

Probably no. items is a predefined name of the variable which is used for extracting each item

and it’s up to the user how they would use item for

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok

mumoshu avatar
mumoshu

i mean the item can be passed as a paramter value, or an optional value

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh, now I see how you’re doing that

mumoshu avatar
mumoshu

or it can even be used to dynamically compute a value, which is eventually passed to a param or option

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that could work.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Maybe depends_on is more like for_each now?

mumoshu avatar
mumoshu

maybe?

mumoshu avatar
mumoshu

but you can omit items at all

mumoshu avatar
mumoshu

so that it will turn into a single dependency

:--1:1
mumoshu avatar
mumoshu

also, you can have more than two kinds of depends_on

mumoshu avatar
mumoshu
job "terraform init all" {
  depends_on {
    run "install terraform" {
    }
  }

  depends_on {
    # this step will be expanded to multiple concurrent steps by replacing `item` in the run body to each item in this tuple:
    items = ["efs", "eks"]

    run "terraform init" {
      project = item
      refresh = true
    }
  }

}
mumoshu avatar
mumoshu

this would firstly install terraform, then concurrently terraform-init efs and eks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha, makes sense

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s nice

mumoshu avatar
mumoshu

would you prefer this?

job "terraform init all" {
  depends_on "install terraform" {
  }

  depends_on "terraform init" {
    # this step will be expanded to multiple concurrent steps by replacing `item` in the args to each item in this tuple:
    items = ["efs", "eks"]

    args = {
      project = item
      refresh = true
    }
  }

}
mumoshu avatar
mumoshu

this way, the dependent job names are known at parsing time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh, I see what you’re doing.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, that’s better and less nesting

mumoshu avatar
mumoshu

while the terraform init targets and args can be still computed at run time

mumoshu avatar
mumoshu

great

mumoshu avatar
mumoshu

which one would you prefer, need or depends_on?

job "terraform init all" {
  need "install terraform" {
  }

  need "terraform init" {
    # this step will be expanded to multiple concurrent steps by replacing `item` in the args to each item in this tuple:
    items = ["efs", "eks"]

    args = {
      project = item
      refresh = true
    }
  }

}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So currently, you don’t use args like this and just pass everything

mumoshu avatar
mumoshu

for runs, yes

1
mumoshu avatar
mumoshu

exec does take arguments like that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ah, true

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(though thought of that more as args to the syscall function)

mumoshu avatar
mumoshu

so in my initial variant2 implementation, it was

job "foo" {
  run {
    job = "anotherjob"
    args = {
      param1 = "param1
    }
  }
}
mumoshu avatar
mumoshu


(though thought of that more as args to the syscall function)
i think that’s valid. so maybe we’re mixing two conceptually different things into the name args. not sure it’s good or bad though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if it made it easier to implement it this way, then we should keep it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

conceptually, this would make more sense to me:

options = {
 ...
}

parameters = {
...
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

vs args where they are lumped together.

mumoshu avatar
mumoshu

i understand

mumoshu avatar
mumoshu

my idea was to explain it like:

Variant reads named args, matches and sets options and parameters values by names

mumoshu avatar
mumoshu

so that we don’t need to distinguish between params and options from the call-side, which makes it more readable(in my op) and easier to refactor later:

run "terraform init" {
  params = ["efs"]
  options = {refresh = true}
}

vs

run "terraform init" {
  project = "fs"
  refresh = true
}
1
joshmyers avatar
joshmyers

Looking to move all make/glue for Terraform runs to variant2?

:--1:1
mumoshu avatar
mumoshu

i believe it’s pretty close to that

mumoshu avatar
mumoshu

i occasionally see people wrapping helmfile, helm, kubectl, and terraform with bash snippets/scripts.

perhaps that’s where variant/variant2 shines

joshmyers avatar
joshmyers

Everyone is pretty much using some wrapper script for Terraform AFAIK, be it Terragrunt/scripts/geodesic

1
joshmyers avatar
joshmyers

make, but I don’t think it is for everyone

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, I think we’ve pushed make to it’s limits

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m really excited about variant2 and the new DSL

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

working on a prototype cli for our clients as a starting off point.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also note, variant2 ships with built-in testing framework.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and slack bot

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s make on beastly steroids.

joshmyers avatar
joshmyers

Nice :–1:

tolstikov avatar
tolstikov

My 2 cents:

I’m not sure if integrations with external services (e.g. Slack) should be inside variant core

As for me, all integrations should be pluggable on demand, and the core should be kept to the very minimal functionality (“Do one thing and do it well”).

Otherwise, it could end like variant1: a lot of half-baked integrations inside the core (Docker runner, Github actions, etc), which are not maintained for the obvious reasons :slightly_smiling_face:

Anyway, that’s a great tool and I’m using variant1 heavily, but it still has some bugs, and I’m not sure if I should invest my time to fix them (as v1 is not supported anymore).

Still thinking about integrating variant2 into my workflows…

@mumoshu thank you for the great work again!

:--1:2
mumoshu avatar
mumoshu

I thought there was no critical bug in variant1 that’s why I switched to spend more time on writing variant2. if you find any, please file issue(s)! I’m not gonna abandon variant1 any time soon

mumoshu avatar
mumoshu


I’m not sure if integrations with external services (e.g. Slack) should be inside variant core
I hear you. It’s just that I couldn’t come up with such pluggable interface at the time of writing variant2.

For example, how would an universal slack bot engine communicate with variant2 to show a dialog to ask the user for filling missing options to the job? I had no idea.

mumoshu avatar
mumoshu


Otherwise, it could end like variant1: a lot of half-baked integrations inside the core (Docker runner, Github actions, etc), which are not maintained for the obvious reasons 
Just curious, but why did you find them half-baked?

I mean, I use most of them in production and no one reported critical issues or submitted PRs to fix them. So I was considering all of them just working as expected.

mumoshu avatar
mumoshu

Anyways, thanks a lot for your feedback!

2020-04-06

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu shouldn’t positional parameters in variant2 should up in help? I see the flags, but not the parameters.

mumoshu avatar
mumoshu

it should! probably i just missed implementing it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Error: accepts between 1 and 1 arg(s), received 0
Usage:
  mycli terraform init [flags]

Flags:
      --cachedir string   Module cache folder
  -h, --help              help for init

Global Flags:
      --env string   Environment to operate on
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
job "terraform init" {
  parameter "project" {
    type = string
    description = "Project to provision"
  }
...
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, it appears it’s not possible to use parameters in defaults for options?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  parameter "project" {
    type = string
    description = "Project to provision"
  }

  option "cachedir" {
    type = string
    default = try(".modules/${param.project}")
    description = "Module cache folder"
  }
mumoshu avatar
mumoshu

Yes it’s not possible. I couldn’t find out how this should work:

job "x" {
  parameter "a" {
    type = string
    default = "foo"
  }
  
  parameter "b" {
    type = string
  }
}

what should ./mycmd x bar do then? is bar intented for specifying a, or b?

mumoshu avatar
mumoshu

I believe in many cases it would be better to use option for anything that can be defaulted.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I discovered variables too - so that’s good.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Using this instead of parameters inside of options defaults

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is there something like locals in terraform? (a more terse way to express a lot of variables)

mumoshu avatar
mumoshu

what would you use locals for?

mumoshu avatar
mumoshu

i mean why variables can’t be used instead?

mumoshu avatar
mumoshu


a more terse way to express a lot of variables
ah so you want it just for a short-hand syntax for a lot of variables?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
locals {
   moduledir = "${opt.cachedir}"
   id = "${opt.namespace}-${opt.stage}-${opt.name}"
....
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

more terse.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

than

variable "moduledir" {
   value = "${opt.cachedir}"
}

variable "id" {
  value = "${opt.namespace}-${opt.stage}-${opt.name}"
}
mumoshu avatar
mumoshu

seems true!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(not a deal breaker, just a nice to have)

mumoshu avatar
mumoshu

i just prefer a TYPE RESOURCE { ATTRS } syntax

mumoshu avatar
mumoshu

is locals widely used in terraform?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh ya! all over the place.

mumoshu avatar
mumoshu

oh really. then we should add it to variant2 as well

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(because in terraform variable default values cannot have interpolations)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so it’s a bit unfair to compare! haha

mumoshu avatar
mumoshu

ah good point

mumoshu avatar
mumoshu

perhaps we can deprecate variant2 variables in favor of locals?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm… so variable could be confusing for those coming from terraform where these are ways to pass settings like option and parameter in variant. But I think it’s pretty quick and easy to see that it’s different.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

another idea:

variable "locals" {
  foo = "bar"
  apple = "delicious"
}

variable "mysql" {
  username = "test"
  host = "localhost"
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so locals is just arbitrary

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but this always for a terse expression of a lot of settings and an easy way to group them.

mumoshu avatar
mumoshu

looks nice

mumoshu avatar
mumoshu

just curious but how would you define a single variable then?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or maybe introduce a new type?

mumoshu avatar
mumoshu

i’m assuming you declared var.locals.foo, var.lobals.apple, var.mysql.username, var.mysql.host in your above example, right?

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
settings "mysql" {
  username = "test"
  host = "localhost"
}
mumoshu avatar
mumoshu

yeah, or just variables would work

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ohhh true

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s better

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

variable and variables

mumoshu avatar
mumoshu

yeah. then i would get a little annoyed on the inconsistency between two

variable "foo" {
  value = "FOO"
}

variables "bar" {
  baz = "BAZ"
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I take it this is not possible:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
variable "foo" {
  value = {
    baz = "BAZ"
  }
}
mumoshu avatar
mumoshu

im not sure either, but this one has more possibility to work:

variable "foo" {
  value = map({
    baz = "BAZ"
  })
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Okay, let’s just punt on this. Not important.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m working on passing a list to another job. I get the following error:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Error: handler for type tuple not implemneted yet

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Note, no line numbers (like variant2 usually outputs)

mumoshu avatar
mumoshu

try list(["foo", "bar"])

mumoshu avatar
mumoshu

in hcl2 tuples and lists are different.

can you use either of them in terraform?(then perhaps terraform has an automatic type conversion between list <-> tuple?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so I think there’s a bug maybe? let me show you….

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
job "shell" {
  description = "Run a command in a shell"
  parameter "commands" {
    description = "List of commands to execute"
    type = list(string)
  }

  exec {
    command = "bash"
    args = ["-c", join("\n", param.commands)]
  }
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so I have defined it as list(string) not a tuple

mumoshu avatar
mumoshu

i didn’t consider it as a bug

mumoshu avatar
mumoshu

in hcl2 ["-c", join("\n", param.commands)] is a tuple

mumoshu avatar
mumoshu

so you need to convert it to a list: list(["-c", join("\n", param.commands)])

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu avatar
mumoshu

ah well, give me a minute

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

no list in the example (e.g. list(["-e", opt.script]) )

mumoshu avatar
mumoshu

i think i misread your example. yeah exec.args can take either types

mumoshu avatar
mumoshu

well, are you trying to call the job shell from another job, or from the command-line?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But I don’t like HEREDOC syntax b/c the closing demark needs to be at character zero on a new line

loren avatar
loren

Not to sidetrack, but does the <<- heredoc syntax not work in variant2? Hard to tell what is a general hcl2 thing vs a terraform-specific thing…

loren avatar
loren

To improve on this, Terraform also accepts an indented heredoc string variant that is introduced by the <<- sequence:

block {
  value = <<-EOT
  hello
    world
  EOT
}

In this case, Terraform analyses the lines in the sequence to find the one with the smallest number of leading spaces, and then trims that many spaces from the beginning of all of the lines, leading to the following result:

hello
  world
loren avatar
loren
Expressions - Configuration Language - Terraform by HashiCorp

The Terraform language allows the use of expressions to access data exported by resources and to transform and combine that data to produce other values.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks for pointing that out!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haven’t tried it yet, but will if it comes up again.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so I thought I’d just convert it to use a list of commands instead

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So your example is:

job "shell" {
  parameter "script" {
    type = string
  }

  parameter "path" {
    type = string
  }

  exec {
    command = "bash"
    args = ["-c", param.script]
    env = {
      PATH = param.path
    }
  }
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I thought I could just do:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
job "shell" {
  parameter "commands" {
    type = list(string)
  }

  parameter "path" {
    type = string
  }

  exec {
    command = "bash"
    args = ["-c", join("\n", param.commands)]
    env = {
      PATH = param.path
    }
  }
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And call it like this:

job "app deploy" {
  option "path" {
    type = string
    default = ".:${abspath("${context.sourcedir}/mocks/kubectl")}:/bin:/usr/bin"
  }
  run "shell" {
    commands = [
    "kubectl -n ${opt.namespace} apply -f ${context.sourcedir}/manifests/"
    ]
    path = opt.path
  }

  assert "path" {
    condition = opt.path != ""
  }
}
mumoshu avatar
mumoshu

yeah but variant2 doesn’t support passing list(string) from command-line args so i was just curious how you tried to call it

mumoshu avatar
mumoshu

ah okay

mumoshu avatar
mumoshu

try

commands = list([
    "kubectl -n ${opt.namespace} apply -f ${context.sourcedir}/manifests/"
    ])
mumoshu avatar
mumoshu

i think we don’t have a type conversion from tuple -> list there today.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, so I changed it to this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…same error

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu avatar
mumoshu

ok i reproduced it on my machine, too

mumoshu avatar
mumoshu

hmm

mumoshu avatar
mumoshu

ahhhh

mumoshu avatar
mumoshu

please try:

commands = list(
    "kubectl", "-n", opt.namespace, "apply", "-f", "${context.sourcedir}/manifests/"
    )
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so what I’m trying to do is create a “script” by joining a list of commands.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not passing commands to exec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(where each arg should be passed like in your example)

mumoshu avatar
mumoshu

ah okay

mumoshu avatar
mumoshu

then

commands = list(
    "kubectl -n ${opt.namespace} apply -f ${context.sourcedir}/manifests/"
    )
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, that was it!

so not: list(["a", "b", "c]), but list("a", "b", "c")

mumoshu avatar
mumoshu

exactly. the former creates a single item list with the type of element being a tuple(string), which isn’t what you want

mumoshu avatar
mumoshu

but hey - i just implemented the automatic type conversion so you can use either, like you would in terraform

mumoshu avatar
mumoshu
feat: passing uni-type hcl2 tuple as the `list(string)` or `list(int)… · mumoshu/[email protected]

args So that this example should just work, without converting the tuple ["kubectl -n …] into a list(string) with list("kubectl -n …"). `` option namespace { descript…

mumoshu avatar
mumoshu

it’s available since v0.18.0

mumoshu avatar
mumoshu

it would also work!

mumoshu avatar
mumoshu

when you need an explicit type conversion

mumoshu avatar
mumoshu
mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

:--1:2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sweet!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I see some test for chdir, but no example

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I noticed that exec is not valid in step "..." { ... } blocks

mumoshu avatar
mumoshu

there’s no chdir in variant2 dsl. how would you use it?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so not all terraform commands support a directory argument

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. terraform output

mumoshu avatar
mumoshu

yes, exec isn’t available under step. use run instead to delegate to a job

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

of course, I can wrap this in a script block, but was hoping not to.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not a blocker, just a nice to have

mumoshu avatar
mumoshu

the general recommendation is to wrap any shell script within exec called from within a job, so that your variant command can be easily tested

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I saw you had a nice example of writing tests for variants

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haven’t looked closer yet though at that.

mumoshu avatar
mumoshu

so you’d better create a terraform job that cd before executing terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if one job cd’s into a directory, will the next job be run there?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(doubt that)

mumoshu avatar
mumoshu
job `terraform` {
  paramter "workdir" {
    type = string
  }

  exec {
    command = "bash"
    args = ["-c", "cd ${param.workdir}; terraform ...."]
  }
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

note the problem with this is now we dont’ free shell-escaping

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It goes from:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  exec {
    command = "terraform"
    args = concat(list(param.subcommand), opt.args)
  }

(safe)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(still working on it)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(the problem I’m working on is that when I’m running exec, I have a bug somewhere, but variant just exits 1 with no output)

mumoshu avatar
mumoshu


now we dont’ free shell-escaping
good point

mumoshu avatar
mumoshu

should we add

exec {
  dir = "workdir"
  command = ...
}

?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(pretty please - that would make things a lot easier in the long run).

mumoshu avatar
mumoshu

i believe i know how shell escaping is hard and that’s why i made exec a combination of command and args

mumoshu avatar
mumoshu

alright, working on it

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we also use a lot of relative paths in helmfile

mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) when would you use relative paths in variant2?

mumoshu avatar
mumoshu

fyi: exec dir and hidden jobs is available since v0.19.0

https://github.com/mumoshu/variant2/releases/tag/v0.19.0

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks!! works as intended. it.

mumoshu avatar
mumoshu


if one job cd’s into a directory, will the next job be run there?
no. and i believe it’s a great thing. previous jobs shouldn’t affect later jobs whereas possible.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

bug: if type = "string" (syntax error) variant exits non-zero, but prints no error messages. obviously, this was my user error.

mumoshu avatar
mumoshu

thanks! i’ve fixed it so that it will emit a nicer message like this:

Error: Invalid type specification

  on example/test.variant line 3, in option "namespace":
   3:   type = "string"

A type specification is either a primitive type keyword (bool, number, string) or a complex type
constructor call, like list(string).
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks!!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

is it deliberate that global parameters and options do not get passed by run automatically? I need to explicitly pass them everywhere.

mumoshu avatar
mumoshu

not really. would you prefer global parameters implicitly inherited to jobs?

that would also mean that job-specific parameters would hide global parameters by names

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


would you prefer global parameters implicitly inherited to jobs
Yes, from a user perspective that seems natural that globals are indeed global. But I admit I don’t know the implications of that.
job-specific parameters would hide global parameters by names
Not sure I grok the implications of this. Sounds not good.

2020-04-04

2020-04-02

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu on variant2, is this where you’re currently headed? looks really nice.

mumoshu avatar
mumoshu

absolutely!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re considering variant for the next phase of a project and wondering if we should invest in variant1 or variant 2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does variant2 support nesting of commands?

mumoshu avatar
mumoshu

I would definitely recommend variant 2 for your next project.

I think it’s almost complete(I can’t come up with anything to add anymore for almost a month) but please feel free to submit feature requests when you find something is still missing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu I didn’t see an example of how to nest commands

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…like in variant 1

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

E.g. “mycli command subcommand”

mumoshu avatar
mumoshu

the last major thing I’d like to decide until releasing variant 2 is wether we add a variant 1 compatibility https://archive.sweetops.com/variant/2020/01/#a3343add-6c11-4f17-8ed0-5b7e3a98e2f0 cc/ @tolstikov

SweetOps #variant for January, 2020

SweetOps Slack archive of #variant for January, 2020.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am okay with no backwards compatibility at this point. It’s a radical departure.

mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) You should use

job "bar" {
  import = "./path/to/dir"`
}

to nest all imported commands under “bar”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmm would about with in a fully self-contained script?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Without importing other files

mumoshu avatar
mumoshu

I don’t generally recommend it as it makes a single variant 2 script too big to read but

mumoshu avatar
mumoshu
mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

mumoshu avatar
mumoshu

job "command subcommand" would define variant run command subcommand.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ohhhhhhhhhh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Easy enough. I assumed it would be expressed with nesting somehow

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@johncblandii (Cloud Posse)

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)
12:11:45 AM

@johncblandii (Cloud Posse) has joined the channel

mumoshu avatar
mumoshu

I didn’t like we had been forced to nest task to define nested commands in variant 1

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I agree that this will look more readable in the long run

mumoshu avatar
mumoshu

also - this syntax makes it more searchable. just grep for job "foo bar" for any subcommand. this wasn’t possible with variant 1

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

True - that’s a plus

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov

Igor Rodionov avatar
Igor Rodionov
12:14:34 AM

@Igor Rodionov has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@mumoshu this begs the question - have you considered HCL2 for Helmfile?

mumoshu avatar
mumoshu

Yeah! Just hesitated to create a feature request for that in the helmfile repo. (I think) there’s so many people who use k8s but not tf and it seemed not appealing at all for non-tf users

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ya could be controversial

mumoshu avatar
mumoshu

also i’m not yet sure on which project i’d build something for managing set of helm releases w/ hcl2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Btw have you see the new kpt tool?

mumoshu avatar
mumoshu

yep

mumoshu avatar
mumoshu

i was a bit disappointed on it - i was expecting something similar to helm but for kustomize

mumoshu avatar
mumoshu

apparently it is not

mumoshu avatar
mumoshu

kpt live apply seems very useful tho

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That was my interpretation of of it - only from reading faq - that it was “helm for kustomize”; surprised it’s not

mumoshu avatar
mumoshu

so a possible feature would be that kpy live apply is integrated into helmfile apply, which allows us to deploy helmfile managed apps without helm…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmmm maybe not time for helmfile? I mean why not build libs for variant2?

mumoshu avatar
mumoshu

yeah i think that’s a valid question

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Could maybe reimplement a lot of this without some of the tech debt

mumoshu avatar
mumoshu

so perhaps i’d enhance variant 2 to add hcl2 syntax on top of something similar to helmfile

mumoshu avatar
mumoshu

and use kpt live apply with that? just brain storming

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yep - but maybe not corner your self by making it generalized

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

E.g. this is the tool to replace Terragrunt for terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Helmfile for helm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

“Kptfile” for kpt

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is a way to declaratively express all of that in one tool

:--1:1
:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What we can do though is make certain things easier to express

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With less boilerplate

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So Helmfile like business logic expressed in a variant2 lib

mumoshu avatar
mumoshu

that makes sense

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The next thing to consider is the common characteristic behind all succesful language alike this: a registry component (ideally just GitHub)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Does the import functionality just use go getter?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So we can import GitHub repos?

mumoshu avatar
mumoshu

its local - but you can use https://github.com/variantdev/mod integrated into variant2 for dependency management

variantdev/mod

Missing package manager for any task runners and build tools e.g. make and variant - variantdev/mod

mumoshu avatar
mumoshu

let’s say you’d want a external variant2 lib hosted under [github.com/cloudposse/variant2libs](http://github\.com/cloudposse/variant2libs), you’ll write a yourapp.variantmod like

module "yourapp" {
  dependency "github_release" "variant2libs" {
    source = "cloudposse/variant2libs"
    version = "> 1.0.0"
  }

   directory "lib/variant2libs" {
     source = "[github.com/cloudposse/variant2libs?ref=${dep.variant2libs.version}](http://github\.com/cloudposse/variant2libs\?ref=\$\{dep\.variant2libs\.version\})"
   }

running variant mod build resolves the latest variant2libs release that mathces the semver constraint > 1.0.0 and downloads the whole tree for the release under lib/variant2libs

:100:1
mumoshu avatar
mumoshu

then in yourcmd.variant, you import the libs like

job "foo" {
  module = "./yourapp.variantmod"
  import = "./lib/variant2libs"
}
mumoshu avatar
mumoshu

:point_up: module = "./yourapp.variantmod" instructs variant2 to load the module from the variantmod file

mumoshu avatar
mumoshu


can we just stick it in the main variant file?
No, not yet. I’m still unsure whether we should allow inlining it or not

mumoshu avatar
mumoshu


Does the import functionality just use go getter?
back to your question - yes, it’s almost like that.

but its worth noting that, under the hood, variant2 delegates dependency management to another tool

mumoshu avatar
mumoshu

so import = "path" doesn’t need go-getter

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, that is pretty rad

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Wow

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You have like generalized terraform. This is magical.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Does it have to be in a separate file like that?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can we just stick it in the main variant file?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha so for right now, I need to let go the drive to have it self contained. I mean, it makes sense as you have it, and we obviously break out out our terraform code. Will wait to use it in practice before making and changes.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For example being able to inline the module map :-)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ahhh HCL probably doesn’t allow mixed types for a key

mumoshu avatar
mumoshu

alright anything else you’d want variant2 to add before you actually use it?

mumoshu avatar
mumoshu

yes. i’ll probably rename module to import_module or something similar and use module for inlined modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Lol haha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In that case, could import then be optional? e.g. it defaults to something like .variant/modules/$sha (just thinking “what would terraform do?”)

mumoshu avatar
mumoshu

hm.. maybe? I do understand your motivation. just not sure how the current variant2 module system fits with that

:--1:1
mumoshu avatar
mumoshu

the variant2 module also allow you to install executable binaries required by your command

mumoshu avatar
mumoshu

perhaps we’d better have a dedicated syntax to import a remote variant2 lib, similar to what we had in variant1

mumoshu avatar
mumoshu

how would you update your variant2 module?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm… good point. mycmd init -refresh ?

mumoshu avatar
mumoshu

the good part of the variant2 module system is that you can run variant2 mod up or mod up to fetch all the dependencies and update respective <yourmodule>.variantmod.lock files

mumoshu avatar
mumoshu

to be pull-requested for reviews. this allows you to manage dependency updates like you would do with go mod for go projects

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya, makes sense.

mumoshu avatar
mumoshu

everything is technically possible.

it is just that i need some time to think about how all these things can fit together.

mumoshu avatar
mumoshu

thx for your feedback anyway! please give me more as you come up. it’s necessary to finish variant2

:--1:1
mumoshu avatar
mumoshu

@Erik Osterman (Cloud Posse) I’ve enhanced import as we discussed above. Please see https://github.com/mumoshu/variant2/blob/master/examples/advanced/import-remote/import.variant#L2

mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Rock on! We will give this a shot

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

this is absolutely lovely

johncblandii (Cloud Posse) avatar
johncblandii (Cloud Posse)

going to dig into variant much more, @mumoshu

1
1
loren avatar
loren

The dependency management is intriguing… One of the problems we always have is managing versions of tools used in a project pipeline… We pin anywhere we can use dependabot to automate the ref update through a pr, but there are a lot of release apis/targets and it just doesn’t work for everything. Any thought to a service or tool that would work with the lockfile and help open a pr for new versions?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

right - maybe piggy back on the gopkg.toml format for compatibility with dependabot

:--1:1
loren avatar
loren

For example, terraform-docs just updated to 0.9.0, introducing the Requirements section. We run tfdocs in our ci to compare diffs and now every pr pulls in the new version and fails. Now, actually, all our tooling is in a container, and we versioned the container. So, as long as we use the earlier container, it keeps working, and it’s just the pr for the container update that fails. But it would be awesome to get the scope down to the specific app update

mumoshu avatar
mumoshu


We pin anywhere we can use dependabot to automate the ref update through a pr, but there are a lot of release apis/targets and it just doesn’t work for everything
i hear you and yeah - that’s why i created variantdev/mod.

run mod up --pull-request periodically from your ci(e.g. circleci scheduled jobs) and it becomes a dependa-like-bot periodically checks for updates and send prs

:100:2
:--1:1
loren avatar
loren

Soooo, what’s involved with that? Support for non-github remotes? E.g. codecommit, gitlab?

loren avatar
loren

Gonna have to dig in…

mumoshu avatar
mumoshu
version: 2.1

jobs:
  mod:
    steps:
      - checkout
      - install_mod
      - run:
          name: Update dependencies
          command: |
            DATE="$(date -u '+%Y-%m-%d')"
            mod up --base master --branch "dep-update" --build --pull-request --title "Dependency updates: ${DATE}"

workflows:
  version: 2
  update_dependecies:
    jobs:
      - run_mod_up
    triggers:
      - schedule:
          cron: "0,10,20,30,40,50 0-7 * * *"
          filters:
            branches:
              only:
                - master
:--1:1
mumoshu avatar
mumoshu


Support for non-github remotes? E.g. codecommit, gitlab?
It supports github only for now. Perhaps it’s not that difficult to add support for more, as long as there’s api?

loren avatar
loren

Yes, there are APIs, we actually added the codecommit support to dependabot. Of course then they were bought by github and have basically stopped merging external contributions, so now we have to maintain a fork and build our own

mumoshu avatar
mumoshu

for github prs, its one line in go

https://github.com/variantdev/mod/blob/15eaa08f7be60b42fafbfb776bae5c7e29fda4e7/pkg/gitrepo/gitrepo.go#L110

please feel free to give me a snippet of go code for codecommit and gitlab or just submit a pr

// btw, does codecommit supoprt pull requests?

variantdev/mod

Missing package manager for any task runners and build tools e.g. make and variant - variantdev/mod

mumoshu avatar
mumoshu


we actually added the codecommit support to dependabot
awesome

mumoshu avatar
mumoshu


Of course then they were bought by github and have basically stopped merging external contributions
oh that’s too bad..

loren avatar
loren

We’re way overcommitted right now, but this definitely sounds interesting… I’ll have to see if I can get someone some time to play here

loren avatar
loren


// btw, does codecommit supoprt pull requests?

Yes, yes it does!

    keyboard_arrow_up