#variant (2020-04)

absolutely!

then in yourcmd.variant, you import the libs like

job "foo" {
  module = "./yourapp.variantmod"
  import = "./lib/variant2libs"
}

:point_up: module = "./yourapp.variantmod" instructs variant2 to load the module from the variantmod file

can we just stick it in the main variant file? No, not yet. I’m still unsure whether we should allow inlining it or not

Does the import functionality just use go getter? back to your question - yes, it’s almost like that.

but its worth noting that, under the hood, variant2 delegates dependency management to another tool

so import = "path" doesn’t need go-getter

For example being able to inline the module map :-)

Ahhh HCL probably doesn’t allow mixed types for a key

alright anything else you’d want variant2 to add before you actually use it?

yes. i’ll probably rename module to import_module or something similar and use module for inlined modules

Lol haha

https://github.com/mumoshu/variant2/issues/3

In that case, could import then be optional? e.g. it defaults to something like .variant/modules/$sha (just thinking “what would terraform do?”)

hm.. maybe? I do understand your motivation. just not sure how the current variant2 module system fits with that

the variant2 module also allow you to install executable binaries required by your command

perhaps we’d better have a dedicated syntax to import a remote variant2 lib, similar to what we had in variant1

how would you update your variant2 module?

hrm… good point. mycmd init -refresh ?

the good part of the variant2 module system is that you can run variant2 mod up or mod up to fetch all the dependencies and update respective <yourmodule>.variantmod.lock files

to be pull-requested for reviews. this allows you to manage dependency updates like you would do with go mod for go projects

ya, makes sense.

everything is technically possible.

it is just that i need some time to think about how all these things can fit together.

thx for your feedback anyway! please give me more as you come up. it’s necessary to finish variant2

@Erik Osterman (Cloud Posse) I’ve enhanced import as we discussed above. Please see https://github.com/mumoshu/variant2/blob/master/examples/advanced/import-remote/import.variant#L2

Rock on! We will give this a shot

right - maybe piggy back on the gopkg.toml format for compatibility with dependabot

For example, terraform-docs just updated to 0.9.0, introducing the Requirements section. We run tfdocs in our ci to compare diffs and now every pr pulls in the new version and fails. Now, actually, all our tooling is in a container, and we versioned the container. So, as long as we use the earlier container, it keeps working, and it’s just the pr for the container update that fails. But it would be awesome to get the scope down to the specific app update

We pin anywhere we can use dependabot to automate the ref update through a pr, but there are a lot of release apis/targets and it just doesn’t work for everything i hear you and yeah - that’s why i created variantdev/mod.

run mod up --pull-request periodically from your ci(e.g. circleci scheduled jobs) and it becomes a dependa-like-bot periodically checks for updates and send prs

Soooo, what’s involved with that? Support for non-github remotes? E.g. codecommit, gitlab?

Gonna have to dig in…

Support for non-github remotes? E.g. codecommit, gitlab? It supports github only for now. Perhaps it’s not that difficult to add support for more, as long as there’s api?

Yes, there are APIs, we actually added the codecommit support to dependabot. Of course then they were bought by github and have basically stopped merging external contributions, so now we have to maintain a fork and build our own

for github prs, its one line in go

https://github.com/variantdev/mod/blob/15eaa08f7be60b42fafbfb776bae5c7e29fda4e7/pkg/gitrepo/gitrepo.go#L110

please feel free to give me a snippet of go code for codecommit and gitlab or just submit a pr

// btw, does codecommit supoprt pull requests?

we actually added the codecommit support to dependabot awesome

Of course then they were bought by github and have basically stopped merging external contributions oh that’s too bad..

We’re way overcommitted right now, but this definitely sounds interesting… I’ll have to see if I can get someone some time to play here

// btw, does codecommit supoprt pull requests?

Yes, yes it does!

it should! probably i just missed implementing it

Using this instead of parameters inside of options defaults

hrm… so variable could be confusing for those coming from terraform where these are ways to pass settings like option and parameter in variant. But I think it’s pretty quick and easy to see that it’s different.

another idea:

variable "locals" {
  foo = "bar"
  apple = "delicious"
}

variable "mysql" {
  username = "test"
  host = "localhost"
}

so locals is just arbitrary

but this always for a terse expression of a lot of settings and an easy way to group them.

looks nice

just curious but how would you define a single variable then?

or maybe introduce a new type?

i’m assuming you declared var.locals.foo, var.lobals.apple, var.mysql.username, var.mysql.host in your above example, right?

yeah, or just variables would work

ohhh true

that’s better

variable and variables

yeah. then i would get a little annoyed on the inconsistency between two

variable "foo" {
  value = "FOO"
}

variables "bar" {
  baz = "BAZ"
}

I take it this is not possible:

im not sure either, but this one has more possibility to work:

variable "foo" {
  value = map({
    baz = "BAZ"
  })
}

Okay, let’s just punt on this. Not important.

Not to sidetrack, but does the <<- heredoc syntax not work in variant2? Hard to tell what is a general hcl2 thing vs a terraform-specific thing…

To improve on this, Terraform also accepts an indented heredoc string variant that is introduced by the <<- sequence:

block {
  value = <<-EOT
  hello
    world
  EOT
}

In this case, Terraform analyses the lines in the sequence to find the one with the smallest number of leading spaces, and then trims that many spaces from the beginning of all of the lines, leading to the following result:

hello
  world

https://www.terraform.io/docs/configuration/expressions.html#string-literals

Thanks for pointing that out!

Haven’t tried it yet, but will if it comes up again.

so what I’m trying to do is create a “script” by joining a list of commands.

not passing commands to exec

(where each arg should be passed like in your example)

ah okay

then

commands = list(
    "kubectl -n ${opt.namespace} apply -f ${context.sourcedir}/manifests/"
    )

ok, that was it!

so not: list(["a", "b", "c]), but list("a", "b", "c")

exactly. the former creates a single item list with the type of element being a tuple(string), which isn’t what you want

but hey - i just implemented the automatic type conversion so you can use either, like you would in terraform

https://github.com/mumoshu/variant2/commit/f355f14760da313328ff6c4e118cb991310757c4

it’s available since v0.18.0

Does tolist work? https://www.terraform.io/docs/configuration/functions/tolist.html

it would also work!

when you need an explicit type conversion

fyi here’s the list of available functions for variant2

https://github.com/mumoshu/variant2/blob/fefd2734ced7def0ea671b1888b843dcbca24965/pkg/conf/funcs.go#L102

sweet!

note the problem with this is now we dont’ free shell-escaping

It goes from:

  exec {
    command = "terraform"
    args = concat(list(param.subcommand), opt.args)
  }

(safe)

to…

(still working on it)

(the problem I’m working on is that when I’m running exec, I have a bug somewhere, but variant just exits 1 with no output)

now we dont’ free shell-escaping good point

should we add

exec {
  dir = "workdir"
  command = ...
}

?

(pretty please - that would make things a lot easier in the long run).

i believe i know how shell escaping is hard and that’s why i made exec a combination of command and args

alright, working on it

we also use a lot of relative paths in helmfile

@Erik Osterman (Cloud Posse) when would you use relative paths in variant2?

fyi: exec dir and hidden jobs is available since v0.19.0

https://github.com/mumoshu/variant2/releases/tag/v0.19.0

Thanks!! works as intended. it.

thanks! i’ve fixed it so that it will emit a nicer message like this:

Error: Invalid type specification

  on example/test.variant line 3, in option "namespace":
   3:   type = "string"

A type specification is either a primitive type keyword (bool, number, string) or a complex type
constructor call, like list(string).

thanks!!

not really. would you prefer global parameters implicitly inherited to jobs?

that would also mean that job-specific parameters would hide global parameters by names

would you prefer global parameters implicitly inherited to jobs Yes, from a user perspective that seems natural that globals are indeed global. But I admit I don’t know the implications of that.
job-specific parameters would hide global parameters by names Not sure I grok the implications of this. Sounds not good.

In this example, terraform init eks resolves to the first job and passes eks as the first positional parameter.

# note this list of “depends on” can be calculated dynamically

Yea, I haven’t yet figured out how to do that, but haven’t tried. I assume there’s some way I could extract that from the yaml configuration.

Worst case I exec out to jq

Think astro by uber: https://github.com/uber/astro

but generalized with variant

so it will work with helm, helmfile, terraform, etc.

Should items maybe be parameters since that’s how it’s used?

Probably no. items is a predefined name of the variable which is used for extracting each item

and it’s up to the user how they would use item for

ok

i mean the item can be passed as a paramter value, or an optional value

Oh, now I see how you’re doing that

or it can even be used to dynamically compute a value, which is eventually passed to a param or option

yea, that could work.

Maybe depends_on is more like for_each now?

maybe?

but you can omit items at all

so that it will turn into a single dependency

also, you can have more than two kinds of depends_on

this would firstly install terraform, then concurrently terraform-init efs and eks

aha, makes sense

that’s nice

would you prefer this?

job "terraform init all" {
  depends_on "install terraform" {
  }

  depends_on "terraform init" {
    # this step will be expanded to multiple concurrent steps by replacing `item` in the args to each item in this tuple:
    items = ["efs", "eks"]

    args = {
      project = item
      refresh = true
    }
  }

}

this way, the dependent job names are known at parsing time

Oh, I see what you’re doing.

Yea, that’s better and less nesting

while the terraform init targets and args can be still computed at run time

great

which one would you prefer, need or depends_on?

job "terraform init all" {
  need "install terraform" {
  }

  need "terraform init" {
    # this step will be expanded to multiple concurrent steps by replacing `item` in the args to each item in this tuple:
    items = ["efs", "eks"]

    args = {
      project = item
      refresh = true
    }
  }

}

So currently, you don’t use args like this and just pass everything

for runs, yes

exec does take arguments like that

Ah, true

(though thought of that more as args to the syscall function)

so in my initial variant2 implementation, it was

job "foo" {
  run {
    job = "anotherjob"
    args = {
      param1 = "param1
    }
  }
}

(though thought of that more as args to the syscall function) i think that’s valid. so maybe we’re mixing two conceptually different things into the name args. not sure it’s good or bad though

if it made it easier to implement it this way, then we should keep it.

conceptually, this would make more sense to me:

options = {
 ...
}

parameters = {
...
}

vs args where they are lumped together.

i understand

my idea was to explain it like:

Variant reads named args, matches and sets options and parameters values by names

so that we don’t need to distinguish between params and options from the call-side, which makes it more readable(in my op) and easier to refactor later:

run "terraform init" {
  params = ["efs"]
  options = {refresh = true}
}

vs

run "terraform init" {
  project = "fs"
  refresh = true
}

Looking to move all make/glue for Terraform runs to variant2?

i believe it’s pretty close to that

i occasionally see people wrapping helmfile, helm, kubectl, and terraform with bash snippets/scripts.

perhaps that’s where variant/variant2 shines

Everyone is pretty much using some wrapper script for Terraform AFAIK, be it Terragrunt/scripts/geodesic

make, but I don’t think it is for everyone

Yea, I think we’ve pushed make to it’s limits

I’m really excited about variant2 and the new DSL

working on a prototype cli for our clients as a starting off point.

Also note, variant2 ships with built-in testing framework.

and slack bot

it’s make on beastly steroids.

Nice

My 2 cents:

I’m not sure if integrations with external services (e.g. Slack) should be inside variant core

As for me, all integrations should be pluggable on demand, and the core should be kept to the very minimal functionality (“Do one thing and do it well”).

Otherwise, it could end like variant1: a lot of half-baked integrations inside the core (Docker runner, Github actions, etc), which are not maintained for the obvious reasons :slightly_smiling_face:

Anyway, that’s a great tool and I’m using variant1 heavily, but it still has some bugs, and I’m not sure if I should invest my time to fix them (as v1 is not supported anymore).

Still thinking about integrating variant2 into my workflows…

@mumoshu thank you for the great work again!

I thought there was no critical bug in variant1 that’s why I switched to spend more time on writing variant2. if you find any, please file issue(s)! I’m not gonna abandon variant1 any time soon

I’m not sure if integrations with external services (e.g. Slack) should be inside variant core I hear you. It’s just that I couldn’t come up with such pluggable interface at the time of writing variant2.

For example, how would an universal slack bot engine communicate with variant2 to show a dialog to ask the user for filling missing options to the job? I had no idea.

Otherwise, it could end like variant1: a lot of half-baked integrations inside the core (Docker runner, Github actions, etc), which are not maintained for the obvious reasons  Just curious, but why did you find them half-baked?

I mean, I use most of them in production and no one reported critical issues or submitted PRs to fix them. So I was considering all of them just working as expected.

Anyways, thanks a lot for your feedback!

running variant run all for this example

https://github.com/mumoshu/variant2/blob/master/examples/depends_on/depends_on.variant

would produce the exact output in

https://github.com/mumoshu/variant2/blob/641eb57ac37198ab82f4a094e9a7a94303f21522/examples/depends_on/depends_on_test.variant#L7-L10

Very clean! I like it. Thanks for doing that so quickly! Going to have a demo probably next weeks #office-hours

Seems like order is not preserved

i guess no - variant doesn’t do any fancy things to preserve key ordering and i thought go’s map doesn’t preserve the key ordering

ya, fair enough. probably just means I’m using the wrong data type for the job.

And in variant3 we’ll get named IDs like for_each to avoid annoying issues with order changes…

haha

i realized I should probably be using a yaml list instead.

please let me recall how I might have designed this to work… anyways, i believe this is due to how HCL differentiates map and objefct.

i thought {} and {k=v} was for the syntax for hcl objects, not maps.

https://github.com/mumoshu/variant2/blob/96bdfd9dd26fa675190d27b0a77f4eca53afdb77/examples/complex/complex.variant#L25-L28

I tried that and it didn’t work. This is for a dynamic map of different keys: map(string).

i think i managed to reproduce it. will try to fix it soon! than ks

@johncblandii this should be fixed since v0.21.1.

you should be able to just use {} as seen in https://github.com/mumoshu/variant2/blob/master/examples/defaults/defaults.variant#L16

Sweet. Will check it out

This should be fixed in v0.21.2 as well

I suppose not. What do you mean by “short options”?

changed -p to --project and the command would run

Whoa! I have no idea how that could happen!

Just curious but you’re enabling the trace like VARIANT_TRACE=1 ./variant run shell 'echo foo' 'echo bar', right?

using a shim

VARIANT_TRACE=1 ../cli/nbo -p acme

seems ok

i’ll try to recreate. 1 sec

Are you sure you’ve set short = "p" within option "project" { ... }?

yeah

I’ll keep an eye on it if i can recreate it

i had a code issue i was trying to trace when i hit that

okay. anyways, i managed to reproduce the handler for type object not implemneted yet error. i’ll shortly fix it and then try to pass run "shell" on my own env

after that there would be much less probability that you’d encounter any issue i hope so

awesome

Re: VARIANT_TRACE issue - I couldn’t reproduce it

$ VARIANT_TRACE=1 PATH=$(pwd):$PATH ./test1/test1 example -d
deploy switch tenant=mytenant item=foo
deploy switch tenant=mytenant item=bar
Done.

I’ve created the shim by modifying an example command contained in the variant2 repo examples/issues/cant-convert-go-str-to-bool to have short = "d" for the example’s dry-run option:

job "example" {
  config "file" {
    source file {
     path = "${context.sourcedir}/conf.yaml"
   }
  }

  option "dry-run" {
    type = bool
    default = false
    short = "d"
  }

# snip

and ran export shim to create the shim:

$ ./variant export shim examples/issues/cant-convert-go-str-to-bool ./test1

I can’t reproduce the issue any longer either, @mumoshu.

Much appreciated!

yq exists in my shell

running this formerly with the shell command (bash -C …) works perfectly fine

here’s the job:

job "helmfile shell" {
  description = "Run a command in a shell targeted specifically at a projects/helmfiles project for Helmfile commands"
  private     = true

  parameter "args" {
    description = "List of args to execute"
    type        = list(string)
  }

  exec {
    args    = param.args
    command = "helmfile"
    dir     = "../projects/helmfiles/${opt.project}"

    env = {
      AWS_PROFILE = "${opt.tenant}-helm"
      KUBECONFIG  = "~/.kube/kubecfg.${opt.tenant}-helm"
      PROJECT     = opt.project
      TENANT      = opt.tenant
    }
  }
}

I think you need to set PATH in exec { }

Perhaps variant2 should just inherit the process envvars when you use exec { env = ... } but omitted PATH in the env attr?

i would expect it to

alright! i’ll see what i can do(probably i’ll just make it so

I couldn’t reproduce the PATH issue with variant2 alone

I ran this successfully:

job "example" {
  exec {
    command = "jq"
    args = ["-h"]
  }
}

Ooops, it does reproduce when env = { whatever } is set

v0.22.2 fixes this.

I think @Erik Osterman (Cloud Posse) mentioned this before. Using his workaround works:

arg     = coalesce(list(""))

Thanks! This should be fixed in v0.21.2

great

or would this be a place for an exec?

Unfortunately it isn’t supported

Would you mind sharing your exact use-case? I’m considering how much we might need it

i’ll share code in a sec. let me give context

I mean, I’m eager to make the depends_on target dynamic if it’s necessar

THx

we have a file with a list like this:

order:
  - terraform.proj1
  - terraform.proj2
  - helmfile.proj3
  - terraform.proj4

Note that HCL2 doesn’t support expressions inside the block label (e.g. NAME in depends_on "NAME" { }

So we need an alternative syntax if we end up implementing it

1. read the file
2. loop over order
3. run internal job for ${var.command} plan or ${var.command} diff

ah, gotcha

so this code outputs a simple echo of each project with the command split:

➜ ../cli/nbo deploy all -t acme
terraform account
terraform cloudtrail
terraform vpc

config file looks like:

order:
  - terraform.account
  - terraform.cloudtrail
  - terraform.vpc

code to output it is like this:

#!/usr/bin/env variant
# vim: filetype=hcl

option "tenant" {
  description = "Tenant to interact with"
  short       = "t"
  type        = string
}

option "project" {
  default     = ""
  description = "Terraform project to process"
  short       = "p"
  type        = string
}

config "file" {
  source file {
    path = "${opt.tenant}.yaml"
  }
}

job "deploy all" {
  description = "Init all projects"

  depends_on "e" {
    items = conf.file.order

    args = {
      a = item
      project = opt.project
      tenant = opt.tenant
    }
  }
}

job "e" {
  option "a" {
    default     = coalesce(list(""))
    description = "args to pass to subcommand"
    type        = string
  }

  variable "asplit" {
    type = list(string)
    value = split(".", opt.a)
  }

  # variable "cmd" {
  #   type = string
  #   value = var.asplit[0]
  # }

  # variable "project" {
  #   type = string
  #   value = var.asplit[1]
  # }

  exec {
    command = "echo"
    args    = var.asplit
  }

  # run "${opt.a} init" {

  # }
  # depends_on "${var.cmd} init" {
  #   args = {
  #     project = var.project
  #     tenant  = opt.tenant
  #   }
  # }
}

some comments still in there

makes sense

and you wanna make e in depends_on "e" dependent on each item?

so that you can run helmfile apply for helmfile.proj3 and terraform apply for items like terraform.proj1?

i technically want to run internal jobs within the variant cli

yes

that makes total sense

i could break it out to individual commands via exec, but plan would need init and workspace

firstly, i don’t think variant2 as of today has a good way to express that

ok

but i do want great support for your use-case

actually i was planning to create a command that looks very similar to yours

myself

nice!

let me dump my ideas

cool

This is the first option. The idea is that you define a deploy job can is able to call helmfile apply or terraform apply depending on the dir it is targeted

so that it can be used from depends_on

The second option is that you make each item of conf.file.order a object which has two atrributes “type” and “dir”

job "deploy all" {
  description = "Init all projects"

  depends_on "deploy" {
    items = conf.file.order

    args = {
      dir = item.dir
      type = item.type
    }
  }
}

job "deploy" {
  description = "deploy the app defined in the directory"
  
  option "dir" {
     type = string
  }

  option "type" {
     type = string
  }

  variable "cmd" {
    value = opt.type
  }

  exec {
    command = var.cmd
    dir = opt.dir
    args = ["apply"]
  }
}

Does either of the two options look good to you? If so we don’t need to enhance variant2 in any way for this. This should just work with today’s variant2.

Otherwise we need something more. Not sure if that’s about making depends_on dynamic. The downside of doing so would be that the deploy all job can be more complex which makes unit-testing it harder

On the other hand, either of the above two options is easily testable. I mean, you can write test for deploy all and deploy separately and independently

the all vs the individual is the plan. so this setup is where we have multiple internal jobs we want to call.

ex: tf plan calls tf init followed by tf workspace then the exec for terraform plan

the deploy will need to call apply and that will consist of init -> workspace -> plan -> apply

Makes sense. As we don’t have any notion of mutliple conditional execs within a single job, that’s not possible

perhaps the only way would be exec another variant command

mmm…so command is the cli with the values passed there

that could be a good workaround

is there a concept of “self” in this case?

or would we need to bake in the cli name in the exec ’s command?

would we need to bake in the cli name in the exec ’s command? yes, that’s the way it works now.

but there’s a potential solution. we already have context that is used for context.sourcedir which basically translates to dirname $($0)

so it would be straight-forward to add context.self or perhaps context.command that represents $0

which one do you prefer? or i’m open to alternatives

The third possible option would be to enhance steps.

Variant2 job can have two or more steps. Each step is able to depend on other step in the same job, run concurrently, and call an another job.

It doesn’t support conditional execution(i.e. if). But if we add support for if, like

job "apply" {
  # params and opts here

  # instead of exec or run, use steps

  step "do terraform apply" {
    if = opt.type == "terraform"
    run "terraform apply" {
      # args here
    }
  }

  step "do helmfile apply" {
    if = opt.type == "helmfile"
    run "helmfile apply" {
      # args here
    }
  }

we neither need self nor dynamic depends_on target, or write/call a lengthy shell script.

that looks spot on, @mumoshu! i definitely love the simplicity of an if statement to allow control of the run as well

that’d be SWEET.

thoughts, @Erik Osterman (Cloud Posse)?

ideally, i’d like each variant2 job to have only a single responsibility, so that we can enforce testability of each job.

that is, a job should only do any of the followings:

• selectively run a job(new)
• run a static graph of jobs(steps. note that steps has no conditions or loops)
• run a job(job with a run. note that you can’t have multiple runs within a single job. use steps if you need to do so)
• run a command(job with a exec. note that you can’t have multiple execs within a job)

so i got to think i would prefer a dedicated block selectively { } for it, like:

job "apply" {
  # params and opts here

  # instead of exec or run, use steps

  selectively {
    run "helmfile deploy" {
      args = {}
      if = opt.type == "helmfile"
    }

    run "terraform deploy" {
      args = {}
      if = opt.type == "terraform"
    }
  }

or change the run syntax to accept an expression for the job name, so that we can write

job "apply" {
  # params and opts here

  # instead of exec or run, use new run syntax

  variable "runs" {
    value = {
       helmfile = {
         name = "helmfile deploy"
         args = {...}
      }
      terraform = {
        name = "terraform deploy"
        args = {...}
      }
    }
  }

  variable "job" {
    value = var.runs[opt.type]
  }
  
  run {
    job = var.job.name
    with = var.job.args
  }

coincidentally, the last solution requires https://github.com/mumoshu/variant2/issues/15

I think I like the latter. Being able to dynamically choose a job in this way could be beneficial.

I do think anyone coming from Terraform (hcl background) would look for a way to use or not use a job (see count on resources) so maybe both?

Got it. run without the job name in a label should now work as documented in https://github.com/mumoshu/variant2/#indirect-run

Just published v0.22.0. It also includes the solution for the issue 15

Perhaps we’d better deprecate the old run syntax because having both seems confusing?

For the consistency reason, I guess we’d better change the step syntax as well

BEFORE

  step "STEP_NAME" {
    run "JOB_NAME" {
      dir = "deploy/environments/${opt.env}/manifests"
    }
  }

AFTER

  step "STEP_NAME" {
    job = "JOB_NAME"
    with = {
      dir = "deploy/environments/${opt.env}/manifests"
    }
  }

Oh, man. That looks solid. Going to dig into this later today and will report back my results

I’m continuously baffled by how fast you implement things @mumoshu

Seriously. It baffles me.

in the past, you used args instead of with ; is there a subtle difference I’m not picking up?

I’ve been long wanted a clear distinction between exec’s args of list(string) and job run’s args of map(any). I thought that picking a different name for job run’s would achieve that.

But I’m open to suggestions

https://github.com/mumoshu/variant2/issues/15

thank you so much!

thanks for all your feedback! it’s helpful and encouraging

John just showed me the working bones of what we wanted to get working!

almost there

It’s so sweet.

Could you share me a fuller example? Is the option dry-run defined in the same job as the depends_on is defined in?

it is a global

full job:

#!/usr/bin/env variant
# vim: filetype=hcl

option "dry-run" {
  default     = false
  description = "simulate an install"
  type        = bool
}

# option "tenant" {
#   description = "Tenant to interact with"
#   short       = "t"
#   type        = string
# }

# option "project" {
#   default     = ""
#   description = "Terraform project to process"
#   short       = "p"
#   type        = string
# }

job "deploy" {
  description = "Init all projects"

  parameter "tenant" {
    type = string
  }

  config "file" {
    source file {
      path = "${param.tenant}.yaml"
    }
  }

  depends_on "deploy switch" {
    items = conf.file.order

    args = {
      item    = item
      tenant  = param.tenant
    }
  }
}

job "deploy switch" {
  private = true

  parameter "item" {
    default     = []
    description = "a config param to deploy in the format: cli.project; e.g. terraform.eks; e.g. helmfile.reloader"
    type        = string
  }

  parameter "tenant" {
    type = string
  }

  # option "dry-run" {
  #   default     = false
  #   description = "simulate an install"
  #   type        = bool
  # }

  variable "item-split" {
    type  = list(string)
    value = split(".", param.item)
  }

  variable "type" {
    type  = string
    value = var.item-split[0]
  }

  variable "project" {
    type  = string
    value = var.item-split[1]
  }

  variable "subcommand" {
    type = string
    # value = opt.dry-run ? "echo ${var.type}" : var.type
    value = "echo ${var.type}"
  }

  run {
    job = "${var.subcommand} deploy"

    with = {
      project = var.project
      tenant  = param.tenant
    }
  }
}

job "echo terraform deploy" {
  parameter "project" {
    description = "args to pass to subcommand"
    type        = string
  }

  parameter "tenant" {
    type = string
  }

  exec {
    command = "echo"
    args    = [opt.dry-run, param.tenant, "terraform deploy", param.project]
  }
}

job "echo helmfile deploy" {
  parameter "project" {
    description = "args to pass to subcommand"
    type        = string
  }

  parameter "tenant" {
    type = string
  }

  exec {
    command = "echo"
    args    = [param.tenant, "helmfile apply", param.project]
  }
}

still refining a bit

so this one runs since i’m not passing it directly to the switch job

it always outputs false, though

but it seems depends_on args are string conversions

but it fails

Thanks for the detailed report! Just spotted the cause and fixed locally.

I’ll publish the next patch release a few hours later

I believe your config is correct. Please just wait for the patch release

This one is working fine after the fix

job "deploy switch" {
  option "dry-run" {
    type = bool
  }

  option "item" {
    type = string
  }

  option "tenant" {
    type = string
  }

  exec {
    command = "bash"
    args = ["-c", "echo deploy switch tenant=${opt.tenant} item=${opt.item}"]
  }
}

job "example" {
  config "file" {
    source file {
     path = "${context.sourcedir}/conf.yaml"
   }
  }

  option "dry-run" {
    type = bool
    default = false
  }

  parameter "tenant" {
    type = string
    default = "mytenant"
  }

  depends_on "deploy switch" {
    items = conf.file.order
    args = {
      dry-run = opt.dry-run
      item    = item
      tenant  = param.tenant
    }
  }

  exec {
    command = "bash"
    args = ["-c", "echo Done."]
  }
}

variant run example
go build -o variant ./pkg/cmd
deploy switch tenant=mytenant item=foo
deploy switch tenant=mytenant item=bar
Done.

There was definitely a bug around type conversion. Having two or more types of values in args was causing the issue.

ahhhh

@johncblandii Just released v0.22.1 with the fix

I like the foo "something" { ... } syntax more than I like foo = "something" … with = { ... }

That said, not definitely not worth keeping both if it makes it harder for you to implement things. This is just a stylistic preference.

Also, we could do a screenshare if you want to see how we’re using it.

This is a “terragrunt killer”

:boom: :boom: :boom: :boom: :boom:

Just verified the dry-run option now works properly as a bool.

I also confirmed the env works perfectly fine with cli shell env and merges internal env with the system env

just that i thought verbosity is important there

perhaps variant could just try to fill in missing option/parameter arguments from the global parameter/option?

well for jobs needing calling jobs calling jobs calling jobs is pushing a global down N levels

but you don’t have to define the option for the > secondary jobs

yeah probably

so i either have to define global options on every single job or hope devs understand i can set args on a run that don’t exist on the job

devs working on the same cli, i mean

I was mostly concerned about a like

option "namespace" {
   type = string
}

job "deploy" {
  option "dir" {
    type = string
  }
  exec {
    command = "kubectl"
    args = ["-n", opt.namespace, "-f", opt.dir]
  }
}

job "all" {
  option "namespace" {
     type = string
     default = ""
  }

  step "app1" {
    run "deploy" {
      namespace = opt.namespace == "" ? "app1" : opt.namespace
      dir = "app1"
   }

  step "app2" {
    namespace = "app2"
    dir = "app2"
  }
}

as a way to override

i gotcha

not sure we should allow that though

it seems conflicts internal to a job to a global might confusing

perhaps just forbidding to shadow a global option with a local option, while

automatically filling missing global args as suggested above would be nice?

duplication-wise, yes. for example:

job "helmfile apply" {
  description = "Apply the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "apply"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}

job "helmfile destroy" {
  description = "Destroy the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "destroy"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}

job "helmfile diff" {
  description = "Diff the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "diff"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}

job "helmfile lint" {
  description = "Lint the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  step "cmd" {
    run "helmfile" {
      command   = "lint"
      namespace = opt.namespace
      region    = opt.region
      project   = param.project
      tenant    = param.tenant
    }
  }
}

job "helmfile sync" {
  description = "Sync the helmfile with the cluster"

  parameter "tenant" {
    description = "Tenant to interact with"
    type        = string
  }

  parameter "project" {
    description = "Terraform project to process"
    type        = string
  }

  run "helmfile" {
    command   = "sync"
    namespace = opt.namespace
    region    = opt.region
    project   = param.project
    tenant    = param.tenant
  }
}

^ those are all duplicated

add a new global, add a new line for every job

that’s indeed painful!

okay then, let’s disallow option/parameter shadowing AND do enhance run to fill on missing args from global opt/param

sweet

@johncblandii Just released v0.23.0 for this.

that’s great! ya, i think this is much better. if a job wants to redefine a global option seems like a bad inconsistency. instead the job should define a new option. this change you made addresses that! I like the exception msg.

Yes he added support for this

Look for keyword interactive

@johncblandii https://github.com/mumoshu/variant2/issues/9

I saw that…
Auto-prompting via interactive messages

One of cool features of the bot is that when you missed to specify values for certain options, it will automatically start a interactive session to let you select and input missing values within Slack. You don’t need to remember all the flags nor repeat lengthy commands anymore.

that’s where you can enter a variant job without required options and it’ll ask you about those options

Ya can’t wait to test the slack bot functionality later

this request was more about a tf destroy asking for yes after you start the process

Aha, though in this case terraform supports that out of the box so we can get around it

yes, if we don’t want to allow a non -auto-approve scenario

so i figured it’d be fine for our scenario, but i’ve hit it a couple times where i wanted to inspect a destroy before running

Ya so maybe a new feature request for “prompt” - but I see this getting complicated with paths

yuppers

i was thinking of just respecting prompts from exec commands

the second was support for our own prompts

can kinda do it now if we set an option as required but not provide it on the cmd

Respecting prompts is supported today, right?

unless i’m missing something, it will fly right through it

for an exec

Right so see the link above :-)

no. that’s not it

Need to add “interactive” flag to the job

that link is interactive for your variant cli

it is not interactive for an exec command

Hrm so see the example in the associated commit?

That is allowing the user to interact with the command in exec and not fly through

interactive = true

^ that’s it

But also a command line arg to rm

rm -i makes it interactive

yes, but i was looking at the fix

https://github.com/mumoshu/variant2/commit/94a6ff9f72a97c6bee6e0df16254c540f5b616cb#diff-637a60a871f8b69622bdfd3d3b2fcad1R671

good to know. thx for the ref. i should’ve read the code in the first place.

So maybe we should dynamically pass auto-approve=false or true based on options to variant

Could probably be done using a ternary

prob can be automated

default to false unless in a deploy command where it passes true

not going to sweat it now, but should be easy

that’s not the intent true! that makes sense

so based on the order above, you end up with:

job terraform deploy account job terraform deploy iam-tenant-roles job terraform deploy cloudtrail job terraform deploy vpc

thx. the job ordering seems correct.

variable "file" {
  value = {
    order = [
      "terraform.account",
      "terraform.iam-tenant-roles",
      "terraform.cloudtrail",
      "terraform.vpc",
    ]
  }
}

job "example" {
  depends_on "deploy switch" {
    items = var.file.order
    args = {
      item   = item
    }
  }
}

job "deploy switch" {
  option "item" {
    type = string
  }

  variable "subcommand" {
    value = split(".", opt.item)[0]
  }

  variable "project" {
    value = split(".", opt.item)[1]
  }

  run {
    job = "${var.subcommand} deploy"
    with = {
      project = var.project
    }
  }
}

job "terraform deploy" {
  option "project" {
    type = string
  }

  exec {
    command = "bash"
    args = ["-c", <<SCRIPT
echo job terraform deploy ${opt.project}; if [ ${opt.project} == "cloudtrail" ]; then echo simulated error 1>&2; exit 1; fi
SCRIPT
    ]
  }
}

stops on the first (simulated) error, as expected, for me:

 VARIANT_DIR=examples/issues/depends_on_stop_on_first_error ./variant run example
job terraform deploy account
job terraform deploy iam-tenant-roles
job terraform deploy cloudtrail
simulated error
Error: command "bash -c echo job terraform deploy cloudtrail; if [ cloudtrail == "cloudtrail" ]; then echo simulated error 1>&2; exit 1; fi
": exit status 1

I’ll revisit, @mumoshu.