#variant

Discuss variant (the “Universal CLI”) https://github.com/mumoshu/variant

Archive: https://archive.sweetops.com/variant/

2019-07-28

davidvasandani

@mumoshu any suggestions to help get me unstuck?

mumoshu
mumoshu/variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow. - mumoshu/variant

mumoshu

basically you need to pass the variant build output into a golang func call of variant.Def()

davidvasandani

davidvasandani
./main.go<i class="em em-35"></i>3: undefined: variant.Def
./main.go<i class="em em-36"></i> unknown field 'fun' in struct literal of type variant.TaskDef
davidvasandani

@mumoshu this is the error I get with the following main.go

davidvasandani
mumoshu

Try removing references to fun fields like fun:(func(variant.ExecutionContext) (string, error))(nil)}}, from your code

mumoshu

And does changing "<http://github.com/mumoshu/variant/pkg>" to variant "<http://github.com/mumoshu/variant/pkg>" in your import ( ) make difference to you?

davidvasandani
./main.go<i class="em em-38"></i>2: syntax error: unexpected ), expecting expression
./main.go<i class="em em-40"></i>2: syntax error: unexpected newline, expecting comma or )
davidvasandani
mumoshu

You seem to have missed closing parenthesis(es) on the way. Do you have VisualStudio Code or something that highlight suspicious parts of your code?

2019-07-27

mumoshu

Ah sry that’s a copypaste issue. Replace r.s with summoner

davidvasandani
./main.go<i class="em em-16"></i>31: undefined: r
./main.go<i class="em em-22"></i>23: cannot call non-function summoner (type *summon.Driver)
davidvasandani

@mumoshu This is my main.go file:

davidvasandani
package main

import (
	"fmt"
	"os"
  "<http://github.com/mumoshu/variant/cmd>"
  "<http://github.com/gobuffalo/packr/v2>"
  "<http://github.com/davidovich/summon/pkg/summon>"
)

func main() {
  assetsDir := "assets"

  box := packr.New("Bundled Assets", assetsDir)

  summoner, err := summon.New(r.box)
  if err != nil {
      fmt.Fprintf(os.Stderr, "%v\n", err)
      os.Exit(1)
  }

  msg, err := summoner(
      summon.All(true),
      summon.Raw(true),
      summon.Dest(assetsDir),
  )
  if err != nil {
      fmt.Fprintf(os.Stderr, "%v\n", err)
      os.Exit(1)
  }

  fmt.Printf("summon: %s\n", msg)

  cmd.YAML(`
		#!/usr/bin/env variant

tasks:
  hello:
    script: \|
      echo Hello $(whoami)!
  whoami:
    script: \|
      assets/whoami.sh
	`)
}
mumoshu

summon.New(r.box) should be summon.New(box)

mumoshu

and msg, err := summoner( should be msg, err := summoner.Summon(

davidvasandani

thanks! testing now.

davidvasandani

Do you recommend I inject the output of variant build ... or use cmd.YAML and inject the output of the variant file?

mumoshu

either is okay!

davidvasandani

When you responded I was using variant build

davidvasandani
davidvasandani

this is the latest main.go

davidvasandani
./main.go<i class="em em-36"></i>17: *variant.TaskDef literal evaluated but not used
./main.go<i class="em em-36"></i> unknown field 'fun' in struct literal of type variant.TaskDef
davidvasandani

I’ll push up my example.

davidvasandani
davidneudorfer/variant-assets-example

Contribute to davidneudorfer/variant-assets-example development by creating an account on GitHub.

2019-07-26

davidvasandani

I’ve added summon and packr2 but these are the errors I get when I build:

./main.go<i class="em em-16"></i>30: undefined: r
./main.go<i class="em em-22"></i>14: undefined: r
davidvasandani

Whats r and s on this line ` msg, err := r.s.Summon(`

2019-07-25

mumoshu

@davidvasandani Hey! I understand your use-case but unfortunately we don’t yet have an example

1
davidvasandani

If I could figure out how to get started I’d be happy to contribute one.

mumoshu

You can add some go code to the go source output by variant build but

davidvasandani

I’m using variant in a couple different projects but all my shellscripts are in a single variant yaml file

1
mumoshu

Perhaps you’d prefer easier ways, like you add a few lines to your variant command yaml and variant build to get the bundled binary..?

davidvasandani

Exactly.

mumoshu

Great. So I was considering to enhance variant or mod to cover that

1
mumoshu
bundle:
  files:
  - ./utils.sh
 - ./helpers.sh

tasks:
  foo:
    script: \|
      {{.moduleDir}}/utils.sh myutilfunc arg1 arg2
1
davidvasandani

exactly this.

mumoshu

Good. I’ll try implementing it soon

mumoshu

In the meantime, you’ll be able to integrate summon like this:

package main

import (
	"fmt"
	"os"
)

func main() {
	assetsDir := "assets"

	box := packr.New("Bundled Assets", assetsDir)

	summoner, err := summon.New(r.box)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%v\n", err)
		os.Exit(1)
	}

	msg, err := r.s.Summon(
		summon.All(true),
		summon.Raw(true),
		summon.Dest(assetsDir),
	)
	if err != nil {
		fmt.Fprintf(os.Stderr, "%v\n", err)
		os.Exit(1)
	}

	fmt.Printf("summon: %s\n", msg)

	// Inject go code generated by `variant build` here...
}
mumoshu

And in your script call it like assets/utils.sh myutilfunc arg1 arg2. packr2 bundles all the files under assets and summon extracts em into assets

davidvasandani

I’ll give this a shot. Thanks @mumoshu

1

2019-07-17

davidvasandani

are there any examples for how to integrate https://github.com/davidovich/summon with variant?

davidovich/summon

Summon: your data on caffeine. Contribute to davidovich/summon development by creating an account on GitHub.

davidvasandani

I’m looking to include additional shell scripts in the variant binary vs trying to write out the each full script within the variant config.

davidvasandani

cc @Erik Osterman wondering if you’ve run into this before

Erik Osterman

Aha yes @mumoshu has somethings he is working on for this

Erik Osterman

I think one is called mod

Erik Osterman

I am away right now but will look it up

davidvasandani
variantdev/mod

Package manager for Makefile and Variantfile. Any set of files in Git/S3/GCS/HTTP as a reusable and parameterized module - variantdev/mod

davidvasandani

I was looking at this and reading the Slack archives but didn’t realize it could be used with shell scripts.

davidvasandani
variant

SweetOps is a collaborative DevOps community. We welcome engineers from around the world of all skill levels, backgrounds, and experience to join us! This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build sweet infrastructure.

2019-07-16

davidvasandani
GoReleaser

Deliver Go binaries as fast and easily as possible

davidvasandani

Has anyone tried to marry variant and goreleaser? I love the idea of automatically building the binary and updating homebrew.

2019-07-02

davidvasandani

When releasing a variant made command has anyone run into:

./main.go<i class="em em-4"></i>5: undefined: cmd.YAML
mumoshu

would you mind sharing the full log?

davidvasandani

updated Gopkg.toml to version = "v0.31.1" and re-ran dep ensure

davidvasandani

currently waiting for that complete.

davidvasandani

Everything works

davidvasandani

Thanks @mumoshu

2019-06-25

2019-06-14

Erik Osterman

sorry all - I haven’t had a chance yet to get my head into this. @mumoshu codes at the speed of hard to keep up

2
1

2019-06-13

mumoshu

Just reviewed and merged your PRs. Thanks @!

1
mumoshu

I built an another tool called variantmod envisioning to integrate it into variant someday, like vgo integrated to go

https://github.com/variantdev/mod/blob/master/examples/basic/variant.mod

The source in this example isn’t meaningful but guess what we can achieve with it…

variantdev/mod

Turn any set of files into a reusable module. Contribute to variantdev/mod development by creating an account on GitHub.

Erik Osterman

why the new org? variantdev?

variantdev/mod

Turn any set of files into a reusable module. Contribute to variantdev/mod development by creating an account on GitHub.

mumoshu

no strong reason, but considered mumoshu/variantmod vs mumoshu/mod, vs variant/mod vs variantdev/mod and realized variant was already taken and mumoshu/variantmod seemed a bit verbose!

mumoshu

while i wasn’t happy with mumoshu/mod because i was planning to integrate it to variant

mumoshu

You can add variant.mod so that running mod fetches templates from remote sources and renders config files from it. Imagine building reusable template of .circleci/config.yml, drone.yml, travis.yml, “import”ing one of them with variant.mod and render it with runtime arguments..

mumoshu

@Erik Osterman Perhaps you’ll be interested

mumoshu

The only type of dependencies is currently file. But I’m considering to add module. That is, any go-getter accessible directory containing variant.mod can be distributed and reused across your projects.

This avoids copy-pasting various YAML files for e.g. CI across projects. But I think there are more use-cases.

@mumoshu That’s a good idea (and I tried to find something like this only yesterday ).

In fact, I was thinking about it (retrieving and processing external files to produce the configuration & code) for the last couple of years and even implemented some version of this idea with “gomplate” and some other tools to retrieve the templates.

In my opinion, you could follow the Unix philosophy principle “Do One Thing and Do It Well” meaning that you can retrieve the templates by “mod” and use other tools, like “gomplate” to generate the result files from the templates (instead of processing the templates by “mod”).

BTW, how are you going to store the “result” files? Do you want to leave this action for the end-user to perform it manually (git commit?) or provide some functionality for it in the tool as well?

mumoshu

@ Thanks! I believe I have similar sentiment.

What if variantmod just provide the unified api/config syntax for the job, and used gotemplate as a library under the hood? Would it still follow the unix philosophy?

mumoshu

Results are rendered to the local paths specified by keys of the files entries. The user would commit the resulting files after manual testing.

For automation, you can later build a CI pipeline to re-render and test the updated files when variant.mod file changes.

Does it sound good to you?

@mumoshu yes for both statements, it fits my “vision” of this tool

1

2019-06-12

feat #81: Do not pollute arguments by aroq · Pull Request #91 · mumoshu/variant

Solves #81 Now you can use the Viper-based hide_extra_cmds configuration entity to control if extra commands entries should be hidden from help, like: VARIANT_HIDE_EXTRA_CMDS=true variant

Erik Osterman

Thanks @!

1

2019-05-23

mumoshu
feat: Markdown-based alternative syntax · Issue #86 · mumoshu/variant

As an ordinary engineer I often write documentation for my one-off scripts in GitHub Flavored Markdown. But it tends to get outdated quickly due to various reasons, including I just forget updating…

2019-05-11

Maycon Santos
07:04:46 PM

@Maycon Santos has joined the channel

2019-05-10

@rohit I think about it like this: makefile is good for the wrapping bash scripts (and other executables), and variant is better than makefile to do it

2
2

2019-05-09

Erik Osterman
1
1
rohit

I just came to know about variant

rohit

i am not sure what advantage does variant have over regular bash scripts

Erik Osterman

consistent interface

Erik Osterman

proper arg validatin

Erik Osterman

very terse

Erik Osterman

most bash scripts have very poor argument handling

mumoshu

yep, and it provides a “upgrade path” to a golang app

mumoshu

you can use variant build and some scripts to produce a single executable that runs without variant

mumoshu

and even add extra commands written in golang to your cmd written in yaml

mumoshu

so if you feel outgrown from bash, variant allows you to gradually/fully migrate to golang.

rohit

it is currently flying over my head, i will have to check it out

1
rohit

i guess i will start with @Erik Osterman intro video

1
rohit
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Erik Osterman

Yes

Erik Osterman

Everything new we do is in variant

3

2019-05-08

mumoshu

@Erik Osterman For variant (and also helmfile potentially) I got to think that we eventually need a tool like this:

https://github.com/mumoshu/versadep/blob/master/README.md

WDYT?

mumoshu/versadep

Versatile application dependency manager. Version, fetch, and generate any files and binaries required to run your application - mumoshu/versadep

mumoshu

the example config doesn’t make sense at all. please take it as purely for showing how the config syntax look like

mumoshu/versadep

Versatile application dependency manager. Version, fetch, and generate any files and binaries required to run your application - mumoshu/versadep

Erik Osterman

Wow, yes, I think you are on to something

Erik Osterman

Sec developing/forming opinion

Erik Osterman

Consider adding semver

Erik Osterman
gruntwork-io/fetch

fetch makes it easy to download files, folders, and release assets from a specific git commit, branch, or tag of public and private GitHub repos. - gruntwork-io/fetch

Erik Osterman

But that is more for GitHub tags/release

mumoshu

thx for sharing!

mumoshu

probably we need to explore more sources other than github tags/releases as well, like curl from any http sources as your deps

mumoshu

probably what need for versadep would look something like a declarative fetch + additional sources?

mumoshu

oh, and i wanted versadep to setup PATH for us, like rbenv does for ruby

Erik Osterman

Aha so you can vendor everything

mumoshu

exactly!

Erik Osterman

Btw have you ever explored terraform vendoring? Not easy

mumoshu

not at all. i’d love to know your experience!

Erik Osterman

For modules.. that nest modules. Only bringing it up because it’s something we’ve been tasked with solving at some point

mumoshu

can you nest terraform modules? i wonder how it works when multiple modules refer to/nest different versions of the module

Erik Osterman

yep, we do extreme nesting

Erik Osterman

sometimes 3-4 levels deep. maybe more.

Erik Osterman

terraform clones every reference of a module to .terraform/modules

Erik Osterman

each one checked out to the version referenced

Erik Osterman

it uses some hashed folder which is presumably the github url with version info

mumoshu
feat: default commands · Issue #85 · mumoshu/variant

Extracted from our official slack channel: have you considered a &quot;default&quot; command? e.g. if none of the args match, pass $* to some command (edited) e.g. terraformctl would intercept any …

2019-05-07

Erik Osterman

yea, not for this stage

Erik Osterman

no deps of deps

Erik Osterman

i think a “package” system would be better then; perhaps why you chose helm

Erik Osterman

@mumoshu have you considered a “default” command? e.g. if none of the args match, pass $* to some command

Erik Osterman

e.g. terraformctl would intercept any task defined, but if not defined, it would pass to terraform

mumoshu

Haven’t tested it with deeply nested commands, but I’ve added the initial support for default commands.

The top-level script is run when if is defined and no sub-task that matches the provided arguments found.

https://github.com/mumoshu/variant#default-command

mumoshu/variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow. - mumoshu/variant

mumoshu

its available since v0.29

Erik Osterman

wow, that’s clever. I like how you kept it consistent with the rest of variant. hadn’t thought to implement it that way, but i like it.

Erik Osterman

basically, using variant to overload commands

1
mumoshu

@Erik Osterman sounds like a great addition to variant!

2019-05-04

mumoshu

@Erik Osterman I believe your approach does work and simpler, which is nice!

Your assumption would be that you don’t need transitive dependencies to be handled, right?

2019-05-03

Erik Osterman
Add deps command by osterman · Pull Request #466 · cloudposse/geodesic

what Add a command to install/uninstall remote dependencies defined by any source url why Mimic the terraform init -from-module=… pattern for anything as a way to initialize a project from rem…

Erik Osterman

any thoughts?

Erik Osterman

this is like your anydeps but uses variant without overloading helm as package manager.

Erik Osterman

(as you can see the motivation is for #helmfile to fetch releases)

Erik Osterman

but will be also used to fetch binaries

Erik Osterman

though I confess i wrote this without looking at anydep and now that I do see a bunch of things I can do better

Erik Osterman
mumoshu/anydep

General-purpose project/application dependency manager - mumoshu/anydep

Erik Osterman

also, I’m bummed that helm-s3 still doesn’t support public repos

mumoshu/anydep

General-purpose project/application dependency manager - mumoshu/anydep

mumoshu

yep, we shouldn’t be tied to helm-s3 nor s3.

anydep can be enhanced to leverage any helm plugin.

so perhaps we can add another backend (http, github, etc) to anydep and serve both use-cases of mine and your deps

2019-05-01

bug: If config-file option is set then environment specific config files can't be loaded · Issue #78 · mumoshu/variant

what If config-file option is set then environment specific config files can&#39;t be loaded. why We want to be able to set the config file through the option and still be able to use environment s…

1

2019-04-30

feat: Allow log level to be customized · Issue #76 · mumoshu/variant

WHAT: Allow log level to be customized through the environment variable, e.g.: VARIANT_LOG_LEVEL=&quot;warn&quot; WHY: We want to allow to set the custom log level for &quot;variant&quot;. For exam…

2019-04-25

@mumoshu hey, please check: https://github.com/mumoshu/variant/pull/74

Add custom log colors by aroq · Pull Request #74 · mumoshu/variant

WHAT: Add custom colors for log messages. WHY: For some cases, we want to allow to set custom colors for every type of log messages (panic, fatal, error, warn, info, debug, trace). For example, man…

mumoshu

thx for the pr! will take a look asap

Add custom log colors by aroq · Pull Request #74 · mumoshu/variant

WHAT: Add custom colors for log messages. WHY: For some cases, we want to allow to set custom colors for every type of log messages (panic, fatal, error, warn, info, debug, trace). For example, man…

2019-04-19

feature request: Allow execute task with not default param inside script section · Issue #73 · mumoshu/variant

Hello, Your tool is great but I faced with some problem. I don&#39;t know may be it already supported but I didn&#39;t find examples how to do it. Let&#39;s me explain. You have next example of usa…

mumoshu

@ Thx for chiming in! Replied.

1

2019-04-17

@mumoshu do you want a motto for variant?

“One glue to glue them all”

mumoshu

Love it!

2019-04-03

mumoshu

thx for sharing

2019-04-02

oscarsullivan_old

How do I ensure the latest version of Variant is on my module?

oscarsullivan_old

Also is this expected?

✅   (healthera-sandbox-iac) pod ➤ variant version -v
loading config file variant.yaml...missing
loading env file .varenv...missing
version
oscarsullivan_old

tried with and without the verbose flag

oscarsullivan_old

On 0.27.1 in ubuntu CLI:

[email protected]:~$ variant version
version
mumoshu

@oscarsullivan_old good catch! something seems wrong with the version command. It should print v0.27.1 at least

mumoshu


How do I ensure the latest version of Variant is on my module?

there’s no way currently. i was thinking that you’ll prefer wrapping the whole variant command up into a docker image, so that you can tie a specific version of variant and your variant command

but that’s not always the case?

mumoshu

i wonder if it helps to add variantVersion that is written in your variant command like:

variantVersion: 0.27.0

so that running ./yourcmd on a machine with an older or newer version of variant immediately fails

wdyt?

oscarsullivan_old

Are you saying every command that is run has a version output?

oscarsullivan_old

Sounds like two ideas there.

oscarsullivan_old

How would you get the latest version for idea 2 to compare against?

mumoshu


Are you saying every command that is run has a version output?

no

mumoshu


How would you get the latest version for idea 2 to compare against?

Not sure I’m following you correctly, but I’d fix variant to properly embed its own version(like 0.27.0). So that when your variant command gets loaded by the variant(as the runtime), variant can see what version of runtime is required by the command wants, and validate if its own version matches the required version

Erik Osterman

@mumoshu you are correct that in our case we can just check what version of the apk is installed

Erik Osterman

though if variant is installed natively without a package manager, there’s really no way to know what version is installed

mumoshu

yes, i have no good idea how we force embed the version number when it is installed natively with go get

Erik Osterman

oh yea, not with go get

Erik Osterman

(though I have never understood why go get is so popular when you cannot even select what eversion you go get)

mumoshu

ah ok

Erik Osterman

it’s like you go get some random version that happens to be at master at the particular instant

mumoshu

i’ve fixed variant version to properly show the version number for releases

$ ./variant version
0.27.2
Erik Osterman

Erik Osterman
Golang application auto build versioning

Is it possible to increment a minor version number automatically each time a Go app is compiled? I would like to set a version number inside my program, with an autoincrementing section: $ myapp -

Erik Osterman


yes, i have no good idea how we force embed the version number when it is installed natively with go get

Erik Osterman

one thing I’ve seen (and considered implementing at cloudposse) is adding a sentinel file for the version that is committed in the repo

Erik Osterman

that way even releases are peer reviewed

Erik Osterman

and then have the CI system apply the release tag from the sentinel file on merge to master

loren

this is what we do, with bumpversion

oscarsullivan_old

I thought you can do go get [email protected]

oscarsullivan_old

Oh wait that’s go_install

Erik Osterman

yea, there are some 3rd party tools that will do it

oscarsullivan_old

I might sound a bit uncertain because I use Ansible to install go packages

oscarsullivan_old
fubarhouse/ansible-role-golang

Installs the Go programming language and packages on Mac & Linux (Ubuntu, CentOS) - fubarhouse/ansible-role-golang

oscarsullivan_old

Can share playbook and examples if Ansible is a path you would be interested in for managing them

Erik Osterman

might be interesting for #ansible ?

loren
gopkg.in - Stable APIs for the Go language

Stable APIs for the Go language

loren

but i think it only pins to major version, which was a major bummer when i looked at it last

1

2019-04-01

mumoshu

fixed in the latest release

oscarsullivan_old

2019-03-30

mumoshu

alright. will address it soon. thx!

2019-03-29

mumoshu

@oscarsullivan_old Try running any of the defined tasks by specifying the name like ./myfirstcommand foo!

I guess - you’d expected to see the command’s help when it was run without any args, right? I think it’s currently failing, by design, because you have no root with script defined. I can make variant just print the help instead, though.

Erik Osterman

@mumoshu I think printing help would make sense in this case

2019-03-28

oscarsullivan_old

Any clue why I get this error when following the tutorial?

 ⧉  sandbox 
 ✓   (-sandbox-admin) variant ⨠ cat myfirstcommand 
#!/usr/bin/variant

tasks:
  bar:
    script: \|
      echo "dude"
  foo:
    parameters:
    - name: bar
      type: string
      description: "the bar"
    - name: environment
      type: string
      default: "heaven"
    script: \|
      echo "Hello {{ get "bar" }} you are in the {{ get "environment" }}"
 ⧉  sandbox 
 ✓   (-sandbox-admin) variant ⨠ which variant 
/usr/bin/variant
 ⧉  sandbox 
 ✓   (-sandbox-admin) variant ⨠ ./myfirstcommand
Error: unknown command "./myfirstcommand" for "myfirstcommand"
Run 'myfirstcommand --help' for usage.
Unexpected type of error *errors.errorString: unknown command "./myfirstcommand" for "myfirstcommand"
ls -la
-rwxrwxr-x  1 oscar oscar  308 Mar 28 13:21 myfirstcommand
antonbabenko

chmod +x ./myfirstcommand ?

2
oscarsullivan_old

oops I missed the ls -la, but yes that’s already been run

oscarsullivan_old
-rwxrwxr-x  1 oscar oscar  308 Mar 28 13:21 myfirstcommand
Joe Presley
05:22:09 PM

@Joe Presley has joined the channel

2019-03-27

ldlework
09:25:06 PM

@ldlework has joined the channel

Erik Osterman
09:26:51 PM

@Erik Osterman set the channel topic: https://github.com/mumoshu/variant

loren
09:28:48 PM

@loren has joined the channel

oscarsullivan_old
11:59:47 PM

@oscarsullivan_old has joined the channel

2019-03-26

2019-03-25

dennybaa
09:43:53 AM

Hello guys, I wonder is it possible to pass the parent task parameter as an argument to the child?

dennybaa

oh shh, sorry it worked when quoting opt1: '{{ get "test_param" }}'. @mumoshu great tool btw, thank you.

3
1
dennybaa

I’ve been playing with variant and in my case I feel like I’m missing toObject template helper. Does anyone now how to treat the output of a task as an object? Is it possible in variant?

mumoshu

@dennybaa hey! you can define an parameter as type: object so that any JSON-formatted stdout content can be parsed as object`

mumoshu
$ cat Variantfile
#!/usr/bin/env variant

tasks:
  getjson:
    script: \|
      echo '{"key1":1, "key2":2}'
  test:
    parameters:
    - name: getjson
      type: object
      properties:
        key1:
          type: integer
        key2:
          type: integer
    script: \|
      cat <<EOS
      {{ get "getjson" \| toYaml }}
      EOS
$ variant test
variant ≫ starting task getjson
variant.getjson ≫ {"key1":1, "key2":2}
variant ≫ starting task test
key1: 1
key2: 2
mumoshu

btw, you can even source from/override the parameter value with a file:

$ cat my.json
{"foo"<i class="em em-11,"bar""></i>22}
$ variant test --getjson=my.json
variant ≫ starting task test
bar: 22
foo: 11
dennybaa

@mumoshu yup, I’ve bumped into it as well. However there is a correlating issue - “get” tasks can not have parameterized arguments. Correct me if I’m mistaken, please.

dennybaa
tasks:
  key:
    parameters:
    - name: count
      type: integer
      default: 1
    script: \|
      echo '["private", "public"]'

  create:
    options:
    - name: key
      required: true
    script: \|
      echo {{ get "key" \| toYaml }}
mumoshu

@dennybaa you’d need to add type: object for create, and return a JSON object, not JSON array from the key task

dennybaa

In fact I already use pattern as above. So I guess parameters should work, sorry. I see. Yaml is not possible with type: object ?

mumoshu

the point would be that you don’t need to explicitly parse json or yaml. type: object do that for you!

mumoshu

would you mind sharing several example variantfiles you’ve tried, so that i can better get what you’re trying and rewrite them for you?

dennybaa

Thank you very much ! It’s just something wasn’t working, but now after a day of trail with variant I feel much more confident. So I will try to brush the code up first and get back to you after and for refactoring advises.

mumoshu

great!

mumoshu

@dennybaa i was in the impression that you wanted to (1) internally generate an parameterized array and (2) print the result as yaml

mumoshu
$ cat Variantfile2
#!/usr/bin/env variant

tasks:
  myarray:
    parameters:
    - name: second
      type: string
      default: "public"
    script: \|
      echo '["private", "{{get "second"}}"]'

  print-json-array-as-yaml:
    options:
    - name: myarray
      type: array
    script: \|
      echo '{{ get "myarray" \| toYaml }}'
$ variant Variantfile2 print-json-array-as-yaml
Variantfile2 ≫ starting task myarray
Variantfile2.myarray ≫ ["private", "public"]
Variantfile2 ≫ starting task print-json-array-as-yaml
- private
- public

dennybaa

@mumoshu why )?

mumoshu

(cont) in that case the above may be of help…

mumoshu

or for type: object, i’d write something like:

$ cat Variantfile2
#!/usr/bin/env variant

tasks:
  myobject:
    parameters:
    - name: second
      type: string
      default: "public"
    script: \|
      echo '{"items":["private", "{{get "second"}}"]}'

  print-json-object-as-yaml:
    options:
    - name: myobject
      type: object
    script: \|
      echo '{{ get "myobject" \| toYaml }}'

which produces:

$ variant Variantfile2 print-json-object-as-yaml
Variantfile2 ≫ starting task myobject
Variantfile2.myobject ≫ {"items":["private", "public"]}
Variantfile2 ≫ starting task print-json-object-as-yaml
items:
- private
- public
dennybaa

I was just trying to say that now I realize that parameterization actually works as well as object. So I should give them a better touch)

dennybaa

@mumoshu here it’s what I’m trying to do

tasks:
  key:
    parameters:
    - name: count
      type: integer
      default: 1
    script: \|
      echo '0: ["private", "public"]'

  create:
    options:
    - name: key
      type: object
      required: true
    script: \|
      echo '{{ get "key" \| toYaml }}'
mumoshu

what do you want to use count for?

dennybaa
tasks:
  key:
    description: Generates eos key-pair(s)
    parameters:
    - name: count
      description: Number of key-pairs to create
      type: integer
      default: 1
    runner:
      image: eosio/eos
      command: bash
      args: [-c]
    script: \|
      {{- range $i, $e := until (.count) }}
      pair=$(cleos create key --to-console \| sed 's/.*: //' \| sed ':a;N;$!ba;s/\n/", "/g')
      printf "{{ $i }}: [\"%s\"]\n" "$pair"
      {{- end }}

  create:
    description: Creates a keypair yaml file

    options:
    - name: key
      type: object
      required: true
    script: \|
      cat <<EOF
      {{ get "key" \| toYaml }}
      EOF
dennybaa

This is the real one ^

dennybaa

first one is a simplification

mumoshu

seems like you want to internally generate a YAML data and print it, right?

dennybaa

yes so far yes. But I’m going to process that data

mumoshu

ah

dennybaa

so the data is not sucked by create…

mumoshu

then i have to go back to yout original question.. you can’t map yaml string to type object

mumoshu

you need to use json as the internal data format

mumoshu

later print it as yaml by pipeing it to toYaml

mumoshu

toYaml is for marshalling objects to YAML, not for parsing YAML data into objects

dennybaa

i understand

dennybaa

that I was saying yaml can not be passed as type: obeject

dennybaa

the point would be that you don’t need to explicitly parse json or yaml. type: object do that for you!

dennybaa

?

dennybaa

can we make it possible too?

dennybaa

Imagine I want to load file a.yml

dennybaa

But first we need to generate that data too… (

dennybaa

while this kind of stuff works

tasks:
  key:
    parameters:
    - name: count
      type: integer
      default: 1
    script: \|
      echo '{"items": ["private", "public"]}'

  create:
    options:
    - name: key
      type: object
      required: true
    script: \|
      echo '{{ index (get "key") "items" }}'
mumoshu

that I was saying yaml can not be passed as type: obeject

mumoshu

can we make it possible too?

dennybaa

Would be awesome! Meanwhile I’m sticking to json

mumoshu

@dennybaa is it possible for you to use jq, so that you can convert it to json before outputting?

https://github.com/mikefarah/yq

mikefarah/yq

yq is a portable command-line YAML processor. Contribute to mikefarah/yq development by creating an account on GitHub.

mumoshu
    script: \|
      bash <<EOC \| yq r -j -
      {{- range $i, $e := until (.count) }}
      pair=$(cleos create key --to-console \| sed 's/.*: //' \| sed ':a;N;$!ba;s/\n/", "/g')
      printf "{{ $i }}: [\"%s\"]\n" "$pair"
      {{- end }}
      EOC
dennybaa

@mumoshu yup, i’m aware of yq bro of jq. Surely it’s possible, though it’s not likely to use option in general.

dennybaa

If we refer to containers, it will be a no go to inject yet another binary into them…

mumoshu

i slightly agree with u i used to think similarly, but started giving up because i can’t complete everything with just variant(of course?

dennybaa

In my case personal case I have a customized container… i can install yq, jq In other cases I’m pretty sure possibility to feed yaml as object should exist. This will allow users to build interesting tools

mumoshu

agreed

mumoshu

it isn’t actually a bad idea to add yaml->object conversion for variant params

mumoshu

as it doesn’t add additional binary size to variant…

dennybaa


but started giving up because i can’t complete everything with just variant(of course?
I believe the grade of your integration issues is above one I can now imagine

mumoshu
tasks:
  key:
    description: Generates eos key-pair(s)
    parameters:
    - name: count
      description: Number of key-pairs to create
      type: integer
      default: 1
    output:
      format: yaml
    script: \|
      {{- range $i, $e := until (.count) }}
      pair=$(cleos create key --to-console \| sed 's/.*: //' \| sed ':a;N;$!ba;s/\n/", "/g')
      printf "{{ $i }}: [\"%s\"]\n" "$pair"
      {{- end }}

maybe it’s a matter of adding output.format to the task definition?

dennybaa

i don’t think it’s needed… better just raw understanding of json or yaml as object, since we don not introduce any syntax or functions (toObject, fromYaml, fromJson )…

dennybaa

anyways. json is a subset of yaml, so I bet just one line of code should be changed somewhere where this conversion happens)

mumoshu

aha, good point. i was implicitly forced to consider other formats like toml, jsonnet, that might be feature-requested in the future

mumoshu

@dennybaa variant v0.27.0 has been published with this feature: type: object parameter sourced from a task output in YAML

dennybaa

wow. amazing !

2019-03-22

2019-03-21

mumoshu

Yeah both use-cases make sense to me! From the author’s perspective, I’d say Variantfile is an alternative to putting Makefile in your project root. It isn’t intended to be reused by any other projects

exactly, that’s the one of my cases! So it is good to have options here!

Erik Osterman

Makes sense….

mumoshu

./yourcmd way, like kopsctl, should be used whenever it is being reused in any way, like (1) imported from another variant command via import: or (2) as a single executable binary

mumoshu

Btw, for the latter, you can go build your variant command like explained in https://github.com/mumoshu/variant#releasing-a-variant-made-command, or in a bit more sophisticated way https://github.com/mumoshu/variant/blob/master/examples/hello/hack/generate-maingo-struct#L1-L15.

mumoshu/variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow. - mumoshu/variant

mumoshu/variant

Wrap up your bash scripts into a modern CLI today. Graduate to a full-blown golang app tomorrow. - mumoshu/variant

Erik Osterman
genuinetools/binctr

Fully static, unprivileged, self-contained, containers as executable binaries. - genuinetools/binctr

Erik Osterman

This plus variant!

Erik Osterman

I have thought about how we could do this with geodesic

Erik Osterman

So you can just run it with out even Docker installed

Erik Osterman

Remember the problem you we’re trying to solve about installing all the deps of variant

Erik Osterman

In this model, you would have a dockerfile of deps, a variant entry point, and a compiled container exe

mumoshu

that sounds very interesting

mumoshu

fyi: if you don’t want the dependency to the bash installed on your machine: https://github.com/progrium/go-basher

progrium/go-basher

Library for writing hybrid Go and Bash programs. Contribute to progrium/go-basher development by creating an account on GitHub.

Erik Osterman

Dude! That’s awesome

Erik Osterman

I think that’s a step in the right direction

2019-03-20

is there any specific reason to separate config file & main Variantfile or config could be included in Variantfile file (as an option)?

Erik Osterman

@ you’re not using the #! mode?

Erik Osterman
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Erik Osterman

to me this is the ideal invocation

Erik Osterman

then the . configs are quite nice

@Erik Osterman probably my question was not clear

I’ve meant that I like the idea to keep everything related to Variant in a single file (including environment specific config files, a main config file, etc) for some specific projects, where some other people will participate (it can be Variantfile or !# file, etc).

It will be probably easier for these people to adopt to the new workflow (with Variant) when everything is in the one place (file).

Also, it will reduce “invasion” (only one new file instead of 2-3 files) to other people’s projects (where I only setup the CICD for example).

Please let me know if I’m not clear!

Erik Osterman

Personally, the Variantfile approach was less appealing than writing a cli with variant

Erik Osterman

for example, we write our cli called kopsctl using the variant DSL

Erik Osterman

and then as a cli tool, I want the configuration separate. that logical separation IMO makes it more manageable than keeping it all in the same script.

yes, it makes total sense in this context

but, I still can see some cases where Variantfile would be more appealing to some people

as for me only, I prefer the CLI approach as well

Erik Osterman

ok,

but still like to have some options

thanks for sharing your vision!

Tim Malone
11:07:47 PM

@Tim Malone has joined the channel

2019-03-19

regarding https://github.com/mumoshu/variant/issues/51

it is helpful to include single tasks, but if I want to include a bunch of tasks with a single statement, what are the options (I didn’t check all the tests & code)?

@mumoshu

Import variant commands from various versioned sources · Issue #51 · mumoshu/variant

So that you can create a common/library variant commands to be versioned and reused. tasks: anothercmd: # Basically runs go-getter <http://github.com/foo/bar?ref=v0.1.0//dir\|github.com/foo/bar?ref=v0.1.0//dir> /tmp/somewhere and then inclu…

found this one - https://github.com/mumoshu/variant/issues/52 seems like this is it

Import config files from various versioned sources · Issue #52 · mumoshu/variant

So that you can create a common config file that is versioned and reused in various variant commands in divergent places. tasks: mytask: parameters: - name: config type: object script: | cat <&l…

nope, that’s only for config files in fact

just import in the main file works OK:

...
import: variant_lib.yaml
...

good stuff!

mumoshu

awesome to see you figured out about that! yes, import can be used for importing both single task or a set of tasks :)

mumoshu

also - if you encouter any difficulty on writing your variant command, pls feel free to share an example code!

1

thanks! I believe this tool is going to save a lot of time for a lot of people!

I’m still going to figure out the differences between “inputs”, “options” & “parameters”

1
mumoshu

those are interchangeable, and parameters is the latest and recommended syntax! the former two will be eventually removed

I see, thanks for the explanation!

2019-02-16

nutellinoit
08:52:16 AM

@nutellinoit has joined the channel

2019-02-14

Erik Osterman
05:23:43 AM

@Erik Osterman set the channel purpose: Discuss variant (the “Universal CLI”) https://github.com/mumoshu/variant

Archive: https://archive.sweetops.com/variant/

2019-02-13

platinumnj
04:31:20 AM

@platinumnj has joined the channel

2019-02-09

2019-02-08

mumoshu

any contribution to docs is welcomed! (well, i’ll write more later :)

mumoshu

FYI: Trigger your Variant tasks via GitHub Actions https://github.com/mumoshu/github-actions/tree/master/variant

mumoshu/github-actions

Contribute to mumoshu/github-actions development by creating an account on GitHub.

1
Erik Osterman

this is exciting!

Erik Osterman

github actions still don’t work on public projects, right?

antonbabenko

Seems so. Unfortunately.

Erik Osterman

Erik Osterman

variant will be so awesome when they add support for public repos

Erik Osterman

run a pipeline locally as easily as remotely

I can see the action button on one the projects i’m contributor in. Not sure if I can use them. Maybe the repo owner has to be subscribed to the beta

dennybaa
12:19:19 PM

@dennybaa has joined the channel

richwine
10:53:51 PM

@richwine has joined the channel

2019-02-07

mumoshu

Wrote up an another issue that fills one of big missing pieces(to me): https://github.com/mumoshu/variant/issues/57

Experimental: Dependency Management(Packages) · Issue #57 · mumoshu/variant

This is even more experimental than other features :) variant should be able to manage package (direct) dependencies of your variant command, along with those of imported commands(=transitive depen…

joshmyers
09:23:20 AM

@joshmyers has joined the channel

joshmyers
mumoshu/variant

Write modern CLIs in YAML. Bash + Workflows + Dataflows + Dependency Injection, JSON Schema for inputs validation - mumoshu/variant

joshmyers

Liking it so far but having to dig through all the tests

Erik Osterman

Unfortunately the integration tests are the best documentation at this time. Fortunately there are a lot of examples there though!

2019-02-06

05:14:47 PM

@ has joined the channel

mallen
09:03:44 PM

@mallen has joined the channel

mumoshu
Dependency Management(Versioned Directories) · Issue #56 · mumoshu/variant

variant should provide a generic way to manage dependencies of your variant command: dependent: directories: - path: ./helmfile-libs/stable # multiple library helmfile.yaml files should be maintain…

Erik Osterman

Ahhh yes, ultimately this becomes an issue.

Erik Osterman

I would like to discuss some idea that @Igor Rodionov had today

1
Erik Osterman

I am afk, so will pick up later

1
mumoshu

btw: https://github.com/mumoshu/variant/issues/51 and https://github.com/mumoshu/variant/issues/52 has been resolved and included in variant v0.20.0 (haven’t done much testing yet. briefly verified to work with github sources though

Import variant commands from various versioned sources · Issue #51 · mumoshu/variant

So that you can create a common/library variant commands to be versioned and reused. tasks: anothercmd: # Basically runs go-getter <http://github.com/foo/bar?ref=v0.1.0//dir\|github.com/foo/bar?ref=v0.1.0//dir> /tmp/somewhere and then inclu…

Import config files from various versioned sources · Issue #52 · mumoshu/variant

So that you can create a common config file that is versioned and reused in various variant commands in divergent places. tasks: mytask: parameters: - name: config type: object script: | cat <&l…

2019-02-05

mumoshu

This feature will reduce repetitions in your scripts https://github.com/mumoshu/variant/issues/53

Reusing only part(s) of scripts · Issue #53 · mumoshu/variant

I tend to organize my variant scripts to the &quot;header&quot; and the &quot;body&quot; like seen in the following example: script: | # header {{ $env := get &quot;environment&quot; }} {{ $state :…

Erik Osterman

that’s awesome!

Erik Osterman

also, like that you support a list under script

Erik Osterman

are those ultimately joined as one script or each run in their own shell?

mumoshu

the former, to cover reuses of both bash variable/func definitions, and template variables!

Erik Osterman

yea, makes sense

Erik Osterman
Import config files from various versioned sources · Issue #52 · mumoshu/variant

So that you can create a common config file that is versioned and reused in various variant commands in divergent places. tasks: mytask: parameters: - name: config type: object script: | cat <&l…

Erik Osterman

this!!

Erik Osterman

I want.

Erik Osterman

basically we can create a variant library, then create a cli that imports various capabiltiies

Erik Osterman

for example, we can create a helm lib, and a helmfile lib and a kops lib and a terraform lib with useful tricks

Erik Osterman

e.g. for terraform, automating force-unlock

Erik Osterman

then in the final implementation, we can include the libs that we want into our cli. for example, one customer might not use kops, so we wouldn’t need to import that lib.

Erik Osterman
Import variant commands from various versioned sources · Issue #51 · mumoshu/variant

So that you can create a common/library variant commands to be versioned and reused. tasks: anothercmd: # Basically runs go-getter <http://github.com/foo/bar?ref=v0.1.0//dir\|github.com/foo/bar?ref=v0.1.0//dir> /tmp/somewhere and then inclu…

Erik Osterman

so cool

Erik Osterman

Erik Osterman

@mumoshu I am trying to run an interactive SSH session inside of variant, should that work?

Erik Osterman
+ ssh -A <mailto:[email protected]>
Pseudo-terminal will not be allocated because stdin is not a terminal.
Warning: Permanently added '<http://bastion.us-west-2.staging.even.io>,35.163.154.194' (ECDSA) to the list of known hosts.

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

mumoshu

interactive: true should work

tasks:
  foo:
    interactive: true
    script: \|
      whatever
Erik Osterman

Erik Osterman
02:45:11 AM
Erik Osterman

hah! that worked, but got this

Erik Osterman

mumoshu

whoa! let me check

mumoshu

whats your variant’s version number?

Erik Osterman

(variant version empty)

Erik Osterman
cloudposse/packages

Cloud Posse installer and distribution of native apps, binaries and alpine packages - cloudposse/packages

Erik Osterman

0.16.1

Erik Osterman

i will try the latest

Erik Osterman

reproducible in 0.18.0

Erik Osterman

in 0.18.0 I see some debug output

Erik Osterman
schema = map[type:object properties:map[] required:[]]
mumoshu

thanks! please try v0.18.2 that fixes the logging and the hanging

Erik Osterman

ALL FIXED!

Erik Osterman
       tasks:
          # Connect to bastion instance
          bastion:
            description: "Connect to the bastion"
            script: \|
              set -e
              source <(chamber exec kops -- sh -c 'export -p')
              eval $(ssh-agent -s)
              ssh-add - <<<${KOPS_SSH_PRIVATE_KEY}
              ssh -A [email protected]${KOPS_CLUSTER_NAME}
              eval $(ssh-agent -k) >/dev/null
Erik Osterman

Erik Osterman

I love it!!!

mixins:
  chamber: &chamber
    source <(chamber exec kops -- sh -c 'export -p' 2>/dev/null)
Erik Osterman

Then anywhere I want chamber, I can just do this:

          bastion:
            description: "Connect to the bastion using ssh-agent forwarding"
            interactive: true
            script:
            - *chamber
            - \|
              set -e
              eval $(ssh-agent -s)
              ssh-add - <<<${KOPS_SSH_PRIVATE_KEY}
              ssh -A [email protected]${KOPS_CLUSTER_NAME}
              eval $(ssh-agent -k) >/dev/null
Erik Osterman

this is shell scripting on steroids

Erik Osterman

Erik Osterman

I tried to change the default runner by adding

Erik Osterman
runner:
  command: "bash"
  args: ["-ex", "-c"]


Erik Osterman

should that work?

Erik Osterman

(in the global scope)

Erik Osterman

my first advanced variant script:

1
Erik Osterman
[kopsctl] add commands to facilitate management of cluster by osterman · Pull Request #378 · cloudposse/geodesic

what Add commands to easily rotate a kops cluster&#39;s ssh keys Add command to easily connect to a kops cluster Add command to see a kops plan why This are routine operations that are complicat…

1
04:37:18 AM

@ has joined the channel

2019-01-17

antonbabenko
08:07:23 AM

@antonbabenko has joined the channel

davidvasandani
05:32:52 PM

@davidvasandani has joined the channel

2019-01-16

Erik Osterman
07:44:30 AM

@Erik Osterman has joined the channel

Erik Osterman
07:44:30 AM

@Erik Osterman set the channel purpose: Discuss variant (the “Universal CLI”) https://github.com/mumoshu/variant

mumoshu
07:44:31 AM

@mumoshu has joined the channel

aknysh
07:44:31 AM

@aknysh has joined the channel

Erik Osterman
mumoshu/variant

Write modern CLIs in YAML. Bash + Workflows + Dataflows + Dependency Injection, JSON Schema for inputs validation - mumoshu/variant

5
Erik Osterman

You updated the description

07:47:54 AM

@ has joined the channel

07:55:48 AM

@ has joined the channel

    keyboard_arrow_up