#build-harness (2019-12)

Help with the Cloud Posse build-harness https://github.com/cloudposse/build-harness

2019-12-28

deftunix avatar
deftunix
02:30:41 PM

@deftunix has joined the channel

deftunix avatar
deftunix

Hi all, do you have in build-harness any switch to deploy helm charts?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No, or at least not maintained. We use #helmfile for deploying charts

2019-12-19

Can Kutlu Kinay avatar
Can Kutlu Kinay
10:46:02 AM

@Can Kutlu Kinay has joined the channel

2019-12-18

Roderik van der Veer avatar
Roderik van der Veer
08:57:14 AM

@Roderik van der Veer has joined the channel

Taras avatar
Taras
12:13:47 PM

@Taras has joined the channel

Kostis (Codefresh) avatar
Kostis (Codefresh)
03:34:57 PM

@Kostis (Codefresh) has joined the channel

Eugene avatar
Eugene
03:38:48 PM

@Eugene has joined the channel

osirisx.tls_cloudposs avatar
osirisx.tls_cloudposs
05:25:37 PM

@osirisx.tls_cloudposs has joined the channel

curious deviant avatar
curious deviant
05:31:20 PM

@curious deviant has joined the channel

2019-12-17

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
08:22:14 PM

@Erik Osterman (Cloud Posse) has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
08:22:15 PM

set the channel description: Help with the Cloud Posse build-harness https://github.com/cloudposse/build-harness

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
08:22:30 PM

@Andriy Knysh (Cloud Posse) has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@imiltchman just created this channel

imiltchman avatar
imiltchman
08:22:39 PM

@imiltchman has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) said you have some problems with the build-harness when running make readme

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’d like to help triage

imiltchman avatar
imiltchman

Yes, thank you. The readme is generated but without terraform output. ie docs/terraform.md is empty.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

I had similar issue (variables/outputs not generated). It was an old version of gomplate in PATH

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

removed it, and then make readme/deps installed the new version, which worked

imiltchman avatar
imiltchman

I have no gomplate in PATH

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but that was on Mac

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, so docs/terraform.md does not have anything to do with gomplate

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we can rule that out

imiltchman avatar
imiltchman

That’s terraform-docs, right?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

docs/terraform.md is generated using the “terraform-docs” or more accurately a shell script that wraps it and munges the terraform code (HCL2) so that it works with terraform-docs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i suspect there’s something we are using in that script that doesn’t work on your ubuntu distro (perhaps missing package)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

one sec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is the script we run

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you test if that script works for you?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. ./build-harness/bin/terraform-docs.sh md . i think

imiltchman avatar
imiltchman

I don’t have terraform-docs in my PATH either

imiltchman avatar
imiltchman

but it seems to be in the /vendor/ folder of the build-harness

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the Makefile sets the search PATH to include the build-harness/vendor folder

imiltchman avatar
imiltchman

Do I need to run make within build-harness itself?

imiltchman avatar
imiltchman

(hides behind the tree)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not technically, you just need those things in your path

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so try this:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(for debugging) export PATH="./build-harness/vendor:$PATH"

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then if you run terraform-docs it should find it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

test that that works first..

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then we’ll proceed to debug

Jeff Young avatar
Jeff Young
08:32:37 PM

@Jeff Young has joined the channel

imiltchman avatar
imiltchman

cannot open /bin/terraform-docs.awk

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha sec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

set export BUILD_HARNESS_PATH=./build-harness (again, just for debugging right now - none of this stuff needs to be set otherwise)

imiltchman avatar
imiltchman

no output from terraform-docs.sh now

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, run the script this way:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

bash -x ./build-harness/bin/terraform-docs.sh md .

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then we should see where it gets

imiltchman avatar
imiltchman
+ which awk
+ which terraform
+ which terraform-docs
++ head -1
++ terraform version
+ [[ Terraform v0.12.18 =~ 0\.12 ]]
++ mktemp /tmp/terraform-docs-XXXXXXXXXX
+ TMP_FILE=/tmp/terraform-docs-JhZVXiFB8g
+ awk -f ./build-harness/bin/terraform-docs.awk ./main.tf ./outputs.tf ./variables.tf ./versions.tf
+ terraform-docs md /tmp/terraform-docs-JhZVXiFB8g

+ rm -f /tmp/terraform-docs-JhZVXiFB8g
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, so it’s working, but appears then to be something wrong either with the awk

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or the output from the awk

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you run awk -f ./build-harness/bin/terraform-docs.awk ./main.tf ./outputs.tf ./variables.tf ./versions.tf

roth.andy avatar
roth.andy
08:43:53 PM

@roth.andy has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we also have a couple open PRs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i haven’t had a chance to test the work yet by @cytopia

cytopia avatar
cytopia

As stated in the PR description, I’ve added regression tests for every issue on GitHub, so it should not be too time-consuming to validate it.

cytopia avatar
cytopia
08:48:29 PM

@cytopia has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

do any of those issues sound like they could be related to your problem?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if so, we can test that

imiltchman avatar
imiltchman

I messed something up in my environment and now curl to get gomplate fails for me

imiltchman avatar
imiltchman

on make readme/deps

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what’s the error?

imiltchman avatar
imiltchman

curl: (23) Failed writing body (0 != 16360)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh, that just means the gomplate binary is not writable

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

probably b/c sudo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

check fs permissions

imiltchman avatar
imiltchman

all good now

imiltchman avatar
imiltchman

trying the awk cmd, one sec

imiltchman avatar
imiltchman

no output

tamsky avatar
tamsky
08:51:48 PM

@tamsky has joined the channel

krueger.andre avatar
krueger.andre
08:51:48 PM

@krueger.andre has joined the channel

cbravo avatar
cbravo
08:51:48 PM

@cbravo has joined the channel

daveyu avatar
daveyu
08:51:49 PM

@daveyu has joined the channel

oscar avatar
oscar
08:51:49 PM

@oscar has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

<https://raw.githubusercontent.com/cloudposse/build-harness/b65f55ec83c4c63d6aff1dfa2201547ffc676262/bin/terraform-docs.awk>

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you download that file

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then replace the awk script with that one

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

re-run see if it works?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if so, I’ll merge this PR

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(also, i assume you can read all the *.tf files? no perms issues, right?)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, @imiltchman which repo?

imiltchman avatar
imiltchman

terraform-aws-ecs-container-definition

imiltchman avatar
imiltchman

What should the expected output be for terraform-docs.awk?

imiltchman avatar
imiltchman

I get nothing with the version above as well

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the terraform variables and outputs formatted in a an old HCLv1 format

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that can be read by terraform-docs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add support for Terraform 0.12's Rich Value Types · Issue #62 · segmentio/terraform-docs

Prerequisites Put an x into the box that applies: This issue describes a bug. This issue describes a feature request. For more information, see the Contributing Guidelines. Description With the upc…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically we wrote an awk script to munge the HCL2 code so it could be parsed by terraform-docs (sufficient for automatic documentation)

imiltchman avatar
imiltchman

If I run terraform-docs directly

imiltchman avatar
imiltchman

I get 2019/12/17 15:58:45 At 3:26: Unknown token: 3:26 IDENT var.environment

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yup

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(due to HCl2)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) I don’t have access to my terminal ATM

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you check if you can run readme on that project right now?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

terraform-aws-ecs-container-definition

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

will check (but I ran it about two weeks ago)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

runs ok

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

README and terraform.md generated

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@imiltchman IDENT error usually happens when you mix different TF versions of TF binary and the module version

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is always going to be the case with terraform-docs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There is no 0.12 support

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That is why the wrapper exists

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

e.g. try to use TF 0.12 version of the module with TF 0.11 binary

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

or other way around

imiltchman avatar
imiltchman

Looks like terraform-docs doesn’t support HCL2

imiltchman avatar
imiltchman

Hence the error

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(And the awk script)

imiltchman avatar
imiltchman

I don’t know enough about awk to troubleshoot why it’s failing, but it’d be great to figure it out so others don’t get stuck with the same problem

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) can you confirm that you can reproduce?

imiltchman avatar
imiltchman
09:35:01 PM

runs ok

imiltchman avatar
imiltchman

I think it’s just me

imiltchman avatar
imiltchman

I’ll try to find time to debug it on my own

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Weird! Something else though I was thinking is we should just run build harness inside of Docker

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And eliminate these issues

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We already have the container

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And already do this in our other pipelines

imiltchman avatar
imiltchman

I am guessing if I run geodesic, it would be more likely to work as well, right?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes try in geodesic

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and yes, I tried it many times, works (with TF 0.12 binary and 0.12 version of the module, master branch)

AutomatedDissonance avatar
AutomatedDissonance
03:18:47 AM

@AutomatedDissonance has joined the channel

    keyboard_arrow_up