#geodesic (2019-02)

@Erik Osterman (Cloud Posse) - how often do you update your APK repository?

on every merge

we cut a release on every merge to master

in other words, if a PR is merged, it’s been deployed

when installing packages, recommend doing somethin glike

apk add --update terraform@cloudposse=0.11.1-r0

for strict pinning, plus updating the apk cache

odd - when I’m building I’m still getting terraform 0.11.10 and i get this if I try to update

ERROR: unsatisfiable constraints: terraform-0.11.7-r0: breaks: world[terraform=0.11.11]

Running this got it to work apk add terraform –update-cache –repository https://apk.cloudposse.com/3.8/vendor –allow-untrusted

hrm

not good.

ah i see why i still get 0.11.10 - layer caching

here’s what I get in geodesic

ah you need the -r0

All good now, thanks

yea, I haven’t dug deeper into why the release is required

we are going to do some more work on the alpine repo over the coming month (time permitting)

among other things, improve the build process so that it only builds changed paths

also, considering rendering the APKBUILD template and committing so it’s more obvious what’s going on. currently, the APKBUILD template wraps make

Also, --allow-untrusted should not be required (we don’t use it); the packages are signed

Are you guys doing any authentication to private git repositories for terraform modules in geodesic? If so, how are you handling authentication?

@me1249 yes, SSH keys

how are you making the SSH keys available to the docker container?

When running geodesic locally your agent gets mounted in, when running via Atlantis, it is writing out an SSH key from chamber

ah

Which docker container exactly?

I totally missed ATLANTIS_SSH_PRIVATE_KEY

Thanks

@rohit.verma I forget, are you using GitLab?

Still having issues after settings ATLANTIS_SSH_PRIVATE_KEY - it seems like the atlantis user doesn’t have access to the SSH auth socket - anyone come across this issue?

hrmmmm

sec

https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh#L73

yes, we should chmod the sock as well.

on line 89, we gosu to the ATLANTIS_USER; prior to that we’re root

@me1249 btw, have you seen the tmate-session util we added to geodesic

if you install tmate in your container, you can call tmate-session and it will output a URL you can use for debugging

Yep - no luck with chmoding the socket either

even tried chowning it

Have to execute the ssh-agent under the atlantis user using gosu

hrmmmm

sec

what did you chown it?

what was the mode?

past experience tells me that it’s not required to run the agent under atlantis.

ohh

wait

yes, the pid probably needs to be same uid

So changing to source <(gosu ${ATLANTIS_USER} ssh-agent -s)

no - that’s not true, since on Linux we bind mount in the agent sock

fixes the issue

that’s a nice fix

however, <(...) won’t work without a TTY

have you deployed that under ECS and it works?

Just deploying it now

ok, lmk

ohhhh we had source <(ssh-agent -s)

hrmmm odd. i had trouble with that under certain circumstances.

if that doesn’t work, try eval $(gosu ${ATLANTIS_USER} ssh-agent -s)

It worked

woohoo!

can you open a PR for that?

we’ll release that right away

Sure can

thanks @me1249!!

merged and released as 0.69.1

No problem

Got another fix coming for you soon regarding timezones

haha, cool

https://github.com/cloudposse/geodesic/pull/378

here’s an example of using the #variant tool for exposing a clean cli UI

much more powerful than make

will have to check it out

nice

so even if not using kops, this pattern can be extended to other things

like myorgctl

that ties together all the tools “my org” uses

In case anyone comes searching in here that’s trying to use atlantis with tfenv and terragrunt - do not set TF_CLI_DEFAULT_NO_COLOR=true

It breaks terraform init

terragrunt passes -no-color by default so it ends up passing it twice - terraform doesn’t like that

yes, with terragrunt that causes problems

I had all kinds of problems with terragrunt and -no-color irrespective of atlantis

(from our soon to be released slack archives!!)

Hmm, I don’t think terragrunt itself has any internal logic for injecting -no-color… What does your terraform.tfvars file look like?

I’ll look at the terragrunt code when I get to a computer to confirm…

ahh, ok, right, auto-init is the trigger… https://github.com/gruntwork-io/terragrunt/blob/master/cli/cli_app.go#L585-L591

huh. usually slack expands the code with those urls

that was done relatively recently… pretty sure related to how terragrunt checks the auto-init output for a particular error message that is not an error (to auto-init) so it knows to ignore the non-zero return code

https://github.com/gruntwork-io/terragrunt/commit/25c3b8db3b8a634a4de8393f62779f63a3f19bd1#diff-56be1e853ecf6e7ad64321692db32678R538

curious if this works: TF_CLI_ARGS_init=-no-color terragrunt ...

either way, i think you have enough here to report as a bug… terragrunt may need to be a little smarter about how/when it appends -no-color in auto-init

There is already an issue for it

i couldn’t find one

there is one saying that -no-color is not being passed to init

https://github.com/gruntwork-io/terragrunt/issues/390

yeah

that’s kinda the opposite though

ah right, hadn’t quite followed all the convo

adding nano to the image is one of those little things that makes life less painful

https://github.com/segmentio/chamber/pull/184

Chamber now supports env export suitable for eval

Nice

Can somebody please approve my PR https://github.com/cloudposse/packages/pull/159

@me1249 can you please rebuild README

make init

make readme/deps

make readme

Done

Thanks

Is there an architectural design for how the reference architecture is setup?

As in a diagram?

@joshmyers yes

Not as far as I’m aware, no.

Do you have any specific questions?

hmm, wonder what it would look like in cloudcraft… maybe sign up for a free trial, spin out the reference architecture, capture it in cloudcraft, and export a diagram?

I think that would be an excellent idea

Have to say I think CloudCraft diagrams look meltingly good, but actually not that easy to follow

I struggled with cloudcraft diagrams where if what I wanted to add something which was not isomorphic

But yes, diagrams are long overdue

there are a couple other similar services i think, could try them all i imagine

We use Lucid Charts

Have a lot of diagrams that we use in our SOWs, just haven’t had the chance to open source something.

FYI The export cloudcraft from aws is a tad iffy. We’re currently paying for it though, its nice to have docs that don’t look arse

hey folks :slightly_smiling_face: I’m checking out the reference architecture setup. great work, it’s really nice. I can’t seem to locate the code where the admin IAM users are created The root-iam module takes a list of user names root_account_admin_user_names, but looking through all the repos, I can’t find where those users are actually created before being passed to root-iam

i think the module that creates the users is cloudposse/terraform-aws-iam-user, but where is this called?

@Abel Luck https://github.com/cloudposse/root.cloudposse.co/blob/master/conf/users/osterman.tf

@Abel Luck https://github.com/cloudposse/reference-architectures is our latest approach. We changed how the users are provisioned. Before, we were adding users to root_account_admin_user_names, but this was not scalable (e.g. when using CI/CD and need to add a user, we’d have to update the user list)

so now, for each new user we create a separate file in https://github.com/cloudposse/root.cloudposse.co/blob/master/conf/users

and the user gets added to the group https://github.com/cloudposse/terraform-root-modules/blob/master/aws/users/main.tf#L36

the group itself gets written to SSM here https://github.com/cloudposse/terraform-root-modules/blob/master/aws/iam/staging.tf#L19

@Abel Luck welcome to the community

Ah interesting.

that clears that up!

where does the local.admin_groups come from?

share the URL where you see it

it’s read from SSM

@Abel Luck it comes from https://github.com/cloudposse/terraform-root-modules/blob/master/aws/users/main.tf#L44

terraform-root-modules/aws/users gets merged with [root.cloudposse.co/aws/users](http://root.cloudposse.co/aws/users)

gotcha

nice use of the ssm

Logic to add users is common to all stages, so goes in your terraform-root-modules but your users maybe different per stage, so we put those per stage

thanks

well, all users get added to the root account in our case, but that is the general idea behind your stage codebase (like [testing.cloudposse.co](http://testing.cloudposse.co)) and terraform-root-modules

took me a little bit to duplicate… the config to end up here is pretty specific

What is the bootstrap module for exactly? Where is it used?

the pr says “ * This is needed by the reference-architectures to provision the subaccounts because it’s not possible to assume-role using master account”

but aren’t the accounts already provisioned by the time the bootstrap module is run?

(I’ve assumed the execution order is defined by thishttps://github.com/cloudposse/reference-architectures/blob/67afdc62ccedb6ab3dcd1f8b6deff85949a6535b/configs/root.tfvars#L71-L81)

Our reference architecture assumes you start with a brand spanning new AWS account

In fact that account is so new there are no existing IAM users

So the role of the bootstrap module is to provision a user and role which can be used by the coldstart process to assume role into the sub accounts that get created

Then at the very end we can destroy this module

gotcha.

i’ve been working through the cold start code by following https://docs.cloudposse.com/reference-architectures/cold-start/ as well as looking at the repos

i’ve noticed that that doc page has diverged from the code, but i didn’t realize it was that much

Following on from my question yesterday about how users get created

The user is added to the admin groups by the users root module, but later in the iam root module the organization-access-group module will REMOVE the user from the group, because the [prod|dev|...]_account_user_names list is empty

yes, so users should get created in the [root.org.com](http://root.org.com) account

and not added to root-modules directly

an old pattern we had, was (1) adding users, then (2) adding users to the group list

that didn’t work well

makes sense yea

the new pattern is (1) adding users and adding them to the groups at the same time

so when you delete a user, they are deleted from the groups too

works well with atlantis

indeed, yes, have the user creation also add them to the group

so the iam root module is run before the users are added

i am not sure if our terraform examples in [root.cloudposse.co](http://root.cloudposse.co) are up to date; what i am describing is how we’re doing it with current customers

because if i run the iam root module after adding users, it removes them

yea, so our root modules are mostly organized into separate “phases”

@Abel Luck btw, feel free to schedule some time with me next week: https://calendly.com/cloudposse

It seems to me that the organization-access-group module should only create the group and apply the policy

not actually manage the members of the group

yes

agreed - this is what i think we should deprecate

that works better when users are decoupled from the group

(btw, if you have time to submit a PR for that, we’ll promptly review)

if not, we’ll probably fix this on our next customer rollout

i think the same applies to the iam-assumed-roles module

yep - indeed

So i’ve been piecing together my own version of the reference architecture using ya’lls great building blocks

Since I started with the outdated docs, I created the accounts by hand, and also created the root admin user by hand

that’s no problem..

Now I have the root account bootstrapping complete as far : tfstate backend bootrap, root-iam policies, account-settings and creating users

however the switch role url for the non-root accounts isn’t working

rather, the url works, but i get a perm denied error

now given what I know about AWS roles, the sub-account (aka the trusting account) needs to be involved, but I haven’t involved the root user of the sub-account at all

how were the child accounts created?

i created them manually and invited them to the root acount

maybe the invited accounts don’t have the OrganizationAccountAccessRole?

ding ding

https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_accounts_access.html#orgs_manage_accounts_create-cross-account-role

correct

they do not

we do have a module you can use to provision that though

https://github.com/cloudposse/terraform-aws-organization-access-role

of course you do

:)

the null-label module is just fantastic

Mind boggling.

who knew string concatenation could be so great

Do you all generally create one log bucket per-thing that ships logs (e.g, an ALB) or do use one bucket to receive multiple services’ logs?

The latter seems more user friendly, but managing multiple policies on a bucket is a PITA

Genercally I try and and <S3://region-account-logs-bucket/$service>

I think we should adopt this convention on our buckets for logs

isolating buckets by region is smart.

hey all!

i’m giving a talk on Geodesic in the beginning of March here in Los Angeles

I don’t have all the details yet.

I’m putting together an abstract for the meetup.

Would love to get your feedback if I’ve captured what we have with #geodesic

Title: Geodesic Cloud Automation Shell
Subtitle: The easy way to automate everything

Geodesic is a cloud automation shell. It’s the superset of all other tools including (terraform, terragrunt, chamber, aws-vault, aws-okta, kops, gomplate, helm, helmfile, aws cli, variant, etc) that we use to automate workflows. You can think of it like a swiss army knife for creating and building consistent platforms to be shared across team environments. It easily versions staging/production/dev environments in a repeatable manner that can be followed by any team member with only a single dependency: docker. Because of this, it works with Mac OSX, Linux, and Windows 10. Learn how you can use the geodesic shell to improve your DevOps workflows! There will be a live demo with Q&A.

Please add any suggestions as a thread to

@Jan @daveyu @Adam @joshmyers @Dombo @chrism @me1249 @loren @Andriy Knysh (Cloud Posse) @Igor Rodionov

The easy way to automate everything

beauty is in the eye of the beholder

now that I’m used to chamber exec in geodesic, I’m looking for a clean way to export secrets for developers. I’d love to put something like this in an app’s .direnvrc: aws-vault exec myco-dev-admin -- chamber exec myapp -- sh -c 'export -p'. I haven’t been able to butcher my way through bash to get this to work, I think because aws-vault needs user input

is it possible?

yep, it’s possible

i’m on a call

we do it though

also, they just added shell export compatibility

not sure if a release has been cut

https://github.com/segmentio/chamber/pull/184

In the ref architecture world is it better to pump flowlogs to cloudwatch or into the audit account S3 bucket.

In case you want/need to do both: https://aws.amazon.com/blogs/security/how-to-facilitate-data-analysis-and-fulfill-security-requirements-by-using-centralized-flow-log-data/

Ta. Need logging of my loggings logging

Terraform network ACLS are crap. Nothing like having to use counts because someone decided subnet CIDR should be a single entry

@Erik Osterman (Cloud Posse) not sure if you’ve tripped over this; https://github.com/nozaq/terraform-aws-secure-baseline

the alarm base is pretty similar to yours (bar you use loops, they’ve used individual entries)

Will take a look. I know ours also needs some love.

the layouts a bit shit in there but such is the joy of having to use N providers

I feel like I’d prefer a module that took a single provider, and let me call it as many times as needed for each region I needed to configure

Looks alright if I use the submodules

yeah the submodules for just that

think the point is that you’re supposed to configure every region (you have enabled) or someone with access via a liberally create iam key could simply spawn shit outside of the region you normally use

Can you restrict regions at the SCP level? seems to all be around feature restriction

Not sure this is baked properly in cloudposse/geodesic:0.71.0

@chrism do you have anything in /localhost/.aws/config ?

works fine on rel 70

I can take a look as well in a few hours

that’ll be https://github.com/cloudposse/geodesic/releases/tag/0.71.0 @chrism

yup; changing to 70 at least stopped me veering off down a rabbit hole to work out that one

@chrism what does your $AWS_CONFIG_FILE look like?

im assuming its something in the function choose_role_interactive() { in aws-vault but it seems sane enough (though ive no idea what the fzf bit tis)

thats running 70

ill give you 71 in about a minute or so when its built again

Cool, it should be the same, works for me (tm) - see https://github.com/cloudposse/geodesic/pull/376

what does your config look like is what I’m asking, is it in the same format as can be seen on that PR?

` ✓ (ivendi-prod-admin) vpc ⨠ echo $AWS_CONFIG_FILE /localhost/.aws//config` looks the same

yeah its pretty much the same

even though i can cat out my config

no diff if I change the path to /localhost/.aws/config rather than double slash

Is this going to show a prompt everytime? because that kinda takes away from having N repos /org split

Everytime you assume a role, or you can pass in a specific role as before assume-role role_name

Needed a way to assume different IAM roles for k8s rbac

i just typed assume-role previously and im assuming it defaults to the one in the env

yeah, it did that if you didn’t give a name.

so both @joshmyers and @chrism are right

@joshmyers is alluding to the fact we support now multiple roles per account; not just one admin role

e.g. one role to become a kubernetes admin, one role for read-only, etc

however, the “one role policy” is probably going to continue to common

so maybe we should still honor AWS_DEFAULT_PROFILE if it is set

unset it for the selector

thoughts @joshmyers?

Seems fair, AWS_DEFAULT_PROFILE is still set

hrm

but not honored by the selector?

(i haven’t looked into the code yet)

https://github.com/cloudposse/geodesic/pull/376

AWS_DEFAULT_PROFILE is set everywhere IIRC so you will never get to selector

previously it worked like this

but now, AWS_DEFAULT_PROFILE is gone

yes, as I alluded to

i guess what I mean, is we (cloudposse) should probably remove AWS_DEFAULT_PROFILE from our dockerfile (*.cloudposse.co)

then we get the benefit we want

but if AWS_DEFAULT_PROFILE is set, the original functionality persists that @chrism is using

crudini --get "${AWS_CONFIG_FILE}" | awk -F ' ' '{print $2}' fails with the same error but the variable is set.

and the file exists

hah, looks like it

even giving it the direct path fails

crudini --version ?

0.9

if i copy the file it works

chrism, remind me: are you on WSL?

yerp

hrm… ok, maybe related

ah ha

file system permission based issue; i’d expect aws-vault to fail though

does cat ${AWS_CONFIG_FILE} >/dev/null

work?

work as in do nothing yes

ok, no errors

yeah it cats out fine

i wonder if we run crudini with config from stdin instead then

https://github.com/cloudposse/geodesic/pull/380

cloudposse/reference-architectures(master)%% make children
make: *** No rule to make target `dev/validate', needed by `children/validate'.  Stop.

so running into this, after a make root. looking at the makefiles, I’d tend to agree with make. which leads me to suspect there’s a missing step somewhere to generate more makefiles?

https://github.com/cloudposse/reference-architectures/issues/10

does this fix your issue?

(sorry, we haven’t had a time to open the PR for this fix albeit very simple)

this should be enough to fix it

that’s where I removed the = (child =)

ok

in your root.tfvars, do you have something like : https://github.com/cloudposse/reference-architectures/blob/master/configs/root.tfvars#L48-L56

really odd - removing the = results in the missing sep. adding it back doesnt fit it. reverting the file does.

draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% cat -ev tasks/Makefile.child |grep "define child"
define child =$
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
rm -rf //build-harness
rm -rf repos accounts .terraform *.tfstate* artifacts/*
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
tasks/Makefile.child:61: *** missing separator.  Stop.
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% cat -ev tasks/Makefile.child |grep "define child"
define child$
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
tasks/Makefile.child:61: *** missing separator.  Stop.
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% cat -ev tasks/Makefile.child |grep "define child"
define child =$
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% gco tasks/Makefile.child
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
rm -rf //build-harness

ah, something sublime is doing on save…

why dont I remember that make is tab sensitive? only change I see is tab > space

nope, same thing if I edit with nano. odd.

(and much smaller diff. aka, none)

oh, no, it’s reading my copy dammit

haha yea, whitespace issues suck

ok let me reset everything and try again

a make root/reset would be useful. (though the world still breaks if a sub account already exists, so I also have to bump my email prefix)

keep in mind, each time yuo do that, you create a new AWS subaccount

subaccounts cannot be easily deleted

yep

that’s why I have a 1k account limit

HAHHA

wow, didn’t even know that was possible

“Accounts as Cattle”

hey everyone, is there any way to import the aws_zone53_zone.parent_dns_zone and aws_zone53_record.parent_dns_zone_soa when using reference-architectures and/or root.cloudposse.co repository directly?

yeah, it took a few weeks to get approved, but my justification of “working on automation tooling to manage creation of accounts” didnt result in any followup questions

at some point, I’d like to have a better idea of how to add new accounts to an existing structure - is there a sane way to do that with what exists now?

yes, and no

@Jan has done this and I know others as well

But before doing that, best to get familiar with the overall strategy

Heya

If you have existing accounts, I suggest just using the base images and writing your own dockerfiles from hand.

[testing.cloudposse.co](http://testing.cloudposse.co) is a good place to look for inspiration

Yea I can give you loads of advice on that

Can also tell you where I would like to start from if I had time

Coming to grips with the reference architecture was a great way to learn

But ultimately it would ha e been better to start with empty dockerfile

And add

Rather than adapt

Once you understand the pattern it’s trivial

in this case, the plan is to start from scratch - but new products, teams, environments, will roll in and out all the time - so a straightforward way to provision a base new sub account is important

Yep

Optimize for change!

indeed.

We have done loads of things that are custom

Busy on a few cool ones

We have heavy deadlines for end of April so have not been able to contribute much back yet

But plan is to do so

in general, the overall framework is not far off from what I would build if I were scratch building again (and using tf as the basis, instead of as a component tool like I built last time.)

If you have more specific sorta constraints / ideas / needs I am happy to help

ok, make children is running! (minus equal)

For example on ours we had existing aws accounts, in an existing org where we setup our “root” in a sub aws org account

With totally different network cidr schemes

On premise vpn, hashicorp vault

We are a gitlab / Jenkins shop

In the end it’s just terraform

fun. did you try to import the existing accounts into the scheme, or migrate away from them? (the only thing I hate more than manually fixing tfstate is trying to craft it for existing resources)

With a nice layer of convention on top

They were empty accounts

Just crated already

ah cool

easier there

Well to an extent

Honestly the virvana state would have been starting a full aws org from scratch

My biggest pain so far with taking the geodesic reference architecture and adapting it has been the env names

Try do a search and replace on “data” in terraform

I dare you

hahahaha

haha

so for the record, the point there is you do not renmae

you create additional ones

since we have accounts_enabled, you can then choose the flavors you want

so make children failed out with auth problems - a number of similar errors scattered through the log

panic: SharedConfigAssumeRoleError: failed to load assume role for arn:aws:iam::894200800587:role/OrganizationAccountAccessRole, source profile has no shared credentials
...
The source_profile "brasstack" must specify either static credentials or an assume role configuration
....
Error: Error running plan: 1 error(s) occurred:

* provider.aws: No valid credential sources found for AWS Provider.
	Please see <https://terraform.io/docs/providers/aws/index.html> for more information on
	providing credentials for the AWS Provider

did you enable the bootstrap module?

the children will only be accessible via the bootstrap role

it’s in root.tfvars yes

only thing I removed was cloudtrail

arn:aws:iam::665716774983:role/brasstack-root-bootstrap

in the artifacts/ folder you should see the temp aws config

yeah

and [profile brasstack]

?

i don’t see that in that output

(jumping on a call)

that’s in credentials with keys

it should exist in the config file

but no `[profile brasstack]’

just an empty def

mm, something didnt put it there.

oh no

it’s at the top

first line blindness

the full config/credentials stack looks correct - keys, iam roles, policies, between the two accounts. I can assume the role on the dev account (using a root account user not created by tf. guess I can try using the tf user)

yep, no issues there either.

failed to load assume role for arn:aws:iam::894200800587:role/OrganizationAccountAccessRole, source profile has no shared credentials seems to be from the go sdk, so I’m presuming tf

this seems to be the origin of the failure (and unhandled)

then it keeps going until tf fails

ah https://github.com/cloudposse/reference-architectures/issues/8

looking much happier. I’m gonna PR that

well, next… Failed to save state: truncate terraform.tfstate: read-only file system

that one is new! haven’t seen that

race condition on s3 create maybe?

module.tfstate_backend.aws_s3_bucket.default: Creation complete after 10s (ID: brasstack-dev-terraform-state)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (10s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (20s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (30s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (40s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (50s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Creation complete after 59s (ID: brasstack-dev-terraform-state-lock)
Failed to save state: truncate terraform.tfstate: read-only file system

though I’m speculating - actually that’s probably still saving to local state, not to remote

so tfstate backend initialization is tricky

it first initializes locally

then imports it

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/tfstate-backend/scripts/init.sh

appearance is this was terraform apply -auto-approve -input=false not the import (first tf run in the container)

so, then, why would the container be read-only. hrm

Which channel we’re we talking about test frameworks? https://opensource.com/article/19/2/testing-bash-bats

Hrmmmmmm can’t recall.

Think it recently was in #terraform

maybe my docker has taken a dive. removed the s3/ddb, reran…and docker error.

module.account.module.docker_build.null_resource.docker_build (local-exec): Sending build context to Docker daemon  27.65kB
module.account.module.docker_build.null_resource.docker_build (local-exec): Error response from daemon: mkdir /mnt/sda1/var/lib/docker/tmp/docker-builder348479862: input/output error

yea, looks like it totally crapped out

@loren are you using bats?

i’m not no, pytest/testinfra, exploring go test some with terratest

but i knew you were, and thought it would be a good read for other folks here…

yep, thanks for the heads up

looks like a good read

I’ve used bats before (in something I PRed. forget what. heard about it a few years before that. then later experimented with it for another project. also forget what)….it wasn’t bad. like anything, it has it’s own concepts

restarted docker-machine, and now children ran ok. so $chaos

are you not using docker for mac?

nope, docker for mac panics 9 out of 10 startups (after 5-10m of complete resource sucking - the clock slows down…), the 10th just sucks all resources until I give up and kill it

go mohave.

so, back to ref-arch - after the first pass through - make root make children make finalize with a defined set of children in accounts_enabled - is there a way to reuse the tooling to create a new child? (I may have asked before, but it’s been a really long last few weeks) I’ve tried a few things (adding a new entry to accounts_enabled, subing out the list to just a new acct, etc), but there’s apparently enough state retained to keep that from happening. using the root generated repo, I can make the tooling create a new account, but that doesnt generate a new repo framework…..

pointers to places that might lead me to a path to be able to do that are similarly welcome… (since there’s pretty much no way to know what sub accounts we’d need right now - I’ve got 3 prods and their required test/dev/stage envs on the books so far…)

So you will want to fork the terraform root modules

Then define all the different account types there

yeah saw that

You can truly have any number of accounts

I think best would be to do a quick call to see if I can help unblock you

https://calendly.com/cloudposse

Sometime later this week..

just hoping to find a way to generate them, config them, and build the repo for each new account as it comes along (it seems like the ref-arch as it sits now is more focused on onetime use than long term generation)

Yes totally correct

ok cool - I was hoping I was missing something. I can work with that.

The current use-case solved was a one time cold start

We don’t try to maintain terraform state for the generation

I think it’s doable, but the dns zone delegation or interaccount linking is complicated

are the discovery dns zones used for anything outside of k8s and friends?

It’s used for discovery things like RDS, elastic ache, elastic search etc

gotcha

I’ll have to look a bit further at how you’re using it with rds in that case

Check out our RDS module

will do

one other clarification - in the current setup, for each repo - root straight out of the ref-arch for example - there’s no converge the entire repo's state right? just cd dir init-t t plan t apply yes; cd ../dir repeat ?

(noting that I completely and totally understand why you’re splitting tfstate.)

I built a chef-solo framework around tf for similar reasons

@keen no converge all.

thanks

yes, which (as you mention) is by design. it’s very easy to manage across accounts in terraform.

however, we made the conscious design decision to not use that pattern. instead, enforcing due process on how changes are introduced to environments using git workflow.

yep

i think that the approaches aren’t mutually exclusive

so for example, a certain set of accounts can be delegated out to BUs

then BUs can have their own accounts that they converge

would be nice to be able to have a single converge prod option (vs manual workflow steps in test, for example)

but that’s easy enough to achieve

(eventually)

btw, see this: https://sweetops.slack.com/archives/CB2PXUHLL/p1551115877045700

(and I know there are others like this)

Not really, you have no idea what you are going to get via a plan if there are dependencies between the states. which almost all places will have.

but this is a small precompiled go binary which is nice

and could be an interesting notion to leverage with geodesic

yeah - looks like it might be slightly saner than some of the other generators I’ve worked with. (at quick glance)

howdy

@Erik Osterman (Cloud Posse) was asked to join the channel, unable to get the make docker/build to do anything other than error out: DOCKER not defined in docker/build make: *** [docker/build] Error 1

https://docs.cloudposse.com/reference-architectures/cold-start/ <– trying to follow this

am I doing something wrong here?

hey @Chris

can you run make init first and then make docker/build

did

make init runs fine

can you try to use the latest geodesic https://github.com/cloudposse/geodesic/tree/0.71.3

clone the repo, make init, make?

do you have Docker installed ?

https://github.com/cloudposse/build-harness/blob/master/modules/docker/Makefile#L1

docker desktop wasn’t running

workingnow

good

you also need to update https://github.com/cloudposse/testing.cloudposse.co/blob/master/Makefile with your settings

(and the Dockerfile)

then

make init

make docker/build

make install

[root.cloudvirga.com](http://root.cloudvirga.com)

Was able to build the image, now trying to login and assume-role but getting AccessDenied: MultiFactorAuthentication failed, unable to validate MFA code. Please verify your MFA serial number is valid and associated with this user. Status code 403

I’ll try a few more times

Did you enable and configure MFA on the account for the user you are trying to login with?

for those of you in Los Angeles (#lax), I’ll be presenting on Geodesic at this meetup: https://www.meetup.com/Los-Angeles-Kubernetes-Meetup/events/258976627/