#geodesic (2019-10)

@Jeremy G (Cloud Posse) A follow up to this, I dumped the previously created sub accounts and cleaned up the root. With the latest commits I started over and the result was the same. The s3 bucket was created, in my case lunarops-io-security-terraform-state. Is there a fix or workaround for this

@Erik Osterman (Cloud Posse) As I look at ref-arch now, I see that while the README lists security and identity as accounts that can be enabled, there is no /config/security.tfvars or /config/identity.tfvars file. What do you want to do about that?

@dalekurt Despite them being listed in the README, we do not have working support for the security or identity accounts. I suggest you stick to the list of accounts enabled by default: https://github.com/cloudposse/reference-architectures/blob/62f20f5bf365944f54e1bbd20a85993ffbae24f6/configs/root.tfvars#L78-L85

Jermey is correct we didn’t finish implementing those 2 accounts

The idea is to offload whAt we have in root to those 2

Yes, that was explained to me. However it is not the only account that is affected. I just picked that from the list.

@dalekurt Also, it looks like you set namespace = "lunarops-io". It should be a single word, and we try to keep it to 4-6 letters. Maybe just lunar or rops?

The other sub accounts are also affected by this error, as previously shared in the gist. Dev, staging, etc…

Ah ok. Could that be an issue?

Is the audit bucket named lunarops-io-audit-terraform-state or lunarops-audit-terraform-state?

It is

the first

I typed better at the keyboard. The audit bucket is named lunarops-io-audit-terraform-state

Would it result in a breaking change if I changed the namespace?

Yes, changing the namespace breaks everything.

We use naming conventions all over the place.

Ok, does it require manual clean up or the tf state would effect that.

I’m early in this deployment as it is, so i can handle a breaking change. I’m just hoping I don’t have to do the clean up.

And the max the namespace should be is 6-char?

The expected name of the terraform state bucket will change when you change the namespace, so terraform will not be able to automatically clean up. I think the best you can do is terraform destroy in the root account to do some of the clean up.

Do that before changing the namespace.

The length of the namespace is not strictly limited, but it is part of every identifier and the total length of some identifiers are limited, so it’s best to keep it short.

okay, can the clean up (terraform destroy) be executed from with the container that did the deployment? Or within the project directory

Actually, now that I think of it, most of the terraform state of the bootstrap code is stored in the root directory of the checked-out repo. How many *.tfstate files do you have there?

Unfortunately, the whole setup is a bit complicated. We run some stuff on your host to get things started, and that includes creating stuff with terraform with the state in local *.tfstate files, then building a Docker container, then we run more terraform inside the Docker container. You probably didn’t get that far, though, because the Docker container would try to use the S3 bucket for the Terraform state.

Worst comes to worse, you can use https://github.com/rebuy-de/aws-nuke to delete everything.

I had to use that myself when I created a set of accounts with the wrong namespace.

@Jeremy G (Cloud Posse) I have 10 *.tfstate files.

I’ve heard of aws-nuke, in fact from @Erik Osterman (Cloud Posse) on Office hours.

@Jeremy G (Cloud Posse) if I choose the nuclear option, should I perform that on each account or just the root?

I’m thinking your errors happened very early on in make children, is that right?

That is my assumption as well, It is shown as an error during the make finalize but if it has a dependency of the s3 buckets which are created in the make children then you are correct.

Wait, you ran make children and it exited successfully?

Yeah

That is going to be difficult to recover from. You cannot just delete the AWS accounts, and we do not have the mechanism (as far as I know) to tell the scripts that they have already been created. @Erik Osterman (Cloud Posse) what do you suggest?

To be accurate, there was no success message just goodbye

chamber_secret_access_key =
chamber_user_arn =
chamber_user_name =
chamber_user_unique_id =
secret_access_key = <sensitive>
user_arn =
user_name =
user_unique_id =
Skipping cloudtrail...
Goodbye

Whereas with the make root I got

* provider.aws: version = "~> 2.31"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
Goodbye

To be safe, I would make a backup copy of your entire repo (which will include a lot of generated artifacts by now, including the 10 *.tfstate files). Some of the artifacts you are going to need, particularly the list of account numbers, but could be erased if you are not careful.

Sounds like some DevOps surgery

I’ve made the backup

Yes, sorry, you are in a bad state. I think you are going to need to have @Erik Osterman (Cloud Posse) help you during office hours. You hit this failure where it is too late to start over but we suspect the problem is the dash in the namespace, and changing the namespace does break an awful lot of stuff.

No worries. I’m wondering if I should just do the nuke on the root and sub accounts. And re-run the make root and make children

My hope is that you can get by with changing the config files and then resuming from make children, but I’m not sure.

The problem is that if you run make root from scratch, it will create another set of AWS accounts and you really want to avoid that.

gotcha

Thank you for the time, I really appreciate it..

If your are feeling daring, I can suggest you edit /configs/root.tfvars to be what you want it to be (maybe post it or a redacted version of it), and then run make root/init-resume.

What I’m counting on here is that the names of the accounts do not change, and init-resume is specifically for picking up after errors, and it avoids cleaning up old state.

What I’m not sure of is whether or not it will really create new AWS accounts or not. It should not, but this is not well tested code.

Thanks for the suggestion

I will tell how I feel tomorrow

OK, I will leave you with this tip. Deleting AWS accounts is a long, painful process, because AWS does not want to be on the hook for deleting stuff that it cannot get back. One key thing is that the email address associated with the account will be _forever_ associated with that account. You will not be able to create a new account with that email address and you will not be able to change the email address later. So before you delete an account, change the email address to something you can consider a throwaway. Gmail and some other providers allow you to add “+anything” to your username to create a unique email that still routes to you, so I suggest you do that, using “+identity-del-201910” or something like that.

I wish I had know I could that a couple days ago before nuking the accounts. Thanks for the tip

Added this https://github.com/cloudposse/reference-architectures/issues/46

I will deploy the changes to the namespace later today, as well update the email addresses assigned to the existing sub-accounts. I may have to crosswalk this out on paper first so that I don’t miss a step for clean up.

If the accounts are still in the waiting period for deletion, you can probably still change the email address. And if you cannot, you can stop the deletion process, change the email address, and start it again, so long as they are not already fully deleted.

I will check the emails for the cancellation

I’ve submitted a case to re-open the suspended accounts, then I will update the email addressed for those including the accounts I recently created. Then have them terminated.

@Jeremy G (Cloud Posse) Once that is done, I will go ahead and re-deploy using the new namespace lunar, this I could get away with lunarops my OCD is going to bug me

Phew I just went through all the accounts replying to the AWS case that was opened. So they should be reinstated soon enough.

@Jeremy G (Cloud Posse) I took your advise and updated the email addresses for the previously suspended AWS accounts. I will do the same from the second set of accounts I had created, then request a termination.

Once that is done I will re-deploy with the new namespace

@dalekurt before deleting the current set of accounts, I suggest you attempt some of the surgical options. If they work, that saves you deleting the accounts. If they mess up horribly, you can still resolve it by nuking and deleting the accounts.

sounds good.

@Jeremy G (Cloud Posse) Here is the plan, update the namespace to lunarops and re-run the make root.

No, don’t rerun make root!

ok

Update the namespace in root.tfvars

Then run make root/init-resume

Ah ah.. okay let me give that a go.

I got a brief output of it.

terraform apply \
		-var-file=artifacts/aws.tfvars \
		-var-file=configs/root.tfvars \
		-auto-approve \
		-state=root.tfstate \
		accounts/root
...
module.account.module.render.local_file.data[1]: Creation complete after 0s (ID: 6c6649ca2a9e8400634a22bb826e5ae0f6d01543)
module.account.module.render.local_file.data[0]: Creation complete after 0s (ID: ca981bf46f94b8366ea80bcf0f00466c62199b2e)

Apply complete! Resources: 2 added, 0 changed, 2 destroyed.

Outputs:

docker_image = cloudposse/root.lunarops.io:latest
terraform output -state=root.tfstate docker_image > artifacts/root-docker-image

make children

...
===============================================================================================
ARN for lunarops-dev-admin not found in /artifacts/.aws/config

* Please report this error here:
          <https://github.com/cloudposse/reference-architectures/issues/new>



Goodbye
make: *** [dev/provision] Error 1

Not entirely relevant, but did you remove security and identity from the accounts_enabled list? If not, please do.

Verify that artifacts/accounts.tfvars has the correct account numbers

You have a backup of everything, right? In which case, delete everything under repos/ and try make root/init-resume again

@Jeremy G (Cloud Posse) Thanks for that. So, I did just that. I removed security and identity from the accounts_enabled in configs/root.tfvars.

As well, I removed them from the artifacts/accounts.tfvars and confirmed that account IDs are correct.

Then removed everything from the repos/ directory path and executed make root/init-resume which create the repos/root.lunarops.io dir

@Jeremy G (Cloud Posse) So i bit the bullet on this and re-deployed make root the accounts got created successfully. make children and make finalize ran successfully as well. This was with the lunarops namespace. So you were right [lunarops.io](http://lunarops.io) namespace created issues.

I’m doing clean up now by deleting the previous accounts created.

@Erik Osterman (Cloud Posse) I am successful! I will update you further

Omg! That’s awesome. You are one persistent guy. Shows it pays off.

Oh yeah. So I have a question however, though the identity and security accounts are not supported.

Would there be any harm in still deploying them.

Yes, we haven’t finished supporting those

I think it should work, but they won’t do much

Here is the deal, I do want to have my users login on the identity account and write my policies in terraform to create the IAM Roles there .

So that I can isolate Root users to the root account and employees to the identity account.

I had it humming before with a manual deployment using the AWS Profile Extender extension

If that is what is it called.

AWS Extend Switch Roles extension for Chrome and Firefox

That sounds interesting

@Erik Osterman (Cloud Posse) Where did this come from exactly? The time zone is half wrong. It should say PDT, GMT-7

for clarification - how to use geodesic with a monorepo, right?

Yes.

Before my monorepo was:

geodesic:

• dev
• staging
• prod
• bastion

but now it is:

geodesic:

• geodesic.acme

With hundreds of repos we wanted a simple one-liner to import the build-harness that was future proofed so we didn’t have to copy pasta a lot of boiler plate and try to keep it up to date.

Also note that git.io no longer supports vanity short links

make geodesic_apkindex.md5 is a safer bet, but either way you have to be in the geodesic root directory

time to go back to terragrunt?

Not sure…still not really a fan of the terragrunt wrapper style

Pick yer poisons:

• terraform lacks interpolation in backend { blocks.
• terraform (in 0.12) has materially altered what’s allowed in the working directory where tf init is issued.

terragrunt takes care of those issues for me.

and I really appreciate the simple two-layer cakes it bakes for me….

• layer 0 is the base: whatever I put in terraform { source=..., typically

     source = "git::[email protected]:myOrg/myTerraformRepo//aws-blueprints/${path_relative_to_include()}?ref=master"
  

• layer 1 is the environment config: all files in the directory where I run terragrunt init get “rsync’ed” on top of layer 0. Symlinks in layer 1 are resolved before rsync. Filenames in layer 1 overwrite identically named files that exist in layer 0.

I’d offer that the ability to remove (w/ a zero-length file) or replace an entire .tf file in your base module is this arrangement’s super power.

It’s way better (imho) than dealing with conditional logic for stuff like: count = var.is_this_module_enabled ? 1 : 0

How does TF cloud help with these types of things? Not using geodesic there?

Yea, I think using terrafile or terragrunt

terragrunt orchestrates the -from-module process

there’s also git subtrees, and git submodules (not a fan)

there’s bazel which clone dependencies

I like the “just clone this external repo at this version” nothing too smart

From there you can still bare terraform, which you wouldn’t be able to with terragrunt

here’s something you can run with:

it still uses the terraform init -from-module=... but that could be replaced with “just clone this external repo at this version” and run with it

I know you are iffy about make

but this looks pretty nice to me…

https://share.vidyard.com/watch/TzZS8W5m8NEYV3qBAL7x81?

Nice video

I’m a big fan of make, but feels like a traditional wrapper script, may as well be a bash script/terragrunt. Realise in this case it is being used just for init

I wonder if a straight up git clone seems a bit more understandable across a big team

that misses the point. You need a root module n times

That needs to be stored as a configuration

A git clone doesn’t have that

Take for example EKS in 2 regions

The current way geodesic is setup this would all be in the same region as far as state goes though, I’m trying to think of the least intrusive way of swapping out cli init from module, and a git clone in the use_terraform function came to mind

IIIRC geodesic doesn’t currently clone a root module more than once https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/cloudtrail/.envrc#L2

You could, but would the extra logic be worth it? Keep it simple

Too much magic, and the balance tips to tools like terragrunt

We clone each individual root module pinned to a version every time we need it

Otherwise is forced upgrades to other projects

A single clone for all projects is not a stable approach

A single clone for each project pinned to a version is the way to achieve stability

Right, agree that is needed

Given the current approach, could a git clone in https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/direnv/rc.d/terraform (or akin) get the job of init from-module, this gets run in the context of a single root module pinned to a release version. Not saying git clone terraform-root-modules/$VERSION for all your root modules at that version.

isn’t that solved with source_up?

https://manpages.debian.org/stretch/direnv/direnv-stdlib.1.en.html

What do you mean “direnv doesn’t support cd”? Of course it does.

That makes sense

Still use direnv and ‘use terraform’ for setting backend etc

But not init from module

Instead have another top level almost stub module…would need to pass vars etc down it, bleurgh

Or something like https://github.com/coretech/terrafile/blob/master/README.md

Which is already in geodesic iirc

we’re mostly debating in good fun, but I am in the camp that the name is gonna matter someday soon

Giving something a name makes it matter. It’s important. You want it to stay. You name your dog Sir Licksalot because of that one hilarious moment when he trampled your aunt Marge and licked her face. You DON’T name your servers because they are cattle, and you will kill them at a moment’s notice

Giving it a descriptive name is pigeon holing it to being one thing and that one thing will morph

You don’t know what that will be today

A descriptive name would be terraform-kops-hemfile-kubectl-etc and get out of date

Build tools only conveys one usage

They won’t use it for triage?

We call geodesic a “cloud automation shell”

Maybe you can have some fun with names of clouds, weather systems, Greek gods of weather (if any?), etc

https://en.wikipedia.org/wiki/Loki

loki - god of mischief (which is the fun part). Does almost everything (which is useful). The name is short and easy to remember and pronounce, and sounds like lock (which is good for security)

the name of my team is Team Vulcan

Roman god of building shit, as we like to say

ok, so you already have one god

I agree with @Erik Osterman (Cloud Posse) that a descriptive name is a terrible idea. (So does the Buddha.) When you label something, you deny everything else that it is. That is bad enough with a fixed thing, but really, really bad with an evolving thing.

I’ve seen this error when the private key is not imported into keybase’s local store. For reference, if you want to import an existing key from the GPG keyring, use keybase pgp select --import

I think you might find your workflow is made easier by adjusting the last profile script to cd to your dev dir:

# echo "cd /localhost/Workspace/my-company/identity.my-company.com/conf" > rootfs/etc/profile.d/zzz-cd-to-workspace.sh
# chmod +x rootfs/etc/profile.d/zzz-cd-to-workspace.sh

And then rebuild your geodesic image with that new file.

That way when you execute the geodesic wrapper, you hop directly to a shell with CWD set to where you want to be.

I think you can also get away with adjusting the Dockerfile

-WORKDIR /conf
+WORKDIR /localhost/Workspace/my-company/identity.my-company.com/conf

Geodesic is highly run-time customizable. I recommend you use the customization hooks rather than editing the Dockerfile.

Also note that by default Geodesic creates a symbolic link to /localhost that matches the host path to that directory, so you can use host pathnames inside Geodesic. Tlide doesn’t work because you are a different user, but fully qualified path names work. More info here:

https://github.com/cloudposse/geodesic/pull/422

I tried and failed to remove Python 2 a while ago.

We rely on crudini quite a bit in managing AWS credentials. Also, as explained in requirements.txt, we are stuck with PyYAML 3.13 until awsebcli upgrades. So we’re keeping Python 2 for a while.

https://github.com/cloudposse/geodesic/blob/aad55273d71b838923620d7779432fd0cff4eb1f/requirements.txt#L7-L12

Yes, I saw those comments – in my case, with crudini out of the picture (my images currently rely on AWS_DEFAULT_PROFILE being set in the Dockerfile) it worked. I also think the boto==2.49.0 requirement was pulling in Python 2.

What are your interactive test results of the above? How are you testing/invoking assume-role?

Once the container is started, typing assume-role returns

* Assumed role arn:aws:iam::1123456789:role/admin-role

which is otherwise correct…. It also sets the env var for TF about the assumed role that we want

but when running terraform apply, terraform itself says it can’t get the credentials.

The authentication flow is saml2aws script gets the temporary credentials and session tokens and returns them to ENV

when running terraform apply, I get authorization response codes that make it fail, as it can’t assume the role.

in my ~~~/.aws/config I have a profile which has the admin role in it, along with the credential source saml (used by saml2aws in ~~~aws/credentials)

if that answers your question

after you run assume-role successfully – are the expected environment vars set?

• https://github.com/Versent/saml2aws#environment-vars

I’m suspecting your functions don’t exec a new $SHELL

you may be 100% correct. Wow let me test that

it seemed like somethign along those lines, as from my native shell it worked , but only within the shell of the terraform run it failed

assume-role $AWS_DEFAULT_PROFILE /bin/bash ?

following up in a few with result

ran env

We’re getting closer

I added an env var that should have picked up the contents of /localhost/.saml2aws/config.json by default. Aside from that, i don’t mind manually doing this inside of the container every time. It just doesn’t seem to work but is close.

definitely seems related to the shell.

now assume-role is not found, leading me to believe the profile.d needs to be sourced

last update for a while. I exited the container shell entirely. Didn’t rebuild the container, notice on container startup it is able to assume to role, but when we switch shells to the workspace, that is when it fails

adding @Tega McKinney as he may have run into this problem before. Good exercise to troubleshoot this @tamsky I’ll look more into this tomorrow, but not going to let this stop me for right now.

Thank you!

Hrmm… maybe a bug? I know we had this working since we support Okta

we didn’t implement GSuite b/c there wasn’t a nifty aws-vault, aws-okta, esque cli tool (in go!)

I haven’t figured out the root cause on this yet. There is some conflict when running saml2aws (v2.18.0) in the container vs on my laptop.

Same command works on my laptop but fails in the container: saml2aws exec --exec-profile <profile> aws sts get-caller-identity

Fails with error: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid. exit status 255

Interestingly enough, I can use aws --profile <profile> sts get-caller-identity in the container and it works.

does saml2aws open up a webserver?

a lot of these tools do, so that the callback url can be <http://localhost:...>

And since the container is not listening on that port, it fails

@Erik Osterman (Cloud Posse) I do not believe this particular cli uses a local server however I believe I have found that the issue within the Geodesic shell is that this tool exports AWS_SECURITY_TOKEN= and for some reason in the shell, it fails. If I unset the token or make it the same value as AWS_SESSION_TOKEN then the commands work.

Are you aware of anything in the Geodesic shell that would cause this? I haven’t been able to replicate that behavior in any other Docker container.

There should always be a pair of session tokens with security tokens when doing STS

Probably easier to scrernshare and see if we can get to the bottom of it

Does $HOME/.saml2aws need to be a symlink to /localhost/.saml2aws?

The maintainers are resolving the AWS_SECURITY_TOKEN issue and adding it to the exports; that was a regression.

I did mount .saml2aws.

I believe this should allow us to continue however I was just wondering why that missing ENV would cause an issue in Geodesic. I created a simplified Docker image with the rootfs scripts but all the Geodesic packages and it worked without the AWS_SECURITY_TOKEN` ENV.

I upgraded to the latest Geodesic shell and it works (without AWS_SECURITY_TOKEN set).

The only problem I am getting now when I run assume-role profileName /bin/bash, I get the following in the terminal:

bash: _kube_ps1_update_cache: command not found
bash: _direnv_hook: command not found
bash: prompter: command not found

Any thoughts?

Appears that none of the tfenv functions are running

The issue seems to happen when I run assume-role profileName /bin/bash. If I run assume-role profileName /bin/bash -l then it works as expected.

all these refs can go, imho:

• https://github.com/search?q=org%3Acloudposse+MAKE_INCLUDES&type=Code

But you missed this: https://github.com/cloudposse/geodesic/blob/65b76a9958a558a1241d0a766ceabd4f79832a75/rootfs/usr/local/bin/make

/me shakes fist at github search

https://github.com/cloudposse/geodesic/search?q=make.conf&unscoped_q=make.conf

that was my search

I was incredibly frustrated that gnu-make didn’t support a global makefile

config

it does. env var MAKEFILES

https://www.gnu.org/software/make/manual/html_node/MAKEFILES-Variable.html

the problem is it doesn’t support wildcards

hrm

well, I guess it would work this way:

kind’a scratching my head though. i feel like i must have tried that and abandoned it for some reason. But hey, if that works

I didn’t include rootfs/usr/local/bin/make in my setup – so this hasn’t come up yet, or bitten me at all, I’m just pointing out it was “odd”.

My setup also doesn’t need the build-harness so any dependency that has on this custom make setup – hasn’t bit me yet either.

yea, zero deps on build-harness inside geodesic

outside, it’s only to make docker/build

are you attempting to create a smaller base image?

kind of. I’m trying to strip it down for simplicity’s sake – hopefully giving it more curbside appeal when introducing it to other team members.

cool

“minimalist”

interested to see what you come up with

It’s working for me. I fell instantly in love with s3 mount yesterday when I discovered it.

nice!

also works with mount -a

good to know

we hook into the fstab so it works the linux-way

Where I’m at now has a dozen or so “blueprint repos” that get deployed by a Jenkins CI server. So the version of each blueprint deployed to an environment sits in a json file in S3. By being able to easily grab and parse the file, I’m able to setup terragrunt.hcl to spec out the correct

terraform { source = "private-git-repo//modules?ref=vXX.YY.ZZ" }

The sanity this provokes is very calming. Thank you & the rest of the CP gang.

are you pushing the terragrunt.hcl to S3 as an artifact?

I build all of them locally, on the fly, via a rootfs/conf/Makefile ( I’m slightly sheepish to admit I chose that path )

Am also doing something similar to this, but have avoided terragrunt so far :D

It’s really cool to see

@rms1000watt is looking to do something with this. @oscar has his way of doing it. @rohit.verma switched it to Ubuntu.

Wish there was a way to reconcile all of it.

one of the hardest things I’ve still got is, having split your TF into multiple states (sometimes 20+ for a single environment), how do you sanely manages the dependencies for these? And don’t say terragrunt apply-all with deps, cause it don’t work

Does terraform enterprise help with this at all?

Is make up to the task of determining the order? You’d have to specify the dependencies manually, but you’d need to know those in advance when deploying from scratch… So it’s probably something you already know.

But that implies all in the same run, as terragrunt apply-all works

You could mix plan/apply logic in make but ….. will get nasty

I don’t beleive TFE helps with the chain of dependencies.

(there’s also astro by Uber)

Yeah, but looks unmaintained and have already seen your use case/issue

Could be worthwhile investing in though

Interestingly, the astro examples give you a clue to how they have laid out thhmeir tf runs, and that’s important

But that implies all in the same run, as terragrunt apply-all works

I’m not seeing it quite the same way. If I’m running apply in any particular state, make should be verifying that all dependent state dirs have a clean plan output… And if not, halt.

Interesting

https://github.com/uber/astro/issues/8 this

astro looks like it could work… but it’s same essential problem, solved the same way:

The operator of the environment is required to explicitly list all dependencies:

• in astro: modules.deps:
• in make: recipe: prerequisite

@joshmyers have you tried the terragrunt dependency features in the latest versions? they’ve done a lot of work around that use case recently, so it might be better than before if you haven’t looked at it in a while

Not in a long time, thanks. Will have another look!

Looks like terragrunt has added some cool new features, not quite that fix this problem though, which is an inherit terraform/state machine thing. Some interesting approaches to help with dependencies though