#geodesic (2021-04)

please don’t start there - that repo is in limbo

https://docs.cloudposse.com is the best place to start for now

multi-account documentation is not available

figured out, how to use reference architecture with atmos, and components. And looks good so far

@Erik Osterman (Cloud Posse) is multi-account support/documentation available in the paid offering?

yes

in fact, one of our most recent additions to the team has run through it in the last week and brought up a new customer environment despite never having touched it before.

Very cool thanks

yes, that is how we use it

on bucket with subfolders basically

Thanks @jose.amengual. Right, that’s what I inferred from running through the tutorial. Just wanted to make sure that is the intended setup.

Does this path look correct to you, for a component state file in an account / env? –

<s3://acme-ue2-tfstate-key-hookworm/env:/ue2-root/>

Seems env: is incorrect?

mmmmm

I’m actually on my way out, so I will look into it later. But it seemed like a missing interpolation for some var.

yes I think you are missing something

we use : “workspace_key_prefix”

ok, i ran the stock tutorial. I found one issue, so perhaps this is another minor one. I will put a PR up once i try that change.

@Igor Rodionov has written a script to help migrate it for one of our customers

Is that something you could share @Erik Osterman (Cloud Posse) ?

@Joe Hosteny Hello. I can share the script on monday - I have to remove project specific settings and edge cases

Thanks @Igor Rodionov. That would be great. I am working on trying to port one of my orgs (built with reference arch) to the new structure, starting with the account module. I am taking notes, and am happy to contribute those back as well if it is helpful to others.

I ran into the above issue as well (with “env:” leaking into the s3 bucket path) – defining workspace_key_prefix, and making sure there was an ‘environment’ and ‘stage’ set in the top level yaml files seemed to do the trick.

fwiw, I also have a bunch of tutorial notes which may be helpful to some – not sure where I should upload them, however, so they do not get in the way.

@marc slayton thanks for your help yesterday. I have some notes too - happy to set up a shared repo if you would like to record those somewhere in the interim, and I can add mine as well

I have an example set of configs that got me to a successful multi-account spin up the first time (beginner’s luck, I expect). I am hoping to finish a procedure which works reproducibly, even if the initial tfstate-backend gets overwritten. This is a common gotcha, I think – an easy mistake to make in the initial config.

On my second multi-account spin up, I made the mistake of using the same bucket and workspace definitions for the tfstate-backend initialization and for the root account spin up. This had the effect of overwriting my initial tfstate and left me needing to reimport the root account.

I think the problem can be corrected by importing the root account definition – but as you know, Atmos has a bug in the terraform-core.variant file, which is blocking me from testing the workaround.

It’s a little embarrassing, but learning how to recompile atmos has taken longer than I thought. Not much of a roadmap there just yet, and there are lots of unfamiliar tools (for me anyway), and fiddly details. As soon as I have a fresh compile of Atmos, I will forward you the procedure that worked, and also the workaround for the gotcha. If I can’t figure that out today, then I will forward what I have and cross my fingers that it works for you.

Hi Marc, I was able to do this with the Dockerfile in the examples directory in the atmos repo. I then just forked the repo for the module changes and pointed it to my branch with the fix for the region issue. Let me see if there are any other changes from that.

The first patch is (obviously) my copy of the dockerfile. Second is main.variant in the cli subdir. You can import with these changes.

The make docker/build will export a binary with the new options, so atmos will have the fix when run in the built container

The tfstate-backend component also has bucket versioning enabled. It saved me once when I manually deleted the wrong workspace’s statefile going through this.

Hey Joe – all the above makes sense to me, but there is still something obvious I seem to be missing. I have a similar repository to the one you rmention, only mine is called ‘gangofnuns’. I use the same name for my forked repository on github:

https://github.com/gangofnuns/atmos/releases/tag/v0.17.2-beta.1

I can build a binary and upload it, given a few extra variables on the command line, however, when I use Geodesic to pull down my version of the docker image, it contains no atmos at all. So clearly, I have the flow incorrect in some way. This is a good exercise though, as I’m sure this is something other people are wondering about. It’s complicated, and there’s no roadmap – I’ll keep digging this afternoon/eve.

Problem appears to be here, when I run the build (NOTE: had to add a variable to get it working with the docker registry.)

DOCKER_IMAGE_NAME=gangofnuns/atmos make docker/build

What do you mean by “use Geodesic to pull down my version of the docker image”?

The flow is a little complicated, but here’s how I understand it right now:

1.) pull down the github repository for cloudposse/atmos.

2.) make the code changes we talked about.

3.) Create a new docker container with the code changes. Upload to a different docker registry.

4.) Change the reference in the Geodesic container build to pull in the new container you just uploaded to the registry.

5.) Run ‘make all’ from within a repo such as the ‘multi account reference’ repo.

With an edit to the Dockerfile, this should pull down the atmos container I built above, and use it to create the Geodesic container.

Even as I write this, it seems way too complicated. I must be missing something obvious about the intended flow.

I’m a bit confused by what your new docker container is, vs. the reference in the “Geodesic container build”. I am only using one container - the one in the example repo. The line RUN variant2 export binary $PWD $CLI_NAME creates atmos, or whatever you call it if you change that name. So I do a make docker/build, sudo make install, and then run [atmos.mydomain.com](http://atmos.mydomain.com), and if you do which atmos when that runs, you will see one in /usr/local/bin/. CLI_NAME in the dockerfile controls the exe name, whereas the installable container name is set by APP_NAME in the Makefile.

That is, I set APP_NAME in the Makefile to atmos.<somedomain>.com, since I will have multiple versions of the geodesic container, one for each set of accounts managed (since we have multiple segregated projects)

So, from my bash command line, I run [atmos.crl.com](http://atmos.crl.com), which starts the container built from that Dockerfile, and then atmos from the command prompt inside that running container

It sounds like you might be using two dockerfiles, and having one based on another? If so, is it possible you didn’t copy the binary to the latter stage?

I’ve got to run for the evening, but if you are around tomorrow, send me a PM and perhaps we can do a quick zoom?

RE: Geodesic container build – I’ve been following the instructions for building a multi-account stack. The instructions say: “Go get this repository, with these specific components, and stack definitions”. That repsitory comes with a Makefile and a Dockerfile for building Geodesic. That’s the container I’m using, and where I’ve been assuming I’d need to install the newly recompiled atmos.

Yes, I will have worked through my confusion by then. (So embarrassed.) And in any case, I will send you what I have in terms of procedures for multi-account spin up. RE: Zoom - have you tried Gather Town instead? If not, no problem – I can do Zoom too. The password for the Gather space in the link is “so-foo-bar”. Works in any Chrome browser. Cheers –

Thanks, that call cleared up what you are doing

Appreciate the extra time, Joe – many thanks.

Sorry I didn’t get to show you Gather Town, but maybe some other day. It is a fun step beyond Zoom, imho. Will send some preliminary spin up notes this eve. I am in California, btw, so it might be late. Cheers –

@Joe Hosteny - here’s my original template for a multi-account run. Notice that initially, I set the tfstate-backend to local, not s3. I later copied it to S3 in the master account. after accounts were spun up. I have more procedural notes, but it’s a bit late. I’ll have to finish organizing those for tomorrow. Enjoy!

Hey Joe – here’s what I have thus far from a notes perspective. Hope to add more this evening after some other weekly chores. Still very much a work in progress. Cheers –

cc: @Andriy Knysh (Cloud Posse)

^^^ quick caveat – these are embarrassingly incomplete. Slowly trying to improve. Very sorry for delays.

Quick update: With a change to terraform-core.variant and a rebuild of atmos, I was able to import the master account, and thus get my second multi-account reference architecture up and running, despite having botched the tfstate-backend setup. So now I have two procedures that seem to work. This is using terraform 0.14.10.

Important note: I ran into this bug after importing my master account. The workaround was to add:

  # There is no AWS Organizations API for reading role_name
  lifecycle {
    ignore_changes = ["iam_user_access_to_billing"]
  }

to

components/terraform/accounts/main.tf 

seems to have worked without issue. I’ve voted up the bug fix. Hopefully that one gets looked at soon.

Still recording my jolly antics here.

Hi @marc slayton - made a lot of progress over the weekend, including finishing deploying core iam / dns / cloudtrail across all accounts, and importing a number of pre-existing resources created via the old ref arch. Happy to sync at some point if it is helpful.

I would most definitely be interested. I’m looking for sort of a ‘hit list’ of things I would need to address while back porting existing modules.

With such a list, or a condensed series of liner notes, I could probably make decent progress on backporting further modules.

@Joe Hosteny all-new-components I believe would be the spot. That is where all the various upstream branches have been merged in the short term.

Yup, thanks @Matt Gowie. I’ve found a number of minor issues that I’ve been able to work around. I haven’t filed tickets, since I am not sure if I am doing something wrong, or where to best put some of them (e.g., one could be a variant2 issue, or the way it is being used by atmos). Additionally, some look to be due to changes between that branch and master. If you would like, I can send you a list when I am done (or perhaps do a quick zoom) to see where to best route some of those issues.

I’ve been able to import my old ref arch org and sub accounts so far, create dns and identity accounts via atmos, create the account and account-map state, and am working through iam-primary-roles.

Good stuff @Joe Hosteny! That’s awesome. Yeah I’d definitely love to chat about what you’ve found and how you’re liking it so far. As I’m sure @Erik Osterman (Cloud Posse) would as well.

As to where to file issues… it does depend. Collecting them and then chatting through it makes sense — Let’s do that as I’m sure I can direct you where to go

I’ll send you a link to grab some time on my calendar whenever is convenient for you.

I can already tell this will make my life about 100x easier…once the migration is done

Yeah, I think this will be a very impressive and sustainable way to build terraform projects once it gains a bit more community steam. I’ve definitely benefited from it a ton already.