#geodesic (2019-08)

Is it possible override the terraform backend s3 key? I created the following top-level folder structure:

root
-- vpc
-- -- terraform.envrc

I would like the key to be : root/vpc/terraform.tfstate so I set TF_CLI_INIT_BACKEND_CONFIG_KEY=root/vpc/terraform.tfstate.

This does not override the default cause with the key ends up being vpc/terraform.tfstate. I sit possible to override the key?

Ran into another issue where I have created additional root-dns entries that have a hyphen in the name. The label is removing the hyphen from the stage however I assumed the regex_replace_chars should not replace hyphens. Is there something else that could be stripping the hyphen?

Regarding helmfiles, is multi-stage dockerfiles the current approach? How does that relate to /templates/conf/helmfiles/... in reference-architectures?

great you ask

no - we’re using remote helmfiles pinned to releases now

It looks like this:

Okay; reason I was asking is I attempted to add /helmfiles/ templates to the configs/prod.tf in reference-architecture. Failed to render templates as it was looking for a few env vars that are not setup (KOPS_CLUSTER_NAME and STAGE). That got me thinking those templates are just reference files for after the initial architecture is setup. Is that the general idea?

So the Helmfiles will require a ton of settings

We do not have those documented. But if you share specific ones I can help.

Usually we set STAGE in the Dockerfile. If you only have one cluster in a stage, then you can also set KOPS_CLUSTER_NAME in the Dockerfile so it’s available globally.

There are no events this week

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Aug 14, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)

Hi! I’ve been reviewing the terraform-root-modules resource in order to help bring a little order to the chaos AWS I’ve inherited but I’m having trouble understanding where real human users are managed. I see that the aws/usersseems to be set up for this, but it seems incomplete or something, the welcome.txt references a username var that doesn’t exist, and isn’t used as a template source as far as I can see anyways.

I’m having trouble building from geodesic with terraform 0.11 since 0.117.0

is there a better way to use both 0.11 and 0.12?

Yes, but I am afk

@Andriy Knysh (Cloud Posse) can you show Dave

Basically install terraform_0.11@cloudposse

And write “use terraform 0.11”

In your .envrc

where/when do I install 0.11? before geodesic 0.117.0, apk add terraform_0.11@cloudposse worked in the Dockerfile

Yep, add that to your Dockerfile

This way we can support multiple concurrent major/minor versions

Yes, but during docker build:

ERROR: unsatisfiable constraints:
  terraform-0.12.0-r0:
    breaks: world[terraform=0.11.14-r0]
The command '/bin/sh -c apk add terraform_0.11@cloudposse terraform@cloudposse==0.11.14-r0' returned a non-zero

Show me your Dockerfile

You need to remove the second package there

RUN apk add terraform_0.11@cloudposse==0.11.14-r0 is what you want

the long & short of it is that since we upgraded to the alpine:3.10 series, theres been no new 0.11 release, so no package for 0.11 was built under terraform.

however, we explicitly build a terraform_0.11 package and a terraform_0.12 package

like a python2 and python3 package

and from alpine:3.10 , the terraform package will be 0.12.x

behind the scenes, we’re installing a symlink to /usr/local/terraform/x.y/bin/terraform that points to /usr/local/bin/terraform-x.y

that way when we write use terraform 0.11, we can set PATH=/usr/local/terraform/0.11/bin:$PATH and it will automatically find the correct version of terraform

without changing code

and not using alias

@dave.yu this is what we used in Dockerfile to install both TF 0.11 and 0.12 under geodesic 0.117

then, for the modules that use TF 0.12, we use

use envrc
use terraform 0.12
use tfenv

and for the modules that use TF 0.11

use envrc
use terraform 0.11
use tfenv

thanks @Andriy Knysh (Cloud Posse) @Erik Osterman (Cloud Posse) RUN apk add terraform_0.11@cloudposse was the trick (instead of RUN apk add terraform_0.11@cloudposse terraform@cloudposse==0.11.14-r0)

#office-hours hours starting in 15m https://zoom.us/meeting/register/dd2072a53834b30a7c24e00bf0acd2b8

Has any configured kiam beyond the helmfiles defaults for cross account resource access from kops pods? I’m running into a situation where my cross account policies are not allowing me access and I’m thinking maybe kiam is not properly setup to assume roles

have not tried to accomplish cross account kiam

generally we try never to cross account boundaries

There are no events this week

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Aug 21, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)

Hello all, I’m playing with geodesic and it doesn’t seem to be generating the dynamic cluster correctly. I get output that is example.foo.bar.

$ export CLUSTER_NAME=test.myco.com

$ docker run -e CLUSTER_NAME \
-e DOCKER_IMAGE=cloudposse> -e DOCKER_IMAGE=cloudposse/${CLUSTER_NAME} \
-e DOCKER_TAG=dev \
cloudposse/geodesic:latest -c new-projec> -e DOCKER_TAG=dev \
> cloudposse/geodesic:latest -c new-project | tar -xv -C .
Building project for example.foo.bar...
./
example.foo.bar/Dockerfile
example.foo.bar/Makefile
example.foo.bar/conf/
example.foo.bar/conf/.gitignore

I’m looking through various source, it’s pretty challenging to piece it all together (feels over abstracted). So wondering if I’m missing something.

Also new to Terraform so the concepts for how it’s organized is a bit confusing atm

Hey Ryan - sorry - our docs are really out of date.

Here’s an example of how we use it: https://github.com/cloudposse/testing.cloudposse.co

There are no events this week

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Aug 28, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)

Any thoughts on why I receive the following error:

 ✗ . (none) backend ⨠ terraform init
Copying configuration from "git::<https://github.com/cloudposse/terraform-root-modules.git//aws/tfstate-backend?ref=tags/0.35.1>"...

Error: Can't populate non-empty directory

The target directory . is not empty, so it cannot be initialized with the
-from-module=... option.

Latest Geodesic.

add export TF_MODULE_CACHE=.module

@oscar It has been discussed here a while back, check through history, as @Andriy Knysh (Cloud Posse) says ^^

hey @Andriy Knysh (Cloud Posse)

Thanks both, I did actually have a scan through 0.12 and geodesic channel but couldn’t find it.

and in Makefile.tasks, change it to:

Ah you know I can see it now

19th July

https://github.com/cloudposse/geodesic/pull/500

Thanks just had a read of that

Makes sense. I must have missed this this afternoon!

is source <(tfenv) required?

@Andriy Knysh (Cloud Posse) I get this from the new Makefile

 ✗ . (none) backend ⨠ make
Makefile:4: *** missing separator.  Stop.
 ⧉  oscar
 ✗ . (none) backend ⨠ make reset
Makefile:4: *** missing separator.  Stop.
 ⧉  oscar
 ✗ . (none) backend ⨠ make deps
\Makefile:4: *** missing separator.  Stop.

Makefile:4: *** missing separator. Stop. is usually causes by spaces (replace with tabs)

Thanks =] All working now

There are no events this week

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Sep 04, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)

How are folks doing vanity domains using geodesic / ref arch ?

Currently the parent dns name (foo.com) lives in the root account, and by default, users don’t have admin in root

So no one can actually delegate example.foo.com down to example.prod.foo.com

Easily solvable but wondering what folks are doing

by design, the branded domain is provisioned in the root account, since it will contain references to the service discovery domain in any account.

e.g. corp/shared account, prod account, data account, etc.

branded = vanity

Right, but using the ref arch, even admin users in the root account don’t have access to do anything to route53 resources

Yes similar to above I would create a R53 role in root account and assume the role using a second provider block and then for the r53 record resource use the root account t provider.

Why not add route53 perms to the root admin group that already exists?

Currently the users in root admin group, in the root account can only do IAM stuff for their own users

Ah, nevermind me

What was the issue/thing for future reference?

Me being stupid and missing the admin role

#office-hours starting now! join us here https://zoom.us/s/508587304

@Erik Osterman (Cloud Posse) what is to stop a team just having one Geodesic module and then having nested .envrc in /conf to simulate global variables? The only disadvantage I spot is the swapping of AWS-crednentials which could be easily solved.

e.g. /conf/prod/.envrc (all the account specific vars)

BAU: /conf/prod/eu-west-1/terraform/my_project/* /conf/prod/eu-west-2/terraform/my_project/*

Likewise an interesting conversation came up internally yesterday:

In the same way we would promote code in git dev –> master… how would one promote new variables etc along Geodesic /conf? Or new version pegging in the .envrc from-module line? The only solution to make it easy is have all the Geodesic modules in one repo so that it would be easily spotted if you missed updating one environment in the PR.

I feel I have not seen the Cloudposse way

but in geodesic you have all your env vars because you are wrapping geodesic in your env specific container, no?

Not sure I follow.. but normally you would have acme.stage.com but I wonder why not have: acme.com

and then: /conf/ ../dev/.envrc (with equivalent variables of Dockerfile) /prod/.envrc (with equivalent variables of Dockerfile)

It’s a bit weird but something like that came up in a meeting when I was introducing Geodesic to another team and I didn’t really have an answer other than “that isn’t the way”, but it could work without losing any features so…?

More of a probing question than ‘how to do this’. Curious to what other’s think, specifically Erik and Jeremy

The first thing I know would be “broken” is role assumption to AWS. It’d be really easy to assume role in dev, do some stuff, then not be able to apply a different stage because you’re still the the dev account.

I think the separation can be healthy even if it’s not DRY. It’s like the age-old “do we deploy centralized logging per stage or put all stages in one place” kind of argument. There’s pros and cons to both, but if you’re trying to prevent leakage between environments, why not prevent it at the tool level too

The separation between stages also allows you to test changes to your tool chain in safe environments, rather than break the production one.

Yup, makes CI/CD a bit cleaner too

I also like that the toolchain follows the same “TRUST THE ENVIRONMENT” we yell to our ~devs~elves for 12-factor style apps, but maybe not everybody follows 12-factor app style.

on the one hand, I’m jealous of companies that take the monorepo approach to infrastructure. you can easily bring up and tear down your entire environment. You can open PRs that fix problems across all environments in one fell swoop. You can easily setup cross account peering because you control all accounts in a single session. You can do all sorts of automation.

but this also is freggin scary. i think it’s optimizing for the wrong use-case where you start from scratch. i think it’s better to optimize for day to day operations and stability.

also, what i struggle with is where do you draw the line?

i’m sure most of the engineers would agree “share nothing” is the right approach

but despite that, tools, state buckets, repos, dns zones, accounts, webhooks, ci/cd, etc are all shared.

we’ve taken the (relatively speaking) extreme position to truly share nothing.

we don’t share the tools, they are in different containers.

we don’t share the state buckets, they are in different accounts.

we don’t share the repos, each account corresponds to a repo

we don’t share DNS zones. each account has it’s own service discovery domain

we don’t share webhooks, because each account has it’s own repo

we don’t share CI/CD (for infra) because each account has it’s own atlantis, and each atlantis receives it’s own webhooks

etc…

yes, all of that looks like very difficult to setup and maintain from the start, but in the end it’s much easier to manage security and access without jumping through many hoops (and still having holes)

in that share nothing architecture, the only point of access to an account is to add a user to the identity account and allow it to assume a role with permissions to access resources in the other accounts

note that we still share all the code (from terraform modules, helmfiles, and the root-modules catalog) in all account repos, so no repetition there (we load them dynamically using tags with semantic versioning)

we just have different settings (vars, ENvs, etc.) for each account

as you can see for example here https://github.com/cloudposse/testing.cloudposse.co/tree/master/conf/ecs, there is no code in the account repos, just settings (not secrets)

all code (logic) is shared and open (if it’s your company secret, make the repo private)

Yes that example is similar to how were doing it now. We have a few levels of abstraction going on. I can go over this next wednesday for 10-15 minutes d