#geodesic (2019-02)

geodesic https://github.com/cloudposse/geodesic

Discussions related to https://github.com/cloudposse/geodesic

Archive: https://archive.sweetops.com/geodesic/

2019-02-03

me1249 avatar

@Erik Osterman (Cloud Posse) - how often do you update your APK repository?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

on every merge

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we cut a release on every merge to master

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in other words, if a PR is merged, it’s been deployed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

when installing packages, recommend doing somethin glike

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

apk add --update terraform@cloudposse=0.11.1-r0

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for strict pinning, plus updating the apk cache

me1249 avatar

odd - when I’m building I’m still getting terraform 0.11.10 and i get this if I try to update

ERROR: unsatisfiable constraints: terraform-0.11.7-r0: breaks: world[terraform=0.11.11]

me1249 avatar

Running this got it to work apk add terraform –update-cache –repository https://apk.cloudposse.com/3.8/vendor –allow-untrusted

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not good.

me1249 avatar

ah i see why i still get 0.11.10 - layer caching

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
apk add terraform@cloudposse==0.11.11-r0
fetch <https://alpine.global.ssl.fastly.net/alpine/v3.8/main/x86_64/APKINDEX.tar.gz>
fetch <https://alpine.global.ssl.fastly.net/alpine/v3.8/community/x86_64/APKINDEX.tar.gz>
fetch <https://apk.cloudposse.com/3.8/vendor/x86_64/APKINDEX.tar.gz>
fetch <https://alpine.global.ssl.fastly.net/alpine/edge/testing/x86_64/APKINDEX.tar.gz>
fetch <https://alpine.global.ssl.fastly.net/alpine/edge/community/x86_64/APKINDEX.tar.gz>
(1/1) Upgrading terraform@cloudposse (0.11.10-r0 -> 0.11.11-r0)
Executing busybox-1.28.4-r2.trigger
OK: 702 MiB in 119 packages
-> Run 'assume-role' to login to AWS
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

here’s what I get in geodesic

me1249 avatar

ah you need the -r0

me1249 avatar

All good now, thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, I haven’t dug deeper into why the release is required

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we are going to do some more work on the alpine repo over the coming month (time permitting)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

among other things, improve the build process so that it only builds changed paths

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, considering rendering the APKBUILD template and committing so it’s more obvious what’s going on. currently, the APKBUILD template wraps make

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also, --allow-untrusted should not be required (we don’t use it); the packages are signed

2019-02-05

me1249 avatar

Are you guys doing any authentication to private git repositories for terraform modules in geodesic? If so, how are you handling authentication?

joshmyers avatar
joshmyers

@me1249 yes, SSH keys

me1249 avatar

how are you making the SSH keys available to the docker container?

joshmyers avatar
joshmyers

When running geodesic locally your agent gets mounted in, when running via Atlantis, it is writing out an SSH key from chamber

me1249 avatar

ah

joshmyers avatar
joshmyers

Which docker container exactly?

me1249 avatar

I totally missed ATLANTIS_SSH_PRIVATE_KEY

me1249 avatar

Thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@rohit.verma I forget, are you using GitLab?

rohit.verma avatar
rohit.verma

yes we are using gitlab

me1249 avatar

Still having issues after settings ATLANTIS_SSH_PRIVATE_KEY - it seems like the atlantis user doesn’t have access to the SSH auth socket - anyone come across this issue?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmmmm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
01:17:27 AM

confesses, i only tested that the key got added and not I could use it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/geodesic

Geodesic is the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! https://slack.cloudposse.com/ - clou…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, we should chmod the sock as well.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

on line 89, we gosu to the ATLANTIS_USER; prior to that we’re root

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@me1249 btw, have you seen the tmate-session util we added to geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if you install tmate in your container, you can call tmate-session and it will output a URL you can use for debugging

me1249 avatar

Yep - no luck with chmoding the socket either

me1249 avatar

even tried chowning it

me1249 avatar

Have to execute the ssh-agent under the atlantis user using gosu

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmmmm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sec

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what did you chown it?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what was the mode?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

past experience tells me that it’s not required to run the agent under atlantis.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ohh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wait

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, the pid probably needs to be same uid

me1249 avatar

So changing to source <(gosu ${ATLANTIS_USER} ssh-agent -s)

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

no - that’s not true, since on Linux we bind mount in the agent sock

me1249 avatar

fixes the issue

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s a nice fix

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

however, <(...) won’t work without a TTY

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you deployed that under ECS and it works?

me1249 avatar

Just deploying it now

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, lmk

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ohhhh we had source <(ssh-agent -s)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmmm odd. i had trouble with that under certain circumstances.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if that doesn’t work, try eval $(gosu ${ATLANTIS_USER} ssh-agent -s)

me1249 avatar

It worked

cool-doge1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

woohoo!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

can you open a PR for that?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ll release that right away

me1249 avatar

Sure can

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thanks @me1249!!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

merged and released as 0.69.1

me1249 avatar

No problem

me1249 avatar

Got another fix coming for you soon regarding timezones

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
[kopsctl] add commands to facilitate management of cluster by osterman · Pull Request #378 · cloudposse/geodesic

what Add commands to easily rotate a kops cluster&#39;s ssh keys Add command to easily connect to a kops cluster Add command to see a kops plan why This are routine operations that are complicat…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

here’s an example of using the #variant tool for exposing a clean cli UI

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

much more powerful than make

me1249 avatar

will have to check it out

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
04:15:10 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
04:15:28 AM
me1249 avatar

nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so even if not using kops, this pattern can be extended to other things

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

like myorgctl

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that ties together all the tools “my org” uses

me1249 avatar

In case anyone comes searching in here that’s trying to use atlantis with tfenv and terragrunt - do not set TF_CLI_DEFAULT_NO_COLOR=true

me1249 avatar

It breaks terraform init

me1249 avatar

terragrunt passes -no-color by default so it ends up passing it twice - terraform doesn’t like that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I found that terragrunt inconsistently passes -no-color

loren avatar

i went ahead and opened an issue for this: https://github.com/gruntwork-io/terragrunt/issues/648

Using extra_args on init to pass -no-color results in duplicate -no-color (auto-init) · Issue #648 · gruntwork-io/terragrunt

Terragrunt&#39;s auto-init is adding -no-color to its terraform command, so when extra_args is used to pass -no-color to terraform init, we end up with two -no-color args on the command. Terraform …

2
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, with terragrunt that causes problems

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I had all kinds of problems with terragrunt and -no-color irrespective of atlantis

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:16:41 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(from our soon to be released slack archives!!)

2019-02-06

loren avatar

Hmm, I don’t think terragrunt itself has any internal logic for injecting -no-color… What does your terraform.tfvars file look like?

loren avatar

I’ll look at the terragrunt code when I get to a computer to confirm…

loren avatar
gruntwork-io/terragrunt

Terragrunt is a thin wrapper for Terraform that provides extra tools for working with multiple Terraform modules. - gruntwork-io/terragrunt

loren avatar

huh. usually slack expands the code with those urls

loren avatar

that was done relatively recently… pretty sure related to how terragrunt checks the auto-init output for a particular error message that is not an error (to auto-init) so it knows to ignore the non-zero return code

loren avatar
Speed up init usage · gruntwork-io/terragrunt@25c3b8d

When we use init to download modules, set -get=false, -get-plugins=false, and -backend=false so that all of those can be handled in the second call to init (if necessary). I tried this befo…

loren avatar

curious if this works: TF_CLI_ARGS_init=-no-color terragrunt ...

loren avatar

either way, i think you have enough here to report as a bug… terragrunt may need to be a little smarter about how/when it appends -no-color in auto-init

joshmyers avatar
joshmyers

There is already an issue for it

loren avatar

i couldn’t find one

loren avatar

there is one saying that -no-color is not being passed to init

joshmyers avatar
joshmyers
Terraform "-no-color" option not passed to terraform init · Issue #390 · gruntwork-io/terragrunt

In a command-line such as terragrunt plan –terragrunt-source-update -no-color the -no-color argument does not get passed to the terraform init call, only to the terraform plan call.

joshmyers avatar
joshmyers

yeah

loren avatar

that’s kinda the opposite though

joshmyers avatar
joshmyers

ah right, hadn’t quite followed all the convo

chrism avatar

adding nano to the image is one of those little things that makes life less painful

1
vim1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add env command to export secrets by jradtilbrook · Pull Request #184 · segmentio/chamber

This resolves #94 by adding a new command that will print the secrets in a format that can be eval’ed to export them as environment variables.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Chamber now supports env export suitable for eval

me1249 avatar

Nice

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@me1249 can you please rebuild README

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

make init

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

make readme/deps

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

make readme

me1249 avatar

Done

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
0.72.0 [terragrunt] upgrade to 0.17.4
me1249 avatar

Thanks

2019-02-07

Micah Martin avatar
Micah Martin

Is there an architectural design for how the reference architecture is setup?

joshmyers avatar
joshmyers

As in a diagram?

Micah Martin avatar
Micah Martin

@joshmyers yes

joshmyers avatar
joshmyers

Not as far as I’m aware, no.

joshmyers avatar
joshmyers

Do you have any specific questions?

loren avatar

hmm, wonder what it would look like in cloudcraft… maybe sign up for a free trial, spin out the reference architecture, capture it in cloudcraft, and export a diagram?

joshmyers avatar
joshmyers

I think that would be an excellent idea

joshmyers avatar
joshmyers

Have to say I think CloudCraft diagrams look meltingly good, but actually not that easy to follow

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I struggled with cloudcraft diagrams where if what I wanted to add something which was not isomorphic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But yes, diagrams are long overdue

loren avatar

there are a couple other similar services i think, could try them all i imagine

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We use Lucid Charts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have a lot of diagrams that we use in our SOWs, just haven’t had the chance to open source something.

chrism avatar

FYI The export cloudcraft from aws is a tad iffy. We’re currently paying for it though, its nice to have docs that don’t look arse

Abel Luck avatar
Abel Luck

hey folks :slightly_smiling_face: I’m checking out the reference architecture setup. great work, it’s really nice. I can’t seem to locate the code where the admin IAM users are created The root-iam module takes a list of user names root_account_admin_user_names, but looking through all the repos, I can’t find where those users are actually created before being passed to root-iam

Abel Luck avatar
Abel Luck

i think the module that creates the users is cloudposse/terraform-aws-iam-user, but where is this called?

joshmyers avatar
joshmyers
cloudposse/root.cloudposse.co

Example Terraform Reference Architecture for Geodesic Module Parent (“Root” or “Identity”) Organization in AWS. - cloudposse/root.cloudposse.co

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Abel Luck https://github.com/cloudposse/reference-architectures is our latest approach. We changed how the users are provisioned. Before, we were adding users to root_account_admin_user_names, but this was not scalable (e.g. when using CI/CD and need to add a user, we’d have to update the user list)

cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so now, for each new user we create a separate file in https://github.com/cloudposse/root.cloudposse.co/blob/master/conf/users

cloudposse/root.cloudposse.co

Example Terraform Reference Architecture for Geodesic Module Parent (“Root” or “Identity”) Organization in AWS. - cloudposse/root.cloudposse.co

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Abel Luck welcome to the community

Abel Luck avatar
Abel Luck

Ah interesting.

Abel Luck avatar
Abel Luck

that clears that up!

Abel Luck avatar
Abel Luck

where does the local.admin_groups come from?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

share the URL where you see it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
# Fetch the OrganizationAccountAccessRole ARNs from SSM
module "admin_groups" {
  source         = "git::<https://github.com/cloudposse/terraform-aws-ssm-parameter-store?ref=tags/0.1.5>"
  parameter_read = "${formatlist("/${var.namespace}/%s/admin_group", local.accounts_enabled)}"
}
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s read from SSM

joshmyers avatar
joshmyers
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

joshmyers avatar
joshmyers

terraform-root-modules/aws/users gets merged with [root.cloudposse.co/aws/users](http://root.cloudposse.co/aws/users)

Abel Luck avatar
Abel Luck

gotcha

Abel Luck avatar
Abel Luck

nice use of the ssm

joshmyers avatar
joshmyers

Logic to add users is common to all stages, so goes in your terraform-root-modules but your users maybe different per stage, so we put those per stage

Abel Luck avatar
Abel Luck

thanks

joshmyers avatar
joshmyers

well, all users get added to the root account in our case, but that is the general idea behind your stage codebase (like [testing.cloudposse.co](http://testing.cloudposse.co)) and terraform-root-modules

2019-02-08

loren avatar

took me a little bit to duplicate… the config to end up here is pretty specific

Abel Luck avatar
Abel Luck

What is the bootstrap module for exactly? Where is it used?

Abel Luck avatar
Abel Luck

the pr says “ * This is needed by the reference-architectures to provision the subaccounts because it’s not possible to assume-role using master account”

Abel Luck avatar
Abel Luck

but aren’t the accounts already provisioned by the time the bootstrap module is run?

Abel Luck avatar
Abel Luck
cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Our reference architecture assumes you start with a brand spanning new AWS account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In fact that account is so new there are no existing IAM users

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So the role of the bootstrap module is to provision a user and role which can be used by the coldstart process to assume role into the sub accounts that get created

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then at the very end we can destroy this module

Abel Luck avatar
Abel Luck

gotcha.

Abel Luck avatar
Abel Luck

i’ve been working through the cold start code by following https://docs.cloudposse.com/reference-architectures/cold-start/ as well as looking at the repos

Abel Luck avatar
Abel Luck

i’ve noticed that that doc page has diverged from the code, but i didn’t realize it was that much

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, almost night and day

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we just added support for “archived” pages

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’m going to need to do that for a bit of our documenation until we can update it.

Abel Luck avatar
Abel Luck

Following on from my question yesterday about how users get created

Abel Luck avatar
Abel Luck

The user is added to the admin groups by the users root module, but later in the iam root module the organization-access-group module will REMOVE the user from the group, because the [prod|dev|...]_account_user_names list is empty

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, so users should get created in the [root.org.com](http://root.org.com) account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and not added to root-modules directly

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

an old pattern we had, was (1) adding users, then (2) adding users to the group list

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that didn’t work well

Abel Luck avatar
Abel Luck

makes sense yea

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the new pattern is (1) adding users and adding them to the groups at the same time

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so when you delete a user, they are deleted from the groups too

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

works well with atlantis

Abel Luck avatar
Abel Luck

indeed, yes, have the user creation also add them to the group

Abel Luck avatar
Abel Luck

so the iam root module is run before the users are added

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i am not sure if our terraform examples in [root.cloudposse.co](http://root.cloudposse.co) are up to date; what i am describing is how we’re doing it with current customers

Abel Luck avatar
Abel Luck

because if i run the iam root module after adding users, it removes them

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think i have seen that behavior

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s a bug we should address.

Abel Luck avatar
Abel Luck

specifically if i run iam a second time after adding users

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i thought it was quite weird though; usually terraform will only try to delete things it created

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so i was not anticipating that behavior. i am not quite sure what the fix is other than deprecating the functionality in the upstream gorup module

Abel Luck avatar
Abel Luck


WARNING: Multiple aws_iam_group_membership resources with the same group name will produce inconsistent behavior!

Abel Luck avatar
Abel Luck

not in this case it seems

joshmyers avatar
joshmyers

yeah, have seen this before

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, so our root modules are mostly organized into separate “phases”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Abel Luck btw, feel free to schedule some time with me next week: https://calendly.com/cloudposse

Abel Luck avatar
Abel Luck

It seems to me that the organization-access-group module should only create the group and apply the policy

Abel Luck avatar
Abel Luck

not actually manage the members of the group

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

agreed - this is what i think we should deprecate

Abel Luck avatar
Abel Luck

that works better when users are decoupled from the group

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(btw, if you have time to submit a PR for that, we’ll promptly review)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if not, we’ll probably fix this on our next customer rollout

Abel Luck avatar
Abel Luck

i think the same applies to the iam-assumed-roles module

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yep - indeed

Abel Luck avatar
Abel Luck

So i’ve been piecing together my own version of the reference architecture using ya’lls great building blocks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, yes, but i think that’s the spirit!

Abel Luck avatar
Abel Luck

Abel Luck avatar
Abel Luck

Geodesic is really cool and I totally see the problem its solving, its just I can’t introduce it into our workflow yet. So I’m back in the land of special directory layouts and copy/pasted HCL

Abel Luck avatar
Abel Luck

using docker image layers to maintain root modules is just brilliant

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so the root modules just helps us be more DRY

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

definitely doesn’t preclude doing things the traditional terraform way

Abel Luck avatar
Abel Luck

Since I started with the outdated docs, I created the accounts by hand, and also created the root admin user by hand

Abel Luck avatar
Abel Luck

that’s no problem..

Abel Luck avatar
Abel Luck

Now I have the root account bootstrapping complete as far : tfstate backend bootrap, root-iam policies, account-settings and creating users

Abel Luck avatar
Abel Luck

however the switch role url for the non-root accounts isn’t working

Abel Luck avatar
Abel Luck

rather, the url works, but i get a perm denied error

Abel Luck avatar
Abel Luck

now given what I know about AWS roles, the sub-account (aka the trusting account) needs to be involved, but I haven’t involved the root user of the sub-account at all

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

how were the child accounts created?

Abel Luck avatar
Abel Luck

i created them manually and invited them to the root acount

Abel Luck avatar
Abel Luck

maybe the invited accounts don’t have the OrganizationAccountAccessRole?

1
Abel Luck avatar
Abel Luck

ding ding

Abel Luck avatar
Abel Luck
Accessing and Administering the Member Accounts in Your Organization - AWS Organizations

Access the accounts that are part of your organization in AWS Organizations. Use the root user or an AWS Identity and Access Management (IAM) role to access the resources of a member account as a user in the organization’s master account.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

correct

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

they do not

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we do have a module you can use to provision that though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-organization-access-role

Terraform module to create an IAM Role to grant permissions to delegated IAM users in the master account to access an invited member account - cloudposse/terraform-aws-organization-access-role

2019-02-09

Abel Luck avatar
Abel Luck

of course you do

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

:)

2019-02-11

Abel Luck avatar
Abel Luck

the null-label module is just fantastic

joshmyers avatar
joshmyers

Mind boggling.

Abel Luck avatar
Abel Luck

who knew string concatenation could be so great

Abel Luck avatar
Abel Luck

Do you all generally create one log bucket per-thing that ships logs (e.g, an ALB) or do use one bucket to receive multiple services’ logs?

Abel Luck avatar
Abel Luck

The latter seems more user friendly, but managing multiple policies on a bucket is a PITA

joshmyers avatar
joshmyers

Genercally I try and and <S3://region-account-logs-bucket/$service>

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think we should adopt this convention on our buckets for logs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

isolating buckets by region is smart.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hey all!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’m giving a talk on Geodesic in the beginning of March here in Los Angeles

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I don’t have all the details yet.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m putting together an abstract for the meetup.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Would love to get your feedback if I’ve captured what we have with #geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Title: Geodesic Cloud Automation Shell
Subtitle: The easy way to automate everything


Geodesic is a cloud automation shell. It’s the superset of all other tools including (terraform, terragrunt, chamber, aws-vault, aws-okta, kops, gomplate, helm, helmfile, aws cli, variant, etc) that we use to automate workflows. You can think of it like a swiss army knife for creating and building consistent platforms to be shared across team environments. It easily versions staging/production/dev environments in a repeatable manner that can be followed by any team member with only a single dependency: docker. Because of this, it works with Mac OSX, Linux, and Windows 10. Learn how you can use the geodesic shell to improve your DevOps workflows! There will be a live demo with Q&A.

4
me1249 avatar

Mention Atlantis in the tool list

Jan avatar

“it’s a super set… “ rather than the?

Jan avatar

Across teams, rather than across team environments?

Jan avatar

Make is also a depedancy

Jan avatar

Otherwise cool

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


Make is also a depedancy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not technically

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
docker run cloudposse/testing.cloudposse.com | bash 
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Please add any suggestions as a thread to

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jan @daveyu @Adam @joshmyers @Dombo @chrism @me1249 @loren @Andriy Knysh (Cloud Posse) @Igor Rodionov

2019-02-12

Jan avatar

The easy way to automate everything

troll2
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

beauty is in the eye of the beholder

2019-02-13

daveyu avatar

now that I’m used to chamber exec in geodesic, I’m looking for a clean way to export secrets for developers. I’d love to put something like this in an app’s .direnvrc: aws-vault exec myco-dev-admin -- chamber exec myapp -- sh -c 'export -p'. I haven’t been able to butcher my way through bash to get this to work, I think because aws-vault needs user input

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this looks good

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

just need to add source<(...)

daveyu avatar

When I try that, it hangs, expecting input for passphrase to unlock awsvault

daveyu avatar

is it possible?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yep, it’s possible

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’m on a call

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we do it though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, they just added shell export compatibility

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not sure if a release has been cut

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add env command to export secrets by jradtilbrook · Pull Request #184 · segmentio/chamber

This resolves #94 by adding a new command that will print the secrets in a format that can be eval’ed to export them as environment variables.

1

2019-02-14

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:19:30 AM

set the channel description: Discussions related to https://github.com/cloudposse/geodesic Archive: https://archive.sweetops.com/geodesic/

2019-02-18

chrism avatar

In the ref architecture world is it better to pump flowlogs to cloudwatch or into the audit account S3 bucket.

loren avatar
How to Facilitate Data Analysis and Fulfill Security Requirements by Using Centralized Flow Log Data | Amazon Web Servicesattachment image

I am an AWS Professional Services consultant, which has me working directly with AWS customers on a daily basis. One of my customers recently asked me to provide a solution to help them fulfill their security requirements by having the flow log data from VPC Flow Logs sent to a central AWS account. This is […]

chrism avatar

Ta. Need logging of my loggings logging

chrism avatar

Terraform network ACLS are crap. Nothing like having to use counts because someone decided subnet CIDR should be a single entry

2019-02-19

chrism avatar

@Erik Osterman (Cloud Posse) not sure if you’ve tripped over this; https://github.com/nozaq/terraform-aws-secure-baseline

nozaq/terraform-aws-secure-baseline

Terraform module to set up your AWS account with the secure baseline configuration based on CIS Amazon Web Services Foundations. - nozaq/terraform-aws-secure-baseline

2
chrism avatar

the alarm base is pretty similar to yours (bar you use loops, they’ve used individual entries)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Will take a look. I know ours also needs some love.

chrism avatar

the layouts a bit shit in there but such is the joy of having to use N providers

loren avatar

I feel like I’d prefer a module that took a single provider, and let me call it as many times as needed for each region I needed to configure

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, I think this would be a better interface

loren avatar

Looks alright if I use the submodules

2019-02-20

chrism avatar

yeah the submodules for just that

chrism avatar

think the point is that you’re supposed to configure every region (you have enabled) or someone with access via a liberally create iam key could simply spawn shit outside of the region you normally use

chrism avatar

Can you restrict regions at the SCP level? seems to all be around feature restriction

loren avatar

Yes, IIRC, looks not unlike restricting regions via IAM policy…

chrism avatar
chrism
03:25:35 PM

Not sure this is baked properly in cloudposse/geodesic:0.71.0

joshmyers avatar
joshmyers

@chrism do you have anything in /localhost/.aws/config ?

chrism avatar

works fine on rel 70

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I can take a look as well in a few hours

joshmyers avatar
joshmyers
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

chrism avatar

yup; changing to 70 at least stopped me veering off down a rabbit hole to work out that one

joshmyers avatar
joshmyers

@chrism what does your $AWS_CONFIG_FILE look like?

chrism avatar

im assuming its something in the function choose_role_interactive() { in aws-vault but it seems sane enough (though ive no idea what the fzf bit tis)

chrism avatar
 ✓   (test-prod-admin) vpc ⨠  echo $AWS_CONFIG_FILE
/localhost/.aws//config
chrism avatar

thats running 70

chrism avatar

ill give you 71 in about a minute or so when its built again

joshmyers avatar
joshmyers

Cool, it should be the same, works for me (tm) - see https://github.com/cloudposse/geodesic/pull/376

Better support for aws-iam-authenticator by joshmyers · Pull Request #376 · cloudposse/geodesic

what This commit adds a command kopsctl login to build a kubecfg that optionally can use aws-iam-authenticator, given we have the KOPS_AWS_IAM_AUTHENTICATOR_ENABLED set to true why We want to enabl…

joshmyers avatar
joshmyers

what does your config look like is what I’m asking, is it in the same format as can be seen on that PR?

chrism avatar

` ✓ (ivendi-prod-admin) vpc ⨠ echo $AWS_CONFIG_FILE /localhost/.aws//config` looks the same

chrism avatar

yeah its pretty much the same

chrism avatar

even though i can cat out my config

chrism avatar
chrism
05:19:54 PM
chrism avatar
 ✗   (none) ~ ⨠  ls -lsa /localhost/.aws//config4 -rwxr-xr-x 1 root root 1019 Feb  6 13:54 /localhost/.aws//config
chrism avatar

no diff if I change the path to /localhost/.aws/config rather than double slash

chrism avatar

Is this going to show a prompt everytime? because that kinda takes away from having N repos /org split

joshmyers avatar
joshmyers

Everytime you assume a role, or you can pass in a specific role as before assume-role role_name

joshmyers avatar
joshmyers

Needed a way to assume different IAM roles for k8s rbac

chrism avatar

i just typed assume-role previously and im assuming it defaults to the one in the env

joshmyers avatar
joshmyers

yeah, it did that if you didn’t give a name.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so both @joshmyers and @chrism are right

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@joshmyers is alluding to the fact we support now multiple roles per account; not just one admin role

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. one role to become a kubernetes admin, one role for read-only, etc

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

however, the “one role policy” is probably going to continue to common

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so maybe we should still honor AWS_DEFAULT_PROFILE if it is set

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

unset it for the selector

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

thoughts @joshmyers?

joshmyers avatar
joshmyers

Seems fair, AWS_DEFAULT_PROFILE is still set

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but not honored by the selector?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(i haven’t looked into the code yet)

joshmyers avatar
joshmyers

AWS_DEFAULT_PROFILE is set everywhere IIRC so you will never get to selector

joshmyers avatar
joshmyers
Still unsure why you don’t have that populated @chrism - ` crudini –get “${AWS_CONFIG_FILE}”awk -F ‘ ‘ ‘{print $2}’`
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:32:19 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

previously it worked like this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but now, AWS_DEFAULT_PROFILE is gone

joshmyers avatar
joshmyers

yes, as I alluded to

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i guess what I mean, is we (cloudposse) should probably remove AWS_DEFAULT_PROFILE from our dockerfile (*.cloudposse.co)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then we get the benefit we want

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but if AWS_DEFAULT_PROFILE is set, the original functionality persists that @chrism is using

1
chrism avatar
chrism
05:34:47 PM

crudini --get "${AWS_CONFIG_FILE}" | awk -F ' ' '{print $2}' fails with the same error but the variable is set.

chrism avatar

and the file exists

joshmyers avatar
joshmyers

hah, looks like it

chrism avatar

even giving it the direct path fails

joshmyers avatar
joshmyers

crudini --version ?

chrism avatar

0.9

joshmyers avatar
joshmyers
05:38:18 PM
chrism avatar

if i copy the file it works

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

chrism, remind me: are you on WSL?

chrism avatar

yerp

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm… ok, maybe related

joshmyers avatar
joshmyers

ah ha

chrism avatar
chrism
05:39:00 PM
chrism avatar

file system permission based issue; i’d expect aws-vault to fail though

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does cat ${AWS_CONFIG_FILE} >/dev/null

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

work?

chrism avatar

work as in do nothing yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, no errors

chrism avatar

yeah it cats out fine

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i wonder if we run crudini with config from stdin instead then

joshmyers avatar
joshmyers
Default to AWS_DEFAULT_PROFILE for IAM assume-role by joshmyers · Pull Request #380 · cloudposse/geodesic

what Default to AWS_DEFAULT_PROFILE for IAM assume-role why If AWS_IAM_ROLE_INTERACTIVE is not set, we default to previous behaviour of using AWS_DEFAULT_PROFILE and not prompting via selector. Rat…

2019-02-21

keen avatar
cloudposse/reference-architectures(master)%% make children
make: *** No rule to make target `dev/validate', needed by `children/validate'.  Stop.

so running into this, after a make root. looking at the makefiles, I’d tend to agree with make. which leads me to suspect there’s a missing step somewhere to generate more makefiles?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Fix Make Error · Issue #10 · cloudposse/reference-architectures

what When trying to hack my way through using the reference architecture with an existing root account, I got prod and audit accounts created. When I run make root, it errors ➜ make children make[1…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does this fix your issue?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(sorry, we haven’t had a time to open the PR for this fix albeit very simple)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:16:36 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this should be enough to fix it

keen avatar
 make children
tasks/Makefile.child:56: *** missing separator.  Stop.
keen avatar
# Define a macro (`child`) for a child init & provisioner
define child

## Validate the child configuration exists
keen avatar

that’s where I removed the = (child =)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

keen avatar
# List of accounts to enable
accounts_enabled = [
  "dev",
  # "staging",
  # "prod",
  # "testing",
  # "data",
  # "corp",
  # "audit",
]
keen avatar

really odd - removing the = results in the missing sep. adding it back doesnt fit it. reverting the file does.

draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% cat -ev tasks/Makefile.child |grep "define child"
define child =$
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
rm -rf //build-harness
rm -rf repos accounts .terraform *.tfstate* artifacts/*
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
tasks/Makefile.child:61: *** missing separator.  Stop.
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% cat -ev tasks/Makefile.child |grep "define child"
define child$
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
tasks/Makefile.child:61: *** missing separator.  Stop.
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% cat -ev tasks/Makefile.child |grep "define child"
define child =$
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% gco tasks/Makefile.child
draistrick-mini:~/.../terraform-prototypes/cloudposse/reference-architectures(master)%% make clean
rm -rf //build-harness
keen avatar

ah, something sublime is doing on save…

keen avatar

why dont I remember that make is tab sensitive? only change I see is tab > space

keen avatar

nope, same thing if I edit with nano. odd.

keen avatar

(and much smaller diff. aka, none)

keen avatar

oh, no, it’s reading my copy dammit

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha yea, whitespace issues suck

keen avatar

ok let me reset everything and try again

keen avatar

a make root/reset would be useful. (though the world still breaks if a sub account already exists, so I also have to bump my email prefix)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

keep in mind, each time yuo do that, you create a new AWS subaccount

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

subaccounts cannot be easily deleted

keen avatar

yep

keen avatar

that’s why I have a 1k account limit

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

HAHHA

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

wow, didn’t even know that was possible

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

“Accounts as Cattle”

Thibault avatar
Thibault

hey everyone, is there any way to import the aws_zone53_zone.parent_dns_zone and aws_zone53_record.parent_dns_zone_soa when using reference-architectures and/or root.cloudposse.co repository directly?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not easily

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The reference architectures (as they stand) are basically designed to take a someone from a coldstart of zero (nothing in an account) to a full fledged architecture

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Basically, this was born out of our own needs to provision architectures for our clients

Thibault avatar
Thibault

fair enough, I ended up commenting out the auto-apply section of the Makefile.child and scripts/provision.sh (TF_CLI_ARGS_apply="") and -auto-approve in Makefile.root

Thibault avatar
Thibault

so it wouldn’t create automatically resources in our root account (which unfortunately already had resources in it)

Thibault avatar
Thibault

so now I’ve got geodesic containers for my different stage and i’m kind of provisioning them manually

Thibault avatar
Thibault

but i’m hitting a road block at the dns config, trying to import our parent (already existing) zone

keen avatar

yeah, it took a few weeks to get approved, but my justification of “working on automation tooling to manage creation of accounts” didnt result in any followup questions

keen avatar

at some point, I’d like to have a better idea of how to add new accounts to an existing structure - is there a sane way to do that with what exists now?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, and no

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jan has done this and I know others as well

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But before doing that, best to get familiar with the overall strategy

Jan avatar

Heya

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you have existing accounts, I suggest just using the base images and writing your own dockerfiles from hand.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

[testing.cloudposse.co](http://testing.cloudposse.co) is a good place to look for inspiration

Jan avatar

Yea I can give you loads of advice on that

Jan avatar

Can also tell you where I would like to start from if I had time

Jan avatar

Coming to grips with the reference architecture was a great way to learn

Jan avatar

But ultimately it would ha e been better to start with empty dockerfile

Jan avatar

And add

Jan avatar

Rather than adapt

Jan avatar

Once you understand the pattern it’s trivial

keen avatar

in this case, the plan is to start from scratch - but new products, teams, environments, will roll in and out all the time - so a straightforward way to provision a base new sub account is important

Jan avatar

Optimize for change!

keen avatar

indeed.

Jan avatar

We have done loads of things that are custom

Jan avatar

Busy on a few cool ones

Jan avatar

We have heavy deadlines for end of April so have not been able to contribute much back yet

Jan avatar

But plan is to do so

keen avatar

in general, the overall framework is not far off from what I would build if I were scratch building again (and using tf as the basis, instead of as a component tool like I built last time.)

Jan avatar

If you have more specific sorta constraints / ideas / needs I am happy to help

keen avatar

ok, make children is running! (minus equal)

Jan avatar

For example on ours we had existing aws accounts, in an existing org where we setup our “root” in a sub aws org account

Jan avatar

With totally different network cidr schemes

Jan avatar

On premise vpn, hashicorp vault

Jan avatar

We are a gitlab / Jenkins shop

Jan avatar

In the end it’s just terraform

keen avatar

fun. did you try to import the existing accounts into the scheme, or migrate away from them? (the only thing I hate more than manually fixing tfstate is trying to craft it for existing resources)

Jan avatar

With a nice layer of convention on top

Jan avatar

They were empty accounts

Jan avatar

Just crated already

keen avatar

ah cool

keen avatar

easier there

Jan avatar

Well to an extent

Jan avatar

Honestly the virvana state would have been starting a full aws org from scratch

Jan avatar

My biggest pain so far with taking the geodesic reference architecture and adapting it has been the env names

Jan avatar

Try do a search and replace on “data” in terraform

Jan avatar

I dare you

keen avatar

hahahaha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so for the record, the point there is you do not renmae

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you create additional ones

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

since we have accounts_enabled, you can then choose the flavors you want

keen avatar

so make children failed out with auth problems - a number of similar errors scattered through the log

panic: SharedConfigAssumeRoleError: failed to load assume role for arn:aws:iam::894200800587:role/OrganizationAccountAccessRole, source profile has no shared credentials
...
The source_profile "brasstack" must specify either static credentials or an assume role configuration
....
Error: Error running plan: 1 error(s) occurred:

* provider.aws: No valid credential sources found for AWS Provider.
	Please see <https://terraform.io/docs/providers/aws/index.html> for more information on
	providing credentials for the AWS Provider
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

did you enable the bootstrap module?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the children will only be accessible via the bootstrap role

keen avatar

it’s in root.tfvars yes

keen avatar

only thing I removed was cloudtrail

keen avatar

arn:aws:iam::665716774983:role/brasstack-root-bootstrap

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in the artifacts/ folder you should see the temp aws config

keen avatar

yeah

keen avatar
# Temporary configuration for AWS bootstrapping
[profile brasstack-root-admin]
region = us-west-2
role_arn = arn:aws:iam::665716774983:role/brasstack-root-bootstrap
source_profile = brasstack


# Temporary configuration for AWS bootstrapping
[profile brasstack-dev-admin]
region = us-west-2
role_arn = arn:aws:iam::894200800587:role/OrganizationAccountAccessRole
source_profile = brasstack
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and [profile brasstack]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i don’t see that in that output

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(jumping on a call)

keen avatar

that’s in credentials with keys

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it should exist in the config file

keen avatar

but no `[profile brasstack]’

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

just an empty def

keen avatar

mm, something didnt put it there.

keen avatar

oh no

keen avatar

it’s at the top

keen avatar

first line blindness

keen avatar

the full config/credentials stack looks correct - keys, iam roles, policies, between the two accounts. I can assume the role on the dev account (using a root account user not created by tf. guess I can try using the tf user)

keen avatar

yep, no issues there either.

keen avatar

failed to load assume role for arn:aws:iam::894200800587:role/OrganizationAccountAccessRole, source profile has no shared credentials seems to be from the go sdk, so I’m presuming tf

keen avatar

this seems to be the origin of the failure (and unhandled)

keen avatar
fetch <https://alpine.global.ssl.fastly.net/alpine/edge/community/x86_64/APKINDEX.tar.gz>
(1/1) Installing assume-role@cloudposse (0.3.2-r1)
Executing busybox-1.28.4-r2.trigger
OK: 709 MiB in 120 packages
panic: SharedConfigAssumeRoleError: failed to load assume role for arn:aws:iam::894200800587:role/OrganizationAccountAccessRole, source profile has no shared credentials

goroutine 1 [running]:
github.com/remind101/assume-role/vendor/github.com/aws/aws-sdk-go/aws/session.Must(0x0, 0x8e8d80, 0xc420019c90, 0x0)
	/Users/ejholmes/src/github.com/remind101/assume-role/vendor/github.com/aws/aws-sdk-go/aws/session/session.go:265 +0x54
main.assumeProfile(0x7ffd32d66db9, 0x13, 0x0, 0x0, 0x8e8340)
	/Users/ejholmes/src/github.com/remind101/assume-role/main.go:148 +0xf7
main.main()
	/Users/ejholmes/src/github.com/remind101/assume-role/main.go:77 +0x178
Handling [--apply-modules]...
Processing tfstate-backend...
keen avatar

then it keeps going until tf fails

keen avatar
Assume Role panic. · Issue #8 · cloudposse/reference-architectures

Command make children Error panic: SharedConfigAssumeRoleError: failed to load assume role for arnawsiam:role/OrganizationAccountAccessRole, source profile has no shared credentials Stat…

keen avatar
That looks like the right fix. Basically, when using the bootstrap user we need to use `AWS_SHARED_CREDENTIALS_FILE=/artifacts/.aws/credentials
keen avatar

looking much happier. I’m gonna PR that

keen avatar

well, next… Failed to save state: truncate terraform.tfstate: read-only file system

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that one is new! haven’t seen that

keen avatar

race condition on s3 create maybe?

module.tfstate_backend.aws_s3_bucket.default: Creation complete after 10s (ID: brasstack-dev-terraform-state)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (10s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (20s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (30s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (40s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Still creating... (50s elapsed)
module.tfstate_backend.aws_dynamodb_table.with_server_side_encryption: Creation complete after 59s (ID: brasstack-dev-terraform-state-lock)
Failed to save state: truncate terraform.tfstate: read-only file system
keen avatar

though I’m speculating - actually that’s probably still saving to local state, not to remote

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so tfstate backend initialization is tricky

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it first initializes locally

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then imports it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” invocations for provisioning reference architectures - cloudposse/terraform-root-modules

keen avatar

appearance is this was terraform apply -auto-approve -input=false not the import (first tf run in the container)

keen avatar

so, then, why would the container be read-only. hrm

loren avatar

Which channel we’re we talking about test frameworks? https://opensource.com/article/19/2/testing-bash-bats

Testing Bash with BATS

The Bash Automated Testing System puts Bash code through the same types of testing processes used by Java, Ruby, and Python developers.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmmmmmm can’t recall.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Think it recently was in #terraform

keen avatar

maybe my docker has taken a dive. removed the s3/ddb, reran…and docker error.

module.account.module.docker_build.null_resource.docker_build (local-exec): Sending build context to Docker daemon  27.65kB
module.account.module.docker_build.null_resource.docker_build (local-exec): Error response from daemon: mkdir /mnt/sda1/var/lib/docker/tmp/docker-builder348479862: input/output error

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, looks like it totally crapped out

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@loren are you using bats?

loren avatar

i’m not no, pytest/testinfra, exploring go test some with terratest

loren avatar

but i knew you were, and thought it would be a good read for other folks here…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yep, thanks for the heads up

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

looks like a good read

keen avatar

I’ve used bats before (in something I PRed. forget what. heard about it a few years before that. then later experimented with it for another project. also forget what)….it wasn’t bad. like anything, it has it’s own concepts

keen avatar

restarted docker-machine, and now children ran ok. so $chaos

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you not using docker for mac?

keen avatar

nope, docker for mac panics 9 out of 10 startups (after 5-10m of complete resource sucking - the clock slows down…), the 10th just sucks all resources until I give up and kill it

keen avatar

go mohave.

2019-02-26

keen avatar

so, back to ref-arch - after the first pass through - make root make children make finalize with a defined set of children in accounts_enabled - is there a way to reuse the tooling to create a new child? (I may have asked before, but it’s been a really long last few weeks) I’ve tried a few things (adding a new entry to accounts_enabled, subing out the list to just a new acct, etc), but there’s apparently enough state retained to keep that from happening. using the root generated repo, I can make the tooling create a new account, but that doesnt generate a new repo framework…..

keen avatar

pointers to places that might lead me to a path to be able to do that are similarly welcome… (since there’s pretty much no way to know what sub accounts we’d need right now - I’ve got 3 prods and their required test/dev/stage envs on the books so far…)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So you will want to fork the terraform root modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then define all the different account types there

keen avatar

yeah saw that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can truly have any number of accounts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think best would be to do a quick call to see if I can help unblock you

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Sometime later this week..

keen avatar

just hoping to find a way to generate them, config them, and build the repo for each new account as it comes along (it seems like the ref-arch as it sits now is more focused on onetime use than long term generation)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes totally correct

keen avatar

ok cool - I was hoping I was missing something. I can work with that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The current use-case solved was a one time cold start

thumbsup_all1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We don’t try to maintain terraform state for the generation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think it’s doable, but the dns zone delegation or interaccount linking is complicated

keen avatar

are the discovery dns zones used for anything outside of k8s and friends?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s used for discovery things like RDS, elastic ache, elastic search etc

keen avatar

gotcha

keen avatar

I’ll have to look a bit further at how you’re using it with rds in that case

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Check out our RDS module

keen avatar

will do

keen avatar

one other clarification - in the current setup, for each repo - root straight out of the ref-arch for example - there’s no converge the entire repo's state right? just cd dir init-t t plan t apply yes; cd ../dir repeat ?

keen avatar

(noting that I completely and totally understand why you’re splitting tfstate.)

keen avatar

I built a chef-solo framework around tf for similar reasons

joshmyers avatar
joshmyers

@keen no converge all.

keen avatar

thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, which (as you mention) is by design. it’s very easy to manage across accounts in terraform.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

however, we made the conscious design decision to not use that pattern. instead, enforcing due process on how changes are introduced to environments using git workflow.

keen avatar

yep

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think that the approaches aren’t mutually exclusive

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so for example, a certain set of accounts can be delegated out to BUs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then BUs can have their own accounts that they converge

1
keen avatar

would be nice to be able to have a single converge prod option (vs manual workflow steps in test, for example)

keen avatar

but that’s easy enough to achieve

keen avatar

(eventually)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
tmrts/boilr

boilerplate template manager that generates files or directories from template repositories - tmrts/boilr

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(and I know there are others like this)

joshmyers avatar
joshmyers

Not really, you have no idea what you are going to get via a plan if there are dependencies between the states. which almost all places will have.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but this is a small precompiled go binary which is nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and could be an interesting notion to leverage with geodesic

keen avatar

yeah - looks like it might be slightly saner than some of the other generators I’ve worked with. (at quick glance)

2019-02-27

Chris avatar

howdy

Chris avatar

@Erik Osterman (Cloud Posse) was asked to join the channel, unable to get the make docker/build to do anything other than error out: DOCKER not defined in docker/build make: *** [docker/build] Error 1

Chris avatar
Chris
07:36:21 PM
Chris avatar

am I doing something wrong here?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

hey @Chris

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

can you run make init first and then make docker/build

Chris avatar

did

Chris avatar

make init runs fine

Chris avatar
Chris
07:38:43 PM
Chris avatar
Chris
07:40:35 PM
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

can you try to use the latest geodesic https://github.com/cloudposse/geodesic/tree/0.71.3

cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Chris avatar

clone the repo, make init, make?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

do you have Docker installed ?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/build-harness

Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more - cloudposse/build-harness

Chris avatar

docker desktop wasn’t running

Chris avatar

workingnow

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

good

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

(and the Dockerfile)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

then

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

make init

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

make docker/build

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

make install

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

[root.cloudvirga.com](http://root.cloudvirga.com)

Chris avatar

Was able to build the image, now trying to login and assume-role but getting AccessDenied: MultiFactorAuthentication failed, unable to validate MFA code. Please verify your MFA serial number is valid and associated with this user. Status code 403

Chris avatar

I’ll try a few more times

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Did you enable and configure MFA on the account for the user you are trying to login with?

2019-02-28

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for those of you in Los Angeles (#lax), I’ll be presenting on Geodesic at this meetup: https://www.meetup.com/Los-Angeles-Kubernetes-Meetup/events/258976627/

Benefits of Multi-Cloud & Evolution of a Build Process & Automate Everythingattachment image

Thu, Mar 7, 2019, 6:00 PM: Hello Los Angeles Kubernauts!We are going to mix things up slightly by having not just two, but three K8s presentations!First, Kevaughn is going to discuss how Steelhouse de

1
    keyboard_arrow_up