#geodesic (2018-11)

What do you guys use to host the packages available at apk.cloudposse.com?

Managed service or self hosted?

Good question

https://github.com/cloudposse/alpinist

It’s a fork

Very cool little lambda deployed via cloud formation

PR is still open because we’ve been using it for demos

With Atlantis

haha dude

I was like “I do not want to host an APK repo”

THen found this https://github.com/forward3d/alpinist

Checked back here to see if you’d hear of it

TUrns out you’ve done the hard part for me already

If you’re going to host one I can finish up the PR next week

I’m happy to jump on with you?

I think it’s ready to merge and release

Then our CloudPosse packages repo is how we generate our packages

Yeah noted

We are going to consolidate the vendor and install folders in packages. Our justification for doing it this way is so we don’t have lock in on apk. We use it to install native binaries (e.g. for local Dev) and maybe one day will support other packages like Debian.

I might be missing something but I’m unsure how consolidating those folders will change anything? That Makefile in install is a manifest and quite useful as a new-comer to the repo.

You can achieve multi-package support by simply adding more steps to the individual Makefiles in the vendor dir

Disclaimer: not exactly a Makefile pro but it’s easy enough to follow

Oh the install folder was just how we used to do it

The vendor folder is the new way

We are going to add a make target to preserve the original behavior

But keep all the logic in the vendor folder.

Oh btw, if there are any packages missing you would like feel free to open an issue or PR (preferred) and we can add them

Yeah was just thinking that - most of my daily tools are there

Does anyone have an example repo that uses the “multi-stage docker build pattern” against geodesic? I’m hoping not to reinvent the wheel for customizations.

https://github.com/cloudposse/prod.cloudposse.co

https://github.com/cloudposse/staging.cloudposse.co

https://github.com/cloudposse/testing.cloudposse.co

oh, I didn’t know those draw from geodesic – but I should have thought of that

tnx!

what do folks do when they need a later version of a package than what is listed in the cloudposse/packages repo? I’m looking for a later version of terraform than 0.11.8 listed at https://github.com/cloudposse/packages/blob/master/vendor/terraform/VERSION

or a later version than what’s available in the apk aports: https://git.alpinelinux.org/cgit/aports/tree/community/terraform/APKBUILD

Submit a PR against our packages repo

The only reason it’s not the latest is we haven’t had a chance to update it yet.

@tamsky

Also, anything you want a package for, feel free to open an issue or submit a PR

There seem to be at least two different version directives for terraform within cloudposse/packages:

diff -r 5c58433395f3 install/Makefile
--- a/install/Makefile  Tue Oct 23 18:32:59 2018 -0700
+++ b/install/Makefile  Sun Nov 11 15:53:45 2018 -0800
@@ -297,7 +297,7 @@
 teleport:
        $(CURL) <https://get.gravitational.com/teleport/$(TELEPORT_VERSION)/teleport-v$(TELEPORT_VERSION)-${OS}-$(ARCH)-bin.tar.gz> -o - | tar -C $(INSTALL_PATH) -zx --wildcards --strip-components=1 --overwrite --mode='+x'  */tsh */tctl */teleport
 
-export TERRAFORM_VERSION ?= 0.11.7
+export TERRAFORM_VERSION ?= 0.11.10
 # Releases: <https://github.com/hashicorp/terraform/releases>
 ## Install Terraform
 terraform:
diff -r 5c58433395f3 vendor/terraform/VERSION
--- a/vendor/terraform/VERSION  Tue Oct 23 18:32:59 2018 -0700
+++ b/vendor/terraform/VERSION  Sun Nov 11 15:53:45 2018 -0800
@@ -1,1 +1,1 @@
-0.11.8
+0.11.10

and there seems to be some conflict between what version of terraform gets installed in geodesic. geodesic appears to use the apk manifest in //packages.txt to install /usr/bin/terraform. in the cloudposse/packages image, it installs /usr/local/bin/terraform binary directly from vendor (no apk), but this binary does not get copied into geodesic during the multi-stage build.

So that may mean there are 3 different version directives of terraform in play?

• //packages/install/Makefile
• //packages/vendor/terraform/VERSION (appears to have no effect)
• //geodesic/packages.txt (link: https://github.com/cloudposse/geodesic/blob/master/packages.txt#L52)

The install folder is preserved for legacy reasons

It will be migrated to use the new make system for backwards compatibility

But for now we just have both

@tamsky i see what you’re saying

//packages/vendor/terraform/VERSION is used by the new system

//packages/install/Makefile we’re going to create a wildcard target to map the old targets to the new targets for backwards compatibility

geodesic appears to use the apk manifest in //packages.txt to install /usr/bin/terraform. in the cloudposse/packages image,

correct - alpine packages tend to get installed in /usr/bin, so we preserved that

in the cloudposse/packages image, it installs /usr/local/bin/terraform binary directly from vendor (no apk)

this is determined by the INSTALL_PATH environment variable that defaults to /usr/local/bin

we didn’t want non-package packages going into /usr/bin

/usr/local/bin should have priority over /usr/bin (but i haven’t verified)

i’m open to discussion/suggestions on better ways of handling this

@yurchenko did you get to the bottom of it?

@yurchenko has joined the channel

/usr/local/bin should have priority over /usr/bin (but i haven’t verified)

geodesic’s Dockerfile only lists two COPY directives related to the packages image

COPY --from=packages /packages/install/ /packages/install/
COPY --from=packages /dist/ /usr/local/bin/

so everything in --from=packages /packages/bin/ is lost.

for now, I’ve fixed my problem of getting v0.11.10 in geodesic by adding +COPY --from=packages /packages/bin/terraform /usr/bin/terraform

@tamsky see make dist

the idea is that you copy everything you want into /dist

then COPY --from=packages /dist /usr/local/bin

by staging things into the /dist folder, allows you to specify the packages in an ENV, so that you only specify a single COPY statement

if it would help, i can do a quick zoom

@tamsky are you unblocked?

I made good progress yesterday, so I feel pretty good about it all.

Thanks for the reply and offer to help.

allows you to specify the packages in an ENV,

this sounds like a key concept. I’ll take a look at make dist today.

Do you have any existing techniques/strategies for caching the .terraform/plugins/linux_amd64/ directory? – maybe as a runtime volume-mount?

i want to do that

i tried it in vane at some point, but the problem is that it’s per terraform project

i wish .terraform/plugins could be mapped to /var/lib/terraform or something

(and without symlinks)

but i haven’t found any envs to do that

i tried it in vane at some point, but the problem is that it’s per terraform project

it could be a greedy/opportunistic algorithm… all local images that match a pattern get mounted somewhere and an init job/script links/copies them into $HOME/.terraform/plugins…

(and without symlinks)

sounds like you know something here. does terraform require the inodes in .terraform/plugins/linux_amd64/ to be regular files and not symlinks?

no, i’m just “anti-symlink” hacks

especially where generating symlinks programmatically

if it’s a one shot deal, like ln -s /lib64 /lib, thats cool

i guess it’s more related to tfstate. we used to link tfstate to a persistent volume

but if the symlinks were stale or acidentally pointing to the wrong place, it can have catastrophic outcomes. .terraform/plugins should be pretty safe to symlink. Just wish there was an ENV for it.

the init-terraform script would be the place to do it.

well, tbh, I’d expect any mistakes in the tree or data below .terraform/plugins to be ignored by the terraform binary…. Seems like it’s an opportunistic cache, and the cache itself does not need versioning… since each provider binary within the cache is versioned.

https://github.com/hashicorp/terraform/pull/16000

Does TF_PLUGIN_CACHE_DIR do what you want?

wha?? i missed that

that’s cool

yea, that’s what i want.

now can we have the same for modules?

only if you trust .terragrunt-cache

oh, maybe we can setup geodesic to bind mount the local user’s .terraform/plugins, and that way the local user’s cache can be shared?

just set TF_PLUGIN_CACHE_DIR=/localhost/.terraform/plugins

we always mount $HOME to /localhost

(so we can use .ssh among other things)

i caution though against using /localhost/.terraform, unless your host os is linux

since you’ll have conflicts (possibly) unless plugins are versioned by arch

maybe use /localhost/.geodesic/terraform

plugins are versioned by arch they are…

in the filenames?

in the tree structure

nice

this works… my mac’s homedir .terraform.d/plugins used to only have a darwin_amd64/ dir, now it also has a new linux_amd64/ dir

now if we can just get rid of the ssh-add passphrase prompt

I think we should add ENV TF_PLUGIN_CACHE_DIR=/localhost/.terraform.d/plugins to the main geodesic

thoughts?

@OScar continuing this thread https://sweetops.slack.com/archives/CB3579ZM3/p1542300152238500

what other events that you want to log are you asking for?

as always, curious if you have any more suggestions

I think the reason I haven’t had many suggestions is because I haven’t been able to use it daily. That’ll change now.

haha, ok, preparing to duck for cover!

@lvh has joined the channel

@tamsky I’ve updated https://github.com/cloudposse/packages

so that now install/ targets are just proxied to the vendor/<package>/install

(all backward compatible)

What are folks thoughts on Terragrunt usage in Geodesic? I note some usage in places but not everywhere…

We haven’t committed to it wholeheartedly. Generally averse to wrappers as they often break interoperability.

But I do like how we can extend root modules (e.g. for adding users)

It could rather trivially be used everywhere in geodesic

To date, we have been Docker multi stage to copy and keep things dry

I know one company moved to git submodules

So I think we have 3 good approaches with various trade offs

But it’s a topic that I like to debate. I don’t feel like it’s 100% solved.

Part of what we want is a solution that is tool agnostic, which is why terragrunt is not ideal

We need to version kops, Helmfiles, and other things. Terraform isn’t the end-all-be-all

Indeed. adding more layers to geodesic such as Terragrunt make things harder to rationalise

Although I totally see why it is helpful >_<

Part of my motivation for adding it was to show how flexible the strategy is that we have

That it works with terragrunt like it works with everything else

There are aspects of Terragrunt I don’d mind leaning on too much. Others like the overrides, I’m not so sold on.

I like the easy state auto init, env mapping, and ability to create some thing like a poor mans overlay file system (are those the overrides?)

In terms of the directory structure and recursively looking up a directory until it reaches a terraform.tfvars, allows only 2 level deep child/parent overrides IIRC. Agree on above likes, and before/after hooks

Feels weird using it when there is some overlap/similar things going on with geodesic itself

Yea so the terragrunt directory structure doesn’t make sense with geodesic reference architectures

We recommend sticking with our poly repo approach

@joshmyers when you say “overrides” are you talking about terraform’s *[override.tf](http://override.tf) feature?

recursively looking up a directory until it reaches a terraform.tfvars, allows only 2 level deep child/parent overrides

I do this, but in a more generic way in the Makefile I use. It just looks for a placeholder filename that indicates the root of the environment: export UPSTREAM_ROOT_PREFIX := $(shell FILE=.environment_root ; for i in . .. ../.. ../../.. ; do test -e $$i/$$FILE && echo "$$i/" && exit 0 ; done ; echo "unable-to-find-$$FILE" ; exit 1 )

• https://github.com/tamsky/terrabase/blob/master/root-account-email%40example.com%3D123456789012/prod/Makefile#L3

https://github.com/gruntwork-io/terragrunt/blob/master/README.md#find_in_parent_folders

We do something similar with the build-harness “autodiscovery”

nice to see someone love make as much as we do

[email protected]=123456789012/

account-specific resources are placed within an AWS account directory.
directory names the root account email and the 12-digit AWS account number.

that’s kind’a nice

https://sweetops.slack.com/archives/CB3579ZM3/p1542762118320000

so there’s often a two-phased approach here since a “virgin” account has no IAM users

so step 1, you use the master AWS account credentials to setup the scaffolding

part of that scaffolding is to provision an IAM user for yourself (and anyone else on the team)

step 2, is you use your personal IAM account to provision everything else

this is the ugly “coldstart” problem…

Ok so here is my scenario, I am thinking of doing this in the master account (without using the actual master account):

Login to the root account with the root credentials and do the following: Create new IAM group cloud_admin Assign AdministratorAccess policy to the group Create an IAM user with the name aws_admin Add the user to the group Enable MFA for the user (we recommend using Google Authenticator as Virtual MFA device) Generate Access Key ID and Secret Access Key for the user (we’ll need them to run scripts)

yea, that should be fine

So in this scenario for root, anyone in the cloud_admin group, including say my individual account can run the Geodesic stuff right?

assuming my individual account has keys and enabled with MFA of course

yep, that should work

Thanks @Erik Osterman (Cloud Posse)!!

this clarification helps me tremendously in explaining what account is needed to run this stuff

np.. yea, in the end, you just need some kind of “administrator” access. then you can go about provisioning all the other stuff.

It looks like this error is only happening when I am inside the Geodesic shell…

I think maybe your AWS_DEFAULT_PROFILE is not set to use that profile?

Or try to remove the default profile from aws/config, if you have one

I don’t have a default profile in ~/.aws/config

Ok one other idea

Run “hwclock -s” in your container

It doesn’t seem to like your MFA token

So maybe clock is out of sync

oh you know what? i think you are right

We have seen that happen a lot in Docker for MAC

one thing about this account is that the guy who set this up, also setup the mfa so i had to use his barcode in order to add this to my mobile

hwclock yields this 2018-11-22 01:41:54.996656+00:00

so that time seems to be different than my time

its tomorrow

for one thing

Though it’s GMT :)

Are the minutes and seconds similar?

✗ (none) ~ ⨠ hwclock -v hwclock from util-linux 2.32 System Time: 1542851097.345229 Trying to open: /dev/rtc0 Using the rtc interface to the clock. Last drift adjustment done at 1542851026 seconds after 1969 Last calibration done at 1542851026 seconds after 1969 Hardware clock is on UTC time Assuming hardware clock is kept in UTC time. Waiting for clock tick... ...got clock tick Time read from Hardware Clock: 2018/11/22 01:44:58 Hw clock time : 2018/11/22 01:44:58 = 1542851098 seconds since 1969 Time since last adjustment is 72 seconds Calculated Hardware Clock drift is 0.000000 seconds 2018-11-22 01:44:56.995687+00:00

but that does not seem like my local time

Did you try hwclock -s?

https://github.com/cloudposse/docs/issues/203

i did, same issue, the date shows Nov 22 as well

Ok, can help dig in more when in front of a keyboard :)

i need a drink!

thanks @Erik Osterman (Cloud Posse)!

i’ll keep trying things

It’s officially o’clock in PST

Maybe some tips in this issue

https://github.com/docker/for-mac/issues/1260

i found this https://github.com/arunvelsriram/docker-time-sync-agent/ worth a try?

Can try it out…

@tamsky this is adjacent to what you were working on https://github.com/hashicorp/terraform/blob/master/tools/terraform-bundle/README.md

@Erik Osterman (Cloud Posse) i tried that agent, no bueno

I added this to the Dockerfile and now the date shows correctly, but it is like 2 seconds off!

Ola!

quick question, is there a known cost to runt he smallest setup of the full reference archetecture

with aws org

@Jan it depends on the project. There is no cost to provision the resources that are just ’metadata` like VPC, subnets, all IAM stuff (users, roles, groups), accounts, etc.

for Route53, you just pay for each DNS zone you have (I believe it’s $2 per zone per month)

then you deploy EC2, RDS, Aurora, Elasticsearch, Elacticache, for which you pay depending on the instance type and count

if you provision k8s using kops or EKS, you pay per instance type and count

then, you pay for S3 storage

and for network traffic

so, if you deploy the initial setup - VPC, subnets, accounts, IAM, Route53, S3 (for TF and kops states) - you pay just for Route53 and S3 storage (which is almost nothing)

yea thanks @Andriy Knysh (Cloud Posse)

im doing an evaluation of full org using geodesic

and figured I would do so on my own account to start with

mmm there is something I am not groc’ing

with aws-vault

following here https://docs.cloudposse.com/tools/aws-vault/

What’s the error?

ah nm

im being stupid

I had the staging account set to the root account id

Ah yes copy pasta get me every time

yarp

so I would almost do this with terraform local-exec and render templates

and pass the credentials for boot strap via a .tfvars file which the local exec kills after it runs

also what the docs dont mention is that you should have 2 accounts setup to start with

@Jan are you able to login to the root account from geodesic?

So my current place is:

1. root account is there
2. aws-vault is setup with root admin user
3. I can login or exec
4. its when I then want to do the IAM make init I get promoted for the unset account id var

I am

so before #4, let’s provision the other accounts

Ah I see

you have https://github.com/cloudposse/terraform-root-modules/tree/master/aws/accounts

cd ...

1. init-terraform

ok 2 secs

1. terraform plan <whatever account you need>

I have 2 dir’s

it will output the account ID

root and testing

I am using the root bin now yea?

then add that account ID to the Dockerfile in the other account’s repo (testing in your case)

let me quickly do a git reset

–hard

I have been making all sorts of changes debugging this

on a train so internet is slow , gimme a minute

ok

so in shoer, you go to the root account and provision all other accounts, they will be added to the org automatically since the root account is the master/billing in this case

that makes sense

then you update all Dockerfile(s) with the account IDs

I knew there must be something I was missing

yea

then follow the docs https://docs.cloudposse.com/reference-architectures/cold-start/

sorry for the confusion, we’ll update the docs and ref architectures for that

alright, let me get back to that point

hehe all good

next iteration will be better

I have read most of the docs and loads of the TF / kops / k8s code now

nice

if you have any questions or concerns, please just open issues, we’ll get to them ASAP

btw is there any reason I cant use the same value for parent domain name and namespace?

Ideally I will send a pull request from fork with the docs stuff

we use the same namespace for all envs

some English corrections, missed word etc

NS yes

ah yes

two cases here:

im using aws.tf as domain and plan to use aws.tf for NS too

1. Use the same parent domain for all envs, e.g. [testing.aws.tf](http://testing.aws.tf), [prod.aws.tf](http://prod.aws.tf), etc.

yea totally with you there

1. Use diff TLS for the envs/accounts, e.g. [example.com](http://example.com) for prod, [example.qa](http://example.qa) for staging etc.

tld’s or tls?

top-level domains, TLDs

kk

figured

im like a human english linter

and since those DNS zones are in diff AWS accounts, we use DNS zone delegations - add Name servers for [prod.aws.tf](http://prod.aws.tf) to the [aws.tf](http://aws.tf) DNS zone

yea

with you there too

so I essentially have done e2e tooling similar to what you have

over the years many time in TF

just with less abstraction and less nice docker exec env glue code

so the entire design idea fits sooooo perfectly with my likes

yea i see

really excited to help contribute back

thanks for testing and using it

please, any issues or improvements, let us know

for sure

have a list I have built up which I will go voer again when there is a new tag for docs

as it hasnt been deployed since last update

o.O

2018/11/23 14:40:40 Request body type has been overwritten. May cause race conditions

might be from my train internet

haha fighting internet here

mmm

I might just give up and do this when im home in a few hours

We never tested that on high speed trains :)

lol

well I am

yea will pick this up later