#geodesic (2018-07)

geodesic https://github.com/cloudposse/geodesic

Discussions related to https://github.com/cloudposse/geodesic Archive: https://archive.sweetops.com/geodesic/

2018-07-31

i5okie avatar
i5okie
04:50:41 PM

@i5okie has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth this is the fix for the make install problem you reported:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add default value for INSTALL_PATH by vadim-hleif · Pull Request #5 · cloudposse/dev.cloudposse.co

what Add default value for INSTALL_PATH why On wsl INSTALL_PATH defaults to build-harness path

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You’ll need to incorporate that into you account repo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@rohit.verma - you might dig this:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Add Support for Minikube · Issue #204 · cloudposse/geodesic

what Add support for Docker for Mac (DFM) Kubernetes or Minikube why Faster LDE, protyping howto I got it working very easily. Here's what I did (manually): Enable Kubnernetes mode in DFM. Disa…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I was able to get geodesic working with “Docker for Mac / Kubernetes” with relatively little effort

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Max Moon @alebabai

2018-07-30

rohit.verma avatar
rohit.verma

no, we have only linux and mac machines

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks! I think we are close to getting it working on windows

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth how about this?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
08:12:06 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Setting Up Docker for Windows and WSL to Work Flawlessly attachment image

With a couple of tweaks the WSL (Windows Subsystem for Linux, also known as Bash for Windows) can be used with Docker for Windows.

Sebastian Nemeth avatar
Sebastian Nemeth

@Erik Osterman (Cloud Posse) I actually followed this guide to set up docker on WSL and it’s a good reference for you to include in your documentation. I have the path set up as per this guide, so that mounting from /c works fine, but our problem is with /home/username.

Sebastian Nemeth avatar
Sebastian Nemeth
Docker on WSL won't bind mount $HOME

I have the strangest situation using Docker on WSL (Windows Subsystem for Linux, Ubuntu 16.04). I’m trying to bind mount /home/username (or just $HOME for convenience) as a volume in a container, and

Docker on WSL won't bind mount $HOME

Originally asked this on StackOverflow, but I thought SuperUser might be more appropriate. I have the strangest situation using Docker on WSL (Windows Subsystem for Linux, Ubuntu 16.04). I’m tryin…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, I was thinking maybe needed to use /c/Users/username instead of /home/username

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Upvoted

Sebastian Nemeth avatar
Sebastian Nemeth

hm

Sebastian Nemeth avatar
Sebastian Nemeth

That’s an interesting idea

Sebastian Nemeth avatar
Sebastian Nemeth

That’s v. interesting. docker run -it --rm -v /C/Users/sebas_000/AppData/Local/lxss/home/martaver:/test alpine sh works

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, I think then that’s why I see suggestions to run something like this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

mount –bind /mnt/c /c

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

To create a “proxy” path

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You could do that to shorten the path above, albeit ugly

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Docker-compose and incorrect absolute paths for volumes · Issue #1854 · Microsoft/WSL

windows 10 build 1503.11 running Xenial 16.04.2 LTS with docker client 17.03.0 connecting to the hyper-V docker-for-windows daemon 17.03.1 and docker-compose 1.11.1 From within wsl things like dock…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Off to bed! Lmk how it goes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth tracking this issue here: https://github.com/cloudposse/geodesic/issues/199

Fix `$HOME` Mounting for WSL · Issue #199 · cloudposse/geodesic

what Mounting $HOME to /localhost is not working on WSL (Windows Shell for Linux) why Explain why this is a problem and what is the expected behavior. Explain why this feature request or enhancemen…

rohit.verma avatar
rohit.verma

@Erik Osterman (Cloud Posse) I have replaced kube-lego with cert-manager, https://github.com/nikiai/dev.niki.ai/blob/ae192f5bc7a4b1287b2d611267d200f15be845e4/aws/kops/helmfile.yaml#L303. In case you want to include this in geodesic

nikiai/dev.niki.ai

Contribute to dev.niki.ai development by creating an account on GitHub.

rohit.verma avatar
rohit.verma

Kube-lego is now depricated

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Awesome! Thanks… we shall incorporate it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Same annotations?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@rohit.verma @Max Moon what are your thoughts on moving helmfiles to a separate repo that can be rev’d separately?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Especially since they can now be decomposed to helmfile.d

rohit.verma avatar
rohit.verma

most likely I would recommend to use it in root-modules

rohit.verma avatar
rohit.verma

you can actually rename terraform-root-modules to root-modules

rohit.verma avatar
rohit.verma

please see my repo, i am kindly doing the same

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ve struggled with mixing the technologies

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Charts are separate

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Terraform is separate

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Seems like helmfiles should be separate

rohit.verma avatar
rohit.verma

I work on principal of environment, what are the things which aggregately define my environment will come in same repo in the end

rohit.verma avatar
rohit.verma

doesn’t matter from where it came

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Environments are to me the account repos

rohit.verma avatar
rohit.verma

we can create a new repo and in docker file add the same syntax

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Those tie everything together

rohit.verma avatar
rohit.verma

yes, environmet = account repo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Packages come from packages

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Terraform code from root modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Charts from charts.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Seems like clean separations no?

rohit.verma avatar
rohit.verma

I agree completely with you

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Cool I’ll track this in a separate issue

rohit.verma avatar
rohit.verma

what i meant is that in dockerfile Assume we have

COPY --from=terraform-root-modules /aws/tfstate-backend/ /conf/tfstate-backend/
COPY --from=charts /aws/basic /conf/basic (includes, dashboard etc...)
COPY --from=charts /aws/monitor /conf/monitor (includes, kube-prometheus etc...)
COPY --from=services /aws/services /conf/services

But when we are overriding anything e.g monitor, we would be overriding in dev.niki.ai only

rohit.verma avatar
rohit.verma

As I explained earlier, to view the root-modules charts in my ide, i end up copying all to my code only. On first setup I did copy from terraform root, but then I moved it to my repo

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ok, makes sense. I recommend your approach as well.

rohit.verma avatar
rohit.verma

also one thing I wanted to ask since started using helmfile, how can we sync only updated services. Helmfile sync always update all the services

rohit.verma avatar
rohit.verma

doesn’t it use any hashing technique for a release

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We use selectors extensively

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Doesn’t that mitigate it?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Basically, I never call sync without a selector

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I treat helmfile like a package manager

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There also helmfile diff

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I know @Daren loves this

rohit.verma avatar
rohit.verma

I saw you recommended upsert

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh yea, on that discussion

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No response yet though, right?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Proposal: Helmfile sync --delete removes any charts not referenced · Issue #194 · roboll/helmfile

This idea comes from the kubectl apply -f –prune where kubectl deletes any resources that aren't referenced. Helmfile should have an option for the sync command that checks what helm charts ar…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(For reference)

rohit.verma avatar
rohit.verma

yes

rohit.verma avatar
rohit.verma

I will start migrating things to helmfile.d, than will hash of individual file

rohit.verma avatar
rohit.verma

will share once done

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@alebabai is converting them

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Pr is undergoing testing

rohit.verma avatar
rohit.verma

i need to do for our micro-services

rohit.verma avatar
rohit.verma

is there a utility (shell 1 liners for this)? or you are doing manually

rohit.verma avatar
rohit.verma

i think yq on name array and echo in name files should do

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, it’s called vim ;)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No fun

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh, you mean for the testing?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That could work, but lots of envs in chamber also need to be set

rohit.verma avatar
rohit.verma

why we need to worry about chamber when migrating from helmfile to helmfile.d

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh we just use that to store the envs used by the files

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Migrating is actually risky

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Let me explain

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

helm —set does automatic type casting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If it sees a string that looks like a number it makes it an int

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If it sees a Boolean sting it makes it a Boolean

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So when converting to inline values it’s critical to maintain cast

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Errors won’t be obvious

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Subtle feature of charts might get accidentally disable if true is a string

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

While certain annotations require values to numbers to be strings

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So the helmfile ‘set’ declaration just called —set

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s why we care

rohit.verma avatar
rohit.verma

got it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth hows it going?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

See these slides? Wonder if it’s related to the HOME debacle

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:06:07 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
06:06:34 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Setup a Dev environment that feels like $HOME on Windows 10

Windows 10 allows you to run native Linux binaries with the WSL. Let’s see how we can use a good development environment for Vagrant and Docker using VMware Wo…

Sebastian Nemeth avatar
Sebastian Nemeth

This gave me an idea and I figure out a workaround

Detailed in https://github.com/cloudposse/geodesic/issues/199

Fix `$HOME` Mounting for WSL · Issue #199 · cloudposse/geodesic

what Mounting $HOME to /localhost is not working on WSL (Windows Shell for Linux) why Explain why this is a problem and what is the expected behavior. Explain why this feature request or enhancemen…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s awesome! Thanks for sharing

Sebastian Nemeth avatar
Sebastian Nemeth

@Erik Osterman (Cloud Posse) is there any update on when this or similar solution will be rolled into geodesic?

Sebastian Nemeth avatar
Sebastian Nemeth

We’d really rather just reference geodesic at the trunk than have to fork/build/push our own image

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Was hoping today, but still not happy with the current PR solution. It appears that depending on the method of installation or version of WSL that the home path varies.

Sebastian Nemeth avatar
Sebastian Nemeth

Ahhh that’s true. I’ve uninstalled WSL and reinstalled it on my system, so my home path might not even be the default one with current Windows.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Fix $HOME Variable on WSL by vadim-hleif · Pull Request #202 · cloudposse/geodesic

what $HOME path now is built dynamically (search wsl dir on windows, getting windows and linux users names) why By default $HOME refers on /usr/local/bin, but in case with WSL, docker installed…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s the current progress. A few more minor nitpicks to be solved and then we’ll merge.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We cannot test your path, however, so might not work

2018-07-29

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@rohit.verma anyone on your team using Geodesic on WSL?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Learn about the Windows Subsystem for Linux

Learn more about how the Windows Subsystem for Linux works.

2018-07-28

Sebastian Nemeth avatar
Sebastian Nemeth

Hey guys, few issues posted under root.cloudposse.co

Sebastian Nemeth avatar
Sebastian Nemeth

Blocker for me atm is geodesic doesn’t seem to be mounting ~/.aws/config correctly

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ping me this week and we’ll screenshare to get to the bottom of it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Checking

2018-07-26

Sebastian Nemeth avatar
Sebastian Nemeth

Hey guys - the cold start docs describe setting up aws-vault on your local machine, but it’s also available in the geodesic container. Does it have to be set up in both places, or does geodesic mount ~/.aws/config or something? Or is it best practice to only use aws-vault from geodesic?

sarkis avatar
sarkis

@Sebastian Nemeth that is correct, geodesic mounts ~/.aws/config: https://github.com/cloudposse/geodesic/blob/a430b746c88fb81be59db8b1a42ce4a38dc4a3bd/Dockerfile#L166

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth it’s only needed for locally when doing native development

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(e.g. docker compose)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. chamber exec app -- docker-compose up

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I agree though - the docs should be updated to make that clear.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you’re the second person in the last day to point that out.

Sebastian Nemeth avatar
Sebastian Nemeth

Glad I’m helping

2018-07-25

Sebastian Nemeth avatar
Sebastian Nemeth

Problems with cold_start doco:

Sebastian Nemeth avatar
Sebastian Nemeth

I’ll pin this here, and post fixes and small problems I find with the cold_start doco here…

Sebastian Nemeth avatar
Sebastian Nemeth

This line under Provision iam Project to Create root IAM role…

“Update the TF_VAR_root_account_admin_user_names variable in Dockerfile for the root account with your own values.”

Doesn’t make sense… in root.cloudposse.co repo, the root account admin user names are no longer kept in the Dockerfile, they’re in the attached terraform templates (it seems)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Sebastian Nemeth thanks. The docs show the older version, we updated the repos since then. We need to update the docs (will do it this week).

Sebastian Nemeth avatar
Sebastian Nemeth

Cool, I’ll wait for the updates then. Will keep posting what I find in this thread…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you can open an issue for that in the docs. And all other problems you find. Thanks

Sebastian Nemeth avatar
Sebastian Nemeth

ah

Sebastian Nemeth avatar
Sebastian Nemeth

gotcha

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Yea thanks, we see the open issues so we don’t forget to fix it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

For now, you need to update the names not in the Dockerfile, but in the .tfvars files

:--1:1
Sebastian Nemeth avatar
Sebastian Nemeth
02:20:04 PM

@Sebastian Nemeth pinned a message to this channel.

Arkadiy avatar
Arkadiy
02:49:30 PM

@Arkadiy has joined the channel

rohit.verma avatar
rohit.verma

hi, i am looking for strategy to use conf with local dev environment, the way i did the workaround is to manuall add dockerargs in wrapper as --volume=$(pwd)/aws:/conf). So when i run it within the repo root, it mounts my conf to wrapper and i can do changes in my IDE and sync it directly from geodesic shell

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrm

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so for local dev, i always cd /localhost

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and work from there

rohit.verma avatar
rohit.verma

the drawback is, I have to copy all standard modules to my repo, cause if I do COPY --from=terraform-root-modules /aws/tfstate-backend/ /conf/tfstate-backend/ the root modules were hidden

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

/localhost = $HOME <– host machine

rohit.verma avatar
rohit.verma

yes that I am aware of

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmm can you elaborate on what you want to achieve beyond IDE?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or why this is insufficieint with IDE.

rohit.verma avatar
rohit.verma

actually you are right with this, it just being habitual I ended up using the default root

rohit.verma avatar
rohit.verma

i just have too much nesting in my local so felt lazy about this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that is a challenge - also with a lot of docker inheritance it’s not a universal solution for all kind of debugging.

rohit.verma avatar
rohit.verma

/localhost/github/ops/new/dev.niki.ai

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i find it works really well with terraform-root-modules

rohit.verma avatar
rohit.verma

hmm

rohit.verma avatar
rohit.verma

but this still lacks 1 thing, not a major glitch, to work with root modules I have to switch to ~ but then to work with my modules I have to switch back to localhost

rohit.verma avatar
rohit.verma

can’t we force docker mount to show the content of container in the mount directory

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that would be nice. something that automatically drops you in the context of your IDE path

rohit.verma avatar
rohit.verma

if you open terminal within ide, it drops you there only

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically, just need it to pass the --workdir arg to docker

rohit.verma avatar
rohit.verma

yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or do a cd /.... upon entering the shell, perhaps with something in /etc/profile.d

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i don’t have a idea right now on what the convention would look like

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

open to suggestions

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

perhaps caller can set a GEODESIC_WORKDIR env?

rohit.verma avatar
rohit.verma

I think I didn’t follow you completely

rohit.verma avatar
rohit.verma

what would be advantage of GEODESIC_WORKDIR

rohit.verma avatar
rohit.verma

it would be placeholder for my localhost directory?

rohit.verma avatar
rohit.verma

to rephrase the question, it would be nice to see in IDE what is coming up from root-modules

rohit.verma avatar
rohit.verma

that way i have a collective view of what is my total infra, root+custom modules in one dir

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the GEODESIC_WORKDIR would be where you would get dropped in the geodeisc shell

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it would be used only if present.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that would spare the caller the need to remember to cd /localhost/dev/.......

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
raymondbutcher/remake

remake - A watcher tool for Make

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha i just came across that tool the other day.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s useful for go or say c++ projects. it will watch a directory and call a make target anytime it changes.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not really applicable in this situation, but thought i’d share anyways

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


collective view of what is my total infra, root+custom modules in one dir

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is really asking for an overlay filesystem

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i have seen fuse filesystems that will mimick this

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but i think that would be HIGHLY experimental

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

stepping out

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

I think @rohit.verma was asking about seeing the files coming from root-modules in the IDE. Basically on the host file system to see the files from the container. If you work with let’s say testing.cloudposse.co, now it’s completely empty in the IDE on the host, you can see the Terraform files only inside geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ohhhh - i see

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but in my case, I would do

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

GEODESIC_WORKDIR=/locahost/Dev/cloudposse/terraform-root-modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then start the container, and it would drop me in that folder where I iterate.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then in my IDE, I operate in /Users/erik/Dev/cloudposse/terraform-root-modules

2018-07-24

rohit.verma avatar
rohit.verma

hi, is anyone using build chart instructions from here https://docs.cloudposse.com/release-engineering/cicd-process/build-charts/

rohit.verma avatar
rohit.verma

it seems the latest build-harness don’t have target make helm/chart/build-all

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov

rohit.verma avatar
rohit.verma
cloudposse/build-harness

build-harness - Collection of Makefiles to facilitate building Golang projects, Dockerfiles, Helm charts, and more

rohit.verma avatar
rohit.verma

i tried with latest, 0.6.13, 0.6.12

Yoann avatar
Yoann

Thanks @Erik Osterman (Cloud Posse) that seems pretty straightforward I wilkl have a look at it when I achieve to deploy my kops cluster. ATM I could not install the staging env due to VPC conflicts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what kind of VPC conflicts?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haven’t encountered that before.

Yoann avatar
Yoann

Well I tried to run both kops-aws-platform and backing-services from the staging image and it complains on module.kops_vpc_peering.module.kops_metadata.data.aws_vpc.kops: data.aws_vpc.kops: no matching VPC found

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Aha, @Andriy Knysh (Cloud Posse) will tell you how to fix it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it comes down to a bad env setting for the cluster name

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think he’s working on a fix to docs for that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i ran into this today

Sebastian Nemeth avatar
Sebastian Nemeth

Hey guys, is there any way to re-use an existing AWS linked account to initialize testing, audit, staging and prod geomodules?

Sebastian Nemeth avatar
Sebastian Nemeth

I have a bunch of old accounts that would be cleaner to re-name and re-purpose.

Igor Rodionov avatar
Igor Rodionov
04:26:11 PM

@Igor Rodionov has joined the channel

alebabai avatar
alebabai
04:26:11 PM

@alebabai has joined the channel

evan avatar
evan
04:26:11 PM

@evan has joined the channel

mcrowe avatar
mcrowe
04:26:11 PM

@mcrowe has joined the channel

sarkis avatar
sarkis
04:26:11 PM

@sarkis has joined the channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes you should be able to reuse them.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You can use ‘terraform import’ command with the resource name and account is to import state

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth

2018-07-23

Yoann avatar
Yoann

Can we automate geodesic terraform to build and destroy a cluster in a CI env ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Yoann We have this on our roadmap for this quarter, but do not have any reference architectures for how to do it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

if you want to get a headstart on it, i can walk you through how we would approach it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

a geodesic module is just a container. so if you run that container in a context which provides either the AWS credentials as environment variables (AWS_ACCESS_KEY_ID, and AWS_SECRET_ACCESS_KEY) to the container or where there’s an AWS instance profile with sufficient access credentials, then you can just run the commands you would normally run inside the container

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

something like:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
docker run --env-file /dev/shm/secrets.env --workdir /conf/myproject/ mycompany/staging.mycompany.com:1.2.3 chamber exec myproject -- terraform apply
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s saying start a docker container with environment variables from a file called secrets.env

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

change directory to /conf/myproject

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

run chamber exec myproject to get SSM secrets from myproject

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and then run terraform apply

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so you see, deploying infrastructure with CI/CD is not really much different than any deployment process, since it’s all bundled in a container.

2018-07-21

Sebastian Nemeth avatar
Sebastian Nemeth
Thanks very much! My q was that each module sets up a hosted zone e.g. [root.company.com> and prod.company.com… how and where would we set up say, CNAME records that map company.com to <http://prod.company.com prod.company.com](http://root.company.com)?
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am afk so cannot really share any details easily, but you are correct. One geodesic module per AWS account. One delegated dns zone per account. The root account is responsible for delegating the zones, but first need to create the zones

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/testing.cloudposse.co

testing.cloudposse.co - Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you look for account-dns that is where the sub account provisions a zone

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/root.cloudposse.co

root.cloudposse.co - Example Terraform Reference Architecture for Geodesic Module Parent (“Root”) Organization in AWS.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you look for root-dns that is where the delegation happens

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So for a cold start, there are a number steps that must be followed in order to get everything linked up.

Sebastian Nemeth avatar
Sebastian Nemeth

Holy shit guys, it’s Saturday!

Sebastian Nemeth avatar
Sebastian Nemeth

V. much appreciate your response on the weekend. Was not expecting that!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/root.cloudposse.co

root.cloudposse.co - Example Terraform Reference Architecture for Geodesic Module Parent (“Root”) Organization in AWS.

cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Haha! We’re mostly around in some capacity. :)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Then we provision all other accounts (testing, staging, prod, etc)
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) has provisioned this stack dozens of times

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Then we come back to root and provision Zone delegation (Name servers)
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

^ that’s a high level description to provision dns on all accounts

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

please ask more questions

Sebastian Nemeth avatar
Sebastian Nemeth

Okay, so parent zone = company.com…? So then what’s root hosted zone for?

I mean, I get that each module has its own zone and the parent delegates to the modules… but then why a separate root hosted zone?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

One more consideration I don’t know if we call out in our docs is you should have an infrastructure domain that is separate from your branded “vanity” domain

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The infrastructure domains are for service discovery and should be canonical, while vanity domains how you expose services publicly.

Sebastian Nemeth avatar
Sebastian Nemeth

what’s diff between parent and root?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

good question @Sebastian Nemeth

Sebastian Nemeth avatar
Sebastian Nemeth

I try

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we use [xxxxx.cloudposse.co](http://xxxxx\.cloudposse\.co) on all accounts so it’s mostly for:

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. To have consistent naming
:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, for root account not strictly required, but we do it for consistency. To date, we haven’t had to provision any sub domains off of the root account since all it handles is identity. But if for some reason we did need to, it would be on that sub domain.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Parent zone [company.com](http://company\.com) is just DNS name, but we name our AWS accounts root, prod, staging, etc.
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so mostly to have the same naming on all accounts, and also the same naming on all AWS resources (e.g. roles cp-root-admin, cp-testing-admin)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and AWS profiles

Sebastian Nemeth avatar
Sebastian Nemeth

okay makes sense… so the logical hierarchy is…

parent

  • root
  • audit
  • prod
  • staging
  • testing
Sebastian Nemeth avatar
Sebastian Nemeth

etc

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

for DNS yes

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

from the point of view of AWS organization:

Sebastian Nemeth avatar
Sebastian Nemeth

so then, is there ever a use case for actually using root’s dns records?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

root (Organization)

  • audit
  • prod
  • staging
  • testing
Sebastian Nemeth avatar
Sebastian Nemeth

mmhmmm

Sebastian Nemeth avatar
Sebastian Nemeth

okay, cool these structures would really help in the documentation

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Agree!

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks, we’ll add that, nice point

Sebastian Nemeth avatar
Sebastian Nemeth

And maybe a line about the concept that each module has its own xxxx.cloudposse.co and parent delegates from the TLD

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yep

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we have something like that in the docs, but agree, it’s not easy to follow unless we make big notes about that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am excited about concentrating these questions in channel. We have a lot of it in #announcements, but with so much chatter it’s lost in the noise. Having it in here will help in the feedback loop, so we can review discussions and update docs.

:--1:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

also @Sebastian Nemeth if you have some input, you can create issues in any of the repos or in the docs

Sebastian Nemeth avatar
Sebastian Nemeth

Gotcha

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

not sure if you saw that in the other channel, so here it is again FIY

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
We just released Cloud Posse reference architectures:

<https://github.com/cloudposse/terraform-root-modules> - Collection of Terraform root module invocations for provisioning reference architectures
<https://github.com/cloudposse/root.cloudposse.co> - Terraform Reference Architecture of a Geodesic Module for a Parent ("Root") Organization in AWS
<https://github.com/cloudposse/prod.cloudposse.co> - Terraform Reference Architecture of a Geodesic Module for a Production Organization in AWS
<https://github.com/cloudposse/staging.cloudposse.co> - Terraform Reference Architecture of a Geodesic Module for a Staging Organization in AWS
<https://github.com/cloudposse/dev.cloudposse.co> - Terraform Reference Architecture of a Geodesic Module for a Development Sandbox Organization in AWS
<https://github.com/cloudposse/audit.cloudposse.co> - Terraform Reference Architecture of a Geodesic Module for an Audit Logs Organization in AWS
<https://github.com/cloudposse/testing.cloudposse.co> - Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS

They show how we provision AWS accounts and what Terraform modules we use.
Complete description is here <https://docs.cloudposse.com/reference-architectures>

We will be improving the repos and the docs. Your input and PRs are very welcome.
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andriy Knysh (Cloud Posse) sorry didn’t fill you in - @Sebastian Nemeth is with the startup in Helsinki. They found us by way of our reference architectures and modules.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Now they are rolling up their sleeves

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

very nice

Sebastian Nemeth avatar
Sebastian Nemeth

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Sebastian Nemeth we’ll be glad to help and answer any questions

Sebastian Nemeth avatar
Sebastian Nemeth

I’m sure we’ll have plenty - feel very lucky to have you guys on the other side of the line. This stuff is painful to get right.

:100:2

2018-07-20

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
08:16:52 PM

@Andriy Knysh (Cloud Posse) has joined the channel

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@Sebastian Nemeth @Yoann if you have questions about geodesic and https://docs.cloudposse.com/reference-architectures, please ask them in this channel

:--1:1
:100:1
Yoann avatar
Yoann

Thanks

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

btw, welcome

Sebastian Nemeth avatar
Sebastian Nemeth
08:21:28 PM

@Sebastian Nemeth has joined the channel

Yoann avatar
Yoann
08:21:28 PM

@Yoann has joined the channel

tamsky avatar
tamsky
08:41:26 PM

@tamsky has joined the channel

2018-07-05

jonathan.olson avatar
jonathan.olson
09:01:36 PM

@jonathan.olson has joined the channel

    keyboard_arrow_up