#geodesic (2019-04)

geodesic https://github.com/cloudposse/geodesic

Discussions related to https://github.com/cloudposse/geodesic Archive: https://archive.sweetops.com/geodesic/

2019-04-29

SweetOps #geodesic avatar
SweetOps #geodesic
04:00:07 PM

There are no events this week

Cloud Posse avatar
Cloud Posse
04:03:52 PM

Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7).

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Mar 20, 2019 11:30AM.
Add it to your calendar
zoom https://zoom.us/j/684901853
slack #office-hours (our channel)

2019-04-26

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Erik Osterman (Cloud Posse) I did not delete any accounts. Not only are there lots of hoops to go through, but also deleting accounts take a long time (90 days, I think) and the account email address remains permanently associated with the “deleted” account, and therefore cannot be used on a new account.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Mike Pfaffroth What I recommend is to reuse the created accounts. Run make root up to the point where terraform complains that it cannot create the accounts, then import the accounts into Terraform using terraform import.

Mike Pfaffroth avatar
Mike Pfaffroth

interesting… I will try that approach. Appreciate the tip.

Mike Pfaffroth avatar
Mike Pfaffroth

hm… even on a brand new account I am not able to create these- it always dies when trying to create the others:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 7de7c003-6846-11e9-9002-752f3d4639b5
Mike Pfaffroth avatar
Mike Pfaffroth

I changed the email address as well, just to make sure it wasn’t trying to create ones that were already there

Mike Pfaffroth avatar
Mike Pfaffroth

I am the root account:

➜  reference-architectures git:(master) ✗ aws sts get-caller-identity
{
    "UserId": "1redacted7",
    "Account": "1redacted7",
    "Arn": "arn:aws:iam::1redacted7:root"
}
Mike Pfaffroth avatar
Mike Pfaffroth

is there a special permissions I need? or is there a documentation process for signing up for an account in the right way within an organization?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Sounds like maybe yuo’re not provisioning from the top-level payer account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is the “root” account

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you are already in a subaccount, AWS does not let you create more subaccounts

Mike Pfaffroth avatar
Mike Pfaffroth

so if I understand it correctly I need to sign up for a brand new account, and then geodesic creates organizations and IAM users inside that account for each environment @Erik Osterman (Cloud Posse)?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, more or less. you can technically use an existing root level account too, but you run the risk of conflicting resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Our reference-architectures are what we use in our consulting to stand up new accounts for our customers. We have a very specific focus.

Mike Pfaffroth avatar
Mike Pfaffroth

yup- totally understood. just wanted to make sure I understood how it worked. Thanks for your help!

:--1:1

2019-04-25

Mike Pfaffroth avatar
Mike Pfaffroth

Hi- if I have started the “quick start” with example.io, had an issue and ran make clean (after terraform had created some things), how can I most easily “reset”?

Mike Pfaffroth avatar
Mike Pfaffroth

I want to run terraform destroy essentially

Mike Pfaffroth avatar
Mike Pfaffroth

I am getting, for example-

* module.staging.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b93365b-67b0-11e9-808b-3fc6f3a60880
* module.dev.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b9335cd-67b0-11e9-b64d-9b693972455f
* module.prod.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b92998f-67b0-11e9-b23d-dd34fe4bf6c3
* module.audit.aws_organizations_account.default: 1 error(s) occurred:

* aws_organizations_account.default: Error creating account: AccessDeniedException: You don't have permissions to access this resource.
	status code: 400, request id: 5b93362c-67b0-11e9-a70f-ffba4d75ead7
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Mike Pfaffroth recommend to start here instead: https://github.com/cloudposse/reference-architectures

cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Mike Pfaffroth avatar
Mike Pfaffroth

@Erik Osterman (Cloud Posse) I am running that (I’m in the make root stage)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so terraform destroy will fail when trying to delete the accounts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can destroy the accounts after jumping through a lot of hoops (password reset, login, accept t&c, etc - for each child account)

Mike Pfaffroth avatar
Mike Pfaffroth

right- I guess what I was trying to say is- is that process documented anywhere?

Mike Pfaffroth avatar
Mike Pfaffroth

I know I’m in a bad position, I am trying to find a similar checklist

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha! yes, good question

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

no, we have not documented it. @Jeremy (Cloud Posse) would it be relatively quick for you to open an issue against the reference-architectures that describes the process you took? It’s been a long enough time that I can’t recall every step.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i just remember doing the password reset on every child account so that I could log in

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then i think there was some kind of terms & conditions I had to accept (checkbox/click)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and after that, I think it’s sufficient enough to retry the terraform destroy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that said, I don’t think it’s possible to reuse the same email address on round 2

github-check-mark1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@oscarsullivan_old or @Jan or @Josh Larsen might have some more recent recollection

2019-04-24

tamsky avatar
tamsky

I was experimenting today with terragrunt vs vanilla terraform init and wanted to point out a few important differences between these two similar methods:

  • terragrunt { source = "<blueprint source>" } and
  • terraform init -from-module=<blueprint source>

terragrunt allows the use of:

  • [override.tf](http://override\.tf) [1] files in the CWD
  • “mix-in” files in the CWD (*.tf files that do not match blueprint filenames), useful for adding one-off resources to a blueprint
  • “upstage” files in the CWD (*.tf files that do match a blueprint filename), these replace the contents of a blueprint source file of the same name), useful for removing blueprint resources This is due to the fact that terragrunt init creates a tmp dir, clones the SOURCE to the tmp dir, and then copies/overwrites (aka “upstages”) all files in the CWD to the tmp dir (overwriting any duplicates).

Whereas terraform init -from-module= requires that the CWD contains zero files matching *.tf or *.tf.json. This prevents all the above techniques: overrides, mix-ins, and upstage files.

Has anyone thought about how to support/implement the override, mix-in and upstage patterns without the use of terragrunt?

[1] https://www.terraform.io/docs/configuration-0-11/override.html

Override Files - 0.11 Configuration Language - Terraform by HashiCorp

Terraform loads all configuration files within a directory and appends them together. Terraform also has a concept of overrides, a way to create files that are loaded last and merged into your configuration, rather than appended.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

See our overrides strategy

Override Files - 0.11 Configuration Language - Terraform by HashiCorp

Terraform loads all configuration files within a directory and appends them together. Terraform also has a concept of overrides, a way to create files that are loaded last and merged into your configuration, rather than appended.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We support that too, only it’s explicit

tamsky avatar
tamsky

is that the overrides/ directory ?

tamsky avatar
tamsky
cloudposse/root.cloudposse.co

Example Terraform Reference Architecture for Geodesic Module Parent (“Root” or “Identity”) Organization in AWS. - cloudposse/root.cloudposse.co

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/root.cloudposse.co

Example Terraform Reference Architecture for Geodesic Module Parent (“Root” or “Identity”) Organization in AWS. - cloudposse/root.cloudposse.co

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not “ideal” imo, but identical strategy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We copy to . Rather than to a “cache” folder

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Terraform just does a basic check for any .tf file

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

IMO that should be optional

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Perhaps a -force-copy arg

tamsky avatar
tamsky

I’m trying to grok the two init calls to terraform there

tamsky avatar
tamsky

the first one I’m guessing makes use of TF_CLI_ARGS_init, and the second one does what?

tamsky avatar
tamsky

loads any new modules that were included as a result of the new files being cped ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s self mutilating code

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

At first you init with no overrides which pulls down the source

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then we overlay our files which changes the source

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then we might introduce new modules

:100:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So we need to rein it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Re init

tamsky avatar
tamsky

yup ok got it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The second one would fail if we tried to init again from a remote module

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So we null it out

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We could also unset it from the env

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Both ok

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Sitting down for early dinner

tamsky avatar
tamsky

ok thanks for the explainer

2019-04-23

Josh Larsen avatar
Josh Larsen

question: let’s say my terraform-root-modules is a private repo… how can i get my github keys into geodesic so the terraform init will be able to retrieve the modules?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Josh Larsen you have a few options

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you running this under CI/CD or as a human?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then we add the ssh public key as a deploy key to the root modules repo

Josh Larsen avatar
Josh Larsen

ideally i’d like to do both

Josh Larsen avatar
Josh Larsen

but let’s start with human for now

Josh Larsen avatar
Josh Larsen

i’m thinking something like cat /localhost/.ssh/id_rsa | ssh-add - would do it, but would just have to do that every time i started the shell. or is there a better way?

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

By default, Geodesic will start an ssh-agent and add ~/.ssh/id_rsa at startup. You can also write your own scripts to be run at startup. See https://github.com/cloudposse/geodesic/pull/422

Enable run-time customization by Nuru · Pull Request #422 · cloudposse/geodesic

what In addition to some small cleanups and additions, provide a capability for users to customize Geodesic at runtime. why Because people vary, their computers vary, what they are trying to accomp…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you on a mac or linux?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

on linux, we mount your ssh agent socket into the container

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we can’t do that on a mac

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the other option is to store your ssh key in SSM

Josh Larsen avatar
Josh Larsen

i’m on mac… but i think i get the idea. thanks.

2019-04-22

SweetOps #geodesic avatar
SweetOps #geodesic
04:00:05 PM

There are no events this week

Cloud Posse avatar
Cloud Posse
04:01:03 PM

Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7).

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Mar 20, 2019 11:30AM.
Add it to your calendar
zoom https://zoom.us/j/684901853
slack #office-hours (our channel)

2019-04-17

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Office Hours Today from 11:30 AM to 12:20 PM at https://zoom.us/j/684901853

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(PST)

2019-04-16

Eugene Korekin avatar
Eugene Korekin

Hello, everyone. I am trying to follow the cold-start procedure described here https://docs.cloudposse.com/reference-architectures/cold-start/ but it seems that it is outdated, right now I have an issues trying to create users, could anybody help me?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Eugene Korekin sorry for the troubles!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, the cold-start docs are quite out of date and refer to a older implementation.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

our current process is being kept up to date here: https://github.com/cloudposse/reference-architectures

cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
geodesic

SweetOps is a collaborative DevOps community. We welcome engineers from around the world of all skill levels, backgrounds, and experience to join us! This is the best place to talk shop, ask questions, solicit feedback, and work together as a community to build sweet infrastructure.

Eugene Korekin avatar
Eugene Korekin

thanks, Eric! I’ll take a look

Eugene Korekin avatar
Eugene Korekin

@ericthortonjohnson could you please tell me what would be the best approach if I already created the master aws account in aws orgs? I think I can’t remove it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Eugene Korekin hiding

Eugene Korekin avatar
Eugene Korekin

I used the older implementation to create it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

soooo you have a couple of options.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

did you create those AWS accounts by hand or using terraform?

Eugene Korekin avatar
Eugene Korekin

I used the existing account as the base for the procedure described in the old cold start, the master account referring to the existing account was created in aws orgs in the process

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, so option (1) is to continue going down the path you were. there’s nothing wrong with that per say.

Eugene Korekin avatar
Eugene Korekin

so the master account was created via terraform, but it already has some ec2 instances etc

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so there’s no one way to bring up this infrastructure. the idea is the geodeisc is just a run time environment.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what you do inside of it is entirely open-ended.

Eugene Korekin avatar
Eugene Korekin

I stuck on that path, as the user creation step requires some contents present in ssm and I cannot find how to create this content with the old procedure

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmmm so mostly likely, the user creation stuff will then need to use older versions of the modules that pre-date the SSM dependencies

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i can’t say what version that would be.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

to start fresh though, would involve the following:

Eugene Korekin avatar
Eugene Korekin
05:51:57 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  1. password reset on each account
Eugene Korekin avatar
Eugene Korekin

here is the error I have

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  1. login to each account, accept t&c
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  1. terraform destroy accounts
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then you have a clean base

Eugene Korekin avatar
Eugene Korekin

is there a way to somehow import existing accounts?

Eugene Korekin avatar
Eugene Korekin

they are already in use

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can import existing accounts, however @Jeremy (Cloud Posse) recently went through this with one of our current clients and if every parameter doesn’t match 100% it wants to recreate them

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(jumping on a )

Eugene Korekin avatar
Eugene Korekin

I see, is there a way to only create a new account (let’s say ‘testing’) and leave the master one as it is (including all the existing users)?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, you can probabblby do that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

use accounts_enabled flag to only select testing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

e.g. get your feet wet with the system

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

there’s a lot of moving pieces, so getting your hands dirty with one account would be a good idea

Eugene Korekin avatar
Eugene Korekin

so, do I just need to skip ‘make root’ and start right from ‘make children’?

Eugene Korekin avatar
Eugene Korekin

I’ve just tried it, and it doesn’t seem to work, the make command doesn’t provide any output

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

You need to set up your configs/root.tfvars with all the right configuration (hopefully it’s reasonably self-explanatory) including accounts_enabled limited to the accounts you want to create. Then set yourself up with AWS admin credentials in your “root” AWS account and run make root. This creates the children accounts and sets up roles and network CIDRs and so forth and creates a bootstrap user you will use for the rest of the configuration.

Eugene Korekin avatar
Eugene Korekin

but in won’t change anything inside the root account in the process, right?

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

It will definitely change things inside the root account. The root account is where your users are created and roles for the children account are created and DNS entries for children DNS subdomains are created.

Eugene Korekin avatar
Eugene Korekin

I don’t want to change any of the existing users, won’t it be possible to proceed without that?

Eugene Korekin avatar
Eugene Korekin

in other words, would it be possible to use the existing user entries in the master account and reuse then in the children ones?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so our reference architectures don’t provide that level of configurability b/c the number of permutations is insurmountable

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

however, all of our terraform modules are compose of other modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can pick and choose exactly what you want to use

2019-04-15

Abel Luck avatar
Abel Luck

ah ok, that explains what i’m seeing

Abel Luck avatar
Abel Luck

on macos it just works, but on our linux workstations the host mounted files are written with root:root

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Use bind mounts

Bind mounts have been around since the early days of Docker. Bind mounts have limited functionality compared to volumes. When you use a bind mount, a file or directory on…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I was hoping they supported uid/gid mapping in bind-mount (by now), but it’s not supported

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we could technically drop to a non-privileged user in geodesic, but haven’t optimized for that.

SweetOps #geodesic avatar
SweetOps #geodesic
04:00:02 PM

There are no events this week

2019-04-14

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Geodesic installs a wrapper script to run the container, which launches the container with something like (but not exactly, and with other options) docker run -it --privileged --volume=$HOME=/localhost. The shell inside the Geodesic container is run as root. The permissions mapping is handled by Docker. I use docker-machine on my Mac, so everything Docker runs runs as me and the root user inside the Docker container has the same permissions as I do on the host. Files created on the host from Geodesic are owned by me and files I cannot read cannot be read from inside Geodesic.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think the behavior is slightly different on linux, where UID/GID within the container are preserved on the host machine, but what @Jeremy (Cloud Posse) describes is correct on OSX.

Alex Siegman avatar
Alex Siegman

This has actually caused a problem with us on our build server. The Jenkins user can no longer clean up some of the workspaces because files end up getting owned by root after unit testing and such =( On my backlog to fix that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Hrmmm okay I can help address that. So when we run geodesic containers in a headless fashion (e.g. cicd or Atlantis), we never use the wrapper script. We always run it in the “native” way for that platform. So for ECS it’s a task and for Kubernetes it’s a pod in a deployment. We never mount localhost, which is only ever recommended for local development and not the general workflow.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If localhost is not mounted the host machine is always insulated from these kinds of errors.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Keep in mind we always build an image that uses geodesic as it’s base and we don’t run geodesic directly

Alex Siegman avatar
Alex Siegman

Ah, I meant the more generic problem of root files inside a container against a local mount becoming root owned files on the host operating system. This isn’t even using geodesic. Though, a lot of our engineers here do use linux as their day-to-day O/S so it would affect them as well.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

On linux, host permissions for processes running in Docker containers should be managed with user namespaces: https://docs.docker.com/engine/security/userns-remap/

Isolate containers with a user namespace

Linux namespaces provide isolation for running processes, limiting their access to system resources without the running process being aware of the limitations. For more information on Linux namespaces, see Linux…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The awkward thing though is I think we want to be able to be root in the geodesic container while developing locally

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but we want files owned by the host user

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

user namespaces give you that: it maps whatever UIDs you user in Docker to whatever you want on the host

oscarsullivan_old avatar
oscarsullivan_old

Yeh ubuntu runs as root and sets files as root:root

2019-04-13

Abel Luck avatar
Abel Luck

how does geodesic handle issues of uid/gid when mounting dirs from the host?

Abel Luck avatar
Abel Luck

could someone point me to the code where that happens?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It does nothing with uid/gid mapping

2019-04-12

2019-04-11

raehik avatar
raehik

Question I forgot I had yesterday: I know that Geodesic opens a port (range?) for handy kubectl port-forwards. How do I use it, and what version was it released in? (in the last month, or older?)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, that’s supported - it’s been in the container for years

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

there’s an env inside the container that contains the port number

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Document Kubectl Proxy in Geodesic · Issue #428 · cloudposse/docs

what We port map a random port into the geodesic container This port is what should be used for proxying kubectl proxy –port=${KUBERNETES_API_PORT} –address=0.0.0.0 –accept-hosts=&#39;.*&#39; wh…

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That said, I think we should rename the port to be more generic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so it can be repurposed/overloaded

raehik avatar
raehik

Thanks, I was looking for $GEODESIC*

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, that would make more sense

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think we should rename it to GEODESIC_PORT

raehik avatar
raehik

that’s exactly what I thought it was

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

what’s confusing is in the wrapper we call it one thing (GEODESIC_PORT) and I guess we end up renaming it in the container

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you open an issue on geodesic, we’ll track it and fix that

raehik avatar
raehik

Will do, cheers!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
GitOps with Terraform on Codefresh (Webinar)

Infrastructure as code, pipelines as code, and now we even have code as code! =P In this talk, we show you how we build and deploy applications with Terraform using GitOps with Codefresh. Cloud Posse is a power user of Terraform and have written over 140 Terraform modules. We’ll share how we handl

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s how we used geodesic with Codefresh to achieve GitOps with terraform on Codefresh

joshmyers avatar
joshmyers

What happened with Atlantis?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re still using it, but customers have asked us to use codefresh instead

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

… so it’s one system they know and understand

:100:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in the end, I think we were able to reproduce a lot of what atlantis does. Still need better locking mechanisms and support of CODEOWNERS for blocking apply

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Create new S3 Bucket by osterman · Pull Request #75 · cloudposse/testing.cloudposse.co

what Demo of adding a new user bucket why GitOps rocks! =)

2019-04-10

chrism avatar
chrism

If you add sops to the geodesic RUN wget <https://github.com/mozilla/sops/releases/download/3.2.0/sops-3.2.0.linux> -O /usr/bin/sops && chmod +x /usr/bin/sops and set the env vars

RUN curl <https://keybase.io/user/pgp_keys.asc> | gpg --import 
ENV SOPS_KMS_ARN="arn:aws:kms:xx-xxx-x:xxxx:key/xxx-xxx-xxx-xxx-xxxx"
ENV SOPS_PGP_FP="APGPKEY,ANOTHERPGPKEY,ETC"

You can use sops encryption before pushing files into storage so they’re only un-encrypted within the container during use

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I believe we provide a sops package

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(so in fact, sops ships with geodesic)

chrism avatar
chrism

You know… Didnt know that

chrism avatar
chrism

chrism avatar
chrism

it’s pretty good; we use it alongside sopstool

chrism avatar
chrism

for keeping our tls certs secure in storage

chrism avatar
chrism

im now going to go and remove my manually pulling in sops

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, if our sops is not current, feel free to submit PR against cloudposse/packages to update it.

:--1:1
SweetOps #geodesic avatar
SweetOps #geodesic
06:00:01 PM

2019-04-09

Josh Larsen avatar
Josh Larsen

i’ve noticed the build-harness has a lot of cloudposse references… is forking this repo also recommended when getting setup with geodesic? it seems to be a dependency and affects commands like readme generation and such.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Josh Larsen yes/no - I think it’s a great idea to fork it for your own org’s needs

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that said, there’s no easy way to take advantage of your fork in our repos

oscarsullivan_old avatar
oscarsullivan_old

Youd have to fork it to use your fork of the readme generatie for example. The readme generator references CP a lot

2019-04-08

SweetOps #geodesic avatar
SweetOps #geodesic
04:00:03 PM

There is 1 event this week

jober avatar
jober

So if I wanted to get starting trying to use geodesic where is the best place to start. I see lots of information about and references to geodesic and reference architectures, just wonder if there is a certain order to follow. Any info would be helpful, thanks

Alex Siegman avatar
Alex Siegman

@oscarsullivan_old recently went through this, made some docs https://github.com/osulli/geodesic-getting-started

osulli/geodesic-getting-started

A getting-started guide for Cloud Posse’s Geodesic. - osulli/geodesic-getting-started

Alex Siegman avatar
Alex Siegman

it’s on my list to investigate using geodesic, i just haven’t gotten there yet~

jober avatar
jober

Awesome thanks so much! I’ll give that a look. Really appreciate that

Alex Siegman avatar
Alex Siegman

The 20 foot view is, you make a dockerfile based off geodesic with some specific stuff in it and then you use the resultant container as your “shell” for the given environment you built it for, if my understanding is correct.

jober avatar
jober

Thats helpful to keep in mind

jober avatar
jober

Im going to document my journey as a noob with this and see if I can come out with some notes/docs for setting this up

oscarsullivan_old avatar
oscarsullivan_old

Yep AMA on Geodesic, will try to answer

oscarsullivan_old avatar
oscarsullivan_old

also tune in on Wednesday and happy to answer and demo then

oscarsullivan_old avatar
oscarsullivan_old

find time in #geodesic

jober avatar
jober

There is 1 event this week

jober avatar
jober

correct?

oscarsullivan_old avatar
oscarsullivan_old

jober avatar
jober

ty

oscarsullivan_old avatar
oscarsullivan_old

That’s local to your time

jober avatar
jober

Awesome!

oscarsullivan_old avatar
oscarsullivan_old

BST

jober avatar
jober

was my next question haha

oscarsullivan_old avatar
oscarsullivan_old

I’ll try and revisit my PR

oscarsullivan_old avatar
oscarsullivan_old

but maybe read this instead of the master branch readme

oscarsullivan_old avatar
oscarsullivan_old
Update getting-started guide by osulli · Pull Request #1 · osulli/geodesic-getting-started

What Update the guides with clearer examples Add Example project that I actually use with Geodesic and Terraform Why Several more weeks worth of experience using the tools Some clear errors in t…

oscarsullivan_old avatar
oscarsullivan_old

it was written 2 weeks later I think

oscarsullivan_old avatar
oscarsullivan_old

(which is p significant at my rate)

oscarsullivan_old avatar
oscarsullivan_old

I also share a TF project that I literally use

jober avatar
jober

Awesome

oscarsullivan_old avatar
oscarsullivan_old

that shows you HOW to use geodesic

jober avatar
jober

thanks for all the info

oscarsullivan_old avatar
oscarsullivan_old

how to leverage it

oscarsullivan_old avatar
oscarsullivan_old

and have one tf project for say your API that can be used for dev/staging/prod without duping files etc

oscarsullivan_old avatar
oscarsullivan_old

np, catch you wednesday

jober avatar
jober

see you then!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @oscarsullivan_old !!

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Note that https://github.com/cloudposse/reference-architectures though a bit hard in itself to grasp, shows how Geodesic was designed to be used. Reference Architectures will actually generate your Geodesic Docker container source repos for you.

cloudposse/reference-architectures

Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

oscarsullivan_old avatar
oscarsullivan_old

@Jeremy (Cloud Posse) a nice feature for reference-architectures would be to just setup the geodesic modules and not affect any AWS accounts

oscarsullivan_old avatar
oscarsullivan_old

I give it my existing account IDs and any other key details like root account ID

oscarsullivan_old avatar
oscarsullivan_old

and it generates the geodesic module repos

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

reference-architectures is very proscriptive, and its primary purpose is to solve the cold-start problem and get you up and running on AWS quickly, starting from nothing. Importing your existing infrastructure and making sense of it automatically is an entirely different concept and a whole other project.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I pretty much agree with Jeremy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s a realllllly hard problem to “generalize” a solution where we are not in control of the configuration

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(maybe one day!)

oscarsullivan_old avatar
oscarsullivan_old

No right I get that. But ref arch does this already right in a linear step? It builds the modules then sets up the accounts or is it all intertwined?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It is linear

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s certainly possible (just not a priority yet)

oscarsullivan_old avatar
oscarsullivan_old

That makes sense. Perhaps you could point me to the file that triggers it.. although I imagine it is the make file

2019-04-05

Jan avatar

hey hey

Jan avatar

danyone know why kops is still 1.10.* in geodesic?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No reason - just haven’t received your PR :-)

Jan avatar

haha :–1:

Jan avatar

We are almost out of the build and migrate madness

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’ve released our version of shared vpc for kops

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We are testing 1.12

Jan avatar

:–1:

Jan avatar

rather than 1.11.0

oscarsullivan_old avatar
oscarsullivan_old

which file??

2019-04-04

oscarsullivan_old avatar
oscarsullivan_old

So should I adjust my calendar invites from 6:30 PM to 7:30PM?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I am not sure :-)

2019-04-03

SweetOps #geodesic avatar
SweetOps #geodesic
06:00:02 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m going to hang out in this zoom for a little bit in case anyone has any questions.

oscarsullivan_old avatar
oscarsullivan_old

Oooh it was later today doh

oscarsullivan_old avatar
oscarsullivan_old

Couldnt make the 6:30 one!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(fwiw, we had daylight savings in the US on March 10)

2019-04-02

Pablo Costa avatar
Pablo Costa

Hello,

I’m rebuilding some packages of the cloudposse/packages container and getting this error: mktemp: failed to create directory via template ‘../../tmp/tmp.XXXXXXXXXX’: No such file or directory

I could observe that the problem is related to this file included at makefile: “/packages/tasks/Makefile.apk”

… export APK_TMP_DIR := $(realpath $(shell mktemp -p ../../tmp -d)) …

As I wasn’t interested in apk packages I’ve just erased this file: echo > /packages/tasks/Makefile.apk to not get errors during the build process, but I thing that It worth’s be mentioned this problem, just to check if I doing something wrong

/packages # echo “1.14.0” > /packages/vendor/kubectl/VERSION /packages # make -C install kubectl make: Entering directory ‘/packages/install’ make[1]: Entering directory ‘/packages/vendor/kubectl’ mktemp: failed to create directory via template ‘../../tmp/tmp.XXXXXXXXXX’: No such file or directory curl –retry 3 –retry-delay 5 –fail -sSL -o /packages/bin/kubectl https://storage.googleapis.com/kubernetes-release/release/v1.14.0 /bin/linux/amd64/kubectl && chmod +x /packages/bin/kubectl make[1]: Leaving directory ‘/packages/vendor/kubectl’ make: Leaving directory ‘/packages/install’

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks for reporting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Pablo Costa can you open an issue for that?

Pablo Costa avatar
Pablo Costa

Sure. It’s done.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ll have @Maxim Mironenko (Cloud Posse) look into it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for now, maybe use an earlier version of cloudposse/packages?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(btw, if you’re using alpine, then we suggest using apk add instead of the make based installer)

Pablo Costa avatar
Pablo Costa

Thanks Erik, btw how is the policy for package updates ? For example helm version is at 2.11 (avaliable 2.13) terraform is 0.11.11 (avaliable 0.11.13)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oh, open policy!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

open a PR and we’ll approve it usually with in a couple of hours

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

only reason our versions lag is (we or you) didn’t submit a PR to update it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

since we can version pin everything, we’re very open to keeping things current.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here are the slides from the “Los Angeles Kubernetes Meetup” where we presented on Geodesic.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Geodesic Cloud Automation Shell (Slides)

Geodesic is a cloud automation shell. It’s the superset of all other tools including (terraform, terragrunt, chamber, aws-vault, aws-okta, kops, gomplate, helm, helmfile, aws cli, variant, etc) that we use to automate workflows. You can think of it like a swiss army knife for creating and building

Alex Siegman avatar
Alex Siegman

Is there a VOD of the talk by any chance?

Geodesic Cloud Automation Shell (Slides)

Geodesic is a cloud automation shell. It’s the superset of all other tools including (terraform, terragrunt, chamber, aws-vault, aws-okta, kops, gomplate, helm, helmfile, aws cli, variant, etc) that we use to automate workflows. You can think of it like a swiss army knife for creating and building

2019-04-01

SweetOps #geodesic avatar
SweetOps #geodesic
04:00:09 PM

There is 1 event this week

    keyboard_arrow_up