#geodesic (2018-08)

geodesic https://github.com/cloudposse/geodesic

Discussions related to https://github.com/cloudposse/geodesic Archive: https://archive.sweetops.com/geodesic/

2018-08-27

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
09:42:25 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s the slide I’ve been looking for that came from 2017 re:Invent talk on architecting security and governance across a multi-account strategy (SID331)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
09:45:07 PM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It’s very close to our reference architecutres.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The “Enterprise Accounts” are more decomposed

2018-08-24

Max Moon avatar
Max Moon

@Erik Osterman (Cloud Posse) Do you know if there are any known issues with upgrading the nginx ingress image using the cloudposse/nginx-ingress chart? Just curious, I have an edge case scenario that results in a (known) race condition issue in 0.11

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So it should be relatively straight forward

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think we recently added a helmfile for the official ingress

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

basically, our ingress gives you the fancy 404/500 pages

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but I can see why you’d want to move to the official one

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(we started ours before there was an official one)

Max Moon avatar
Max Moon

right right

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/helmfiles

helmfiles - Comprehensive Distribution of Helmfiles. Works with helmfile.d

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so long as the ingress class is the same it, should be a drop in replacement

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but definitely test in staging first!

Max Moon avatar
Max Moon

HAH

Max Moon avatar
Max Moon

who has time for that!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

the stable ingress comes with prometheus exporters for monitoring

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so it integrates nicely with grafana

Max Moon avatar
Max Moon

i don’t mind being on the CP chart (we use the error pages), was moreso curious if you knew anyone using say… 0.13 nginx ingress

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

not yet =/ - as in i don’t know if anyone has tried upgrading

Max Moon avatar
Max Moon

okay cool, no worries!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, what did you guys decide to do for monitoring?

Max Moon avatar
Max Moon

still auditioning companies, i was in Ireland for two weeks in July so the monitoring got re-prioritized for.. next week

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ve gotten grafana working with the autodiscovery of dashboards in configmaps

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s so freggin sweet

Max Moon avatar
Max Moon

we’ve got the portal up and rocking in both stage and prod, it’s soooo nice

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i think that alone is a BIG motivator to use it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

maybe not for everything, but definitely as a first line of monitoring

Max Moon avatar
Max Moon

right right, i dig it!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

have you tried using the portal with argo yet?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

should be really easy

Max Moon avatar
Max Moon

not yet, we actually found our first use case for argo last week

Max Moon avatar
Max Moon

so that should be coming along… swiftly

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
[cloudflare-ingress-controller] Add helmfile by osterman · Pull Request #17 · cloudposse/helmfiles

what Add cloudflare ingress controller (aka argo / acsess) why Expose services inside kubernetes securely and speedily using Argo tunnels

Max Moon avatar
Max Moon

as per usual, saving me time!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s slightly out of date

Max Moon avatar
Max Moon

also, re: race condition, will put some deets in #kubernetes for other folks?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, that would be cool

Max Moon avatar
Max Moon

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

or even open up an issue in our helmfiles or charts repos

Max Moon avatar
Max Moon

sounds good

Max Moon avatar
Max Moon

will do that in a bit

2018-08-23

tarrall avatar
tarrall

Latest “probably a n00b mistake I’m making” issue — init-terraform in terraform-root-modules/aws/ecr is erroring…

tarrall avatar
tarrall

will cut/paste the error here in a sec

tarrall avatar
tarrall
 ✓   (flowtune-prod-admin) ecr ⨠  init-terraform
Mounted buckets
Filesystem                    Mounted on
flowtune-prod-terraform-state /secrets/tf
Initializing modules...
- module.kops_ecr_app
  Getting source "git::<https://github.com/cloudposse/terraform-aws-kops-ecr.git?ref=tags/0.1.0>"
- module.kops_ecr_user
  Getting source "git::<https://github.com/cloudposse/terraform-aws-iam-system-user.git?ref=tags/0.3.0>"
- module.kops_ecr_app.label
  Getting source "git::<https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.3.3>"
- module.kops_ecr_app.kops_metadata
  Getting source "git::<https://github.com/cloudposse/terraform-aws-kops-metadata.git?ref=tags/0.1.1>"
- module.kops_ecr_app.kops_ecr
  Getting source "git::<https://github.com/cloudposse/terraform-aws-ecr.git?ref=tags/0.2.6>"
- module.kops_ecr_app.kops_ecr.label
  Getting source "git::<https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.3.1>"
- module.kops_ecr_user.label
  Getting source "git::<https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.3.1>"

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Error: output 'registry_url': "repository_url" is not a valid output for module "kops_ecr"
Error: output 'repository_name': "name" is not a valid output for module "kops_ecr"
Error: output 'kops_ecr_app_registry_url': "repository_url" is not a valid output for module "kops_ecr_app"
Error: output 'kops_ecr_app_repository_name': "name" is not a valid output for module "kops_ecr_app"
tarrall avatar
tarrall

this is with terraform-root-modules:0.5.3

tarrall avatar
tarrall

and cloudposse/geodesic:0.16.0

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tarrall I think this is fixed in an upcoming PR

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Disable default ecr by goruha · Pull Request #35 · cloudposse/terraform-root-modules

What Disabled default ecr That can be BREAKING CHANGES for some projects that use default ecr. Why Default ecr does not make sense for custom projects, that needs names for ecr

tarrall avatar
tarrall

yeah I was thinking that was likely, despite the misleading title

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, the PR should be updated

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

code review changed the nature of the PR

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, definitely recommend forking or creating your own root modules sooner rather than later

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

are you guys on codefresh?

tarrall avatar
tarrall

and/or split out bugfix work from “I want to change the functionality” work

:--1:1
tarrall avatar
tarrall

yeah

tarrall avatar
tarrall

not on codefresh no

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I updated the PR description

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

repository_name was renamed (easy fix on your side)

tarrall avatar
tarrall

and yeah time to fork. Kinda in a “chicken and egg” situation where I’m just starting to set up services — no build server yet, we have bitbucket for code (shockingly bad but this should surprise no one, it’s atlassian after all), etc. All of this “build and publish a Dockerfile” workflow would be easier if I wasn’t in the middle of trying to set up ECR to … publish Dockerfiles

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, coldstart problems..

tarrall avatar
tarrall

I’m just glad I already have a few years of experience in arguing with Terraform, or I’d probably be outside yelling at the cloud

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(you can possibly use dockerhub automated builds)

tarrall avatar
tarrall

since I’m comfortable with vanilla terraform, including magic tricks like data sources / finding stuff by tags, I’m gonna just do a combination of “copy and modify” on your code and just rolling my own from scratch to get this going. Some experience writing Dockerfiles but less with the day-to-day workflow stuff like dockerhub, compose etc

2018-08-22

Adam avatar
Adam
07:01:01 PM

@Adam has joined the channel

tarrall avatar
tarrall

OK carrying on from #announcements

tarrall avatar
tarrall

And yeah I’m picking up where Jonathan left off, or at least trying to

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


the Current Dockerfile is basically an intermingling of configuration and code, making updating it to track the “upstream” versions awkward

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so there are a lot of versioning going on

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(binaries, images, charts, helmfiles, modules)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which versions are you referring to?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

hi @tarrall, welcome

tarrall avatar
tarrall

maybe I asked this the wrong way, let me rephrase. When I got here, they had a copy of prod.cloudposse.co that was obviously from several commits ago, and things didn’t seem quite right, so I figured hey let me check out the latest version before I try to troubleshoot too far.

tarrall avatar
tarrall

However our Dockerfile (and yours) have lines like this

tarrall avatar
tarrall
ENV TF_VAR_account_id="12345"
ENV TF_VAR_namespace="example"
ENV TF_VAR_stage="prod"
ENV TF_VAR_domain_name="[prod.example.com](http://prod\.example\.com)"
ENV TF_VAR_zone_name="prod.example.com."
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha, gotcha!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, so the reference architectures IMO are designed to be hardforked

tarrall avatar
tarrall

which means I need to examine your Dockerfile, copy the relevant changes into ours, instead of having a portable Dockerfile with that stuff elsewhere & pulled in

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I really don’t think it makes sense to try to use ours verbatim

tarrall avatar
tarrall

OK gotcha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

they are an example of how to use all of our tools

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

it’s an example of how we do and how we do it for our customers

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

additionally, terraform-root-modules is also more of highly functioning examples

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

you can definitely reference them and use them, but what you run in AWS will be different from what other people run. they are basically examples for how to invoke all our terraform modules. . . a demonstration of how we use it.

tarrall avatar
tarrall

and then question #2 is … let’s say I want to slap an RDS instance in here — maybe RDS postgres, maybe Aurora. I see cloudposse/terraform-aws-rds-cluster and that’s likely where I’d start. What would the recommended approach be here? I could copypasta that into prod.example.com/rds-cluster and then have the Dockerfile pull that in, but that’s kinda no bueno — if my stage and prod envs both end up with RDS clusters, I should be sharing the same code for both.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, good question

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so there are a few concepts here. let me try to explain.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • geodesic is our base image. that distributes the tools. so that’s our “opinionated” toolchain.
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • then there are “geodesic modules” which are basically those reference architectures. those implement some architecture using the tool chain.
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • then there are terraform-root-modules. those are basically a collection of patterns. usually those patterns are highly specific to your organization. for example, you would have a way of defining the infrastructure for your “API service”
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this is basically our “MVC” of infrastructure.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

let’s take your example.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You want to add an RDS cluster. How you do this is specific to your organization. You may choose postgres or mysql. You have some opinions on the parameter groups. You have some requirements for security groups, etc.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You add that to terraform-root-modules. The root module have no “identity”.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

root modules are versioned. also, we like to build a container for the root modules so we can easily copy that stuff around between images.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Now, to invoke that RDS database, you pull that into [prod.example.com](http://prod\.example\.com) image. this achieves many things:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  1. it’s super DRY
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  1. it’s versioned infrastructure
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  1. separation of concerns. the [prod.example.com](http://prod\.example\.com) repo defines all parameters to run in that environment. this is basically the “identity” layer.
tarrall avatar
tarrall

yup I like the versioning approach there — saves drama around your TF module’s interface changing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then there’s the question: how do we develop?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i mean, if you have to push changes to terraform-root-modules, rebuild image, then rebuild your current account repo ([staging.example.com](http://staging\.example\.com)) everytime you make a change, you’ll NEVER get done.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

for this reason, when we develop, we cd /localhost/path/to/my/root-modules/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and do all iteration there until we achieve the desired outcome

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then commit/push that, open PR against master in terraform-root-modules and then merge that after approval, tag a release, and subsequently distribute that release across the various stages as needed.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also, if you use [dependabot.com](http://dependabot\.com) it’s pretty cool - you can get these updates as PRs automatically

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(we’re also working on CI/CD of everything - but it will be a bit before that’s fully baked)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and here how we pull it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/prod.cloudposse.co

prod.cloudposse.co - Example Terraform/Kubernetes Reference Infrastructure for Cloud Posse Production Organization in AWS

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/staging.cloudposse.co

staging.cloudposse.co - Example Terraform Reference Architecture for Geodesic Module Staging Organization in AWS.

tarrall avatar
tarrall

OK makes sense. I’m inclined here, I think, to have our own root modules container which is separate from yours, and develop in there…

:100:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

exactly. that’s what I would recommend.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/helmfiles

helmfiles - Comprehensive Distribution of Helmfiles. Works with helmfile.d

tarrall avatar
tarrall

@Andriy Knysh (Cloud Posse) thanks! Somehow all my google-fu was able to turn up was your TF module, not the one that was in root-modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

You might like to fork these too

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(also, we welcome all PRs - so if you find/fix bugs, develop cool new things, would love to see it)

tarrall avatar
tarrall

I’m waaaaaaay too much of a k8s n00b to want to fork someone’s “young” repo and develop on that. Good odds y’all will find and fix bugs & improve workflow faster than I can, which means I’m better off being able to follow your repo rather than blazing my own trail

tarrall avatar
tarrall

so yeah PRs fersure, once I’ve got a vague clue of what I’m doing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, sounds good! you’re on a fast track.

tarrall avatar
tarrall

BTW one other minor thing — I think AWS well-architected (or whatever they’re calling it these days) normally recommends a separate “identities” account where the humans are managed, rather than managing those out of the master account. At least, that’s what I did at my last place, and I kinda liked that because IMO the master account should be locked down hard. Might be something to consider adding to the reference architecture, though I realize it may be overkill for many places.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think what we call [root.example.com](http://root\.example\.com) is that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(though I think we should rename ours to [identity.example.com](http://identity\.example\.com))

tarrall avatar
tarrall

Aaaaah. Yeah :–1: to renaming, because I think we ended up with root.example.com == master. Not positive.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmmmm ok, so I need to re-review the well-architected doc

tarrall avatar
tarrall

Heh I was at that re:Invent talk…

tarrall avatar
tarrall

one of only like 3-4 I managed to make last year

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Perhaps we have something we should rethink there. I’ve always treated root = identity = master

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

but deploy nothing other than identity in it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

then prod, staging, audit (~security), dev, testing accounts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

which share nothing

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and identity delegates to those accounts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i need to adjust my mental model for how it would look if master != identity

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so is master just billing?

tarrall avatar
tarrall

yeah :–1: on identity delegating, fersure. I like having the “master account” (payer account, and where the service control policies are defined) separate from the “identities account” (where humans are defined)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

master is root/Org

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

identity is our current [root.cloudposse.co](http://root\.cloudposse\.co)

tarrall avatar
tarrall

ok cool so mostly just I got confused by the naming convention

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

something like that. yea, naming is hard.

tarrall avatar
tarrall

2 hard things in CS, right? Naming things, cache invalidation, and counting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ve had a lot of discusssion around this internally (fwiw) - we know we need to change root to something or to rename terraform-root-modules (for which there’s no relationship)

tarrall avatar
tarrall

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’m inclined to rename [root.cloudposse.co> to <http://identity.cloudposse.co|identity.cloudposse.co](http://root\.cloudposse\.co)

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and introduce a new [master.cloudposse.co> or <http://billing.cloudposse.co|billing.cloudposse.co](http://master\.cloudposse\.co)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

so we do DNS zone deletation from identity as well

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

where would that belong?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yea, there we have two diff hierarchies: AWS and DNS

tarrall avatar
tarrall

Re DNS, oh man, that’s one of those that’s gonna be super company dependent right? I mean, for some places using Route53 maybe they do a zone per account, other places maybe have a single shared zone with cross-account access…

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

root of DNS is [cloudposse.co](http://cloudposse\.co)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


I mean, for some places using Route53 maybe they do a zone per account, other places maybe have a single shared zone with cross-account access…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

=(

tarrall avatar
tarrall

LOLOLOL but when you migrated to the cloud LOOONG before AWS had orgs…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha - yea, we’re probably going remain very strict about “share nothing”

tarrall avatar
tarrall

you might happen to be proud of having almost everything migrated out of Classic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(we even recommend using different TLDs per account in some situations)

tarrall avatar
tarrall

Yup, reasonable fersure. I’ve always liked at least separating “customer-facing” TLD from “internal ops” TLD

tarrall avatar
tarrall
[example.com> / <http://example.net example.net](http://example.com) kinda thing
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes - that’s a must

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we call the customer facing one the “vanity domain”

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so we provision the root DNS zone (e.g. [cloudposse.co>) in <http://master.cloudposse.co|master.cloudposse.co](http://cloudposse\.co)?

tarrall avatar
tarrall

An alternative there might be to have an account dedicated to “shared infrastructure.” I can’t decide if it is just massive overkill to have that as a separate account from the “identities” account or not… identities is certainly an instance of “shared infra”.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

yes that’s possible

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

what we’ve found out after many iterations is that there are no perfect solution for this

tarrall avatar
tarrall

Yup

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you touch/change something in one place, you get a lot of issues in other places

tarrall avatar
tarrall

And k8s is new enough that I’m confident that in a year or two, the “best practices” there today will be a laughingstock. This just based on my past experience with seeing workflows mature on Chef and Terraform…

:100:1
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

that’s actually one of the main reasons we created https://github.com/cloudposse/terraform-root-modules and https://github.com/cloudposse/helmfiles - to introduce some patterns for TF and k8s

cloudposse/terraform-root-modules

terraform-root-modules - Collection of Terraform root module invocations for provisioning reference architectures

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

they are not perfect, but at least we have the same structure between projects and consistent naming (which is hard)

2018-08-21

tarrall avatar
tarrall
01:23:29 AM

@tarrall has joined the channel

2018-08-19

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

some discussion right now in #release-engineering related to #geodesic

2018-08-17

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Lots of great UX fixes were merged today

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • check for duplicate syslog-ng
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • fancier banner
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • prompt line-wrapping
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
  • ^C ssh-agent no longer aborts subsequent scripts

2018-08-15

Dylan avatar
Dylan
07:38:23 PM

@Dylan has joined the channel

2018-08-09

2018-08-08

2018-08-07

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth just cut 0.13.0 which adds WSL (Windows) support

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can you give that a shot?

Sebastian Nemeth avatar
Sebastian Nemeth

Will do!

Sebastian Nemeth avatar
Sebastian Nemeth

Hey man - so this still isn’t solving the problem convincingly, I think… There’s just one problem I can see…

The first is that it’s common for WSL users to change the mount path for their local drives from /mnt/c to just /c (for example), which makes a lot of things easier for us. But this causes the new geodesic to fail with:

/usr/local/bin/root.potato.com: line 91: /mnt/c/Windows/System32/cmd.exe: No such file or directory

E.g. line here: https://github.com/cloudposse/geodesic/commit/a096ddf28314f0d7c9423f61b8516853663b4d24#diff-499f40d14b68a5dc159a3d3ebc5c4870R91

Looks like it’s looking for cmd.exe under /mnt - however, cmd.exe is something that should always be in PATH in WSL, so it might be fine to omit the path and just use cmd.exe everywhere?

Add Support for Windows Subsystem for Linux (WSL) (#202) · cloudposse/[email protected]
  • Add user environment preserving * fix(*): add env variables for changing $HOME varaible(for wsl) * fix(wrapper-on-wsl): Now windows and linux usernames get dynamically * refactor(wrapper…
Sebastian Nemeth avatar
Sebastian Nemeth

The location of the mounted drives can be obtained from /etc/wsl.conf under [automount] > root. https://docs.microsoft.com/en-us/windows/wsl/wsl-config#set-wsl-launch-settings

Manage Linux Distributions

Reference listing and configuring multiple Linux distributions running on the Windows Subsystem for Linux.

Sebastian Nemeth avatar
Sebastian Nemeth

@Erik Osterman (Cloud Posse)

Sebastian Nemeth avatar
Sebastian Nemeth
Support non-default WSL root mount path by Martaver · Pull Request #214 · cloudposse/geodesic

Uses regex to look up the correct root mount path from wsl.conf. I tested the script on my system, and it works - however wasn’t able to test the whole build.

2018-08-06

tamsky avatar
tamsky

doh, pstree -p ; jobs -l; for i in $(jobs -p) ; do echo $i ; ls -l /proc/$i/fd ; done;

tamsky avatar
tamsky

$p was in my example – should have been $i

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

oops, I missed that too.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

will try again next time.

2018-08-04

tamsky avatar
tamsky

[2] Stopped indicates the process received a SIGTSTP

tamsky avatar
tamsky

I should smile along with my crazy talk.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

tamsky avatar
tamsky

https://github.com/cloudposse/geodesic/pull/210#pullrequestreview-143392008

so in marked contrast to my bash shell’s default shopt, geodesic does have checkwinsize on….

so unless that setting changed only recently – I think that the shopt should have been solving this problem all along.

Strategically signal bash to resize by osterman · Pull Request #210 · cloudposse/geodesic

what Only call kill -WINCH within s…

tamsky avatar
tamsky

And I’m thinking: “where have you been all my life checkwinsize?” Not having it, forced me to learn the kill -WINCH trick in the meantime.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The checkwinsize wasn’t working for me, but maybe that was masked by other problems

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For example the ones you previously fixed

tamsky avatar
tamsky

maybe docker run doesn’t pass WINCH to subshells?

tamsky avatar
tamsky

because the terminal’s window change signal needs to propagate down this entire chain: (iterm/xterm) -> local shell -> docker run -> geodesic shell

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh fascinating. Didn’t know that either

tamsky avatar
tamsky

I guess it does propagate… here’s one where the signal was killing apache if docker run -it was used: https://github.com/docker-library/php/issues/64

SIGWINCH issue · Issue #64 · docker-library/php

When making a derived image from php:5.6-apache, when the server starts, at the slightest movement it stops, and gives me this message: [Wed Jan 21 20:24:05.736731 2015] [mpm_prefork:notice] [pid 1…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Regarding the other problem, with sigstop it seems to happen when AWS session expires. Probably related to aws-vault usage.

tamsky avatar
tamsky

yes, that sounds like aws-vault trying to read/write stdin/out and being blocked

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes, I think it is related to stdin

tamsky avatar
tamsky

I just scanned their issue queue and didn’t see anything related.

tamsky avatar
tamsky

next time this happens to someone, can they please run and report back: pstree -p ; for i in $(jobs -p) ; do echo $p ; ls -l /proc/$i/fd ; done;

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:18:55 AM
tamsky avatar
tamsky

as well as the [N] Stopped aws-vault exec ...

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Will do!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Man… can’t say how much I appreciate your insights. In the short time since you’ve joined, learned a lot of little tricks from you.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:19:31 AM
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tamsky this is the output.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

pid 8 is aws-vault server mode (mock metadata api)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:21:55 AM

2018-08-03

2018-08-02

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

damn, I spun up an AWS Workspace for Windows 10 not realizing it doesn’t support Hyper-V, so no Docker. =(

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

heh, and no “Windows Server 1709” support yet for “AWS Workspaces”, so no WSL even.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

=(

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Sebastian Nemeth i wanted to try this to test Geodesic on WSL/Docker

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@rohit.verma going back to your question of simplifying IDE integration with geodesic

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

does running ` mount –bind /localhost/Dev/cloudposse/terraform-root-modules/ /conf` inside of geodesic make things any better for you?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ve been using this today and it really has helped for this specific use-case

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For example, I’m working on some kops automation. So I run:

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

mount --bind /localhost/Dev/cloudposse/terraform-root-modules/aws/kops /conf/kops

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(replace Dev/cloudposse/terraform-root-modules/ with the path to your root-modules folder)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
04:58:01 AM
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’ve started getting these errors. I suspect it could be related to bash not responding to kill -WINCH $$ within some kind of deadline

tamsky avatar
tamsky

That’s crazy talk. POSIX signals were designed to be deadlock free. I’ve never heard of a signal deadline. Have you?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I don’t know the internals well enough anymore. Not strictly speaking of posix signals, thought I have experienced that if a process doesn’t acknowledge a signal that something happened to it, but I confess it’s been 15 years since I was that low level and probably am entirely off.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I just don’t know what to attribute this behavior to

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think what I might be thinking of (confusing it with) is where signal handling has been used for heartbeating a process. If it doesn’t respond, then the process is reaped.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@tamsky Have you seen this before?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

i started getting the errors too (after some period of inactivity in geodesic)

sarkis avatar
sarkis

I’ve noticed strange behavior after long period of inactivity but was assuming it had to do with my aws-vault session expiring

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
strategically signal bash to resize by osterman · Pull Request #210 · cloudposse/geodesic

what Only call kill -WINCH within s…

2018-08-01

    keyboard_arrow_up