#geodesic (2020-10)

geodesic https://github.com/cloudposse/geodesic

Discussions related to https://github.com/cloudposse/geodesic

Archive: https://archive.sweetops.com/geodesic/

2020-10-07

Stan M avatar

what’s the place to start with geodesic? example repos like root.cloudposse.co haven’t been updated for a while. is reference-architectures repo the latest example to follow?

Joe Niland avatar
Joe Niland

It’s deprecated. What is your goal?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So we still use geodesic in of our engagements, but we don’t practice the “one repo per account” strategy any more

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

geodesic is really just an image that bundles all the tools we need. That said, even though it’s so simple, we’ve managed to open 500 PRs to get there

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re actively working on releasing our next generation of reference architecture, but it probably won’t be available until Q1 2021.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re releasing bits and pieces as we go. the next important piece we’re releasing is our automation library built on top of #variant

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’re releasing some more opinionated modules around datadog, opsgenie, and terraform cloud too.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

https://github.com/cloudposse/reference-architectures is also in transition, so there’s nothing to follow right now

cloudposse/reference-architectures

[WIP] Get up and running quickly with one of our reference architecture using our fully automated cold-start process. - cloudposse/reference-architectures

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we’ve deprecated the original reference architecture (as we’ve abandoned it)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and we’re launching a new one that will be found here.

Stan M avatar

Ok, thanks for the clarification. looking for a starter template to follow for a multi-account setup. using something like geodesic to package all the dependent tools looks like a nice approach to combine with it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m happy to jump on a quick call to give you the lay of the land and pointers to getting started

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Stan M avatar

sure, i appreciate it. i’ll look for some time in the next few days

2020-10-08

2020-10-11

Joe Niland avatar
Joe Niland

A while ago it was mentioned that geodesic will move to being based on Debian. At least I’m pretty sure I remember that!

Is there a timeline for this yet?

2020-10-12

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No timeline, but a big step to being able to support that was migrating our repos to Cloudsmith which is done

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It is something we may need to do for one of our current customers, because some things (e.g. teleport shell tsh) no longer run on alpine

Joe Niland avatar
Joe Niland

Thanks Erik!

Joe Niland avatar
Joe Niland

I’d be happy to test it, if you need testers.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok! follow along in cloudsmith - we’ll be transparent in the process there for adopting deb

1

2020-10-27

chris avatar

I am trying to use Geodesic and following the cold start documentation (https://docs.cloudposse.com/reference-architectures/cold-start/) Things were going ok until I got to the PROVISION IAM PROJECT TO CREATE ROOT IAM ROLE section. I believe that maybe the cd command here is supposed to be cd root-iam because the module identified is not used in the iam directory and the plan identifies nothing to perform. So I moved over to the root-iam directory and performed the same steps but when I try to apply I only get 403 errors.

* module.organization_access_group_root.aws_iam_group.readonly: 1 error(s) occurred:

* aws_iam_group.readonly: Error creating IAM Group instinct-root-readonly: InvalidClientTokenId: The security token included in the request is invalid
	status code: 403, request id: b06f7a1d-bd11-49d9-9fd9-b0b69dfebd02
....
....

Does anyone have ideas what might be going on here? I have tried all combinations of using the actual root AWS account, the admin account setup at the top of that page, and have disabled MFA in case it was getting in the way. I have been stuck for way to long on this step and can’t seem to find anything in the archives (though i could be missing something)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Looks like your token might have expired?

chris avatar

I thought so as well. I fiddled with reauthenticating, removing the credentials from aws-vault and replacing them with no luck. I am not 100% clear on exactly how the token is acquired so I may be missing a step to try to re-authenticate. I was able to perform the earlier step and make the S3 bucket and setup the DynamoDB for the TF backend.

chris avatar

FYI, I did get past this after much trial and error. I ended up not running assume-role before trying to apply and instead waiting for the assume role prompt and entering the role arn.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Sorry! that documentation is out of date. We have a new reference arch coming out by the end of the year that is revamped for 0.13+ (we’ve used this for the past year with our customers but haven’t open sourced it yet)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
05:02:15 AM

2020-10-28

chris avatar

Yes, I did see the notice at the top so I did expect to bump into some issues due to outdated docs. After getting stuck with Geodesic I also tried using the reference-architecture repo and got stuck trying to run make root. I am very excited to hear about a new release coming out, but unfortunately I am unable to wait to setup this architecture. I am committed to using your tooling as I believe in your design and approach to dev ops and the amazing infrastructure you produce for your clients. Is there a path forward at this point?

1
Nathan Margaglio avatar
Nathan Margaglio

We’re in a similar boat as chris. We’ve used the reference-architecture repo in the past with success, but we are now in the process of setting up a new project and want to be prepared to transition to the new arch when it’s made available.

Do you have any tips, @Erik Osterman (Cloud Posse), in terms of setting up a new project so that transitioning into the new architecture is as painless as possible? We’re prepared to refactor things when the time comes, but are hesitant to follow the current/previous approach since it would seem the new approach is going to differ significantly.

So we still use geodesic in of our engagements, but we don’t practice the “one repo per account” strategy any more

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i’m sorry - I really wish we could release the new ref arch faster. it leverages entirely new root modules (plus all of our public modules), uses yaml based configuration, runs in geodesic, works with terraform cloud… but unfortunately, we just don’t have the available hands right now. I started working with a community member (@Matt Gowie) to release it faster, but realized there’s too much tacit knowledge required to setup. we are starting a new engagement in the next few weeks. as part of this, we will be taking this opportunity to publish more of the reference architecture.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

strategically, this is very important for us - it’s definitely happening - but is a large initiative

1
Joe Hosteny avatar
Joe Hosteny

Hi Erik, I had Nate reach out on here to find out what the update on the new ref arch was. I’ve been fairly under water, and he just joined us to start work on large a new project, so we wanted to align to the new arch if possible. I 100% understand the tacit knowledge comment, and the overhead of having someone help, but feel free to reach out to us if we can assist (e.g., focused PR reviews, contributions or alpha testing) in any way.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What I really need to do is start a list - because I am losing count

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Andy Miguel let’s discuss setting that up

Andy Miguel avatar
Andy Miguel

@Erik Osterman (Cloud Posse) ack

Nathan Margaglio avatar
Nathan Margaglio

thanks for the update!

chris avatar

@Nathan Margaglio I tried to use reference-architecture but got stuck there… if you have notes that you could share it would be much appreciated. I am currently working on figuring out the next steps on moving from setting up the accounts to setting up services (e.g. EKS) and the correct way to structure the repo… lots more reading to do.

chris avatar

In reading my blocker may be a terraform version issue. But I also got parts of the accounts setup going through the cold-start documentation… So I am going to do a lot more research to see if I should continue down the path outlined in cold-start or back out and use the reference-architecture design.

Joe Hosteny avatar
Joe Hosteny

I’m not sure if you saw this, but it was invaluable when we used the reference arch. I don’t think there was much else other than the official docs + this that we needed: https://gist.github.com/joe-niland/b96150bfc13828c2a58751dfca7ffe7e

chris avatar

Thanks that looks like a lot of great information. The next part I have to figure out is going from cold start to deployed services (eg VPC, EKS, etc) so that is what I am digging into now

chris avatar

@Joe Hosteny Thanks for the link it did get me unstuck and I made a lot of progress! Unfortunately, I just bumped into another issue and am digging into it now

terraform init
Copying configuration from "git::<https://github.com/cloudposse/terraform-root-modules.git//aws/accounts?ref=tags/0.58.1>"...
Initializing modules...
- module.audit
  Getting source "stage"
- module.corp
  Getting source "stage"
- module.data
  Getting source "stage"
- module.dev
  Getting source "stage"
- module.identity
  Getting source "stage"
- module.prod
  Getting source "stage"
- module.security
  Getting source "stage"
- module.staging
  Getting source "stage"
- module.testing
  Getting source "stage"

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Checking for available provider plugins on <https://releases.hashicorp.com>...
- Downloading plugin for provider "aws" (2.70.0)...

The following providers do not have any version constraints in configuration,
so the latest version was installed.

To prevent automatic upgrades to new major versions that may contain breaking
changes, it is recommended to add version = "..." constraints to the
corresponding provider blocks in configuration, with the constraint strings
suggested below.

* provider.aws: version = "~> 2.70"

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
parse error: Invalid numeric literal at line 1, column 2
Goodbye
Joe Hosteny avatar
Joe Hosteny

I don’t recall seeing that, but the last time I did this was probably 8 months ago, with a different version of the provider

chris avatar

I am really at a cross roads of determine what is the best way to get started at this point and get something working that also has some path forward.

2020-10-29

Joe Niland avatar
Joe Niland

FWIW, I set up one client using the reference-architecture last year. It’s going well, but there are a few limitations with using the full terraform featureset (due to the -from-module approach.)

For all clients since then I’ve simplified things and have been using Terragrunt and Terraform natively. To be fair, all of these clients are on Mac/Linux or I am managing it all for them (on Mac.) None of them need CI/CD with Terraform. So the simple approach is working well.

For authentication, i’m using aws-vault with my AWS account assuming roles in the clients’ and the awscli’s credential_process option.

That said, I am really looking forward to see what the Cloudposse team have been working on for their updated reference architecture!

This Terragrunt reference architecture could help some of you in the meantime. It’s not exactly how I’m using it, but pretty close.

antonbabenko/terragrunt-reference-architecture

Terragrunt Reference Architecture (upd: May 2020). Contribute to antonbabenko/terragrunt-reference-architecture development by creating an account on GitHub.

3

2020-10-30

chris avatar

Thanks for the information. I will look at that as well. I have pushed through and it appears I have got everything setup using the reference-architecture repo. I am going to make a gist of my notes and where I bumped into issues for anyone else that is trying to use it at this time.

Joe Niland avatar
Joe Niland

@chris it slipped my mind earlier but I created a gist like this myself. Perhaps it’s out of date now. Not sure.

https://gist.github.com/joe-niland/b96150bfc13828c2a58751dfca7ffe7e

chris avatar

Thanks. Someone else also linked this to me… very useful and helped me past a few issues! Thanks again.

1
    keyboard_arrow_up