#geodesic (2020-04)

geodesic https://github.com/cloudposse/geodesic

Discussions related to https://github.com/cloudposse/geodesic Archive: https://archive.sweetops.com/geodesic/

2020-04-28

Mario Feliz avatar
Mario Feliz

Hi all, I am new to geodesic . Does anyone know where can I find some video tutorial/docs I can get myself up and running more in-depth?

2020-04-27

Cloud Posse avatar
Cloud Posse
04:00:30 PM

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is May 06, 2020 11:30AM.
Register for Webinar
slack #office-hours (our channel)

2020-04-24

omerfsen avatar
omerfsen

Hi all. Can i ask how much will it cost to deploy reference-architectures monthly assuming nothing ran/deployed on it ?

omerfsen avatar
omerfsen

i will play with that with my personal account

omerfsen avatar
omerfsen

so i can get a heads up

omerfsen avatar
omerfsen

Just read Erik’s comment about reference-architecture on this channel

2020-04-23

Alex Siegman avatar
Alex Siegman

I went from using Geodesic somewhere around version 0.124.0 to the newest 0.129.1 a week or two ago and have noticed that my assume-role doesn’t seem to be refreshing it’s session.

I have export AWS_VAULT_SERVER_ENABLED=true in the environment, I do see the message at startup

* assume-role will start EC2 metadata service at <http://169.254.169.254/latest>

But roles time out multiple times daily for me. The max duration for a session on the assumed role is 1 hour, but that used to be auto-refreshed for me. Now I have to backout and reassume the role every hour, rather than every 12.

Something change/break here?

Alex Siegman avatar
Alex Siegman

I just saw that shortly after I did that, 0.129.2 came out. Can update to test that, but question still stands.

Joe Niland avatar
Joe Niland

@Alex Siegman not an expert but the latter behaviour always happened to me since I started using geodesic from about 0.123.0.

I resolved it by setting an env var in ~/.geodesic/preferences.d/aws-vault

AWS_VAULT_DURATION=4h

You can also set

AWS_VAULT_ASSUME_ROLE_TTL
AWS_VAULT_SESSION_TTL

but honestly I haven’t spent the time to figure out exactly which one has which effect.

Joe Niland avatar
Joe Niland

Actually, sorry! Just realised you were talking about auto-refresh - I have never seen that working, but would like to!

Alex Siegman avatar
Alex Siegman

I had it working; the key being allowing login sessions to your root account to last 12 hours, and assume role sessions being 1-12 hours, it should auto refresh. That’s the whole point of running the aws-vault server is so that it does that. It worked great for me in 0.124.X and before

Alex Siegman avatar
Alex Siegman

I know Jeremy from cloudposse is kind of the guru on these things, hoping he chimes in

:--1:1
Joe Niland avatar
Joe Niland

@Alex Siegman that is good to know. I would love to get that working over here too. Looking at the commits, I don’t see anything obvious around that time.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy Grodberg (Cloud Posse) anything standout?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Is this related to the deprecation of server command?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Promote proxy server to first class feature · Issue #361 · 99designs/aws-vault

There are a few open issues regarding the proxy server already, and I am opening this new one with new information/issues/use cases and will try to consolidate some of the existing issues here. To …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

i seem to recall there was something that changed outside of our control.

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

This is all due to changes in aws-vault

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)
99designs/aws-vault

A vault for securely storing and accessing AWS credentials in development environments - 99designs/aws-vault

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

So the trick is probably for @Alex Siegman and @Joe Niland to pin aws-vault to an earlier release inside of their Dockerfile

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

We might consider reverting Geodesic to aws-vault 4.7.1 until we and/or they figure out how to get caching and auto-renewing working again.

Joe Niland avatar
Joe Niland

Thanks @Erik Osterman (Cloud Posse) and @Jeremy Grodberg (Cloud Posse) - I’ll try pinning.

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

I haven’t looked into it much because I haven’t been using aws-vault at all myself lately. I’ve been using aws-google-auth since just after aws-vault moved to v5

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

There were a lot of complaints about v5, including that it asks for MFA before password and doesn’t cache MFA, and they were going to fix some of that. Been waiting to see.

:--1:1
Alex Siegman avatar
Alex Siegman

Thanks, if it’s a known issue with aws-vault I can pin it back to the newest 4.x

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

Yes, I say go back to 4.7.1 and let us know how it works out. If it’s good, I think we should roll back globally until v5 has the kinks worked out.

:--1:1
Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)
Assume role credentials are not cached · Issue #552 · 99designs/aws-vault

I am using the latest release of AWS Vault I have provided my .aws/config (redacted if necessary) I have provided the debug output using aws-vault –debug (redacted if necessary) When aws-vault ass…

Alex Siegman avatar
Alex Siegman

Added the following to my Dockerfile:


\# Pin aws-vault to a version <5.0

\# There are bugs with aws credential caching that make version 5 more annoying to use; see:

\# <https://github.com/99designs/aws-vault/issues/552>
RUN apk del aws-vault && apk add [email protected]==4.7.1-r0

Will see if that helps

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

@Alex Siegman before rolling back aws-vault to v4, make sure you have set up the new TTL variables

Alex Siegman avatar
Alex Siegman

in v5? i can try that

Alex Siegman avatar
Alex Siegman

I assume you mean these?

AWS_SESSION_TOKEN_TTL: Expiration time for the GetSessionToken credentials. Defaults to 1h
AWS_CHAINED_SESSION_TOKEN_TTL: Expiration time for the GetSessionToken credentials when chaining profiles. Defaults to 8h
AWS_ASSUME_ROLE_TTL: Expiration time for the AssumeRole credentials. Defaults to 1h
AWS_FEDERATION_TOKEN_TTL: Expiration time for the GetFederationToken credentials. Defaults to 1h

The session token one is probably the big hoser here, should be 12h to operate like it did previously

Alex Siegman avatar
Alex Siegman

That said, if the credentials aren’t cached, if assume_role_ttl is 1h, it will still ask every hour, so you’d have to change all of your profiles in AWS to allow >1h timing

Alex Siegman avatar
Alex Siegman

But maybe bumping the session is enough

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

Yes, I was looking for where they were defined. You need to set them all, except maybe FEDERATION

Alex Siegman avatar
Alex Siegman

I’ll play with those before reverting to 4.7.1 and report back. I just logged in with them set, should know in an hour or so I’d imagine

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

The idea with aws-vault is that you log in and it caches your session, including your MFA, so you can keep using it to assume roles.

Alex Siegman avatar
Alex Siegman

Yeah, but if it doesn’t cache your session, it can’t re-assume roles without new MFA input I’d imagine

Alex Siegman avatar
Alex Siegman

case in point, i just did an aws-vault --debug list it decided my existing session was expired and deleted it, but my session is still valid (can use awscli) yet aws-vault shows no sessions for my profile

Alex Siegman avatar
Alex Siegman

seems funky, now that I know where to look though I’ll play with it

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

I was excited about aws-vault v5 because the fixed some of the problems with running aws-vault exec directly, which running role-server was a workaround. But as long as we have role-server, if changing the TTLs in the env vars doesn’t work, then yes, go back to 4.7.1 and role-server.

Joe Niland avatar
Joe Niland

Just a note - I couldn’t pin because –duration is an unsupported option in 4.7 (https://github.com/cloudposse/geodesic/blob/59d9637c32c29d3097859ad3d2c76fef0fb802b9/rootfs/etc/profile.d/aws-vault.sh#L132)

cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

@Alex Siegman @Joe Niland Please check if you have AWS_CHAINED_SESSION_TOKEN_TTL set as well.

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

@Alex Siegman @Joe Niland @Erik Osterman (Cloud Posse) PR for Geodesic to re-enable support for aws-vault version 4, plus clean up some issues that may have been causing problems with aws-vault version 5. I suggest trying the new Geodesic (when it is released) with v5 before reverting to v4. https://github.com/cloudposse/geodesic/pull/579

[aws-vault] Simultaneous support for aws-vault v4 and v5 by Nuru · Pull Request #579 · cloudposse/geodesic

what [aws-vault] Simultaneous support for aws-vault v4 and v5 [helmfile] update to v0.111.0 Update other tools per cloudposse/packages why aws-vault released a major upgrade, going from version …

Joe Niland avatar
Joe Niland

@Jeremy Grodberg (Cloud Posse) thank you - this is great!

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

OK, Geodesic version 0.130.0 is released. Give it a try and please report back.

:--1:1
Alex Siegman avatar
Alex Siegman

Using just 0.130.0 with no changes, I still had timeouts due to aws-vault still not properly caching credentials it seems.

Will use your method to revert to aws-vault v4 and retest as able

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

How’s it is going?

Alex Siegman avatar
Alex Siegman

I haven’t been able to get v5 to work refreshing stuff. I have not yet reverted as it was the weekend and I didn’t bother to test it yet

Alex Siegman avatar
Alex Siegman

I just did the reversion, and initial debugging looks like it will work fine. I can see it caching sessions properly as well. Will use over next day or two to be sure. Should know later today if it properly renews sessions after 1hr timeout

Alex Siegman avatar
Alex Siegman

Still doesn’t look like it’s working quite right. I have a valid session, the first hour has gone by so my role assumption should be expired and thus need refreshing, but using something that needs it to be refreshed does not cause it to be.

 √ : [spoton-staging-admin] customer-auth-service ⨠ aws-vault --debug list
2020/04/28 04:58:35 [keyring] Considering backends: [file]
2020/04/28 04:58:35 Using AWS_CONFIG_FILE value: /localhost/.aws/config
2020/04/28 04:58:35 Loading config file /localhost/.aws/config
2020/04/28 04:58:35 Parsing config file /localhost/.aws/config
2020/04/28 04:58:35 [keyring] Expanded file dir to /conf/.awsvault/keys/
2020/04/28 04:58:35 Looking up all keys in keyring
2020/04/28 04:58:35 [keyring] Expanded file dir to /conf/.awsvault/keys/
2020/04/28 04:58:35 Session "session,c3BvdG9u,YXJuOmF3czppYW06OjE0ODQ4NzQwMjkxOTptZmEvYXNpZWdtYW5Ac3BvdG9uLmNvbQ,1588089054" expires in 10h52m18.485944511s
Profile                  Credentials              Sessions
=======                  ===========              ========
default                  -
spoton                   spoton                   1588089054 (mfa)
spoton-corp-admin        spoton
spoton-staging-admin     spoton
 ⧉  staging
 √ : [spoton-staging-admin] customer-auth-service ⨠ aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".

Container startup seemed normal


\# Exposing port 34060
* aws-vault: overriding AWS_SESSION_TTL () with AWS_VAULT_SESSION_TTL (12h)
* assume-role will start EC2 metadata service at <http://169.254.169.254/latest>
Alex Siegman avatar
Alex Siegman

It does not, however, reask for MFA to get new session credentials if I exit out and rerun assume-role so it is properly caching that.

2020-04-20

Cloud Posse avatar
Cloud Posse
04:00:51 PM

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Apr 29, 2020 11:30AM.
Register for Webinar
slack #office-hours (our channel)

2020-04-16

2020-04-14

Alex Siegman avatar
Alex Siegman

Is the newest package for terraform really 0.12.10 in the cloudposse packages? Or is it recommended to go back to the community packages for terraform?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We have been in tech debt hell.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

GitHub actions changed somethings that broke our automations.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Maxim Mironenko (Cloud Posse) is working to fix that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Upstream maintainers keep changing their release versions and artifacts breaking our auto updates

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I hope we get the auto updates working again in the next day or so. @Maxim Mironenko (Cloud Posse) has been working on it all week.

Alex Siegman avatar
Alex Siegman

No worries, was just in the process of updating all my images for a kubernetes update and was going to bump that too. Don’t have a need for it at the moment.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Think @Jeremy Grodberg (Cloud Posse) updates this this week

Jeremy Grodberg (Cloud Posse) avatar
Jeremy Grodberg (Cloud Posse)

I updated packages (and Geodesic) yesterday, current Terraform is 0.12.24

:--1:2

2020-04-13

Cloud Posse avatar
Cloud Posse
04:00:35 PM

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Apr 22, 2020 11:30AM.
Register for Webinar
slack #office-hours (our channel)

2020-04-07

cada avatar

Hello. Im new to geodesic and attempting to install. During make process “ERROR: Not committing changes due to missing repository tags. Use –force-broken-world to override.” causes process to fail. My tool set is windows docker desktop with a cloned geodesic repo shared from windows dir to alpine Linux container. Testing install from cloned geodesic dir directly on alpine container does complete successfully. Ive attached error with full details. Any assistance is appreciated. Thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Wierd, in this case makes me think there’s something up with the line endings (perhaps related to windows?)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

see how the ' character is at the beginning of the line? makes me thing something up with \r

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Master successfully built 4 days ago, so I think it’s a windows incompatbility. Our windows support came from community contributions (We don’ thave any way to test it)

cada avatar

Thanks for the quick response Erik. I agree with you that it’s a windows environment problem. I’ll continue working the error out and will update the channel if able to resolve.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that would be great

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

we accept most PRs too

2020-04-06

Cloud Posse avatar
Cloud Posse
04:00:32 PM

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about geodesic, get live demos and learn from others using it. Next one is Apr 15, 2020 11:30AM.
Register for Webinar
slack #office-hours (our channel)

2020-04-03

Alex Siegman avatar
Alex Siegman

Yeah, i’ve hit this problem in other scenarios and never solved it, was hoping someone smarter than me had

2020-04-01

Alex Siegman avatar
Alex Siegman

So, I started using Geodesic on a linux box (Amazon Linux 2 specifically, based on CentOS) rather than my normal OSX setup, and I’m noticing a lot of issues with being root in the container causing files to be root on my local file system and screwing with stuff. there a docker configration i’m missing somewhere? Looking in the startup scripts if you do a make install I see it passing in USER_ID and GROUP_ID as environment variables in to the docker container on the command line, but not sure how geodesic is consuming those or if it has an effect on file system mounts

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, no practical way to do that.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

in geodesic you want to be able to install packages, so you want to be root.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

on the local fs, it’s just mounted and there’s no way to set the uid/mapping

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(e.g. like on NFS)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

open to suggestions

    keyboard_arrow_up