#geodesic (2020-04)

Think @Jeremy G (Cloud Posse) updates this this week

I updated packages (and Geodesic) yesterday, current Terraform is 0.12.24

I just saw that shortly after I did that, 0.129.2 came out. Can update to test that, but question still stands.

@Alex Siegman not an expert but the latter behaviour always happened to me since I started using geodesic from about 0.123.0.

I resolved it by setting an env var in ~/.geodesic/preferences.d/aws-vault

AWS_VAULT_DURATION=4h

You can also set

AWS_VAULT_ASSUME_ROLE_TTL
AWS_VAULT_SESSION_TTL

but honestly I haven’t spent the time to figure out exactly which one has which effect.

Actually, sorry! Just realised you were talking about auto-refresh - I have never seen that working, but would like to!

I had it working; the key being allowing login sessions to your root account to last 12 hours, and assume role sessions being 1-12 hours, it should auto refresh. That’s the whole point of running the aws-vault server is so that it does that. It worked great for me in 0.124.X and before

I know Jeremy from cloudposse is kind of the guru on these things, hoping he chimes in

@Alex Siegman that is good to know. I would love to get that working over here too. Looking at the commits, I don’t see anything obvious around that time.

@Jeremy G (Cloud Posse) anything standout?

Is this related to the deprecation of server command?

https://github.com/99designs/aws-vault/issues/361

i seem to recall there was something that changed outside of our control.

This is all due to changes in aws-vault

Big changes in aws-vault v5 https://github.com/99designs/aws-vault/releases/tag/v5.0.0

So the trick is probably for @Alex Siegman and @Joe Niland to pin aws-vault to an earlier release inside of their Dockerfile

We might consider reverting Geodesic to aws-vault 4.7.1 until we and/or they figure out how to get caching and auto-renewing working again.

Thanks @Erik Osterman (Cloud Posse) and @Jeremy G (Cloud Posse) - I’ll try pinning.

I haven’t looked into it much because I haven’t been using aws-vault at all myself lately. I’ve been using aws-google-auth since just after aws-vault moved to v5

There were a lot of complaints about v5, including that it asks for MFA before password and doesn’t cache MFA, and they were going to fix some of that. Been waiting to see.

Thanks, if it’s a known issue with aws-vault I can pin it back to the newest 4.x

Yes, I say go back to 4.7.1 and let us know how it works out. If it’s good, I think we should roll back globally until v5 has the kinks worked out.

https://github.com/99designs/aws-vault/issues/552

Added the following to my Dockerfile:

# Pin aws-vault to a version <5.0
# There are bugs with aws credential caching that make version 5 more annoying to use; see:
# <https://github.com/99designs/aws-vault/issues/552>
RUN apk del aws-vault && apk add aws-vault@cloudposse==4.7.1-r0

Will see if that helps

@Alex Siegman before rolling back aws-vault to v4, make sure you have set up the new TTL variables

in v5? i can try that

I assume you mean these?

AWS_SESSION_TOKEN_TTL: Expiration time for the GetSessionToken credentials. Defaults to 1h
AWS_CHAINED_SESSION_TOKEN_TTL: Expiration time for the GetSessionToken credentials when chaining profiles. Defaults to 8h
AWS_ASSUME_ROLE_TTL: Expiration time for the AssumeRole credentials. Defaults to 1h
AWS_FEDERATION_TOKEN_TTL: Expiration time for the GetFederationToken credentials. Defaults to 1h

The session token one is probably the big hoser here, should be 12h to operate like it did previously

That said, if the credentials aren’t cached, if assume_role_ttl is 1h, it will still ask every hour, so you’d have to change all of your profiles in AWS to allow >1h timing

But maybe bumping the session is enough

Yes, I was looking for where they were defined. You need to set them all, except maybe FEDERATION

I’ll play with those before reverting to 4.7.1 and report back. I just logged in with them set, should know in an hour or so I’d imagine

The idea with aws-vault is that you log in and it caches your session, including your MFA, so you can keep using it to assume roles.

Yeah, but if it doesn’t cache your session, it can’t re-assume roles without new MFA input I’d imagine

case in point, i just did an aws-vault --debug list it decided my existing session was expired and deleted it, but my session is still valid (can use awscli) yet aws-vault shows no sessions for my profile

seems funky, now that I know where to look though I’ll play with it

I was excited about aws-vault v5 because the fixed some of the problems with running aws-vault exec directly, which running role-server was a workaround. But as long as we have role-server, if changing the TTLs in the env vars doesn’t work, then yes, go back to 4.7.1 and role-server.

Just a note - I couldn’t pin because –duration is an unsupported option in 4.7 (https://github.com/cloudposse/geodesic/blob/59d9637c32c29d3097859ad3d2c76fef0fb802b9/rootfs/etc/profile.d/aws-vault.sh#L132)

@Alex Siegman @Joe Niland Please check if you have AWS_CHAINED_SESSION_TOKEN_TTL set as well.

@Alex Siegman @Joe Niland @Erik Osterman (Cloud Posse) PR for Geodesic to re-enable support for aws-vault version 4, plus clean up some issues that may have been causing problems with aws-vault version 5. I suggest trying the new Geodesic (when it is released) with v5 before reverting to v4. https://github.com/cloudposse/geodesic/pull/579

@Jeremy G (Cloud Posse) thank you - this is great!

OK, Geodesic version 0.130.0 is released. Give it a try and please report back.

Using just 0.130.0 with no changes, I still had timeouts due to aws-vault still not properly caching credentials it seems.

Will use your method to revert to aws-vault v4 and retest as able

How’s it is going?

I haven’t been able to get v5 to work refreshing stuff. I have not yet reverted as it was the weekend and I didn’t bother to test it yet

I just did the reversion, and initial debugging looks like it will work fine. I can see it caching sessions properly as well. Will use over next day or two to be sure. Should know later today if it properly renews sessions after 1hr timeout

Still doesn’t look like it’s working quite right. I have a valid session, the first hour has gone by so my role assumption should be expired and thus need refreshing, but using something that needs it to be refreshed does not cause it to be.

 √ : [spoton-staging-admin] customer-auth-service ⨠ aws-vault --debug list
2020/04/28 04:58:35 [keyring] Considering backends: [file]
2020/04/28 04:58:35 Using AWS_CONFIG_FILE value: /localhost/.aws/config
2020/04/28 04:58:35 Loading config file /localhost/.aws/config
2020/04/28 04:58:35 Parsing config file /localhost/.aws/config
2020/04/28 04:58:35 [keyring] Expanded file dir to /conf/.awsvault/keys/
2020/04/28 04:58:35 Looking up all keys in keyring
2020/04/28 04:58:35 [keyring] Expanded file dir to /conf/.awsvault/keys/
2020/04/28 04:58:35 Session "session,c3BvdG9u,YXJuOmF3czppYW06OjE0ODQ4NzQwMjkxOTptZmEvYXNpZWdtYW5Ac3BvdG9uLmNvbQ,1588089054" expires in 10h52m18.485944511s
Profile                  Credentials              Sessions
=======                  ===========              ========
default                  -
spoton                   spoton                   1588089054 (mfa)
spoton-corp-admin        spoton
spoton-staging-admin     spoton
 ⧉  staging
 √ : [spoton-staging-admin] customer-auth-service ⨠ aws s3 ls
Unable to locate credentials. You can configure credentials by running "aws configure".

Container startup seemed normal

# Exposing port 34060
* aws-vault: overriding AWS_SESSION_TTL () with AWS_VAULT_SESSION_TTL (12h)
* assume-role will start EC2 metadata service at <http://169.254.169.254/latest>

It does not, however, reask for MFA to get new session credentials if I exit out and rerun assume-role so it is properly caching that.