#refarch (2023-02)

@Ben Smith (Cloud Posse) @Dan Miller (Cloud Posse)

I chatted with him yesterday

@Christopher Pieper let us know if you need more help on this

@Andriy Knysh (Cloud Posse) @RB @Jeremy White (Cloud Posse)

@johncblandii I belive to use SSO with terraform, you have to use AWS profiles, not roles

for example, each component in this repo has https://github.com/cloudposse/terraform-aws-components/blob/master/modules/aurora-mysql/providers.tf

are you using profiles or roles?

I did set profiles_enabled: true on account map

let me make sure i ran everything with that in place

we actually using IAM roles to work with TF in geodesic. I beleive we did it with SSO and profiles a few times and encountered a lot of issues (I don’t remember all the details, it was long time ago)

how would i use roles + ssm? it seems when i use assume-role it selects a profile, but assuming again always shows my SSO user

and would we have the same issue on spacelift?

nm…it wouldn’t use SSO…ignore last question

we use roles with Spacelift

for the same reason that the roles generated by SSO are dynamic

the dynamic role above does not have the permissions b/c the aws-teams, aws-team-roles components don’t know anything about it - so the role arn:aws:iam::9876543210:role/xy-core-gbl-audit-admin does not have permissions for the dynamic roles to assume it

if you use profiles, you need to make sure you use profiles everywhere

for that reason, we don’t use profiles in geodesic and Spacelift

we provision the IAM roles which we use with Spacelift and to work with TF in geodesic

we can also provision SSO for access to AWS console

so this seems to say it can assume the role

console access w/ sso does work here

keep in mind that for cross-account access, you need to have permissions on both sides

just need to be able to run this locally to verify plans/create some infra for spacelift deploy

the primary role has to have a trust policy to allow the delegated (or dynamic from SSO) roles to assume it

the delegated (or dynamic SSO) roles need to have permission to assume the primary role

meaning the identity-admin, right?

if so, that works

or do you mean the [account]-admin?

identity-admin needs to have a trust policy to allow the dynamic SSO roles to assume it

yes, that’s there

same aws-reserved principal

the dynamic SSO roles need to have permissions to assume identity-admin

i would go to AWS console in the two account and check the trust policy in one and the permissions in the other (it’s also easier to manually change it until it works)

that’s what i was verifying and both sides look accurate

identity: trusts IdentityAdminRoleAccess assume + gbl-identity-admin

audit: trusts same from current account and identity

maybe you did not show everything, but this

aws:PrincipalArn": [
                        "arn:aws:iam::123456789:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_AdministratorAccess_*",
                        "arn:aws:iam::123456789:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_IdentityAdminRoleAccess_*",
                        "arn:aws:iam::987654321:role/xy-core-gbl-identity-admin",
                        "arn:aws:iam::987654321:role/xy-core-gbl-identity-spacelift"
                    ]

does not include anything about this

User: arn:aws:sts::0123456789:assumed-role/AWSReservedSSO_IdentityAdminRoleAccess_xzypdq/[email protected] is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::9876543210:role/xy-core-gbl-audit-admin

identity sso -> xy-core-gbl-identity-admin -> audit

^ those assumes would take the last principal there, right?

but assume seems to ignore the 1st assume

i would just allow the account’s roots on both sides to assume and trust to check if it works in principal

ok. i’ll try that real quick

yeah, forcing the sso user on there from identity works so i’ll likely need to update this module to support that

there is aws_saml_login_enabled but not sso-specific.

so we basically need to duplicate this and set the account to the identity acct:

format("arn:%s:iam::%s:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_%%s_*", local.aws_partition, module.account_map.outputs.full_account_map[acct]),

you can try it first in the AWS console. Once it’s working, update the component. (you can DM me or somebody else from CP to review the code together)

i did

i can upstream this as well

i have another PR i need to do as well to address aws-sso so it supports core-identity as an account name and still works

i will fill out the specifics in a little bit: https://github.com/cloudposse/terraform-aws-components/pull/567.

gotta handle a meeting

updated description

not sure if this is the right channel btw

Yes, we’ve done this using context filters

See the upstreamed spacelift component

https://github.com/cloudposse/terraform-aws-components/tree/master/modules/spacelift

Cool! So if we’re going from a single admin stack to multiple admin stacks (per tenant), would we manually initialize the new admin stacks via the Spacelift UI?

Or perhaps… the existing admin stack could create and manage child admin stacks??? e.g.

• infrastructure (original admin stack): ◦ tenant0-infrastructure ◦ tenant1-infrastructure ◦ . . . ◦ tenantN-infrastructure

We’ve implemented the infra admin stack which creates a specific set of child stacks, policies, and spaces

We have discussed an admin stack of admin stacks which is probably the best way to do this

Oh and you can also split up admin stacks by tags as well which is a recent feature added to the spacelift automation module

(congrats on hitting 1200 stacks!)

We have another customer up to 1500 stacks and we broke each space into it’s own admin stack, and organized spaces by team.

@RB were you going to be working on creating a component for each spacelift-space? I think this was a new optimization we recently talked about. This way the spaces themselves have their own terraform state, which should further improve performance.

I believe Sean mentioned he was working on one. Id be happy to create one too

Yeah, I wrote one which creates all of our spaces based on data out of our account-map remote-state

Oh nice! I’d love to check it out. Have you considered upstreaming to terraform-aws-components?

Yeah I definitely thought about it, but was afraid it might be too specific to our org since I also rolled in some Okta group management stuff into it as well

Feel free to take a look (assuming y’all still have access to our monorepo)

If you’re open to it @Sean Nguyen we can maybe generalize it. I think it relates to some work that @RB might be undertaking.

That’s fine with me. None of what we’ve implemented is particularly fancy.

FWIW, it might be a little more ergonomic to allow users to specify Space membership using the spacelift_space_by_path datasource ( perhaps as an alternative rather than in-lieu of space_id).

It seems that user-created spaces (not legacy and not root ) spaces have some suffix attached to them. For example, a user-created space with name core-dev might have a space_id which is called core-dev-01ABCDEF

@Erik Osterman (Cloud Posse) https://github.com/cloudposse/terraform-aws-components/pull/587

Thanks a lot!

for context, we do not see PRs triggering admin stack runs

It needs one more policy

https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation/blob/master/catalog/policies/git_push.administrative.rego

ah, ok. i figured. the repo doesn’t have it referenced, though

testing it out

that worked. sweet. i’ll push a PR to have that updated in the README/etc (unless there is one already)

pretty much a stock deploy here. nothing extra special done with it

yup. removed the swap lines and the worker starts up just fine

it seems like the first line is the culprit

dd if=/dev/zero of=/swapfile bs=128M count=32 2>/var/log/spacelift/error.log

@Jeremy White (Cloud Posse) any ideas?

i tried adjusting the size of the swap and the size of the instance. neither worked

If you ssh to the node, and run it manually what do you see?

Maybe a small node type does not have 4GB of free space?

(Oh disregard, I see you already tried that)

I haven’t set it up to ssh yet. I may do that today

permission error on the spacelift log file @Erik Osterman (Cloud Posse)

does user-data run in sudo mode by default?

well that didn’t work with sudo either.

sudo dd if=/dev/zero of=/swapfile bs=128M count=32 2>/var/log/spacelift/error.log
sh: /var/log/spacelift/error.log: Permission denied

took off the log and get this:

sh-4.2$ sudo dd if=/dev/zero of=/swapfile bs=128M count=32
dd: error writing '/swapfile': No space left on device

(increasing the size, btw; should be good after)

Ah! makes sense. Can you upstream any fixes?

no fix needed. it already supported a block device mapping so just added a bigger volume

well, i could update the README example so that’s clear. i’ll do that