#refarch (2023-03)

Ignore. The referenced account id was wrong.

ok so the provider is using the assume role from the same role into the same role within the same account. that’s the issue

Any thoughts here @Andriy Knysh (Cloud Posse) @RB?

1. This looks strange arn:aws:sts::1234567890 - 1234567890 is not a real account (search for it in the repo, might be hardcoded)

i just replaced the actual account id

1. If #1 is fixed and still, not working, maybe the gbl-identity-admin does not have permissions to assume gbl-dns-terraform

1. Or gbl-dns-terraform does not have a trust policy to enable gbl-identity-admin assuming it (#2 and #2 can be checked in AWS console in the two accounts)

it is using module.iam_roles.dns_terraform_profile_name to assume role on module.iam_roles.terraform_profile_name and both are the -dns-terraform role

1. Also check if [providers.tf](http://providers.tf) is configured correctly for dns-primary and dns-delegated

it is using module.iam_roles.dns_terraform_profile_name to assume role on module.iam_roles.terraform_profile_name and both are the -dns-terraform role

check both roles in AWS console, and check if the account-map is provisioned correctly (atmos terraform output … should show all the accounts and all the roles)

1. it trusts gbl-identity-admin and gbl-identity-spacelift

1. it is from the latest from tf-aws-modules

it is essentially a self-referential assume

-terraform to -terraform

my initial assume to -dns-terraform works, but it assumes at the provider level and doesn’t

you need to run atmos terraform apply dns-delegated/primary after you assume gbl-identity-admin role

my initial assume to -dns-terraform works, but it assumes at the provider level and doesn’t

that’s what you don’t need to do in geodesic

you have to assume gbl-identity-admin and then execute commands to apply dns-delegated

clarity: i’m not assuming i mean me using my identity account works to access the terraform role

RequestID: a-b-c-d-e, api error AccessDenied: User: arnsts:assumed-role/sm-core-gbl-dns-terraform/aws-go-sdk-1677817372266350530 is not authorized to perform: sts:AssumeRole on resource: arniam:role/sm-core-gbl-dns-terraform

that’s from spacelift there

it says that you are not using gbl-identity-admin , something else

Effective AWS Role Arn is arnsts:assumed-role/sm-core-gbl-identity-spacelift/Spacelift

yeah, this being true is what triggers the failure… module.iam_roles.profiles_enabled because module.iam_roles.terraform_profile_name is used

adjusting the assume_role section of the [provider.tf](http://provider.tf) in both dns-delegated and dns-primary

@Andriy Knysh (Cloud Posse) should I upstream this or is this not the proper fix?

if the code is from terraform-aws-components, then yes, please upstream when you get a moment ([providers.tf](http://providers.tf) should be the same for all components)

it is and will do

ok so the provider is using the assume role from the same role into the same role within the same account. that’s the issue Until recently, AWS IAM implicitly allowed this. They recently removed this in the interest of security, and in most cases Cloud Posse does not add rules to allow it.

Ah, that makes sense

Sorry to necro this, but I actually just ran into this myself. I cloned down an older ref-arch repo and went through all the initialization steps, and when I run terraform-plan, I’m met with:

User: arn:aws:sts::<account-id>:assumed-role/OrganizationAccountAccessRole/<session-id> is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<account-id>:role/OrganizationAccountAccessRole

Is there a quick fix I can apply within the refarch repo, or do I need to wait for fixes to be upstreamed to the terraform modules?

@jwood that role cannot perform assume so you need to run it with one of the other roles (ie - -terraform-admin)

Right, what’s happening is the IAM user has the ability to assume the AccountAccess role, which happens when I run assume-role <name>-staging-admin from geodesic. But then when I run terraform plan, the aws provider tries to assume the role it is already in, set by the env var TF_VAR_aws_assume_role_arn

For context, this is a pretty old clone of the refarch, circa June ‘21. I’d like to get the project updated and fixed up to use newer geodesic versions, but I’m not sure if there’s a good upgrade path. Are there any docs on how to update the refarch-generated project?

@jwood You should be in the <namespace>-identity-admin role when running Terraform, or just in general under the current and previous refarch.

Unfortunately, upgrading from iam-primary-roles (old) to aws-teams (current) is so painful we have put off documenting how to do it. Self-serving as it is, this is a task I would strongly recommend you pay Cloud Posse to do for you. (That is, unless you are in a state where you can completely tear down all your deployments and infrastructure and essentially start over from the point of having empty accounts.)

Thanks @Jeremy G (Cloud Posse)! It’s in a state where a teardown is possible, since I want to re-architect our stuff in AWS anyway.

aws-sso (for AWS SSO) and aws-saml are the latest versions

I just deployed aws-sso @Michael Dizon so if i can help…i will

cool, i’m going thru it now, will update with progress soon

i’m not using okta, is that a requirement? @johncblandii

no, but a provider is required

google, auth0, etc

provider groups/users + their assignments get synced to AWS SSO

yeah i am using google

I used Okta with my setup so can’t give the Google details, but I could help with the configs

@johncblandii ran into this error when trying to deploy

╷
│ Error: creating SSO Permission Set (PowerUserAccess): ConflictException: PermissionSet with name PowerUserAccess already exists.
│ 
│   with module.permission_sets.aws_ssoadmin_permission_set.this["PowerUserAccess"],
│   on .terraform/modules/permission_sets/modules/permission-sets/main.tf line 4, in resource "aws_ssoadmin_permission_set" "this":
│    4: resource "aws_ssoadmin_permission_set" "this" {
│ 
╵

also, i’m not sure how to go about setting up google workspace with sso. i did this last year with the previous sso component. but don’t recall what i did back then lol

you need to do the ClickOps section first in Identity Center https://github.com/cloudposse/terraform-aws-components/blob/master/modules/aws-sso/README.md

how’s your yaml config? did you create any manual permission sets?

no, the yaml is direct from the README

i’ve gone through the click ops steps

is there a particular order that these modules need to be applied in? i’ve deployed aws-teams, aws-team-roles, and aws-saml

• teams

• team-roles

• aws-sso

don’t need aws-saml

you vendored the component direct from terraform-aws-components?

when you go to Identity Center in the console do you see permission sets already in there?

yeah from terraform-aws-components

and yeah the permission sets are there

i commented out local.poweruser_access_permission_set,and local.terraform_update_access_permission_set and was able to deploy

i see TerraformUpdateAccess but it says it’s not provisioned

PowerUserAccess is there too, and it is provisioned

@johncblandii there is a chance that someone had created those permission sets before me

absolutely

@johncblandii confirmed that was the case

That’s correct. There is also the aws-sso component which does the AWS SSO (IAM Identity Center) instead of the IAM SAML integration.

Ah maybe this is already covered in the above thread – My bad.

hah, nah, mostly my bad. i don’t know why i keep mixing those components up

maybe you have some Service Control policies provisioned that require instances with encryption in transit only

like this one https://github.com/cloudposse/terraform-aws-service-control-policies/blob/master/catalog/ec2-policies.yaml#L69

whoa

look in the Org’s root account

yep, found it

thanks again!

a root account is in use, but it is assuming the terraform user first

NOTE: profile = null is a solution that prevents the [stage]-terraform user being set, but the provider uses this by default for us sense profiles are enabled.

provider "aws" {
  region = var.region

  profile = null

  dynamic "assume_role" {
    for_each = var.import_role_arn == null ? (module.iam_roles.org_role_arn != null ? [true] : []) : ["import"]
    content {
      role_arn = coalesce(var.import_role_arn, module.iam_roles.org_role_arn)
    }
  }
}

so for profiles_enabled scenarios, we’re blocked without changing the actual provider code

also, it seems to be requesting increases even when increases were already requested

any thoughts @Andriy Knysh (Cloud Posse)?

sorry, I’m not in the context what was done and how, so can only guess. Are you using profiles or roles?

aws sso so profiles

@Dan Miller (Cloud Posse)

I don’t have an example on hand, but can you share your implementation s.t. we can point you towards what might be missing?

That attribute needs to be a list of objects in this format:

    kubernetes_taints = list(object({
      key    = string
      value  = string
      effect = string
    }))

https://github.com/cloudposse/terraform-aws-components/blob/master/modules/eks/cluster/variables.tf

i passed an empty array and that seemed to deploy. i will look at it in the morning to see if everything went ok.

Relevant code:

locals {
  enabled  = module.this.enabled
  zone_map = zipmap(var.zone_config[*].subdomain, var.zone_config[*].zone_name)

  private_enabled = local.enabled && var.dns_private_zone_enabled
  public_enabled  = local.enabled && !local.private_enabled

  private_ca_enabled = local.private_enabled && var.certificate_authority_enabled

  aws_route53_zone = local.private_enabled ? aws_route53_zone.private : aws_route53_zone.default

  vpc_environment_names = toset(concat([var.vpc_primary_environment_name], var.vpc_secondary_environment_names))

  aws_partition = join("", data.aws_partition.current.*.partition)
}

resource "aws_route53_zone" "default" {
  for_each = local.public_enabled ? local.zone_map : {}

  name    = format("%s.%s", each.key, each.value)
  comment = format("DNS zone for %s.%s", each.key, each.value)


  tags = module.this.tags
}

@Ben Smith (Cloud Posse) @Dan Miller (Cloud Posse)

I believe this is a bug with how we’ve written the Terraform component. However, if you wanted to work around it, you could try defining a separate component like this:

components:
  terraform:

    dns-primary:
      vars:
        domain_names:
           - acme-dev.com

    dns-delegated:
      vars:
        zone_config:
          - subdomain: dev.plat
            zone_name: acme-svc.com

    dns-delegated-alt:
      metadata:
        component: dns-delegated
        inherits:
          - dns-delegated
      vars:
        zone_config:
          - subdomain: dev.plat
            zone_name: acme-svc-alt.com

yeah, that’s exactly what i ended up doing

ah okay great!

I think a separate component instance is a better long-term solution too and the direction should advise.

That way the state isn’t tightly coupled.

and along those lines, any plans to detach it from eks?

I already had these changes so made them. I didn’t include my removal of EKS, though.

https://github.com/cloudposse/terraform-aws-components/pull/602

I went ahead and did these changes in the core module too since it still breaks due to old versions:

https://github.com/cloudposse/terraform-aws-mq-broker/pull/54

I’m unsure if the tests are complete here to validate it, but it definitely needs validating before merging

#602 (refarch) is updated to add more outputs and available vars

I tested both of these (local refarch usage) with a local install and the plan worked

@Linda Pham (Cloud Posse)

@johncblandii reviewed, and changes were requested

Everything was addressed except a separate repo needing a new version.

Also, this PR is still pending https://github.com/cloudposse/terraform-aws-components/pull/602

Thanks! Let me get another review

@johncblandii, @Andriy Knysh (Cloud Posse) requested additional changes

Done

This one was updated too:

https://github.com/cloudposse/terraform-aws-mq-broker/pull/54

and unrelated but this one too https://github.com/cloudposse/terraform-aws-components/pull/602

oops…meant this one https://github.com/cloudposse/terraform-aws-components/pull/597

I have a concept I think could work here. I’m just curious if there are some already defined patterns

i’d def love to see this

this disables the required sops integration and adds support for valueFrom.

this update to the var was needed too

which module does this modify?

ssm-parameters

NOTE: this was following these steps:

• deploy w/ CHANGEME value

• clickops change value

• deploy same code

• error

obviously the clickops shouldn’t be there, but this is largely testing out boundaries as to what could happen here

if overwrite: false and terraform wants to update the parameter, the error will be thrown

yes

shouldn’t it ignore it?

create once, ignore after?

sidebar: have any good SOPS examples/tutorials? I’m not familiar w/ it and we might go that route

yeah, setting overwrite: true makes the value CHANGEME again; as expected

it makes me wonder if we should include this:

  lifecycle {
    ignore_changes = [value]
  }

seems to work as expected

but it might have higher ramifications

we are testing it. You don’t have to do anything to use it, just enable it. You use the same settings.spacelift.depends_on section to describe deps. We are looking into the workflows and how it all should be used. You can try it as well

Thanks! I’m hoping this might make CI checks for Spacelift stacks a little faster on our monorepo considering the old way is definitely a little eager

@Jeremy White (Cloud Posse)

Just be aware that the dependency resource will prevent deletion of stacks if you don’t remove the depends_on . That is, they also function as a sanity check to avoid deleting a stack that is “depended on” by another stack.

@Ben Smith (Cloud Posse) @Jeremy White (Cloud Posse) @Dan Meyers

was your session expired?

i don’t think so. the weird thing is that i switched to (what i think) is an identical stack for a different project, and switched to that iam profile, and it works.

Leapp refreshed the session

hmm, even toggling between different sessions?

its able to pull the state though

i get this in both projects with the tfstate-backend component:

╷
│ Error: reading IAM Role (xxx-gbl-xxx-tfstate): InvalidClientTokenId: The security token included in the request is invalid
│       status code: 403, request id: 853af46f-83d4-4c9b-b925-a4e7f52768bb
│ 
│   with aws_iam_role.default["duetti-gbl-duetti-tfstate"],
│   on iam.tf line 66, in resource "aws_iam_role" "default":
│   66: resource "aws_iam_role" "default" {
│ 
╵

Are you using the SuperAdmin user?

my iam user does have Administrator privileges, is that what you mean?

update: i was able to resolve this by adding MFA to my IAM user. I don’t *think* I’ve ever had to do this before.

also thanks @Andrea Cavagna

No problem @Michael Dizon

AWS for security reasons does not allow IAM user without MFA to call any IAM APIs.

Thanks why the problem sound like token expired.

All those corner case and IAM knowledge is being collected into the IAM blog