#terraform-aws-modules (2023-07)
Terraform Modules
Discussions related to https://github.com/terraform-aws-modules
Archive: https://archive.sweetops.com/terraform-aws-modules/
2023-07-03
Hi, seeing various patterns like source = "../account-map/modules/iam-roles" (eg https://github.com/cloudposse/terraform-aws-components/blob/a63341e8fb54178294c4f42be78a9b2e0c205870/modules/vpc/providers.tf#L17), basically one module referring to modules in another module.
This seems to introduce some versioning / consistent state when not all modules are updated at the same time. (and besides introduce implicit dependencies)
2023-07-04
Hi Everyone! About a month ago we have rolled out the new github shared workflows to all our terraform modules. Here are the docs covering this change: https://docs.cloudposse.com/community/automated-testing/
tl;dr
feature-branch/*is requried and runs automatically on every commit to a PR. You can also run it locally withmake precommit/terraform. This will do auto-format and build readme for you. We no longer do auto-commits since it’s a bad security practice.test/terratestis required and runs on request, using the/terratestcomment on a PR. You need to have at leasttriagelevel of access on a repo to trigger it.
Here’s a proposal to include this docs ref into a PR template https://github.com/cloudposse/build-harness/pull/353
2023-07-11
Hi,
I’d like to add (optional) alarms to cloudfront-cdn/aws, do you prefer those as a new module or to be added directly?
2023-07-12
2023-07-19
2023-07-26
I may be wrong, but I dont believe account-map (at this current time) is not ready for account names in the {tenant}-{stage} format. The issue isn’t with deploying the account-map component. Issues arise at lookup-time for the account-map component via cloudposse/stack-config/yaml//modules/remote-state .
e.g. If account-map dns_account_account_name variable is dns, terraform will error (The given key does not identify an element in this collection value) whenever it tries to get information about the dns account from account-map becuase the key for the dns account should be core-dns.
This may also be as simple as updating the descriptions of *_account_account_name variables in account-map. Currently, the description says it should be the stage name, not the {tenant}-{stage} name if using that format.
Yep. I am pretty sure the description is outdated. Ill submit PR to fix with others.
@Brian Were you able to address this issue? If yes, do you mind sharing your solution? I am running into the same problem.
Did you figure it out?
May be :slightly_smiling_face:. I don’t know if this is the correct solution or not, but it worked. I updated the names in the account-map component definition on the catalog like this.
components:
terraform:
account-map:
metadata:
component: infra/account-map
vars:
enabled: true
# Set profiles_enabled to false unless we are using AWS config profiles for Terraform access.
# When profiles_enabled is false, role_arn must be provided instead of profile in each terraform component provider.
# This is automatically handled by the component's `provider.tf` file in conjunction with
# the `account-map/modules/iam-roles` module.
profiles_enabled: false
# root_account_aws_name: "aws-root"
# root_account_account_name: root
root_account_aws_name: "xxxxxx"
root_account_account_name: xxxxxx
identity_account_account_name: core-identity
dns_account_account_name: core-dns
audit_account_account_name: core-audit
artifacts_account_account_name: core-artifacts
# The following variables contain `format()` strings that take the labels from `null-label`
# as arguments in the standard order. The default values are shown here, assuming
# the `null-label.label_order` is
# ["namespace", "tenant", "environment", "stage", "name", "attributes"]
# Note that you can rearrange the order of the labels in the template by
# using [explicit argument indexes](<https://pkg.go.dev/fmt#hdr-Explicit_argument_indexes>) just like in `go`.
# `iam_role_arn_template_template` is the template for the template [sic] used to render Role ARNs.
# The template is first used to render a template for the account that takes only the role name.
# Then that rendered template is used to create the final Role ARN for the account.
iam_role_arn_template_template: "arn:%s:iam::%s:role/%s-%s-%s-%s-%%s"
# `profile_template` is the template used to render AWS Profile names.
profile_template: "%s-%s-%s-%s-%s"
This seems to get me through this issue. I have also had to add updated descriptor_formats to my stack defs:
terraform:
vars:
descriptor_formats:
account_name:
format: "%v-%v"
labels:
- tenant
- stage
stack:
format: "%v-%v-%v-%v"
labels:
- namespace
- tenant
- environment
- stage
Yep. The descriptor_formats was the change that fix the error.
I will say putting SSO in the dedicated account called identity seemed like a good idea on paper, but it became a pain.
CloudPosse position on using a non-root account to control SSO reverted. They use root for SSO as do I now.
… at least that’s what I read in one of the README or comments in one of related component modules.
My understanding was that SSO needs to be deployed into root account. However, I haven’t been able to successfully accomplish that yet.
Yes. It should. If you’re having issues and that account-map you pasted is your current config, then that’s the issue.
identity_account_account_name should be be core-root (or root if you dont prefix with tenant).
Also, make sure the feature is enabled in your root AWS account.
The feature must be manually enabled.
The feature is enabled - but the account itself is called something else though. For instance, let’s say the primary/root account I created is called “Rabbit” - I used that name in the settings I described above. I am indeed getting an error that says “core-root” account is not found. Does that mean that the actual name of the account in AWS should be “core-root”?
That’s how I have mine configured. But it’s possible to configure so that “Rabbit” maps to the root account too. I don’t recall how to do it off the top of my head
Bruises from multiple implementations tell me that this account structure step needs careful planning Do you know if the “root” account can be something other than the master account we create when we start off with AWS? OR does it need to be the same? I will try to look up the forum on how to map a custom account name to root account.
I have implemented CloudPosse’s SweetOps (as best as I can infer since I am not a paying client so not able to review their docs) patterns for the AWS organizations for my previous startup employer, current startup employer, self-owned consulting company, and a startup client.
I will say the cold start process (deploy of tfstate-backend, account, account-map, aws-sso, aws-teams, aws team-roles, and a few other) must be thought out. However, once its in place, everything else is smooth.
That’s what I am hoping to do on my end. This would be the first time I am trying out the SweetOps framework, along with Atmos. Good to hear you had success with the framework though. Appreciate you sharing your thoughts and experiences.
@Brian these are perhaps the variables you were referring to. I am assuming one of them points to the ACTUAL name of the root account on AWS and the other one maps to the “formatted” root account name.
root_account_aws_name: MasterAccount
root_account_account_name: core-root
Goes in account-map component definition
Yep.
@Brian Quick (hopefully) follow up question. I think the above changes fixed some of the issues I had with aws-sso component. However, I am now getting the following error. It looks like the terraform user I created in the root account (terraform) account is not authorized to assume the role mentioned in the error. The challenge is that the role shown in the error actually does not exist in the root account. Am I missing a step in creating aws-teams and aws-tem-roles?
Error: Cannot assume IAM Role
│
│ with provider["registry.terraform.io/hashicorp/aws"],
│ on providers.tf line 28, in provider "aws":
│ 28: provider "aws" {
│
│ IAM Role (arn:aws:iam::xxxxxxxxxx:role/hct-core-gbl-root-admin) cannot be assumed.
│
│ There are a number of possible causes of this - the most common are:
│ * The credentials used in order to assume the role are invalid
│ * The credentials do not have appropriate permission to assume the role
│ * The role ARN is not valid
│
│ Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID:
│ d383ead8-bcae-4e47-aa57-a25ee5aea728, api error AccessDenied: User: arn:aws:iam::xxxxxxxxxx:user/terraform is not
│ authorized to perform: sts:AssumeRole on resource: arn:aws:iam::xxxxxxxxxx:role/hct-core-gbl-root-admin
In the root account, things are more restrictive. Generally speaking, the cicd/terraform role cannot assume the admin role. At most, it can assume poweruser role.
I dont use users. All identities in my org are roles, including the one used for cicd/terraform.
Don’t you need a user that assumes these roles though? My approach was to create an IAM user with enough permissions to execute terraform scripts in the rest of the accounts - until I am able to create some users via aws-sso. Are you saying there is way to use roles and not create users at all?
For the cold start, I generally use root account aws tokens until I deployed these components:
• tfstate-backend
• account
• account-map
• aws-sso
• aws-teams
• aws team-roles
Once aws-team-roles is in place, then the AWS SSO paired with the role created by the aws-team-roles (applied to all account except root account) will working together.
… And I delete the tokens of the root user.
Durning the cold start, you will need to disable using a role to interact with the remote state in S3. This example of disabling it for aws-teams
components:
terraform:
aws-teams:
backend:
s3:
role_arn: null # <--------
vars:
# ...
Once cold start is complete, I delete the backend override for the 6 components.
This makes sense @Brian and that is where I was heading. My plan was to stop using the terraform user I created and start using the SSO and roles framework. However, my issue is that I am not able to move past the aws-sso module due to error posted above. I did not make many changes to example provided in the documentation - was trying to provision as-is. It looks like the script is trying to have the terraform user assume a role in root account that cannot be assumed due to permissions.
Also, did you say aws-team-roles creates roles in the rest of the accounts? I see one role created in root but no roles in the rest of the accounts, especially the workload accounts.
aws-sso only recreates roles that can be assume by human users. aws-teams and aws-team-roles create roles that can be assume by any type of identity.
aws-teams is applied in root account with aws-sso roles. SSO roles can be allowed assume aws-teams roles.
aws-team-roles is deploy in all the the other accounts.
aws-sso roles, aws-team roles, and other identities (e.g. cicd role/user) in the root are configured to assume roles created in the non-root accounts created by aws-team-roles.
aws-teams and aws-team-roles are CloudPosse’s 2nd iteration (at least) to solve this problem. 2+ years ago, during my first implementation of CP SweetOps, they had another model/pattern that I didn’t like so I created my own. However, this new pattern I do like so I have used it for the past 3 implementation of CP’s SweetOps.
components:
helmfile: {}
terraform:
aws-sso:
vars:
enabled: false
privileged: true
account_assignments:
core-root:
groups:
role.cloud-engineer-superadmin:
permission_sets:
- AdministratorAccess
- PowerUserAccess
- ReadOnlyAccess
- IdentityAdminTeamAccess
- IdentityDevopsTeamAccess
role.cloud-engineer:
permission_sets:
- ReadOnlyAccess
- IdentityDevopsTeamAccess
core-audit:
groups:
role.cloud-engineer-superadmin:
permission_sets:
- AdministratorAccess
- PowerUserAccess
- ReadOnlyAccess
role.cloud-engineer-admin:
permission_sets:
- PowerUserAccess
- ReadOnlyAccess
role.cloud-engineer:
permission_sets:
- ReadOnlyAccess
core-dns:
groups:
role.cloud-engineer-admin:
permission_sets:
- AdministratorAccess
- PowerUserAccess
- ReadOnlyAccess
role.cloud-engineer:
permission_sets:
- PowerUserAccess
- ReadOnlyAccess
core-tools:
groups:
role.cloud-engineer-admin:
permission_sets:
- AdministratorAccess
- PowerUserAccess
- ReadOnlyAccess
role.cloud-engineer:
permission_sets:
- PowerUserAccess
- ReadOnlyAccess
plat-dev:
groups:
role.cloud-engineer:
permission_sets:
- AdministratorAccess
- PowerUserAccess
- ReadOnlyAccess
role.software-engineer:
permission_sets:
- AdministratorAccess
- PowerUserAccess
- ReadOnlyAccess
plat-staging:
groups:
role.cloud-engineer:
permission_sets:
- AdministratorAccess
- PowerUserAccess
- ReadOnlyAccess
role.software-engineer:
permission_sets:
- PowerUserAccess
- ReadOnlyAccess
aws_teams_accessible:
- "admin"
- "devops"
aws-teams:
vars:
enabled: false
teams_config:
viewer: &user-template
max_session_duration: 43200 # 12 hours in seconds
role_policy_arns:
- "arn:aws:iam::aws:policy/job-function/ViewOnlyAccess"
role_description: Team restricted to viewing resources in the identity account
aws_saml_login_enabled: false
trusted_permission_sets: [ ]
denied_permission_sets: [ ]
trusted_teams: [ ]
denied_teams: [ "viewer" ]
trusted_role_arns: [ ]
denied_role_arns: [ ]
admin:
<<: *user-template
role_description: Team with PowerUserAccess permissions in `root` and AdministratorAccess
role_policy_arns: [ "arn:aws:iam::aws:policy/PowerUserAccess" ]
trusted_teams: [ admin ]
trusted_permission_sets: [ AdministratorAccess, IdentityAdminTeamAccess ]
devops:
<<: *user-template
role_description: Team with PowerUserAccess permissions in `root` and AdministratorAccess
role_policy_arns: [ "arn:aws:iam::aws:policy/PowerUserAccess" ]
trusted_teams: [ "devops" ]
trusted_permission_sets: [ AdministratorAccess, IdentityDevopsTeamAccess ]
terraform:
<<: *user-template
role_description: Team with PowerUserAccess permissions in `root` and AdministratorAccess
role_policy_arns: [ "arn:aws:iam::aws:policy/PowerUserAccess" ]
trusted_teams: [ admin, devops ]
trusted_permission_sets: [ AdministratorAccess, IdentityAdminTeamAccess, IdentityDevopsTeamAccess ]
trusted_github_repos:
terraform: ["nxt-big-startup/*"]
aws-team-roles:
vars:
enabled: true
roles:
template: &user-template
enabled: false # the role will not be created in this account
role_description: Template role, should not exist
max_session_duration: 7200 # 2 hour in seconds
aws_saml_login_enabled: false
role_policy_arns: []
trusted_permission_sets: []
denied_permission_sets: []
trusted_teams: []
trusted_role_arns:
- arn:aws:iam::000000000000:role/aws-reserved/sso.amazonaws.com*/AWSReservedSSO_AdministratorAccess_*
denied_teams: []
denied_role_arns: []
admin:
<<: *user-template
enabled: true
role_policy_arns:
- arn:aws:iam::aws:policy/AdministratorAccess
role_description: Full administration of this account
trusted_teams: [admin]
terraform:
<<: *user-template
enabled: true
role_policy_arns:
- arn:aws:iam::aws:policy/AdministratorAccess
role_description: Role for Terraform administration of this account
trusted_teams: [admin, devops]
github-action-oidc:
metadata:
component: github-action-oidc
vars:
enabled: false
superadmin: true
That’s the configuration I recently used except I replaced the GH org name with nxt-big-startup.
@Brian Thank you for the detailed response and sample code. I really appreciate it. Assuming the above definition is coming from one of your stack defs, which specific stack definition file does that belong in? My understanding is that aws-teams must be execited within the scope of identity account where as aws-sso is provisioned in the scope of root account and aws-team-roles in the scope of each of the other accounts. Here is how my core-gbl-root stack def looks like:
import:
- mixins/region/global
- orgs/xxx/core/root/_defaults
- catalog/terraform/account
- catalog/terraform/account-map
- catalog/terraform/aws-sso
terraform:
vars:
descriptor_formats:
account_name:
format: "%v-%v"
labels:
- tenant
- stage
stack:
format: "%v-%v-%v-%v"
labels:
- namespace
- tenant
- environment
- stage
And here is my core-gbl-identity stack def:
import:
- mixins/region/global
- orgs/xxx/core/identity/_defaults
- catalog/terraform/account-map
- catalog/terraform/aws-teams
terraform:
vars:
descriptor_formats:
account_name:
format: "%v-%v"
labels:
- tenant
- stage
stack:
format: "%v-%v-%v-%v"
labels:
- namespace
- tenant
- environment
- stage
I do not have core-gbl-identity . That stack would be if you were deploying sso to a dedicated account called identity . However, CP no longer recommends it (but thier docs does not reflect it everywhere yet). Therefore, I do not have any account named identity. This requires that identity_account_account_name for your account-map points to your root account.
Also, this block should be imported in all stacks (e.g. catalog/global) because it can be relevant to other components after the cold start process.
terraform:
vars:
descriptor_formats:
account_name:
format: "%v-%v"
labels:
- tenant
- stage
stack:
format: "%v-%v-%v-%v"
labels:
- namespace
- tenant
- environment
- stage
Yes - I went in the direction of identity account based on the docs. I will try eliminating it (removing accounts in AWS is a pain!) and leveraging the root account for aws-teams and aws-sso.
@Brian What are you using as the IDM/SAML Provider for AWS SSO? I am assuming you are syncing the groups you defined in the YAML above in the provider unless you are creating them using TF or manually in AWS SSO.
Okta w/ SCIM for 1, the other are AWS SSO w/o an external IDP. For SSO w/o external IDP, I had to manually create the groups prior to deploying the aws-sso component.