#refarch (2023-03)

Cloud Posse Reference Architecture

2023-03-01

johncblandii avatar
johncblandii

We have the ECR repo set to allow full access from auto

        read_write_account_role_map:
          core-identity:
            - admin
            - cicd
          core-auto:
            - "*"

…yet we’re still getting this error:

Error pulling Docker image: Error response from daemon: pull access denied for 123456789.dkr.ecr.us-east-1.amazonaws.com/infrastructure, repository does not exist or may require 'docker login': denied: User: arn:aws:sts::654321987:assumed-role/xy-core-ue1-auto-spacelift-worker-pool/i-0f47ad8c786c60585 is not authorized to perform: ecr:BatchGetImage on resource: arn:aws:ecr:us-east-1:123456789:repository/infrastructure because no resource-based policy allows the ecr:BatchGetImage action
johncblandii avatar
johncblandii

Ignore. The referenced account id was wrong.

2023-03-02

2023-03-03

johncblandii avatar
johncblandii

DNS applies are failing over assume role. Other components work just fine. Is there any nuance to the dns-primary/dns-delegated components?

│ Error: configuring Terraform AWS Provider: IAM Role (arn:aws:iam::1234567890:role/sm-core-gbl-dns-terraform) cannot be assumed.
│ 
│ There are a number of possible causes of this - the most common are:
│   * The credentials used in order to assume the role are invalid
│   * The credentials do not have appropriate permission to assume the role
│   * The role ARN is not valid
│ 
│ Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: a-b-c-d-e, api error AccessDenied: User: arn:aws:sts::1234567890:assumed-role/sm-core-gbl-dns-terraform/aws-go-sdk-1677817372266350530 is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::1234567890:role/sm-core-gbl-dns-terraform
johncblandii avatar
johncblandii

ok so the provider is using the assume role from the same role into the same role within the same account. that’s the issue

johncblandii avatar
johncblandii

Any thoughts here @Andriy Knysh (Cloud Posse) @RB?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. This looks strange arn:aws:sts::1234567890 - 1234567890 is not a real account (search for it in the repo, might be hardcoded)
johncblandii avatar
johncblandii

i just replaced the actual account id

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. If #1 is fixed and still, not working, maybe the gbl-identity-admin does not have permissions to assume gbl-dns-terraform
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Or gbl-dns-terraform does not have a trust policy to enable gbl-identity-admin assuming it (#2 and #2 can be checked in AWS console in the two accounts)
johncblandii avatar
johncblandii

it is using module.iam_roles.dns_terraform_profile_name to assume role on module.iam_roles.terraform_profile_name and both are the -dns-terraform role

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
  1. Also check if [providers.tf](http://providers.tf) is configured correctly for dns-primary and dns-delegated
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


it is using module.iam_roles.dns_terraform_profile_name to assume role on module.iam_roles.terraform_profile_name and both are the -dns-terraform role

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

check both roles in AWS console, and check if the account-map is provisioned correctly (atmos terraform output … should show all the accounts and all the roles)

johncblandii avatar
johncblandii
  1. it trusts gbl-identity-admin and gbl-identity-spacelift
johncblandii avatar
johncblandii
  1. it is from the latest from tf-aws-modules
johncblandii avatar
johncblandii

it is essentially a self-referential assume

johncblandii avatar
johncblandii

-terraform to -terraform

johncblandii avatar
johncblandii

my initial assume to -dns-terraform works, but it assumes at the provider level and doesn’t

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you need to run atmos terraform apply dns-delegated/primary after you assume gbl-identity-admin role

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


my initial assume to -dns-terraform works, but it assumes at the provider level and doesn’t

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

that’s what you don’t need to do in geodesic

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you have to assume gbl-identity-admin and then execute commands to apply dns-delegated

johncblandii avatar
johncblandii

clarity: i’m not assuming i mean me using my identity account works to access the terraform role

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)


RequestID: a-b-c-d-e, api error AccessDenied: User: arnawssts:assumed-role/sm-core-gbl-dns-terraform/aws-go-sdk-1677817372266350530 is not authorized to perform: sts:AssumeRole on resource: arnawsiam:role/sm-core-gbl-dns-terraform

johncblandii avatar
johncblandii

that’s from spacelift there

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it says that you are not using gbl-identity-admin , something else

johncblandii avatar
johncblandii


Effective AWS Role Arn is arnawssts:assumed-role/sm-core-gbl-identity-spacelift/Spacelift

johncblandii avatar
johncblandii

yeah, this being true is what triggers the failure… module.iam_roles.profiles_enabled because module.iam_roles.terraform_profile_name is used

johncblandii avatar
johncblandii

adjusting the assume_role section of the [provider.tf](http://provider.tf) in both dns-delegated and dns-primary

1
johncblandii avatar
johncblandii

@Andriy Knysh (Cloud Posse) should I upstream this or is this not the proper fix?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if the code is from terraform-aws-components, then yes, please upstream when you get a moment ([providers.tf](http://providers.tf) should be the same for all components)

1
johncblandii avatar
johncblandii

it is and will do

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)


ok so the provider is using the assume role from the same role into the same role within the same account. that’s the issue
Until recently, AWS IAM implicitly allowed this. They recently removed this in the interest of security, and in most cases Cloud Posse does not add rules to allow it.

johncblandii avatar
johncblandii

Ah, that makes sense

jwood avatar

Sorry to necro this, but I actually just ran into this myself. I cloned down an older ref-arch repo and went through all the initialization steps, and when I run terraform-plan, I’m met with:

User: arn:aws:sts::<account-id>:assumed-role/OrganizationAccountAccessRole/<session-id> is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::<account-id>:role/OrganizationAccountAccessRole
jwood avatar

Is there a quick fix I can apply within the refarch repo, or do I need to wait for fixes to be upstreamed to the terraform modules?

johncblandii avatar
johncblandii

@jwood that role cannot perform assume so you need to run it with one of the other roles (ie - -terraform-admin)

jwood avatar

Right, what’s happening is the IAM user has the ability to assume the AccountAccess role, which happens when I run assume-role <name>-staging-admin from geodesic. But then when I run terraform plan, the aws provider tries to assume the role it is already in, set by the env var TF_VAR_aws_assume_role_arn

For context, this is a pretty old clone of the refarch, circa June ‘21. I’d like to get the project updated and fixed up to use newer geodesic versions, but I’m not sure if there’s a good upgrade path. Are there any docs on how to update the refarch-generated project?

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

@jwood You should be in the <namespace>-identity-admin role when running Terraform, or just in general under the current and previous refarch.

Unfortunately, upgrading from iam-primary-roles (old) to aws-teams (current) is so painful we have put off documenting how to do it. Self-serving as it is, this is a task I would strongly recommend you pay Cloud Posse to do for you. (That is, unless you are in a state where you can completely tear down all your deployments and infrastructure and essentially start over from the point of having empty accounts.)

jwood avatar

Thanks @Jeremy G (Cloud Posse)! It’s in a state where a teardown is possible, since I want to re-architect our stuff in AWS anyway.

2023-03-04

SlackBot avatar
SlackBot
08:22:58 PM

This message was deleted.

Michael Dizon avatar
Michael Dizon

the stack is based on the examples/complete directory in the atmos repo

2023-03-05

2023-03-06

SlackBot avatar
SlackBot
06:22:55 PM

This message was deleted.

2023-03-07

Michael Dizon avatar
Michael Dizon
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

aws-sso (for AWS SSO) and aws-saml are the latest versions

1
johncblandii avatar
johncblandii

I just deployed aws-sso @Michael Dizon so if i can help…i will

1
Michael Dizon avatar
Michael Dizon

cool, i’m going thru it now, will update with progress soon

Michael Dizon avatar
Michael Dizon

i’m not using okta, is that a requirement? @johncblandii

johncblandii avatar
johncblandii

no, but a provider is required

johncblandii avatar
johncblandii

google, auth0, etc

johncblandii avatar
johncblandii

provider groups/users + their assignments get synced to AWS SSO

Michael Dizon avatar
Michael Dizon

yeah i am using google

1
johncblandii avatar
johncblandii

I used Okta with my setup so can’t give the Google details, but I could help with the configs

Michael Dizon avatar
Michael Dizon

@johncblandii ran into this error when trying to deploy

╷
│ Error: creating SSO Permission Set (PowerUserAccess): ConflictException: PermissionSet with name PowerUserAccess already exists.
│ 
│   with module.permission_sets.aws_ssoadmin_permission_set.this["PowerUserAccess"],
│   on .terraform/modules/permission_sets/modules/permission-sets/main.tf line 4, in resource "aws_ssoadmin_permission_set" "this":
│    4: resource "aws_ssoadmin_permission_set" "this" {
│ 
╵
Michael Dizon avatar
Michael Dizon

also, i’m not sure how to go about setting up google workspace with sso. i did this last year with the previous sso component. but don’t recall what i did back then lol

johncblandii avatar
johncblandii

Component: aws-sso

This component is responsible for creating AWS SSO Permission Sets and creating AWS SSO Account Assignments, that is, assigning IdP (Okta) groups and/or users to AWS SSO permission sets in specific AWS Accounts.

This component assumes that AWS SSO has already been enabled via the AWS Console (there isn’t terraform or AWS CLI support for this currently) and that the IdP has been configured to sync users and groups to AWS SSO.

Usage Clickops

  1. Go to root admin account
  2. Select primary region
  3. Go to AWS SSO
  4. Enable AWS SSO
  5. Click Settings > Management
  6. Delegate Identity as an administrator

Once identity is delegated, it will take up to 20 to 30 minutes for the identity account to understand its delegation.

Atmos

Stack Level: Global
Deployment: Must be deployed by root-admin using atmos CLI

Add catalog to gbl-identity root stack.

account_assignments

The account_assignments setting configures access to permission sets for users and groups in accounts, in the following structure:

<account-name>:
  groups:
    <group-name>:
      permission_sets:
        - <permission-set-name>
  users:
    <user-name>:
      permission_sets:
        - <permission-set-name>

• The account names (a.k.a. “stages”) must already be configured via the accounts component. • The user and group names must already exist in AWS SSO. Usually this is accomplished by configuring them in Okta and syncing Okta with AWS SSO. • The permission sets are defined (by convention) in files names policy-<permission-set-name>.tf in the aws-sso component. The definition includes the name of the permission set. See components/terraform/aws-sso/policy-AdminstratorAccess.tf for an example.

identity_roles_accessible

The identity_roles_accessible element provides a list of role names corresponding to roles created in the iam-primary-roles component. For each names role, a corresponding permission set will be created which allows the user to assume that role. The permission set name is generated in Terraform from the role name using this statement:

format("Identity%sTeamAccess", title(role))

Example

The example snippet below shows how to use this module with various combinations (plain YAML, YAML Anchors and a combination of the two):

prod-cloud-engineers: &prod-cloud-engineers
  Production Cloud Infrastructure Engineers:
    permission_sets:
      - AdministratorAccess
      - ReadOnlyAccess

components:
  terraform:
    aws-sso:
      vars:
        account_assignments:
          audit:
            groups:
              <<: *prod-cloud-engineers
              Production Cloud Engineers:
                permission_sets:
                  - ReadOnlyAccess
          corp:
            groups: *prod-cloud-engineers
          prod:
            groups:
              Admininstrators:
                permission_sets:
                  - AdministratorAccess
                  - ReadOnlyAccess
              Developers:
                permission_sets:
                  - ReadOnlyAccess
          dev:
            groups:
              Admininstrators:
                permission_sets:
                  - AdministratorAccess
                  - ReadOnlyAccess
              Developers:
                permission_sets:
                  - AdministratorAccess
                  - ReadOnlyAccess
        aws_teams_accessible:
        - "admin"
        - "ops"
        - "poweruser"
        - "observer"
        - "reader"
        - "support"
        - "viewer"

Requirements Providers Modules Resources Inputs Outputs References

cloudposse/terraform-aws-sso

https://camo.githubusercontent.com/0e29bf3da9c47ee0be2cd91223cfc56e9dade2adaee3b5d63625be0406089fc9/68747470733a2f2f636c6f7564706f7373652e636f6d2f6c6f676f2d3330307836392e737667

johncblandii avatar
johncblandii

how’s your yaml config? did you create any manual permission sets?

Michael Dizon avatar
Michael Dizon

no, the yaml is direct from the README

Michael Dizon avatar
Michael Dizon

i’ve gone through the click ops steps

Michael Dizon avatar
Michael Dizon

is there a particular order that these modules need to be applied in? i’ve deployed aws-teams, aws-team-roles, and aws-saml

johncblandii avatar
johncblandii

• teams

• team-roles

• aws-sso

johncblandii avatar
johncblandii

don’t need aws-saml

johncblandii avatar
johncblandii

you vendored the component direct from terraform-aws-components?

johncblandii avatar
johncblandii

when you go to Identity Center in the console do you see permission sets already in there?

Michael Dizon avatar
Michael Dizon

yeah from terraform-aws-components

Michael Dizon avatar
Michael Dizon

and yeah the permission sets are there

Michael Dizon avatar
Michael Dizon

i commented out local.poweruser_access_permission_set,and local.terraform_update_access_permission_set and was able to deploy

1
Michael Dizon avatar
Michael Dizon

i see TerraformUpdateAccess but it says it’s not provisioned

Michael Dizon avatar
Michael Dizon

PowerUserAccess is there too, and it is provisioned

Michael Dizon avatar
Michael Dizon

@johncblandii there is a chance that someone had created those permission sets before me

johncblandii avatar
johncblandii

absolutely

Michael Dizon avatar
Michael Dizon

@johncblandii confirmed that was the case

1

2023-03-08

Michael Dizon avatar
Michael Dizon

Following up on my question from yesterday, I previously used the now deprecated “iam-primary-roles” and “iam-delegated-roles” modules, as well as “sso,” which allowed me to use Google Workspace to authenticate into my environment with Leapp. At a glance, it seems that in order to replicate this setup, I need to use “aws-teams,” “aws-team-roles,” and “aws-saml.” Does this sound correct?

1
Matt Gowie avatar
Matt Gowie

That’s correct. There is also the aws-sso component which does the AWS SSO (IAM Identity Center) instead of the IAM SAML integration.

1
Matt Gowie avatar
Matt Gowie

Ah maybe this is already covered in the above thread – My bad.

1
Michael Dizon avatar
Michael Dizon

hah, nah, mostly my bad. i don’t know why i keep mixing those components up

np1

2023-03-09

2023-03-10

Michael Dizon avatar
Michael Dizon

has anyone encountered this error DenyEC2InstancesWithoutEncryptionInTransit when trying to deploy a cluster using terraform-aws-components/modules/eks/cluster/

Michael Dizon avatar
Michael Dizon

i am using the configuration in the example

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

maybe you have some Service Control policies provisioned that require instances with encryption in transit only

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
- sid: "DenyEC2InstancesWithoutEncryptionInTransit"
Michael Dizon avatar
Michael Dizon

whoa

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

look in the Org’s root account

Michael Dizon avatar
Michael Dizon

yep, found it

Michael Dizon avatar
Michael Dizon

thanks again!

johncblandii avatar
johncblandii

Once we’ve applied account-settings then adjusted the tfstate-backend, how do we apply updates to account-settings?
Error: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 51ea4cf5-95bc-4c8c-891c-3f06155fda5a, api error AccessDenied: User: arnawssts:assumed-role/sm-core-gbl-artifacts-terraform/aws-go-sdk-1678477476576489221 is not authorized to perform: sts:AssumeRole on resource: arnawsiam:role/OrganizationAccountAccessRole

johncblandii avatar
johncblandii

a root account is in use, but it is assuming the terraform user first

johncblandii avatar
johncblandii

NOTE: profile = null is a solution that prevents the [stage]-terraform user being set, but the provider uses this by default for us sense profiles are enabled.

provider "aws" {
  region = var.region

  profile = null

  dynamic "assume_role" {
    for_each = var.import_role_arn == null ? (module.iam_roles.org_role_arn != null ? [true] : []) : ["import"]
    content {
      role_arn = coalesce(var.import_role_arn, module.iam_roles.org_role_arn)
    }
  }
}
johncblandii avatar
johncblandii

so for profiles_enabled scenarios, we’re blocked without changing the actual provider code

johncblandii avatar
johncblandii

also, it seems to be requesting increases even when increases were already requested

johncblandii avatar
johncblandii

any thoughts @Andriy Knysh (Cloud Posse)?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

sorry, I’m not in the context what was done and how, so can only guess. Are you using profiles or roles?

johncblandii avatar
johncblandii

aws sso so profiles

2023-03-15

Michael Dizon avatar
Michael Dizon

does anyone have an example of the values to provide for kubernetes_taints for this module? https://github.com/cloudposse/terraform-aws-components/tree/master/modules/eks/cluster

the example has it empty, but I get an error saying that a list of object is required

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Dan Miller (Cloud Posse)

Dan Miller (Cloud Posse) avatar
Dan Miller (Cloud Posse)

I don’t have an example on hand, but can you share your implementation s.t. we can point you towards what might be missing?

That attribute needs to be a list of objects in this format:

    kubernetes_taints = list(object({
      key    = string
      value  = string
      effect = string
    }))

https://github.com/cloudposse/terraform-aws-components/blob/master/modules/eks/cluster/variables.tf

``` variable “region” { type = string description = “AWS Region” }

variable “availability_zones” { type = list(string) description = «-EOT AWS Availability Zones in which to deploy multi-AZ resources. If not provided, resources will be provisioned in every private subnet in the VPC. EOT default = [] }

variable “availability_zone_abbreviation_type” { type = string description = “Type of Availability Zone abbreviation (either fixed or short) to use in names. See https://github.com/cloudposse/terraform-aws-utils for details.” default = “fixed” validation { condition = contains([“fixed”, “short”], var.availability_zone_abbreviation_type) error_message = “The availability_zone_abbreviation_type must be either "fixed" or "short".” } }

variable “managed_node_groups_enabled” { type = bool description = “Set false to prevent the creation of EKS managed node groups.” default = true }

variable “oidc_provider_enabled” { type = bool description = “Create an IAM OIDC identity provider for the cluster, then you can create IAM roles to associate with a service account in the cluster, instead of using kiam or kube2iam. For more information, see https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html” }

variable “cluster_endpoint_private_access” { type = bool default = false description = “Indicates whether or not the Amazon EKS private API server endpoint is enabled. Default to AWS EKS resource and it is false” }

variable “cluster_endpoint_public_access” { type = bool default = true description = “Indicates whether or not the Amazon EKS public API server endpoint is enabled. Default to AWS EKS resource and it is true” }

variable “cluster_kubernetes_version” { type = string default = null description = “Desired Kubernetes master version. If you do not specify a value, the latest available version is used” }

variable “public_access_cidrs” { type = list(string) default = [“0.0.0.0/0”] description = “Indicates which CIDR blocks can access the Amazon EKS public API server endpoint when enabled. EKS defaults this to a list with 0.0.0.0/0.” }

variable “enabled_cluster_log_types” { type = list(string) default = [] description = “A list of the desired control plane logging to enable. For more information, see https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html. Possible values [api, audit, authenticator, controllerManager, scheduler]” }

variable “cluster_log_retention_period” { type = number default = 0 description = “Number of days to retain cluster logs. Requires enabled_cluster_log_types to be set. See https://docs.aws.amazon.com/en_us/eks/latest/userguide/control-plane-logs.html.” }

variable “apply_config_map_aws_auth” { type = bool default = true description = “Whether to execute kubectl apply to apply the ConfigMap to allow worker nodes to join the EKS cluster” }

variable “map_additional_aws_accounts” { description = “Additional AWS account numbers to add to aws-auth ConfigMap” type = list(string) default = [] }

variable “map_additional_worker_roles” { description = “AWS IAM Role ARNs of worker nodes to add to aws-auth ConfigMap” type = list(string) default = [] }

variable “primary_iam_roles” { description = “Primary IAM roles to add to aws-auth ConfigMap”

type = list(object({ role = string groups = list(string) }))

default = [] }

variable “delegated_iam_roles” { description = “Delegated IAM roles to add to aws-auth ConfigMap”

type = list(object({ role = string groups = list(string) }))

default = [] }

variable “map_additional_iam_roles” { description = “Additional IAM roles to add to config-map-aws-auth ConfigMap”

type = list(object({ rolearn = string username = string groups = list(string) }))

default = [] }

variable “map_additional_iam_users” { description = “Additional IAM users to add to aws-auth ConfigMap”

type = list(object({ userarn = string username = string groups = list(string) }))

default = [] }

variable “allowed_security_groups” { type = list(string) default = [] description = “List of Security Group IDs to be allowed to connect to the EKS cluster” }

variable “allowed_cidr_blocks” { type = list(string) default = [] description = “List of CIDR blocks to be allowed to connect to the EKS cluster” }

variable “subnet_type_tag_key” { type = string default = null description = “The tag used to find the private subnets to find by availability zone. If null, will be looked up in vpc outputs.” }

variable “color” { type = string default = “” description = “The cluster stage represented by a color; e.g. blue, green” }

variable “node_groups” { # will create 1 node group for each item in map type = map(object({ # EKS AMI version to use, e.g. “1.16.13-20200821” (no “v”). ami_release_version = string # Type of Amazon Machine Image (AMI) associated with the EKS Node Group ami_type = string # Additional attributes (e.g. 1) for the node group attributes = list(string) # will create 1 auto scaling group in each specified availability zone availability_zones = list(string) # Whether to enable Node Group to scale its AutoScaling Group cluster_autoscaler_enabled = bool # True to create new node_groups before deleting old ones, avoiding a temporary outage create_before_destroy = bool # Desired number of worker nodes when initially provisioned desired_group_size = number # Enable disk encryption for the created launch template (if we aren’t provided with an existing launch template) disk_encryption_enabled = bool # Disk size in GiB for worker nodes. Terraform will only perform drift detection if a configuration value is provided. disk_size = number # Set of instance types associated with the EKS Node Group. Terraform will only perform drift detection if a configuration value is provided. instance_types = list(string) # Key-value mapping of Kubernetes labels. Only labels that are applied with the EKS API are managed by this argument. Other Kubernetes labels applied to the EKS Node Group will not be managed kubernetes_labels = map(string) # List of objects describing Kubernetes taints. kubernetes_taints = list(object({ key = string value = string effect = string })) # Desired Kubernetes master version. If you do not specify a value, the latest available version is used kubernetes_version = string # The maximum size of the AutoScaling Group max_group_size = number # The minimum size of the AutoScaling Group min_group_size = number # List of auto-launched resource types to tag resources_to_tag = list(string) tags = map(string) })) description = “List of objects defining a node group for the cluster” default = {} }

variable “node_group_defaults” { # Any value in the node group that is null will be replaced # by the value in this object, which can also be null type = object({ ami_release_version = string ami_type = string attributes = list(string) availability_zones = list(string) # set to null to use var.availability_zones cluster_autoscaler_enabled = bool create_before_destroy = bool desired_group_size = number disk_encryption_enabled = bool disk_size = number instance_types = list(string) kubernetes_labels = map(string) kubernetes_taints = list(object({ key = string value = string effect = s…

Michael Dizon avatar
Michael Dizon

i passed an empty array and that seemed to deploy. i will look at it in the morning to see if everything went ok.

1

2023-03-16

2023-03-17

johncblandii avatar
johncblandii

dns-delegated uses a for_each based on the subdomain. because of this, if you have multiple domains with the same subdomain, the state gets jacked up (last one wins).

is this the desired to handle it?

Solution: I’m using a separate stack component to reference the private zone anyway (my mistake on putting in the same dns-delegated instance.

johncblandii avatar
johncblandii

Relevant code:

locals {
  enabled  = module.this.enabled
  zone_map = zipmap(var.zone_config[*].subdomain, var.zone_config[*].zone_name)

  private_enabled = local.enabled && var.dns_private_zone_enabled
  public_enabled  = local.enabled && !local.private_enabled

  private_ca_enabled = local.private_enabled && var.certificate_authority_enabled

  aws_route53_zone = local.private_enabled ? aws_route53_zone.private : aws_route53_zone.default

  vpc_environment_names = toset(concat([var.vpc_primary_environment_name], var.vpc_secondary_environment_names))

  aws_partition = join("", data.aws_partition.current.*.partition)
}

resource "aws_route53_zone" "default" {
  for_each = local.public_enabled ? local.zone_map : {}

  name    = format("%s.%s", each.key, each.value)
  comment = format("DNS zone for %s.%s", each.key, each.value)


  tags = module.this.tags
}
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Ben Smith (Cloud Posse) @Dan Miller (Cloud Posse)

Dan Miller (Cloud Posse) avatar
Dan Miller (Cloud Posse)

I believe this is a bug with how we’ve written the Terraform component. However, if you wanted to work around it, you could try defining a separate component like this:

components:
  terraform:

    dns-primary:
      vars:
        domain_names:
           - acme-dev.com

    dns-delegated:
      vars:
        zone_config:
          - subdomain: dev.plat
            zone_name: acme-svc.com

    dns-delegated-alt:
      metadata:
        component: dns-delegated
        inherits:
          - dns-delegated
      vars:
        zone_config:
          - subdomain: dev.plat
            zone_name: acme-svc-alt.com
johncblandii avatar
johncblandii

yeah, that’s exactly what i ended up doing

Dan Miller (Cloud Posse) avatar
Dan Miller (Cloud Posse)

ah okay great!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think a separate component instance is a better long-term solution too and the direction should advise.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That way the state isn’t tightly coupled.

this1
1

2023-03-20

2023-03-21

johncblandii avatar
johncblandii

Any plans or active work to upgrade mq-broker in refarch and terraform-aws-mq-broker?

johncblandii avatar
johncblandii

and along those lines, any plans to detach it from eks?

johncblandii avatar
johncblandii

I already had these changes so made them. I didn’t include my removal of EKS, though.

https://github.com/cloudposse/terraform-aws-components/pull/602

#602 chore(mq-broker): upgrade versions

what

• Upgraded to the latest terraform-aws-mq-broker • Updated the providers to match the common pattern • Updated the module versions

why

• The component was dated

references

johncblandii avatar
johncblandii

I went ahead and did these changes in the core module too since it still breaks due to old versions:

https://github.com/cloudposse/terraform-aws-mq-broker/pull/54

I’m unsure if the tests are complete here to validate it, but it definitely needs validating before merging

#54 Upgrade versions

what

• Upgrade versions to the latest

why

• The module fails when used with the latest https://github.com/cloudposse/terraform-aws-components

references

cloudposse/terraform-aws-components#602

johncblandii avatar
johncblandii

#602 (refarch) is updated to add more outputs and available vars

johncblandii avatar
johncblandii

I tested both of these (local refarch usage) with a local install and the plan worked

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Linda Pham (Cloud Posse)

1
Linda Pham (Cloud Posse) avatar
Linda Pham (Cloud Posse)

@johncblandii reviewed, and changes were requested

johncblandii avatar
johncblandii

Everything was addressed except a separate repo needing a new version.

Also, this PR is still pending https://github.com/cloudposse/terraform-aws-components/pull/602

#602 mq-broker: upgrade versions

what

• Upgraded to the latest terraform-aws-mq-broker • Updated the providers to match the common pattern • Updated the module versions

why

• The component was dated

references

Linda Pham (Cloud Posse) avatar
Linda Pham (Cloud Posse)

Thanks! Let me get another review

Linda Pham (Cloud Posse) avatar
Linda Pham (Cloud Posse)

@johncblandii, @Andriy Knysh (Cloud Posse) requested additional changes

1
johncblandii avatar
johncblandii

Done

johncblandii avatar
johncblandii
#54 Upgrade versions

Breaking Changes

This module includes breaking changes due to upgrading from terraform-aws-security-group v1 to v2. You can read the full details of the security group changes and how to migrate in migration notes linked below under “references”, but the short story is this:

• If you were using this modules default value of true for security_group_create_before_destroy then you need not make any changes. If you were explicitly setting it to false, we strongly advise you to read the migration notes because that setting previously did not work, raising the question of whether you want to pay the price of converting to the new module with working false behavior or perform the recommended upgrade to the true. • If you are referring by ID to the security group created by this module in other security group’s rules outside the Terraform plan that controls this one, then you should read the security group migration notes discussion of the new input preserve_security_group_id and probably set it to true

what

• Upgrade versions to the latest

Key changes in v2.0

• create_before_destory default changed from false to true • preserve_security_group_id added, defaults to false • Terraform version 1.0.0 or later required

why

• The module fails when used with the latest https://github.com/cloudposse/terraform-aws-components

references

cloudposse/terraform-aws-components#602 • Migration Notes for Security Group v2.0

johncblandii avatar
johncblandii
#602 mq-broker: upgrade versions

what

• Upgraded to the latest terraform-aws-mq-broker • Updated the providers to match the common pattern • Updated the module versions

why

• The component was dated

references

1
johncblandii avatar
johncblandii
#597 spacelift: Update README.md example login policy

what

• Added support for allowing spaces read access to all members • Added a reference for allowing spaces write access to the “Developers” group

why

• Spacelift moved to Spaces Access Control

references

https://docs.spacelift.io/concepts/spaces/access-control

johncblandii avatar
johncblandii
#597 spacelift: Update README.md example login policy

what

• Added support for allowing spaces read access to all members • Added a reference for allowing spaces write access to the “Developers” group

why

• Spacelift moved to Spaces Access Control

references

https://docs.spacelift.io/concepts/spaces/access-control

johncblandii avatar
johncblandii

Are there any patterns for using remote state references in stack configs?

Context: I’m looking at some solutions for referencing outputs in a config so we can create SSM parameters in a file to reference all params in 1 place vs them littered in TF components

johncblandii avatar
johncblandii

I have a concept I think could work here. I’m just curious if there are some already defined patterns

Michael Dizon avatar
Michael Dizon

i’d def love to see this

johncblandii avatar
johncblandii
johncblandii avatar
johncblandii

this disables the required sops integration and adds support for valueFrom.

johncblandii avatar
johncblandii

this update to the var was needed too

Michael Dizon avatar
Michael Dizon

which module does this modify?

johncblandii avatar
johncblandii

ssm-parameters

1

2023-03-22

Linda Pham (Cloud Posse) avatar
Linda Pham (Cloud Posse)
05:30:14 PM

@Linda Pham (Cloud Posse) has joined the channel

johncblandii avatar
johncblandii

In ssm-parameters, setting overwrite: false results in an error on the next run:
ParameterAlreadyExists: The parameter already exists. To overwrite this value, set the overwrite option in the request to true.

johncblandii avatar
johncblandii

NOTE: this was following these steps:

• deploy w/ CHANGEME value

• clickops change value

• deploy same code

• error

johncblandii avatar
johncblandii

obviously the clickops shouldn’t be there, but this is largely testing out boundaries as to what could happen here

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

if overwrite: false and terraform wants to update the parameter, the error will be thrown

johncblandii avatar
johncblandii

yes

johncblandii avatar
johncblandii

shouldn’t it ignore it?

johncblandii avatar
johncblandii

create once, ignore after?

johncblandii avatar
johncblandii

sidebar: have any good SOPS examples/tutorials? I’m not familiar w/ it and we might go that route

johncblandii avatar
johncblandii

yeah, setting overwrite: true makes the value CHANGEME again; as expected

johncblandii avatar
johncblandii

it makes me wonder if we should include this:

  lifecycle {
    ignore_changes = [value]
  }
johncblandii avatar
johncblandii

seems to work as expected

johncblandii avatar
johncblandii

but it might have higher ramifications

2023-03-23

2023-03-29

2023-03-30

Sean Nguyen avatar
Sean Nguyen

@RB @Andriy Knysh (Cloud Posse) Question: New(-ish) Spacelift Dependencies feature

Are there any special considerations that have to be made if we were to migrate from the old deps: label method for declaring explicit component/stack dependencies to Spacelift’s new native stack dependency feature?

I do see that y’all released support for it in v0.55.0 of your spacelift-cloud-infrastructure-automation module.

Stack dependencies - Spacelift Documentation

Collaborative Infrastructure For Modern Software Teams

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we are testing it. You don’t have to do anything to use it, just enable it. You use the same settings.spacelift.depends_on section to describe deps. We are looking into the workflows and how it all should be used. You can try it as well

Stack dependencies - Spacelift Documentation

Collaborative Infrastructure For Modern Software Teams

Sean Nguyen avatar
Sean Nguyen

Thanks! I’m hoping this might make CI checks for Spacelift stacks a little faster on our monorepo considering the old way is definitely a little eager

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy White (Cloud Posse)

1
Jeremy White (Cloud Posse) avatar
Jeremy White (Cloud Posse)

Just be aware that the dependency resource will prevent deletion of stacks if you don’t remove the depends_on . That is, they also function as a sanity check to avoid deleting a stack that is “depended on” by another stack.

1
1
Jeremy White (Cloud Posse) avatar
Jeremy White (Cloud Posse)
05:33:01 PM

@Jeremy White (Cloud Posse) has joined the channel

2023-03-31

Luke avatar

Hi, has anyone run into the following error: creating CloudTrail: InsufficientEncryptionPolicyException: Insufficient permissions to access S3 bucket or KMS key. (Service: AWSCloudTrail; Status Code: 400; Error Code: InsufficientEncryptionPolicyException ? The cloudtrail-bucket component has been deployed in audit account, account itself has been made a CloudTrail delegated administrator and while I am trying to deploy cloudtrail component I am getting the above error during creation of module.cloudtrail.aws_cloudtrail.default[0] . CloudTrail trail can be successfully created in the console if I’ll modify KSM policy to allow kms:* for [cloudtrail.amazonaws.com](http://cloudtrail.amazonaws.com) principal.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Ben Smith (Cloud Posse) @Jeremy White (Cloud Posse) @Dan Meyers

Michael Dizon avatar
Michael Dizon

running into a weird error when trying to run a plan using the aws-team-roles component. i’m logged in using leapp with my iam user with admin privileges

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: reading IAM Role (xxx-core-gbl-network-admin): InvalidClientTokenId: The security token included in the request is invalid
│       status code: 403, request id: 5a33f9f0-d0c3-465b-842f-de33f3cabfff
│ 
│   with aws_iam_role.default["admin"],
│   on main.tf line 71, in resource "aws_iam_role" "default":
│   71: resource "aws_iam_role" "default" {
│ 
╵
╷
│ Error: reading IAM Role (xxx-core-gbl-network-terraform): InvalidClientTokenId: The security token included in the request is invalid
│       status code: 403, request id: aad87fdc-ac41-4276-8070-a39f70e8ac24
│ 
│   with aws_iam_role.default["terraform"],
│   on main.tf line 71, in resource "aws_iam_role" "default":
│   71: resource "aws_iam_role" "default" {
│ 
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

was your session expired?

Michael Dizon avatar
Michael Dizon

i don’t think so. the weird thing is that i switched to (what i think) is an identical stack for a different project, and switched to that iam profile, and it works.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

Leapp refreshed the session

Michael Dizon avatar
Michael Dizon

hmm, even toggling between different sessions?

Michael Dizon avatar
Michael Dizon

its able to pull the state though

Michael Dizon avatar
Michael Dizon

i get this in both projects with the tfstate-backend component:

╷
│ Error: reading IAM Role (xxx-gbl-xxx-tfstate): InvalidClientTokenId: The security token included in the request is invalid
│       status code: 403, request id: 853af46f-83d4-4c9b-b925-a4e7f52768bb
│ 
│   with aws_iam_role.default["duetti-gbl-duetti-tfstate"],
│   on iam.tf line 66, in resource "aws_iam_role" "default":
│   66: resource "aws_iam_role" "default" {
│ 
╵
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you using the SuperAdmin user?

Michael Dizon avatar
Michael Dizon

my iam user does have Administrator privileges, is that what you mean?

Michael Dizon avatar
Michael Dizon

update: i was able to resolve this by adding MFA to my IAM user. I don’t *think* I’ve ever had to do this before.

Michael Dizon avatar
Michael Dizon

also thanks @Andrea Cavagna

Andrea Cavagna avatar
Andrea Cavagna

No problem @Michael Dizon

AWS for security reasons does not allow IAM user without MFA to call any IAM APIs.

Thanks why the problem sound like token expired.

All those corner case and IAM knowledge is being collected into the IAM blog

Leapp blogattachment image

A place for improving your Cloud security posture and the Developer experience of cloud developers. A community of passionate about Cloud access and IAM

1
1
    keyboard_arrow_up