#terraform (2022-02)

@Shrivatsan Narayanaswamy thanks. Fork the repo, make the changes, then run from the root of the repo

make init
make github/init
make readme

then open a PR against our repo, we’ll review, approve and merge

Between 0.12 => 1.1 there are very few changes to the syntax. The only ones that I can think of off the top of my head are the usage of the map and list functions which are now toMap and toList. https://www.terraform.io/language/functions/map https://www.terraform.io/language/functions/list

There likely are others, but that is one of the main ones. In all upgrades of this nature you typically just need to go for it and then deal and work through the errors that show up in your code from the new version.

That said — Of course you should start managing your infra via TF You’re only getting about 30% of the benefit TF has to offer by just creating the infrastructure.

Hope to eventually lol

I highly, highly recommend using these upgrade guides. There are breaking changes to look out for https://www.terraform.io/language/upgrade-guides/0-13

Terraform supports upgrade tools and features only for one major release upgrade at a time, so if you are currently using a version of Terraform prior to v0.14 please upgrade through the latest minor releases of all of the intermediate versions first, reviewing the previous upgrade guides for considerations relevant to your use case.

I working in support and see many sad users who did yolo upgrades and their infra and state files are spaghetti.

Reading your comment again, you are not concerned with state since you use manual destroy process. Hmm. Well, still good practice to use the upgrade guides in case you plan to enhance your terraform usage and fully commit to infra-as-code.

I didn’t check the release notes for iam-system-user to see the breaking change. Doy.

awsutils provider is required now

I think this happens when you are missing a provider configuration for the awsutils provider

https://sweetops.slack.com/archives/CB6GHNLG0/p1642600240130700?thread_ts=1642600040.130400&cid=CB6GHNLG0

thanks for your reply I have this

provider "awsutils" {
  region = "us-east-2"
}

which is the region i am using

I have both

both pointing to the same region

ahh OK found it.. thanks for the hint I am assuming roles on the aws provider so I just added the same assume role to the awsutil provider and seems to work

Glad that helped!

Negative. We generally provision a single backend bucket and use workspace key prefixes instead.

If it’s a must, use the first tfstate backend as the backed state for all other state buckets, like a tree

It’s supported by atmos too

You can use #Terragrunt to run all the backends once for all and avoid repeating pieces of code.

I haven’t gotten there yet, but I was curious how the subaccounts access the root accounts bucket/dynamodb tables. Or is TF smart enough to use different credentials for the state than you are using in the current plan?

Backends support role assumption

@Erik Osterman (Cloud Posse) – we are using Control Tower to manage the accounts, so all our administrator/user roles have been auto setup (so I was trying to avoid using iam-primary-roles/iam-delegated-roles if possible. If I have both profiles active in Leapp, is it possible to specify the profile for the state and that be different for than the profile for the component? (or is that a stupid question because I haven’t wrapped my head around this yet?)

I’m not sure the best way. @Jeremy G (Cloud Posse)?

btw, have you checked out AFT yet?

AFT? No, because I don’t recognize that acronym

Role or profile for backend can be specified in [backend.tf](http://backend.tf) . In our case we always set it to ...-root-gbl-terraform. Role or profile for provisioning AWS resources is set in [providers.tf](http://providers.tf) in the AWS provider configuration. We usually set it to <account>-gbl-terraform , although there are exceptions (see privileged=true and account-map/modules/iam-roles)

We create a single state bucket in the root account, accessed via the ...root-gbl-terraform role and a separate Terraform role (with Admin privileges, since it has to be able to create IAM roles) in each stage account.

We log in via Leapp and set AWS_PROFILE to a role in the identity account that is allowed to assume the various terraform roles. Then everything else happens automatically.

AWS SDK will not go looking for credentials to assume a role ARN. Instead, it will use whatever it considers to be the “current” or “active” credentials to try to assume that role. You can, instead of using Role ARNs, use Profile names, and then the SDK will go looking for credentials as specified in the profile. This way you can log into multiple roles via Leapp and have them used automatically, but it can require extra setup in that you may have to define all the profiles in ~/.aws/config

no one?

@matt should have our lambda module coming out soon

we have a code deploy module, but we haven’t tied the two together yet

or ar you thinking canaries with API gateway?

or canaries with lambda alias weighting? There are a lot of options, I’m curious how you can do any of them with Terraform

that is the problem

there is many ways

there is the alias weight but you seems to need that with codedeploy

and there is a lambda you can deploy to do this

a lambda that controls the weight

we are thinking on doing in TF but use codedeploy to do the canary

We tried managing partial deploys with publish = true and aws_lambda_alias in pure Terraform, it was a big failure. Terraform can’t handle publish = true’s side effects and the alias weights were non-deterministic

since the artifact is already there

ahhh interesting to know

so it seems that it will be like deploy once, do not touch terraform again and use codedeploy

how do you measure if the new lambda is succeeding?

there seems to be a cloudwatch alert you can tight in with the deployment but I’m unsure yet if will give you errors base on the versions deployed

that cloudwatch alert supposed to feed codedeploy for automatic rollback

I’d heard about this after re:Invent but had only spent a fleeting amount of time looking into it. Haven’t looked at implementing in my own CT multi-org deployment

What benefit does CT offer over and/or with orgs?

CT is complimentary to operating in AWS with orgs. CT is for managing your accounts and provisioning a secure baseline. It makes it very easy to follow the many foundational best practices for organizing AWS accounts with Cloud Trails, OUs, SCP, SecurityHub, etc. Cloud Posse typically provisions all this with terraform rather than CT, but it’s taken us years to build out the modules and tools to be able to do it.

Unfortunately, CT itself has no API and therefore no terraform support.

the new CT AFT is targeted at bridging that gap and allowing you to make use of CT with the ease of git commits and using Terraform. I just haven’t dived to deeply into it yet. As I understand the implementation it is meant to be deployed after CT and replace the standard method of using Cloudformation through Service Catalog

Yes, is right, though I believe it is not mutually exclusive with CFT, it’s in addition to.

CT seems overly complicated to me, and I’m not sure I’d advocate it when you have a reasonable TF solution that manages accounts/roles similarly

That’s helpful to know as I continue to wade through all of this

@Mike Crowe it’s targeting a different demographic / use-case

imagine you’re an larger enterprise, with a sizable CT presence already. Teams are constantly pushing for using terraform and you want to get out of their way. This is how you can make amends.

You still have all the guardrails of CT and can continue to manage the enterprise the way you have been, but can open up the option of provisioning baseline infra inside of individual accounts with terraform.

TBH, we usually deploy the autoscaler with helm, so this might not be thoroughly tested.

You can’t represent the AWS SSO connection with your SSO provider in Terraform code.

But you can represent users, groups and permission sets in Terraform. You probably want to

Okay, that makes sense. So basically we’ll need to set up some initial SSO configuration, partly on the IDP side, before being able to dive in. Thanks!

you don’t normally need to specify users or groups. I mentioned it only for completion. I meant to say “you probably want to create permission sets with Terraform or some similar tool”

My company and my own method is to have an AWS account with your users. We us a deploy user. We then have n accounts for services. Each account for services leverages role based authentication.

Our deploy users credential’s are associated to the build pipeline via protected env vars. In our provider , we specify an assume_role block with the desired deploy role.

Hey, thanks for the response. Sorry if this is sounds silly, but what parts must be done by hand before I can go “full” IaC? Does it make sense to create an account dedicated to all the things that are self-managed and needed for terraform? prerequisite roles, buckets for state, dynamodb, etc.

So generally, use of IAM users is discouraged. There are a few exceptions, such us necessary integrations with third party services which crudely support AWS credentials. Generally, we’d recommend going with AWS SSO or Federated IAM, however, as newcomers to AWS these will be a bit more to setup.

If you are looking for some guidance, AWS publishes the Well-Architected Framework which is what we generally like to follow.

Our deploy users credential’s are associated to the build pipeline via protected env vars. This is where the answer will depend on your implementation. The ideal way is to aovid any hardcoded ENVs and instead use instance profiles or OIDC. GitHub Actions supports this https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services

Does it make sense to create an account dedicated to all the things that are self-managed and needed for terraform? We wouldn’t recommend this for any business because there’s poor audibility. You want to be able to trace everything back to an individual via Cloud Trail logs.

We then have n accounts for services. What you really want are IAM Roles that are assigned to services (not IAM users). ECS and EKS make this very easy.

(actually, I misread, you’re talking about AWS accounts; yes, you should have multiple AWS accounts broken out depending on how you need to isolate your workloads)

Awesome. So one aws account for images and logs + IAM roles? Then one account per environment? Get short lived access tokens through SSO, aws-vault, etc. and assume a role when needing to deploy to a specific environment (AWS account)?

Well, that’s one way - and there’s no right way.

https://docs.aws.amazon.com/controltower/latest/userguide/how-control-tower-works.html https://docs.aws.amazon.com/prescriptive-guidance/latest/designing-control-tower-landing-zone/account-structure-ous.html

We typically have root (top level), audit for all logs, security which operates on audit, identity for federated IAM, network for transit gateways, artifact for ECR and S3 artifacts, dns as the registrar, auto for automation, and then platform level accounts like dev, staging, prod, and sandbox. This is pretty much the minimal number of accounts we would provision.

Ah. Control Tower. Of course, I replied to a thread about this elsewhere and now it comes full circle. Makes sense. And wow, what a list. A great reference

https://sweetops.slack.com/archives/CB6GHNLG0/p1643949425613539?thread_ts=1643944436.230679&channel=CB6GHNLG0&message_ts=1643949425.613539

So I get the SSO and aws_iam_instance_profile thing for humans. What about service accounts? Imagine in house software that reads from an s3 bucket. I was planning to manage in terraform via IAM user and download the keys to set as env variables. Is there a more secure or compliant way?

aws_iam_instance_profile is for “service accounts” at the EC2 instance level (not humans)

Here’s how it works for EKS (aka IRSA) https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

And for ECS https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using-service-linked-roles.html

Hmm. Ok, so the ecs instance gets the credentials needed automatically

Or eks, or ec2. I guess for local development one could generate a dedicated key pair

Or wait, maybe I’m missing the point. Sounds like if one logs in through AWS-SSO, gets the right role, that solves the local development key issue. Nice

Zero to devops in 60 minutes. Really appreciate your time and insight

ecs instance * ECS task

Solid advice here. Theres a good amount to grok but you’ll capture that through more use.

If the service in question is not currently on AWS, is a programmatic (terraform-managed) IAM user an okay compromise? Hopefully it’s on ECS soon enough

If the service in question is not currently on AWS Yes, this is more often where you might make a compromise. Nonetheless, the same best practices apply surrounding regular key rotation.

can you comment out everything in the component and only leave the remote state, then output the entire remote state?

my guess is that you haven’t deployed dns delegated since adding this new output so when you try to use the remote state output in your new component, it fails

usually we use only role arn in the backend configuration so it’s more explicit. but i don’t think using either profile or role in the backend would change the above results

Good idea – [main.tf](http://main.tf) is now blank, and [outputs.tf](http://outputs.tf) now has:

output "dns_delegated" {
  value = module.dns_delegated
}

Error:

│ Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable
│ 
│   with module.dns_delegated.data.utils_component_config.config,
│   on .terraform/modules/dns_delegated/modules/remote-state/main.tf line 1, in data "utils_component_config" "config":
│    1: data "utils_component_config" "config" {

looks like atmos.yaml is not correct, or atmos can’t find it

You can DM me your code to take a look

Also, ATMOS_BASE_PATH ENV var needs to be set

To the absolute path to the root of the repo

this is what we do in geodesic

cd /localhost/
cd <path to the infrastructure repo on localhost>
export ATMOS_BASE_PATH=$(pwd)

@Andriy Knysh (Cloud Posse) – found the issue: I needed to add stack to my [remote-state.tf](http://remote-state.tf) file:

  stack = join("-", [var.environment, var.stage])

Does this seem right?

@Mike Crowe it does not sound correct. I’ll look at the code that you DM-ed me

Asking the obvious from the initial error: Aren’t you missing an outputs?

module.dns_delegated.outputs.wildcard_certificate_arn

That’s usually how the remote-state modules work. The Terraform error messages sometimes are misleading.

@Mike Crowe was missing context in the remote state (which provides environment and stage)

module "dns_delegated" {
  source  = "cloudposse/stack-config/yaml//modules/remote-state"
  version = "0.22.0"
  component = "dns-delegated"
  context = module.this.context
}

Correct, that was the root of my problem, however even with that context I’m still having the issue. I think it’s cuz I’m using something similar to the standard setup but not exactly because I’m using control tower for the account management

so it works like this:

will call the CloudPosse utils provider (which uses atmos code)

the provider will get the context from context (environment, stage, etc.) + the component name, and using atmos.yaml it will find the stack config for the provided component/environment/stage

Yeah, I switched to this inside and still have the issue

then from the stack config, it will get the s3 backend config and call TF remote backend data source to get the outputs for the component

Is there any way to print the context to make sure I’m getting it in the module correctly?

you can output it

output "context" {
  value = module.this.context
}

That only works if you can plan. This won’t plan

I’ve even tried TF_LOG=debug but that doesn’t really show me what is wrong

what’s the issue you re facing now @Mike Crowe ?

(you can comment out the malfunctioning resources/data sources and only comment in a specific output)

At a high level, my core issue is: • My accounts/environment/roles were set up by Control Tower, so I’m using multiple AWS_PROFILE entries to point to different accounts/roles (I am not using delegated-roles). So root points to the main account (profile=root in [backend.tf](http://backend.tf) at the top level) where the state is stored, and dev is the current environment (for the two components above) that I’m developing • I have one component deployed in a stack (ue1-dev) working fine and storing state remotely in s3 properly in the root S3 bucket/dynamodb tables just as expected. I’ve downloaded the JSON from this table for this component and it looks perfect • The second component is attempting to access the first’s component’s state using the [remote-state.tf](http://remote-state.tf) pattern • I have fixed all my context statements (as far as I can tell), but I cannot seem to get [remote-state.tf](http://remote-state.tf) working unless I add stack = join("-", [var.environment, var.stage]) (which should not be needed) in the second component. When I do a plan w/o stack=..., I get:

Error: stack name pattern must be provided in 'stacks.name_pattern' config or 'ATMOS_STACKS_NAME_PATTERN' ENV variable

I’ve confirmed (with Andriy’s help) my base_path and I (believe) everything is setup correctly in atmos (after all, it’s working properly in the first component). Even setting ATMOS_STACKS_NAME_PATTERN manually in the environment doesn’t resolve the error. Here’s the working [remote-state.tf](http://remote-state.tf) in the second component that I’m using:

module "dns_delegated" {
  source  = "cloudposse/stack-config/yaml//modules/remote-state"
  version = "0.22.0"
  component = "dns-delegated"
  stack = join("-", [var.environment, var.stage])
  context = module.this.context
}

I’m not technically blocked (per se), but it concerns me that something in my setup isn’t quite right

the error above means the utils provider could not find atmos.yaml file

FYI:

 ✗ . [none] (HOST) infrastructure ⨠ echo $ATMOS_BASE_PATH 
/localhost/Programming/Pinnacle/infrastructure

whats the output of this

find . -name atmos.yaml

It’s in the root of this folder

To be clear, atmos and the first component work fine and are properly configured. The core issue is the second component accessing the first components remote state

@Mike Crowe please place it at this path https://github.com/cloudposse/atmos/blob/master/examples/complete/rootfs/usr/local/etc/atmos/atmos.yaml , or in home dir (~/.atmos)

and test

the issue is, the utils provider gets called from the components’ folders

it can’t see the root of the repo anymore

(and remote-state calls the utils provider to process YAML config to get the remote state for the component in the stack)

it did not find it in /usr/local/etc/atmos

it did not find it in home dir (~/.atmos)

and it did not find it in the current directory since the current dir is the component folder

So, it can go in /conf in geodesic?

in geodesic, we put it into rootfs/usr/local/etc/atmos/atmos.yaml

so it’s visible to any process from any folder

Ding, Ding, Ding – we have a winner. Looks like you need atmos.yaml in both the root of your project and at one of the global locations referenced above. Adding it to geodesic at rootfs/usr/local/etc/atmos/ fixed my problem. Thanks @Andriy Knysh (Cloud Posse) @RB for excellent help

you need it only in one location, which should be visible to all processes in all folders (the root of the repo is not one of them; although atmos itself worked when atmos.yaml was in the root of the repo b/c you ran atmos commands from the root and it was the “current dir”)

it should be deployed once per account

cc @

oh my, that looks so great!

In practice, is it just a distinction between creating new users vs a data lookup of existing users?

in short, it is

but… companies can create users in Opsgenie using SAML

so those users are “existing” to the terraform code

if you want to use those users in other code, you need to read them using data source

Thank you @Andriy Knysh (Cloud Posse) That is exactly our case. An IT team adds users to our Confluence/Jira/Opsgenie (and other apps), but I want to manage the teams, rotations, services, and relationships between which teams ‘own’ which services etc. So in this case, I should use existing_users over users, right?

yes

existing_users will be looked up using the data source

And then just match the username (email address) of each user as their .id in the list of members in the teams . Is that right?

yes

we usually do

data "opsgenie_user" "team_members" {
  for_each = local.enabled ? {
    for member in var.members :
    member.user => member
  } : {}

  username = each.key
}

module "members_merge" {
  source  = "cloudposse/config/yaml//modules/deepmerge"
  version = "0.8.1"

  count = local.enabled ? 1 : 0

  maps = [
    # Existing members
    data.opsgenie_user.team_members,
    # New members
    local.members
  ]

}

module "team" {
  source  = "cloudposse/incident-management/opsgenie//modules/team"
  version = "0.15.0"

  team = {
    name           = module.this.name
    description    = var.description
    members        = try(module.members_merge[0].merged, [])
    ignore_members = var.ignore_team_members
  }

  context = module.this.context
}

https://github.com/cloudposse/terraform-opsgenie-incident-management/blob/master/modules/team/main.tf#L13

I believe there’s no way to have optional parameters as of terraform 1.1 without using the experimental default(...) function which is not advisable yet for production. maybe try passing null to the parameters not needed.

Thanks, @Erik Osterman (Cloud Posse) tried that, but still the same result; no worries appreciate the response!

@Yonatan Koren

@Josh B. can you please give me a snippet of what you’re trying to do?

Or permalink to the lines in [variables.tf](http://variables.tf) that you think are problematic?

So I am trying to do something along these lines with ordered cache.

ordered_cache = [
  {
     path_pattern     = "/avatars/*"
    allowed_methods  = ["GET", "HEAD", "OPTIONS"]
    cached_methods   = ["GET", "HEAD", "OPTIONS"]
    target_origin_id =  module.s3_bucket_assets.bucket_domain_name

   trusted_signers    = ["self"]
    trusted_key_groups = []
    min_ttl                = 0
    default_ttl            = 0
    max_ttl                = 0
    compress               = true
    viewer_protocol_policy = "redirect-to-https"

    forward_query_string              = false
    forward_header_values             = [""]
    forward_cookies                   = ""
    forward_cookies_whitelisted_names = [""]

    cache_policy_id          = ""
    origin_request_policy_id = ""
    viewer_protocol_policy = "allow-all"
    min_ttl                = 0
    default_ttl            = 3600
    max_ttl                = 86400
  response_headers_policy_id = ""

    lambda_function_association = [{
      event_type   = "viewer-request"
      include_body = false
      lambda_arn   = ""
    }]

    function_association = [{
      event_type   = "viewer-request"
      function_arn = null
    }]

}
]

It seems function_arn is the main culprit, but would need to to dig deeper.

@Josh B. in your original message you wrote
I think mainly the function arn which I am not using any sort of lambda or cloud function so confused a tad. So you’re not using Lambda at all? You should set lambda_function_association and function_association both to []

Correct no lambda at all. Okay let me give that a try! (could have sworn it made me add it but maybe not)

Let me know how it goes

and yes intuitively you’d just want to omit anything having to do with lambda if you’re not using Lambda. However as Erik mentioned default() aka optional object attributes is opt-in only still (I was surprised they didn’t throw it into Terraform 1.0.0 as fully supported out of the box).

If it was enabled by default in Terraform, then what we’d do is give those aforementioned variable a default of [], so users with use cases such as yours just wouldn’t have to think about those variables and wouldn’t run into this problem.

Ahh, okay, that makes total sense. Thanks for the explanation. So I think that for sure fixed this issue; I think I am running into something like https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/pull/188 but can work through it now. Thanks so much for the help Cloudposse

Thanks @Yonatan Koren

@Josh B.
I think I am running into something like https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/pull/188 but can work through it now Yes going to try to merge some of these open PRs soon

This is the part its referring to

resource "aws_wafv2_web_acl_association" "default" {
  count = module.this.enabled && length(var.association_resource_arns) > 0 ? length(var.association_resource_arns) : 0

  resource_arn = var.association_resource_arns[count.index]
  web_acl_arn  = join("", aws_wafv2_web_acl.default.*.arn)
}

maybe the web acl arn input is empty?

resourcearn is <20chars so either its empty or a bad validation error got added making your life difficult.

Can you create and output {} and dump the the attribute to make sure it’s populated/valid?

Thanks for the reply folks, I will check and report.

The changes to the aws_s3_bucket resource are huge, in particular. That’s likely a reasonably big rewrite of most modules using s3 buckets

Ugh, plus needing to import all the new, separate s3 resources

oh boy

maybe we need to set our versions from >= 3.0 to ~ 3.0

cc: @Erik Osterman (Cloud Posse)

I wouldn’t, in reusable modules. I’d instead encourage users to do that in their root config

ohhhh my……those are big changes

tons of refactor in many resources

and there it is… https://github.com/hashicorp/terraform-provider-aws/releases/tag/v4.0.0

https://www.hashicorp.com/blog/terraform-aws-provider-4-0-refactors-s3-bucket-resource

ohhhhh my…..I wasn’t expecting this the NEXT DAY……

These changes, along with other minor updates, are aimed at simplifying your configurations and improving the overall experience of using the Terraform AWS provider.

But at what cost? AT WHAT COST

The cost of makings us indispensable

haha. yay, I guess.

You can pin the provider version by updating [versions.tf](http://versions.tf) in the component to:

required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

maybe this example will help https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/tree/master/examples/vpc-only

Am I understand right, I should define VPCs and subnets explicitly? And do this with your modules?

Thank you for the link, but without any additional comments it is not really helpful

I did as described in the registry:

Provision Instructions Copy and paste into your Terraform configuration, insert the variables, and run terraform init:

module "vpc-peering-multi-account" {
  source  = "cloudposse/vpc-peering-multi-account/aws"
  version = "0.5.0"
  # insert the 4 required variables here
}

but it doesn’t work

I put 4 required variables

What am I doing wrong?

Am I understand right your example is for one region? I need cross region

use diff providers for the regions

provider "aws" {
  region = var.second_region
  alias  = "second"
}

then use the provider for the resources in that region

or look at this example https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/master/examples/complete/main.tf

you provide requester_region accepter_region

and role ARNs for the accepter and requester

I did exactly this way

And now I got this error. Everything is working except peering

@Kirill I. if you share the code (in Slack or by email), we’ll take a look (it’s difficult to say anything w/o looking at the code)

variable “accepter_region” { default = “us-west-1” }

variable “requester_region” { default = “eu-west-2”

terraform { required_providers { aws = { version = “3.74.1” } } }

provider “aws” { # London region = “eu-west-2” }

provider “aws” { alias = “eu-west-2” # London region = “eu-west-2” }

provider “aws” { alias = “us-west-1” # California region = “us-west-1” }

#Create VPC in us-west-1 resource “aws_vpc” “vpc_uswest” { provider = aws.us-west-1 cidr_block = “10.10.0.0/16” enable_dns_support = true enable_dns_hostnames = true }

#Create VPC in eu-west-2 resource “aws_vpc” “vpc_euwest” { provider = aws.eu-west-2 cidr_block = “10.11.0.0/16” enable_dns_support = true enable_dns_hostnames = true }

resource “aws_iam_role” “test_role” { name = “cross_region_vpc_peering”

# Terraform’s “jsonencode” function converts a # Terraform expression result to valid JSON syntax. assume_role_policy = jsonencode({ Version = “2012-10-17” Statement = [ { Action = “sts:AssumeRole” Effect = “Allow” Sid = “” Principal = { “AWS” : “arniam:user/kiril” } }] }) }

module “vpc-peering-multi-account” { source = “cloudposse/vpc-peering-multi-account/aws” // version = “0.5.0” # insert the 4 required variables here

accepter_region                          = var.accepter_region
requester_aws_assume_role_arn             = aws_iam_role.test_role.arn
requester_region                          = var.requester_region

}

resource “aws_subnet” “subnet_1_us” { provider = aws.us-west-1 # availability_zone = element(data.aws_availability_zones.azs.names, 0) vpc_id = aws_vpc.vpc_uswest.id cidr_block = “10.10.1.0/24” }

resource “aws_subnet” “subnet_1_eu” { // provider = aws.eu-west-2 vpc_id = aws_vpc.vpc_euwest.id cidr_block = “10.11.1.0/24” availability_zone = “eu-west-2a” }

resource “aws_route_table” “route_table_us” { provider = aws.us-west-1 vpc_id = aws_vpc.vpc_uswest.id // route { // cidr_block = “0.0.0.0/0” // gateway_id = aws_internet_gateway.igw-us.id // } // route { // cidr_block = “10.11.1.0/24” // vpc_peering_connection_id = aws_vpc_peering_connection.eu-us-peering.id // }

lifecycle { ignore_changes = all } tags = { Name = “US-Region-RT” } }

resource “aws_main_route_table_association” “set-us-default-rt-assoc” { provider = aws.us-west-1 vpc_id = aws_vpc.vpc_uswest.id route_table_id = aws_route_table.route_table_us.id }

resource “aws_route_table” “route_table_eu” { provider = aws.eu-west-2 vpc_id = aws_vpc.vpc_euwest.id /* route { cidr_block = “0.0.0.0/0” gateway_id = aws_internet_gateway.igw-eu.id } route { cidr_block = “10.0.1.0/24” vpc_peering_connection_id = aws_vpc_peering_connection.eu-us-peering.id } */ lifecycle { ignore_changes = all } tags = { Name = “EU-Region-RT” } }

resource “aws_main_route_table_association” “set-eu-default-rt-assoc” { provider = aws.eu-west-2 vpc_id = aws_vpc.vpc_euwest.id route_table_id = aws_route_table.route_table_eu.id }

terraform plan ╷ │ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.accepter[0], │ on .terraform/modules/vpc-peering-multi-account/accepter.tf line 67, in data “aws_route_table” “accepter”: │ 67: data “aws_route_table” “accepter” { │ ╵ ╷ │ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.accepter[1], │ on .terraform/modules/vpc-peering-multi-account/accepter.tf line 67, in data “aws_route_table” “accepter”: │ 67: data “aws_route_table” “accepter” { │ ╵ ╷ │ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.requester[1], │ on .terraform/modules/vpc-peering-multi-account/requester.tf line 121, in data “aws_route_table” “requester”: │ 121: data “aws_route_table” “requester” { │ ╵ ╷ │ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.requester[0], │ on .terraform/modules/vpc-peering-multi-account/requester.tf line 121, in data “aws_route_table” “requester”: │ 121: data “aws_route_table” “requester” { │ ╵ ╷ │ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.requester[2], │ on .terraform/modules/vpc-peering-multi-account/requester.tf line 121, in data “aws_route_table” “requester”: │ 121: data “aws_route_table” “requester” {

@Andriy Knysh (Cloud Posse) any thoughts?

when we deploy this module, we usually have the VPCs already provisioned

I think it’s not possible to provision the VPCs at the same time as the VPC peering

try to separate the VPCs into a separate component and provision it first

then use remote state to get the VPC IDs and provide them to the peering component

also, in your code

module "vpc-peering-multi-account" {
  source  = "cloudposse/vpc-peering-multi-account/aws"
//  version = "0.5.0"
  # insert the 4 required variables here
    accepter_region                          = var.accepter_region
    requester_aws_assume_role_arn             = aws_iam_role.test_role.arn
    requester_region                          = var.requester_region
}

I don’t see

requester_vpc_id                          = var.requester_vpc_id
accepter_vpc_id                          = var.accepter_vpc_id

if you don’t provide the VPC IDs, then that’s the issue

if you just omitted it in the code snippet, try to provision the VPCs first (either in a separate component, or using terraform plan/apply --target=xxx

take a look at the example, all variables need to be provided to the module https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/master/examples/complete/main.tf

role (which could specify the same account or diff account), region and VPC ID - for both accepter and requester

@Kirill I.

I looked into github page and put only those settings which was listed as mandatory.

Does your module support ‘depends’ directive?

Can I use explicit declaration instead of variables?

the module accepts the variables as shown in the example

you need to provide all those variables for the module to work

you have all the info in your code, you just did not provide some vars to the module (e.g. VPC IDs were not provided)

Add the variables VPC IDs to the module with the values from the VPCs that are already configured in your code

@Kirill I. can you add the variables to the module (check which one is the accepter and which one is the requester, the code below is an example), and test it

requester_vpc_id = aws_vpc.vpc_uswest.id
accepter_vpc_id  = aws_vpc.vpc_euwest.id

sure will do and get back to you

│ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.accepter[0], │ on .terraform/modules/vpc-peering-multi-account/accepter.tf line 67, in data “aws_route_table” “accepter”: │ 67: data “aws_route_table” “accepter” { │ ╵ ╷ │ Error: error reading EC2 VPC: UnauthorizedOperation: You are not authorized to perform this operation. │ status code: 403, request id: a70a1d6b-15b0-4df2-9f39-4277450eb88d │ │ with module.vpc-peering-multi-account.data.aws_vpc.requester[0], │ on .terraform/modules/vpc-peering-multi-account/requester.tf line 99, in data “aws_vpc” “requester”: │ 99: data “aws_vpc” “requester” { │ ╵

errors anyway

fixed policy and got the same error :

│ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.accepter[0], │ on .terraform/modules/vpc-peering-multi-account/accepter.tf line 67, in data “aws_route_table” “accepter”: │ 67: data “aws_route_table” “accepter” { │ ╵ ╷ │ Error: query returned no results. Please change your search criteria and try again │ │ with module.vpc-peering-multi-account.data.aws_route_table.requester[0], │ on .terraform/modules/vpc-peering-multi-account/requester.tf line 121, in data “aws_route_table” “requester”: │ 121: data “aws_route_table” “requester” {

instead of manually creating all those VPCs, subnets and route tables (prob something is missing when creating all of that manually), I’d recommend using some existing modules to create them, for example as shown here https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/master/examples/vpc-only/main.tf

then the VPC IDs outputs from the modules can be used as inputs to the peering module as shown here https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/master/examples/complete/main.tf

use the first example as one component and provision it first

then use remote state to get the VPC IDs outputs outputs and provision the peering component

(the issue could be 1) something is missing when creating all of the VPC resources; or 2) terraform can’t provision the peering and the VPC in one step (so either provision the two components separately, or use -target; or 3) both)

this terratest https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/master/test/src/examples_complete_test.go provisions the first component to create the VPCs/subnets/route tables, then provisions the second component to create the peering connection (and it’s a working test that deploys the resources on AWS)

A nice upgrade guide. https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade

https://cloudposse.slack.com/archives/CB6GHNLG0/p1644448421073489

we’ll have to review and fix the module to support the provider v4. You can also pin the provider in your code to ~> 3

oh Matt already posted the above here https://sweetops.slack.com/archives/CB6GHNLG0/p1644519534463299?thread_ts=1644519506.553189&cid=CB6GHNLG0

+1 I get the same thing

there isn’t anything super clean, but you can use this approach: https://github.com/plus3it/terraform-null-validate-list-item/blob/master/main.tf#L1-L13

hm interesting, thank you - will play around with it

another idea might be to use an object for the variable type that includes both attributes? then your variable validation block would have access to both values

That’s so weird I was just taking a break for coffee and was thinking about doing that, grouping the top level arguments together in an object just as I got your message ping

I think that’s much nicer, gives the API caller a bit more of an idea of what’s going on without having to read descriptions and just looking at how it’s called

heh, great minds!

Yeah but objects cannot have default values for particular attributes (for now)

https://www.terraform.io/language/functions/defaults is a thing but requires expermintal flags which output warnings on every plan/apply

The object and the attributes can have default values, but if the user specifies the object at all, then they must specify all values

Yup which is very annoying

Doesn’t bother me much. I find I prefer the strong typing and improved error messages, and clear and easy documentation

It’ll be nice if the optional experiment makes it GA, of course

I used the optional type and enabled the experiment for it, so far so good it has not eaten my lunch

Aye, but them warnings were

yeah big yellow blob definitely annoying, luckily most of my terraforming is automated these days so I don’t see it that often

https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/tree/0.8.3

@cloudposse-team

Probably related to https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/blob/0.8.3/docs/migration-0.7.x-0.8.x%2B.md

In a meeting, will respond more later.

@Alyson Please post the output of terraform plan

also, what TF state are you using? If you are using local state, it could be wiped out and TF will want to create everything again. You need to use s3 for remote state

@Jeremy G (Cloud Posse)

follow the result of my terraform plan

https://pastebin.com/JzXGy1g7

@Andriy Knysh (Cloud Posse) I’m using TF remote!

This is the issue, but I wonder if it is an AWS provider bug. What do you think, @Andriy Knysh (Cloud Posse)?

      ~ client_authentication {                                                                                        

          + tls { # forces replacement}                                       

Terraform thinks that client_authentication has not been set up on the target cluster.

@Alyson Have you successfully applied the changes once, so that the cluster was destroyed and recreated with client TLS authentication enabled?

@Jeremy G (Cloud Posse)

Yes, “terraform apply” was successfully executed!

Then I ran “terraform plan” again and it said that the cluster would be destroyed and recreated again.

I’m not sure what the issue is, but this code looks inconsistent to me (might be not related to the issue at all)

dynamic "client_authentication" {
    for_each = var.client_tls_auth_enabled || var.client_sasl_scram_enabled || var.client_sasl_iam_enabled ? [1] : []
    content {
      dynamic "tls" {
        for_each = var.client_tls_auth_enabled ? [1] : []

@Andriy Knysh (Cloud Posse) The code looks OK to me. The tls section is a subsection of client_authentication .

@Alyson We have not tested this code much because we do not have a cluster to test on. Please review this open issue on the Terraform AWS provider and let us know if you think it applies to your situation.

like this? https://github.com/cloudposse/terraform-aws-service-control-policies/blob/master/examples/complete/main.tf#L1-L21

leverages https://github.com/cloudposse/terraform-yaml-config

I’ll check that out. I got the pattern of pulling in YAML files via locals from you, but this looks easy. I’d like to validate the input YAML files directly — or pass the product of yamldecode(file(xxx)) as a TF variable in the root mod. I generally pull the yaml in the root and pass all or some of what’s in those files as resource or module params

I’m thinking — or recall your having said — that you pull yaml files in via locals and write them to the root dir as auto.tfvars.json files. Does that work?

our yaml config module supports local files and remote files

for customers who want their configs local, we copy and commit them. it’s not something we use terraform for.

Please have a look in the history ;)

I tried it, let me check again

https://cloudposse.slack.com/archives/CB6GHNLG0/p1644448421073489

https://cloudposse.slack.com/archives/CB6GHNLG0/p1644578070833829

https://sweetops.slack.com/archives/CB6GHNLG0/p1644577293783099

Thank you @Erik Osterman (Cloud Posse). It fixed the issue!

you are the best

:-) glad it was an easy fix.

Hackerman!

We will be releasing a new module but it will be breaking changes

Is there any chance of getting the module updated prior to that, just saying it needs provider <4? That way there’s a working version prior to the new “breaking changes” one. I’ve thrown this onto github the other day: https://github.com/cloudposse/terraform-aws-cloudfront-cdn/issues/83

looks like the s3 website module depends on the https://github.com/cloudposse/terraform-aws-s3-log-storage module which uses raw s3 bucket resources instead of our upstream s3 bucket module. we’d have to convert that to use our module, then get that merged, then bump the s3 website module to use the latest log storage to allow s3 website module to work with v4 aws provider

https://github.com/cloudposse/terraform-aws-s3-website/issues/71 provider v4 making things fail

Why it just use the account alias

Sorry, this was for being able to assume a role in the target account; we SSO to our management account, then assume roles into the relevant accounts via an account_id variable that we just substitute in the ARN of the providers assume-role block. I guess I’m looking for a way to replace this with a “friendlier” name but then do the mapping to the account_id in some way, to then be used in the provider config

Ah I did this based off workspaces, with the workspace being the account alias and then a map of aliases and account ids would get the correct account I’d from workspace. But that’s pretty opinionated. You could just use a var and do same thing

you could have a remote state with an output map for all your accounts where each key is the account name

that’s how we do it with our account-map component

https://github.com/cloudposse/terraform-aws-components/tree/master/modules/account-map

yes this is the way, then it can be updated in one place.

Trying to avoid workspaces; I’ve shot myself in the foot with it a few times. We’ve standardised on terragrunt which is working really well.

Thanks @RB will check it out

oh the foot has been shot, workspaces sounds like a good idea until its not

we use workspaces using atmos but it’s completely automated so we never have to even select the workspace

if you haven’t used atmos yet, it’s a pretty nice tool that converts deep merged yaml into terraform module input vars

I actually just checked it out for first time a couple weeks ago, need to take a deeper dive.

@pjaudiomv see these resources

https://github.com/cloudposse/atmos/tree/master/examples/complete https://docs.cloudposse.com/tutorials/atmos-getting-started/

Thanks

you can generate the password using the random provider and store it as an ssm parameter

thank you very much for your advise

np. here’s how we do it for postgres

https://github.com/cloudposse/terraform-aws-components/blob/95ade5b36b61d2432179399bd0e9fa8639eeb899/modules/aurora-postgres/main.tf#L54

Hi @hasinireddybitla2404 it would be easier to help if you could clean up the terraform code in your post by formatting it using triple backticks

This is the module we use to provision kafka / msk https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster

Not sure what you mean by unauthenticated access

https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/blob/cd11751cb4c999f3c17013dd32336659a277744e/main.tf#L134

If you set all of these to false, authentication looks like it will be disabled

var.client_tls_auth_enabled
var.client_sasl_scram_enabled
var.client_sasl_iam_enabled

Hi @RB thanks for the reply. Im trying to say abt the “Access control methods” where we select two access controls one is “Unauthenticated access” and the other one is “IAM role-based authentication”. I want to enable above two access control methods. But with my script i can only enable the “IAM role-based authentication” not the “unauthenticated access”. Im stucked here. Im sure Im missing some logic in the script, but im not getting it.

Please use codeblocks

Can you show an example of the name you want output?

Try #terragrunt instead