#terraform (2021-03)

yes

the issue i have is each time it needs to execute against a different account

I think for loops are totally random

so every time you execute them it will pick a different account

I do not know if there is a way to for an order

maybe @loren knows?

i’m guessing getting the provider right is the issue, more than order?

there is not yet a way to use an expression on a module’s “providers” block… it is static, even if the module is using for_each . so i don’t think you can do what you are thinking yet…

you could maybe template the tf files from outside terraform, not using for_each, but creating an instance of the module per account and setting the provider

good point I was assuming he had that working

We have an IAM policy aggregator module that maybe can help?

On the aws_iam_policy_document, use source_json and/or override_json in pre-3.28.0 of aws provider. Or with >=3.28.0, use source_policy_documents and/or override_policy_documents

Meant to open an issue @Erik Osterman (Cloud Posse), with 3.28.0, that aggregator module can probably be archived…

ah cool, is this related to that AWS provider release that made you cry tears of joy?

Yeah, haha, that was the first of them… The second was managing roles with exclusive policy attachments

wow. I wish I waited to rewrite this policy statement in HCL/jsonencode

Great tool for if you ever need to do that in the future @Alex Jurkiewicz: https://github.com/flosell/iam-policy-json-to-terraform

There’s just not enough to go on. The cloudposse/terraform-aws-ecs-codepipeline doesn’t even create the tasks. You must be using other modules. Provide as much context as possible. We have hundreds of modules.

(and use threads)

i think i found the culprit actually, im running a next js deployment, and im doing a build on the container once its tarts. that build appears to trigger the autoscale based on cpu usage

the reason the deployment isn’t removing the previous deployment looks like a host capacity issue

that makes sense

i gotta figure out how to get my app autoscale policy to scale up more ec2s to fit the service needs

Yes! this is currently used in all the cloudposse tf modules

and in my company modules

Ah they launched a docs site though — that looks fresh.

nice

(didn’t know it supported a config)

Yes, and you can get header from a file like [doc.tf](http://doc.tf) and add it to README.md

WAnt to share some cool stuff about this today during announcements in #office-hours?

https://github.com/cloudposse/terraform-yaml-config is more or less generic module to work with YAML files and deep-merge them (and using imports to be DRY)

we have a more opinionated one for YAML stack configs (Terraform and helmfile vars defined in YAML files for stacks of any level of hierarchy ) https://github.com/cloudposse/terraform-yaml-stack-config

ahh, crud, thats what i meant to post.

Thanks for chiming in and correcting that.

it uses this TF provider https://github.com/cloudposse/terraform-provider-utils/blob/main/examples/data-sources/utils_stack_config_yaml/data-source.tf to deep-merge YAML configs (TF deep-merging was very slow, go is much faster)

(having said that, we did not try to convert TG to TF )

That makes complete sense. The stack module which has the child modules seems like a great fit to the default actions of TG. It also seems like it can solve my remote_state configuration issue, which was the second thing i was concerned about.

Its all in a branch, of course, and if i can wipe enough of company data and still be useful to others, i’ll make sure i do a write up and share here.

It’s all about Terraform modules that you are using, start with one module at a time, replace terragrunt blocks with terraform block:

• backend block >> backend.tf

• provider block >> provider.tf

• inputs block >> terraform.tfvars

• locals block >> locals.tf and for each env in Terragrunt create a env folder in terrraform

check this sample layout https://github.com/mhmdio/terraform-infracost-atlantis

so you need to map back the abstraction from Terragrunt into Terraform, let me know if you need any help

Thanks!

delete them!

can you do that via terraform though?

nope, try cloud-nuke

my favourite tool

AWS SSO would be better alternative

I think managing credentials that rotate with terraform is a bad idea.

Thanks for talking the time to leave comments. I’m always open to learning new ways to do things, so could you expand on your answers to suggest how I’d best manage things such as SMTP authentication details within applications?

I can’t always use an instance role as the application requires a username and password to be configured or they won’t start. Typically I create a user and then user data grabs the ID and ses_smtp_password_v4 details from secret store on boot up. This allows us to change the details every time we flatten and build the environment. My challenge comes when I get to a Prod system where I need to leave the access keys in place while a new ASG spins up and replaces the old machines/keys. I can simply do this by removing the current key from state, run the TF apply again but this never feels overly right. Hence my original thoughts about two variables that I use but I’m interested to hear an alternative approach.

Anybody got further thoughts on this?

I think you should move this out of Terraform, use lambda function with boto3 to provide temp credentials to you ASG when needed, otherwise use something like HashiCorp vault to implement that.

Thanks Mohammed, interesting idea, I’ll give it some further thought as I can see how that could work.

can hashicorp vault actually produce IAM secret key & access keys? doesn’t seem so with the kv engine. You’d have to re-gen the new keys, store them in the kv engine as a new set.. basically the multi-step plan @Gareth was intiailly thinking about

@Steve Wade (swade1987) This might help: https://docs.aws.amazon.com/eks/latest/userguide/troubleshooting_iam.html

S3 with dynamodb locking

Versioned s3 bucket

yeah, cloudposse has a module for this

https://github.com/cloudposse/terraform-aws-tfstate-backend

which works fine

but you cannot use a variable in the file it generates

I want to use a variable here because the profile can change, and the key as well

last time i checked, that module cannot reuse an s3 bucket unfortunately

what I can do is copy over the backend.tf file it generates and replace the key there

per thing that I need

I don’t want to put everything in a single state file

but I want to use a single bucket

i run this script

# bucket name
export tfstateBucket=mybucket
# get repo name
export repoName=$(git config --get remote.origin.url | cut -d '/' -f2 | cut -d '.' -f1)
# get current directory
export repoDir=$(git rev-parse --show-prefix | rev | cut -c 2- | rev)
# create backend
cat <<EOF > backend.tf
terraform {
  required_version = ">=0.12"
  backend "s3" {
    encrypt        = true
    bucket         = "$tfstateBucket"
    dynamodb_table = "TerraformLock"
    region         = "us-west-2"
    key            = "$repoName/$repoDir/terraform.tfstate"
  }
}
EOF
# see the newly created file just to be safe
cat backend.tf

it always gives me a unique key based on the repo name and repo path

youll have to change tfstateBucket

that’s easy, we have multiple customers

and I use the customername as a single identifier

hah you can use it

https://github.com/cloudposse/terraform-aws-tfstate-backend/blob/master/main.tf#L237

they generate file based on variables

so a combo of init, apply and init again should work

¯_(ツ)_/¯

hmmm interesting

it wont work for me yet as the bucket is not configurable but im glad it works for you

don’t you use a single bucket to put all the state files in per project ?

sepparated by keys ?

yep

that module would have to be consumed in everyone of our modules for the backend.tf to be generated, which would then create a new bucket and dynamodb table for each module

what id want is to reuse a single dynamodb and state bucket for every terraform module

yeah

same here

but i dont believe the cloudposse tf module supports that (yet)

I want to make the location of the state file configurable

the rest can stay as is

im sure everyone would be open to a PR. probably no one has gotten around to it.

because I get in trouble as well when I have a single AWS account that we use as a internal account and we want to add a new project

then you need to import the current bucket and dynamodb table

and then it gets a bit messy

gross

i wouldnt import a bucket and dynamodb table to more than one module

i also wouldnt import any resource to more than one module

thats an antipattern

thanks for the suggestion

I am still a bit newbie in terraform

there’s a lot to learn but basically resources should be stored in a single state. if they are stored in multiple states then multiple modules may try to modify the same resource and then have conflicting states

keep shared resources in their own modules.

I do

what I am puzzled about is what the best suggestion to:

I have one AWS account

we have several machines under that account

that we use internally

for all external customers, they have their own account

which makes handling a state file easy

for the multiple projects under our own internal account

I am puzzled how to handle this the best

a bucker per internal project ?

or prefixes in one big s3 bucket

this is what i do with multiple accounts.

1 tfstate bucket 1 dynamodb reuse the bucket with a key of the repo name inside the repo name key, use a repo path key

same shared bucket across all accounts

same shared dynamodb across all accounts

the unique identifier is the key / prefix

so you dynamically create the s3state.tf that pushes the state to the backend right ?

no. i dynamically create the backend.tf using the script i shared earlier

then it magically works

nice

it’s more clear now, thanks a lot

in the past I was the main creator of the terraform configs, now I need share work with co’s, that is better in one way but gives me way more work

but in the end, we will benefit all (I hope so )

divide and conquer!

We currently have the concept of a centralized aws account that manages buckets & dynamo (via terraform) as well as other things unrelated to this conversation (logging, monitoring, etc..). Using terraform workspaces & assume_role in the provider - you can do exactly what you’re looking for. This may be a bit more advanced and makes assumptions about having SSO and other policies in place already IIRC. I’ll try to provide a scrubbed example shortly.

nice thanks MattyB

Here’s an example I found for the provider portion: https://gist.github.com/RoyLDD/156d384bd33982e77630b27ff9568f63

Will try to get back to you about the terraform backend. Gotta run to a meeting..

Thanks for the convo

Back to it…using the examples above and a backend config like so:

terraform {
  backend "s3" {
    # Replace this with your bucket name!
    bucket = "prod-terraform-state-bucket"
    key    = "terraform.tfstate"
    region = "us-west-1"

    # Replace this with your DynamoDB table name!
    dynamodb_table = "prod-terraform-state-lock-table"
    encrypt        = true
  }
}

When you use terraform workspace with this config it stores the state in the bucket path at :env/<environment>/tfstate in the centralized account

I avoided workspaces when i first started b/c they were rather opaque. Do you use these as a go to?

Yep. We use it with every account. 1 coworker copy/pasted between staging and prod directories instead…and that account is a mess. He also didn’t use CloudPosse modules, and developed his own instead. It’s seen as the red headed step child that nobody wants to touch. lol

Fair enough. I plan on using the modules plus some tips here and in #office-hours today. trying to get as many opinions as possible before i start changing everything.

Gotcha. It’s straightforward and IMO helps keep environments as close to the same as possible. If you want a smaller DB in staging than you need in prod just set a different variable. Super simple. I don’t know what copy/pasting between directories (dev/stage/prod) buys you.

Not something that drives me. I’m more interested in keeping environments clean and reduce copy/paste.

right now i have none

Nice! Bug free

yep, and that’s what i’m trying to do with plain TF, with some CP module assistance.

now all abstractions are handled via TG.

terraform “static” typing is annoying in many cases

maybe that ^ will work

Just add iam_policies = [] to each item. Simplifies all the logic away

for example (now that i’m back at a keyboard…)

locals {
  lambda_functions = {
    foo = { memory=256, iam_policies = [...] }
    bar = { memory=128, iam_policies = [] }
  }
}

...

dynamic "inline_policy" {
    for_each = local.lambda_functions[each.key].iam_policies
    content {
      name   = "ExtraPermissions${md5(jsonencode(inline_policy.value))}"
      policy = jsonencode(inline_policy.value)
    }
  }

you could set it like this

enabled_cloudwatch_logs_exports = ["audit", "error", "general", "slowquery"]

the doc are related to cloudwatch log groups for RDS

that is where you can find the info and details of the valid loggroups

Thanks @jose.amengual that works great. Actually what i didn’t get is that I don’t have to declare any Cloudwatch Log group. So I used enabled_cloudwatch_logs_exports = [“postgresql”] and everything has been fine! Thanks again.

np

ahhhh yes in postgress you only have one

indent-rainbow FTW

Hi @maarten

what resources do you need to backup?

dynamodb (cmk)

did you look at https://aws.amazon.com/blogs/database/cross-account-replication-with-amazon-dynamodb/ ?

I don’t think this module https://github.com/cloudposse/terraform-aws-backup supports https://docs.aws.amazon.com/aws-backup/latest/devguide/cross-account-backup.html

this https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/backup_global_settings needs to be added (and maybe something else)

having said that, you can add the above to the top-level module

and then yes, you create a separate aws_backup_vault and use https://github.com/cloudposse/terraform-aws-backup/blob/master/variables.tf#L42 to copy into it

(IAM permissions must be in place)

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/efs_access_point

Word.

still not sure how to pass a map of maps like that

I’ve tried tons of variations on..

  access_points = {
    jenkins = {
      posix_user = {
        uid = 1000
        gid = 1000
      }
      root_directory = {
        path = "/jenkins"
        creation_info = {
          owner_gid = 1000
          owner_uid = 1000
          permissions = 777
        }
      }
    }
  }

but they all error out with things similar to

The given value is not suitable for child module variable "access_points"
defined at .terraform/modules/efs/variables.tf:12,1-25: element "jenkins":
element "root_directory": all map elements must have the same type.

this is the issue with the latest TF versions

they introduced so-called “strict” type system

all items in a map must have the same types even for all complex items

try to make all map entries having the same fields, set those not in use to null

something like that ^

that looks horrible.

it’s not pretty (and might not work, in which case we’ll have to redesign the module)

we can separate those into two vars

PRs are always welcome

If I was able to understand what this was doing, i wouldnt be here asking

yea, but you are trying to use the feature of the module which was tested with TF 0.12

TF 0.13 and up introduced a lot of changes

so the feature needs to be updated to support TF 0.13/0.14

and the only ways around are: 1) use the ugly syntax above; 2) open a PR to separate the map into two variables

@Justin Seiser would you like to open a PR with the changes? We’ll help with it and review

I am not capable of writing the PR.

I see you opened an issue, thanks (we’ll get to it)

In regards to CloudPosse Terraform modules or with Terraform in general?

1. Start your morning with an extra hit of caffeine.
2. Be thankful it’s not as bad as 0.11 to 0.12

it should work right away

Terraform In general

including Cloudposse modules

most of the modules are 0.13-0.14 compatible

I am running the following steps tfenv use latest:^0.13

terraform 0.13upgrade

terraform init terraform apply

in your modules?

yes

that is fine

then you need to update the source of cloudposse modules if one of them does not work

Is this mandatory to run terraform 0.13upgrade?

Even if run these it upgrades tfenv use latest:^0.13 terraform init terraform apply

in yours modules probably

you need to change few thing

the providers and such

so is better to run it

For upgrading 0.13.6–>0.14.5 tfenv use latest:^0.14 terraform init terraform apply

Are these steps good?

you still need to run the upgrade command I think

..and if our modules are in their own git repo, once the terraform init and terraform apply succeed, we’d have to commit in a featurebranch, then merge to master, correct?

git checkout -b upgrade_to_terraform013
tfenv use latest:^0.13
terraform init
terraform apply
git commit -m 'upgrade module to terraform 0.13.6'
git tag -a "v1.5" -m "v1.5"  #(or whatever your next v is)
git push --follow-tags

https://www.terraform.io/upgrade-guides/0-14.html

Looks like upgrade step is not needed for 13–>14 upgrade

here’s my method https://gist.github.com/nitrocode/1d5811a9dbc74720f63fc35372d82674

i use this all over the place at my current job and it works well

Leverage the serverless paradigm for all it’s worth! Skip the vpc, it is gloriously freeing

Thanks @loren you’re a fountain of knowledge as usual.

Just pay attention to your resource policies, control them tightly and actively with terraform, so you don’t expose anything you don’t mean to expose

VPC endpoints are needed only when you want to connect to your services from VPC without using public Internet (via AWS PrivateLink): https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints.html

I’m using the cloudposse/eks-cluster with the cloudposse/named-subnets module mostly configured like the examples:

After creation, terraform plan works fine.

When I change the existing us_east_1a_private_subnets/subnet_names and us_east_1b_private_subnets/subnet_names to be ["eks", "mysql"] and do terraform plan, I see:

Error: Get "<http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth>": dial tcp [::1]:80: connect: connection refused

Releasing state lock. This may take a few moments...

With the subnet_names change, the debug output contains a WARNING: Invalid provider configuration was supplied. Provider operations likely to fail: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

2021-03-05T10:10:43.078-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:10:43 [WARN] Invalid provider configuration was supplied. Provider operations likely to fail: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable: timestamp=2021-03-05T10:10:43.078-0800
2021-03-05T10:10:43.079-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:10:43 [DEBUG] Enabling HTTP requests/responses tracing: timestamp=2021-03-05T10:10:43.078-0800
2021/03/05 10:10:43 [INFO] ReferenceTransformer: reference not found: "local.enabled"
2021/03/05 10:10:43 [INFO] ReferenceTransformer: reference not found: "var.apply_config_map_aws_auth"
2021/03/05 10:10:43 [INFO] ReferenceTransformer: reference not found: "var.kubernetes_config_map_ignore_role_changes"
2021/03/05 10:10:43 [INFO] ReferenceTransformer: reference not found: "local.map_worker_roles"
2021/03/05 10:10:43 [INFO] ReferenceTransformer: reference not found: "var.map_additional_iam_roles"
2021/03/05 10:10:43 [INFO] ReferenceTransformer: reference not found: "var.map_additional_iam_users"
2021/03/05 10:10:43 [INFO] ReferenceTransformer: reference not found: "var.map_additional_aws_accounts"
2021/03/05 10:10:43 [DEBUG] ReferenceTransformer: "module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0]" references: []
module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0]: Refreshing state... [id=kube-system/aws-auth]
2021-03-05T10:10:43.083-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:10:43 [INFO] Checking config map aws-auth: timestamp=2021-03-05T10:10:43.083-0800
2021-03-05T10:10:43.083-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:10:43 [DEBUG] Kubernetes API Request Details:
---[ REQUEST ]---------------------------------------
GET /api/v1/namespaces/kube-system/configmaps/aws-auth HTTP/1.1
Host: localhost
User-Agent: HashiCorp/1.0 Terraform/0.14.7
Accept: application/json, */*
Accept-Encoding: gzip


-----------------------------------------------------: timestamp=2021-03-05T10:10:43.083-0800
2021-03-05T10:10:43.084-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:10:43 [DEBUG] Received error: &url.Error{Op:"Get", URL:"<http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth>", Err:(*net.OpError)(0xc000e67cc0)}: timestamp=202
1-03-05T10:10:43.084-0800

Without the change, there is no WARNING.

2021-03-05T10:07:59.545-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:07:59 [DEBUG] Enabling HTTP requests/responses tracing: timestamp=2021-03-05T10:07:59.545-0800
2021/03/05 10:07:59 [INFO] ReferenceTransformer: reference not found: "local.enabled"
2021/03/05 10:07:59 [INFO] ReferenceTransformer: reference not found: "var.apply_config_map_aws_auth"
2021/03/05 10:07:59 [INFO] ReferenceTransformer: reference not found: "var.kubernetes_config_map_ignore_role_changes"
2021/03/05 10:07:59 [INFO] ReferenceTransformer: reference not found: "local.map_worker_roles"
2021/03/05 10:07:59 [INFO] ReferenceTransformer: reference not found: "var.map_additional_iam_roles"
2021/03/05 10:07:59 [INFO] ReferenceTransformer: reference not found: "var.map_additional_iam_users"
2021/03/05 10:07:59 [INFO] ReferenceTransformer: reference not found: "var.map_additional_aws_accounts"
2021/03/05 10:07:59 [DEBUG] ReferenceTransformer: "module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0]" references: []
module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0]: Refreshing state... [id=kube-system/aws-auth]
2021-03-05T10:07:59.548-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:07:59 [INFO] Checking config map aws-auth: timestamp=2021-03-05T10:07:59.548-0800
2021-03-05T10:07:59.548-0800 [INFO]  plugin.terraform-provider-kubernetes_v2.0.2_x5: 2021/03/05 10:07:59 [DEBUG] Kubernetes API Request Details:
---[ REQUEST ]---------------------------------------
GET /api/v1/namespaces/kube-system/configmaps/aws-auth HTTP/1.1
Host: [REDACTED]
User-Agent: HashiCorp/1.0 Terraform/0.14.7
Accept: application/json, */*
Authorization: Bearer [REDACTED]
Accept-Encoding: gzip

Versions:

Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/aws v3.28.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.0.2
+ provider registry.terraform.io/hashicorp/local v2.0.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.1
+ provider registry.terraform.io/hashicorp/template v2.2.0

@johncblandii

I’m not seeing any solutions out there, but lots of similar references https://github.com/cloudposse/terraform-aws-eks-cluster/issues/104#issuecomment-792520725

@Andriy Knysh (Cloud Posse) is still actively working on this, but no silver bullet yet.

yes, this is a race condition

Error: Get "<http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth>": dial tcp [::1]:80: connect: connection refused

between TF itself and the kubernetes provider

many people see similar error, but in different cases (some on destroy, some on updating cluster params)

(if you just provision the cluster, all seems ok, only updating/destroying it causes those race condition errors)

Hey, same issue here, apply a fresh eks cluster all good bu when I want to update or destroy:

╷
│ Error: Get "<http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth>": dial tcp 127.0.0.1:80: connect: connection refused
│ 
│ 

I cleaned the state with terraform state rm module.eks.kubernetes_config_map.aws_auth_ignore_changes[0] and it worked

if you do it by path or port you can do it

but you can’t use the same path and port

you could do same port as 443 and path /app1 /app2 / (default app)

Thank you

I’m going to give it a try

Can you expand on the context / your migration a bit Bart? Might be able to provide some guidance, but I’m not sure what you’re referring to.

hey Matt

like a build iam policies with terraform before

now I used mostly the cloudposse modules, like this one for example:

https://github.com/cloudposse/terraform-aws-iam-role

the configuration does exist already

another example, I have existing s3 buckets

https://github.com/cloudposse/terraform-aws-s3-bucket

so when I deploy the module, you get conflicts because the resource does exist already

so what I want to do is, lay the module configuration over the existing resources

As you in created those resources via ClickOps already?

yes and creating the resources in terraform myself

now I want to base my config on one codebase, mostly build by modules

If you created resources via ClickOps then you only have a few options:

1. Delete the resources and let them be recreated by your Terraform code. This only works if the resources are not critical and are not in a production environment.
2. Import the resources using terraform import
3. Accept that those resources are managed outside of IaC, document them, and then work to never do ClickOps again so you don’t run into that issue again in the future.

yeah, but I herited quite a legacy installed base

You look into terraformer as another option… but I haven’t used it myself so I’m not sure how if that will work out for you or not.

I’m writing a blog post about how to do this and am looking to review the different methods people have used out there.

wouldn’t that force some sort of code inspection? I’m familiar with tools that’ll scan traffic in order to figure least privilege of an app, and you might be able to have that as part of a complex integration test.. or are you\ talking about something else?

Load balancer takes time to become available after it gets created, so TF calls the API which can’t find the LB in the “ready” state. At the time when we created the module, we could not find a workaround, except 1) using two-stage apply with -target;2) separate the LB into diff folder and apply it first, then all the rest.

Can you add a depends_on argument directly to the alb service task module that depends on the alb? I believe that’s available on tf 14 and should work.

that prob would work. Can you test and open a PR? We’ll review promptly, thanks

The EFS resources:

resource "aws_efs_file_system" "ecs01" {
  tags = {
    Name = "ecs-efs-01"
  }
}

resource "aws_efs_mount_target" "main" {
  file_system_id  = aws_efs_file_system.ecs01.id
  subnet_id       = aws_subnet.private2.id
  security_groups = [aws_security_group.ecs_efs.id]
}

Networking:

resource "aws_security_group" "ecs_efs" {
  name   = "ecs-efs"
  vpc_id = aws_vpc.ecs-service-vpc2.id

  egress {
    from_port   = 2049
    to_port     = 2049
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

resource "aws_security_group_rule" "ecs_efs_to_cluster" {
  from_port                = 2049
  to_port                  = 2049
  protocol                 = "tcp"
  security_group_id        = aws_security_group.ecs_efs.id
  source_security_group_id = aws_security_group.ecs_cluster.id
  type                     = "egress"
}

resource "aws_security_group" "ecs_cluster" {
  name   = "ecs-cluster"
  vpc_id = aws_vpc.ecs-service-vpc2.id

  egress {
    cidr_blocks = ["0.0.0.0/0"]
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
  }

  ingress {
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_alb.id]
  }

  egress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_efs.id]
  }

  ingress {
    from_port       = 2049
    to_port         = 2049
    protocol        = "tcp"
    security_groups = [aws_security_group.ecs_efs.id]
  }

  lifecycle {
    create_before_destroy = true
  }
}

ECS task definition:

resource "aws_ecs_task_definition" "service_task" {
  family                = "service"
  execution_role_arn    = aws_iam_role.ecs_task_execution_role.arn
  container_definitions = <<TASK_DEFINITION
    [
        {
            "essential": true,
            "name": "service",
            "image": "${module.service_ecr.repository_url}:latest",
            "command": [
                "sh",
                "-c",
                "service serve -api-key $service_API_KEY -tls=false -server-url $service_FQDN -http-addr :80"
            ],
            "environment": [
                  {
                    "name": "SERVICE_FQDN",
                    "value": "https://${aws_route53_record.ecs01.fqdn}"
                  },
                  {
                    "name": "SERVICE_API_KEY",
                    "value": "foo"
                  }
                ],
            "logConfiguration": {
              "logDriver": "awslogs",
              "options": {
                "awslogs-group": "${aws_cloudwatch_log_group.ecs_service.name}",
                "awslogs-region": "${data.aws_region.current.name}",
                "awslogs-stream-prefix": "ecs"
              }
            },
            "mountPoints": [
              {
              "containerPath": "/var/db/service",
              "sourceVolume": "service-db"
              }
            ],
            "portMappings": [
                {
                    "containerPort": 80,
                    "protocol": "tcp",
                    "hostPort": 80
                },
                {
                    "containerPort": 2049,
                    "protocol": "tcp",
                    "hostPort": 2049
                }
            ]
        }
    ]
TASK_DEFINITION

  volume {
    name = "service-db"

    efs_volume_configuration {
      file_system_id = aws_efs_file_system.ecs01.id
      # root_directory = "/opt/db/service/01/"
      root_directory = "/"
      # transit_encryption      = "ENABLED"
      # transit_encryption_port = 2999
      # authorization_config {
      #   access_point_id = aws_efs_access_point.test.id
      #   iam             = "ENABLED"
      # }
    }
  }
  cpu                      = 256
  memory                   = 512
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
}

Have you set the platform version for the ecs service to 1.4.0?

# Set platform version to 1.4.0 otherwise mounting EFS volumes won't work with Fargate. # Defaults to LATEST which is set to 1.3.0 which doesn't allow efs volumes. platform_version = "1.4.0"

yeah I have this in my service hah platform_version = "1.4.0" # 1.3.0 doesn't support EFS mounts

I was missing an ingress rule in resource "aws_security_group" "ecs_efs" 

:yes

I’m in somewhat the same boat. I’ve never heard of TF trying to delete the snapshot (I mean, terraform doesnt even control the snapshot as a resource). Do you mean delete the option group as you stated earlier? because that HAS happened to me too. my brainstorming would be that we’d have to pre-create a second paramter group and option group for the NEW-INCOMING version. Then the upgrade path would be to change engine_version, option_group_name and parameter_group_name all at the same time from the 5.6’s to the 5.7 equivalents:

  engine_version       = "5.7"
  option_group_name    = module.db_option_group.my_5.7_optgrp.name
  parameter_group_name = module.db_parameter_group.my_5.7_paramgrp.id

???

Yeh the option group for 5.6 couldn’t be deleted as it is used by previous snapshots (which is fine)

I have no idea why it’s trying to be deleted in the first place it’s too tighter coupling really

I agree - terraform is really ugly when handling RDS in my opinion. I thought i was going crazy until i happened on a youtube talk saying the same things. In theory if you switch the RDS instance to a new option group, then it won’t try to delete it, is my thoughts..

I am doing this with the upstream module I am wondering if the issue is I’m using the option group module and passing it to the rds module rather than directly to the instance module inside the Ed’s module

@mikesew i managed to hack this by deleting the option group and parameter group from terraform remote state so that it does not get deleted, but it works well

that’s great but a non-optimal solution.. the thing is, I haven’t seen this side of terraform management (stateful managed databases like RDS or AzureSQL) mentioned nearly enough in the blogs . I think I”m honestly doing it wrong.

• Do you have 1 option group per database, or 1 common option group for a group of DB’s? (we had been doing a common option group but are seeing how inflexible it has become with the snapshots being tied to it)

• how about paramter groups? 1 per db, or a common one?

I’m creating a parameter, option and subnet group per database as all our databases are different.

let’s chat in this thread

can you share your full terraform ?

cat *.tf

and dump it here ?

$ cat context.tf module “this” { source = “cloudposse/label/null” version = “0.24.1” # requires Terraform >= 0.13.0

enabled = var.enabled namespace = var.namespace environment = var.environment stage = var.stage name = var.name delimiter = var.delimiter attributes = var.attributes tags = var.tags additional_tag_map = var.additional_tag_map label_order = var.label_order regex_replace_chars = var.regex_replace_chars id_length_limit = var.id_length_limit label_key_case = var.label_key_case label_value_case = var.label_value_case

context = var.context }

variable “context” { type = any default = { enabled = true namespace = null environment = null stage = null name = null delimiter = null attributes = [] tags = {} additional_tag_map = {} regex_replace_chars = null label_order = [] id_length_limit = null label_key_case = null label_value_case = null } description = <<-EOT Single object for setting entire context at once. See description of individual variables for details. Leave string and numeric variables as null to use default value. Individual variable settings (non-null) override settings in context object, except for attributes, tags, and additional_tag_map, which are merged. EOT

validation { condition = lookup(var.context, “label_key_case”, null) == null ? true : contains([“lower”, “title”, “upper”], var.context[“label_key_case”]) error_message = “Allowed values: lower, title, upper.” }

validation { condition = lookup(var.context, “label_value_case”, null) == null ? true : contains([“lower”, “title”, “upper”, “none”], var.context[“label_value_case”]) error_message = “Allowed values: lower, title, upper, none.” } }

variable “enabled” { type = bool default = true description = “Set to false to prevent the module from creating any resources” }

variable “namespace” { type = string default = null description = “Namespace, which could be your organization name or abbreviation, e.g. ‘eg’ or ‘cp’” }

variable “environment” { type = string default = null description = “Environment, e.g. ‘uw2’, ‘us-west-2’, OR ‘prod’, ‘staging’, ‘dev’, ‘UAT’” }

variable “stage” { type = string default = null description = “Stage, e.g. ‘prod’, ‘staging’, ‘dev’, OR ‘source’, ‘build’, ‘test’, ‘deploy’, ‘release’” }

variable “name” { type = string default = “redis-blue-green” description = “Name for the cache subnet group. Elasticache converts this name to lowercase.” }

variable “delimiter” { type = string default = null description = «-EOT Delimiter to be used between namespace, environment, stage, name and attributes. Defaults to - (hyphen). Set to "" to use no delimiter at all. EOT }

variable “attributes” { type = list(string) default = [] description = “Additional attributes (e.g. 1)” }

variable “tags” { type = map(string) default = { Name = “redis-blue-green” }

description = “Additional tags (e.g. map('BusinessUnit','XYZ')” }

variable “additional_tag_map” { type = map(string) default = {} description = “Additional tags for appending to tags_as_list_of_maps. Not added to tags.” }

variable “label_order” { type = list(string) default = null description = «-EOT The naming order of the id output and Name tag. Defaults to [“namespace”, “environment”, “stage”, “name”, “attributes”]. You can omit any of the 5 elements, but at least one must be present. EOT }

variable “regex_replace_chars” { type = string default = null description = «-EOT Regex to replace chars with empty string in namespace, environment, stage and name. If not set, "/[^a-zA-Z0-9-]/" is used to remove all characters other than hyphens, letters and digits. EOT }

variable “id_length_limit” { type = number default = null description = «-EOT Limit id to this many characters (minimum 6). Set to 0 for unlimited length. Set to null for default, which is 0. Does not affect id_full. EOT validation { condition = var.id_length_limit == null ? true : var.id_length_limit >= 6 || var.id_length_limit == 0 error_message = “The id_length_limit must be >= 6 if supplied (not null), or 0 for unlimited length.” } }

variable “label_key_case” { type = string default = null description = «-EOT The letter case of label keys (tag names) (i.e. name, namespace, environment, stage, attributes) to use in tags. Possible values: lower, title, upper. Default value: title. EOT

validation { condition = var.label_key_case == null ? true : contains([“lower”, “title”, “upper”], var.label_key_case) error_message = “Allowed values: lower, title, upper.” } }

variable “label_value_case” { type = string default = null description = «-EOT The letter case of output label values (also used in tags and id). Possible values: lower, title, upper and none (no transformation). Default value: lower. EOT

validation { condition = var.label_value_case == null ? true : contains([“lower”, “title”, “upper”, “none”], var.label_value_case) error_message = “Allowed values: lower, title, upper, none.” } }

Compare to terraform-aws-elasticache-redis/examples/complete/context.tf. Changes I made: 62c84 < default = true —
default = null 86,87c108,109 < default = “redis-blue-green” < description = “Name for the cache subnet group. Elasticache converts this name to lowercase.” —
default = null
description = “Solution name, e.g. ‘app’ or ‘jenkins’” 107,110c129 < default = { < Name = “redis-blue-green” < } < —
default = {}

I did not make much changes in context.tf.

could you use triple backticks to format your code ? it makes it easier to read

also could you provide a minimal viable reproducible example ?

could you provide a minimal example ?

minimal as in the minimum required to reproduce the same error message

oof… youre doing this one at a time, one file per…

if you can create a minimal reproducible example, i’ll be able to help

but at this point, this is a lot of code to trudge through

the tests pass, for now, id point you to the current tf code in the example..

see this https://github.com/cloudposse/terraform-aws-elasticache-redis/blob/master/examples/complete/main.tf#L28

@melissa Jenner ^

I copied code at https://github.com/cloudposse/terraform-aws-elasticache-redis/blob/master/examples/complete/main.tf#L28 and added a few more lines.

@melissa Jenner - This is a bug due to the overall configuration of what’s being passed into the CloudPosse module. If you’d like to do some pair programming I can spend some time helping you out since I just went through and configured this module myself a couple of weeks ago.

if you 2 could figure out what the issue is, perhaps a PR can be submitted to make the module easier to use

@MattyB As of now, do you have ideas of how to fix it?

@MattyB You said, “I just went through and configured this module myself a couple of weeks ago.”. Did you actually fix it?

@melissa Jenner Not without seeing more of the variables that are being passed in. There are a few gotchas depending on how you set it up. Clustered mode, and other settings.

@MattyB I posted context.tf and variables.tf. Do you see anything?

good luck! let us know when you pr

Without additional context (missing variables being passed into here) I’m unable to help you.

Can you share your code?

I don’t think multi AZ is valid for Redis. It only applies to Memcached, right?

Redis instead has the concept of “cluster mode”

Unfortunately not. I can tell you we’re running in clustered mode and that to use the module in any fashion you’ll need to thoroughly understand the CloudPosse implementation and to do some reading here https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticache_replication_group. for example:

number_cache_clusters - (Optional) The number of cache clusters (primary and replicas) this replication group will have. If Multi-AZ is enabled, the value of this parameter must be at least 2. Updates will occur before other modifications. One of number_cache_clusters or cluster_mode is required.

It is valid. I can manually configure it in AWS console.

@Hank One important thing to note: The terraform-aws-modules organization is not built by AWS. It’s built by Anton Babenko who is an AWS Hero I believe, but that does not mean those modules are actively maintained by AWS.

The large majority of Cloud Posse modules are kept up to date with best practices and new features. Particularly a module surrounding EKS since Cloud Posse and the community here use them very heavily.

Thank u very much

https://github.com/cloudposse/terraform-aws-eks-node-group

We also have managed Spot.io node groups

@Erik Osterman (Cloud Posse), im interested on how this works on spot.io

Awe shoot, someone forgot release notes.

nah, their release automation always does this… creates release, then follows up a bit later to post the description and any artifacts. the rss integration isn’t smart enough to handle that

oh nice, i assumed since it said release notes that it was going to be the release notes haha

I’m not exactly sure this is natively possible with TFC as of now. It wasn’t possible before, and I don’t see any documents updating the support for it.

I know we support this directly using a variable in env0 that passes the target flag to the apply:

https://docs.env0.com/docs/additional-controls#terraform-partial-apply

Disclaimer: I’m the DevOps Advocate for env0.

Hey, sorry. We did some digging, and you can do this with TFC now. Using CLI runs, you can use resource targeting. It is still not available in the UI.

Thanks for your response. I just got introduced to a customer who have ruined their TF State with weird cyclic dependencies and deleted the resources manually. TF target and state remove are the only saviors

Oof! Totally understandable. Hopefully you can get it all cleaned up.

I managed to fix the bad terraform state issue-what I could have fixed in an hour with Terraform CLI took me 8 hours with Terraform Cloud

there is no such thing as revert in TF

but if you saved the previous plan you might be able to revert

if you screw up the state but you have not apply you could maybe restore to and old version fo the state

do you know any tools to export existing AWS resources to terraform style?

sure

https://github.com/GoogleCloudPlatform/terraformer

the best there is

yes, that one is good as i know so far but for example: how to generate a tfstate file from the AWS existing resources with it?

there is plenty of examples in the doc

you need o do it per product manually

there is no such thing as scan all a give me a state and tf files

although you can use a for loop and go trough all the products supported by terraformer

remove the interpolation syntax:

alarm_actions = [data.aws_sns_topic.cloudposse-hosting.arn]

ha ok, that worked

thx

Ok looks like I won’t be using cloudposse

in the context.tf file itself there is comments

you might find this helpful https://www.youtube.com/watch?v=V2b5F6jt6tQ

Thank you very much (both). I’ll watch the video now. I just need a demonstration of it’s use.

it takes a bit to get around it and understand it well but when it clicks your are like “I should have started using this last week”

Got it sorted, looking forward to cleaning up my code with this, thanks guys!

I would split Iac codebase pipeline and App codebase pipeline, deploy Terraform using Spacelift. then deploy app code using Github CI by building the image, upload to ECR, the ECR should triggered Codepipeline which use codedeploy for deployments ( like blue/green)

Ya, that would be a conventional approach what I don’t like about it is that is we have multiple systems modifying the ECS task. We wouldn’t be able to deploy ECS task changes (E.g. new SSM parameters) along side a new image deployment.

We’ve managed to recreate an argocd style deployment strategy for infra using spacelift. So really just want to solve the specific problem of running migrations.

Yes, true you need to ignore ecs task definition and lb values, lot of pain

You seen https://engineering.resolvergroup.com/2020/09/running-database-migrations-on-deployment-for-fargate-containers/ for example?

Ideally I prefer not coupling deployments and migrations

They need to be backwards compatible anyways

Or just have the migration task def sitting around and trigger a run-task with the task def for migration?

Not sure exactly which part of the migration is causing issues? Where to run the migration from? Coupling it with CI/CD ?

trigger a run-task would be perfect if it was supported natively in terraform, but requires local exec

We need to run the migration as part of the CD

the CD is calling terraform when files in a git repo change (~like argocd). By design, in this part of the CD, there’s no workflow associated with calling terraform, so there’s no way to run other steps like in a CI pipeline. Terraform’s job is simply to synchronize the git repo with AWS.

Ideally I prefer not coupling deployments and migrations

agree

trigger a run-task
would be perfect if it was supported natively in terraform, but requires local exec
Can you maybe post to an s3 object or ddb item with terraform, and have that event trigger a lambda that invokes the run-task?

one fo the common questions asked here, there are no single best approach for this, but I will list them:

• repo level: each env on standalone repo

• branch level: each env on standalone branch

• folder level: ( I prefer this) each env on standalone folder My selection would be repo level for enterprises clients - branch and folder for startups and medium-small clients Also you need to split you terraform codebase into stacks:

• network stack

• data stack

• app stack this will help you with making your state file small and fast, and small blast radius. So IMHO my current approach for TF cloud project:

• tf-xyz repo

• env/dev/us-east-1 folder contains TF codebase

• create workspace on TF cloud called dev-us-east-1 points to that folder I hope this gives some light

Thank you from another observer. I (a DBA) have been advocating for a separate data-layer, but my current org seems to only split between core and application, grouping the database alongside the app. We also have the env inside folder. I like your use of the region as a sub-dir underneath. do you only have .tfvars inside ./env/dev/us-east-1, or are there actual full-on [main.tf](http://main.tf) etc.?

here all in one approach

.
├── README.md
├── Taskfile.yml
└── env
    └── dev
        └── eu-central-1
            ├── README.md
            ├── data.tf
            ├── dev.auto.tfvars
            ├── doc.tf
            ├── eks.tf
            ├── elasticache.tf
            ├── helm.tf
            ├── kms.tf
            ├── kubernetes.tf
            ├── locals.tf
            ├── mq.tf
            ├── outputs.tf
            ├── provider.tf
            ├── random.tf
            ├── rds.tf
            ├── sg.tf
            ├── ssm.tf
            ├── terraform.tf
            ├── variables.tf
            └── vpc.tf

3 directories, 22 files

here separated stacks

├── Makefile
├── README.md
├── Taskfile.yml
├── app
│   ├── backend.tf
│   ├── cfn
│   │   └── mq.yaml
│   ├── codedeploy-ecs-ingest.tf
│   ├── codedeploy-ecs.tf
│   ├── codedeploy-iam.tf
│   ├── cross-account-access.tf
│   ├── data.tf
│   ├── ecs-alb.tf
│   ├── ecs-cloudwatch.tf
│   ├── ecs-cluster.tf
│   ├── ecs-container-definition-ingest.tf
│   ├── ecs-container-definition.tf
│   ├── ecs-iam.tf
│   ├── ecs-ingest-nlb.tf
│   ├── ecs-route53.tf
│   ├── ecs-service-ingest.tf
│   ├── ecs-service.tf
│   ├── ecs-sg.tf
│   ├── ecs-variables.tf
│   ├── locals.tf
│   ├── mq-nlb.tf
│   ├── mq-route53.tf
│   ├── mq-sg.tf
│   ├── mq.tf
│   ├── outputs.tf
│   ├── provider.tf
│   ├── random.tf
│   ├── secretsmanager.tf
│   ├── terraform.tfvars
│   └── vars.tf
├── data
│   ├── backend.tf
│   ├── data.tf
│   ├── provider.tf
│   ├── random.tf
│   ├── rds.tf
│   ├── reoute53.tf
│   ├── secret.tf
│   ├── sg.tf
│   ├── terraform.tfvars
│   └── vars.tf
└── network
    ├── acm.tf
    ├── backend.tf
    ├── cvpn-cloudwatch.tf
    ├── cvpn-endpoint.tf
    ├── cvpn-sg.tf
    ├── cvpn-tls-users.tf
    ├── cvpn-tls.tf
    ├── outputs.tf
    ├── plan.out
    ├── provider.tf
    ├── route53-resolver.tf
    ├── route53.tf
    ├── terraform.tfvars
    ├── vars.tf
    └── vpc.tf

Thank you for response. Did u mean you reset the backend between local and TF cloud to rotate the config?

that’s a different topic, I have a one time set up for the remote backend states

then i have a main(global) config and a per env one

so I make conditionals because my envs are not very different

like this actually: https://github.com/ozbillwang/terraform-best-practices

Which item in that list? Not quite getting what you mean sorry. But it sounds interesting

backend config section

this module was not converted to the new syntax yet

pull requests are welcome

ok. Thanks

Interesting. Do you have eu-central-1 used anywhere in your terraform code? I can’t find that string used in the module itself

The region in the module is using the region data source so it should use the region from the provider

Hey Rb, thanks for replying. Nope, I don’t think I’m specifying that anywhere in my code. I attached a gist to the bug.

I suppose I could create a new directory and copy bit by bit to be completely sure.

same error in a fresh directory/fresh terraform init

weird

I do see eu-west-1 explicitly defined in .terraform\modules\terraform_state_backend.dynamodb_table_label\examples\autoscalinggroup but that’s an example file and also not eu-central-1

I gotta assume the module properly creates unique bucket names, right? https://github.com/hashicorp/terraform/issues/2774

gah… this solves it I think:

s3_bucket_name = random_string.random.result

the bucket name wasn’t unique somehow

You can use any public PGP key so perhaps you should use a ‘shared’ one in the first place?

Hi Bart, did you find a solution to your problem? I think a ‘shared’ private would lead to the situation that a new key would eventually be required which would also bring re-encryption of secrets to the table. You would have to implement your own rotation method to handle this regularly and without breaking stuff.

I am currently facing the same issue and would like to learn how others are dealing with this challenge.

@hkaya have a listen to the latest office hours. It’s discussed on there.

@Joe Niland thanks, will surely do.

From the network diagram it should be open on port 443, https

@Zach Thanks for the quick respond, I’ve tried all them, 443, 80, 8080 on Route53 DNS, ALB DNS name, EFS hostname, efs_dns_name

still not hitting it.

I’ve not used this module myself but you probably just need to hit all the basic troubleshooting then - go see what the R53 record is pointed at, check any ALBs and TG health, etc

like when you say yoiu can’t reach it… whats happening? Is it timing out? (thats probably a SG issue) 404 (reaches jenkins but something is screwy so its not found)? 503 (hitting the LB but jenkins isn’t responding)?

@Zach route53 zone name just returned no such endpoint I believe.

ALB DNS however showing this, however not that jenkins login page nor a place where I can navigate to it.

@Mohammed Yahya Not sure what you mean with that, but I have that output and its not responding via web request,

@Andriy Knysh (Cloud Posse) @Maxim Mironenko (Cloud Posse), sorry for the ping, but seeing the great work from both of you, if could provide some insight as to what is happening here, I would greatly appreciate this.

just to provide some info, I am using the complete module where I go with all the default info provided in the example tfvars file except different dns_zone_id and github_oauth_token and jenkins usename and pw of course.

it gets deployed to Elastic Beanstalk, so this output should work

output "elastic_beanstalk_environment_hostname" {
  value       = module.elastic_beanstalk_environment.hostname
  description = "DNS hostname"
}

unless it did not get deployed successfully

please login to the AWS console and look at the CodePipeline

if it had any errors, it would show the failed steps in red

@Andriy Knysh (Cloud Posse) Thank you for the quick respond, everything is deployed successfully.

there’s an error like you said, but I have a question, would failed build stop me from accessing jenkins server?

the pipeline could not access the repo or the branch

it either does not exist, or is private

Fixing this now. Thanks Andriy

for test, try to use https://github.com/cloudposse/jenkins

once working, you can switch the repo

Thanks @Andriy Knysh (Cloud Posse). Looks like all good now.

I have another question, after looking through all the root modules, is changing from github repo to Bitbucket repo possible using this module?

Bitbucket is prob not supported by the current version (but should be possible to switch to it)

Thanks for clarifying that. Appreciate it.

re-posting this question

i don’t think so… not literally, anyway, which i imagine would look like this:

resource aws_lambda_function main {
  for_each = local.functions
  
  <attr> = aws_lambda_function.main["<label>"].<attr>
}

only option i can think of is to split it into as many resource blocks as you need to link the dependencies

i’d be interesting in following if you open a feature request…

Yeah. It seems like when you try and create a lot of functions at the same time, AWS will return rate limit failures for most of them as the Lambda control plane limit is 15/sec, and you can only create 1 Lambda in any one VPC at a time. So this results in Terraform creating about one Lambda every 2mins if they are all in the same VPC

oh, that should definitely be opened as a bug

terraform has retry handlers for rate limiting

Well, that’s the thing. The internal retry logic works and it eventually succeeds. It just takes forever

lulz ok

So it’s not really a Terraform bug, and I doubt Hashicorp would do anything to address this. For now we work around it with -parallelism=1 but that slows down the whole configuration

Another use-case is AppSync resources, you can only modify each AppSync API in serial, so if you have for_each on appsync resolvers or data source you get failures (AWS provider has no rate limiting retry logic as it’s a new service and this is always forgotten )

the provider might be able to be smarter about how it schedules them… is there an api to check the rate limit?

for_each is still relatively new also… i feel like this is a good issue to open either way

yeah. That would be ideal. There’s no AWS API to check rate limit. I wonder the AWS provider uses the AWS SDK’s built in retry logic or roll their own. The AWS SDK retry logic is really dumb, it’s plain exponential backoff w/ jitter. So you end up with cases like this where you are retrying every 120 seconds when you could be retrying every 10

That’s fair. I’ll open a bug. I’m not hopeful though.

resource aws_resource foo {
  count = n
  depends_on = [ aws_resource.foo[n-1] ]
}

I wonder if it works with count

heh. yeah, but if there’s some discussion it would be worth it

maybe another approach would be to expose retry logic to every resource as a core feature so you can override it some… retry_method and retriable_errors or somesuch

I hastily wrote up this feature request – per-resource parallelism https://github.com/hashicorp/terraform/issues/28152

fwiw, you can avoid the deadlock in your example by threading the bucket name to the public access block from the bucket policy…

resource aws_s3_bucket default {
  name = mybucket
}
resource aws_s3_bucket_policy default {
  name = aws_s3_bucket.default.name
  policy = ...
}
resource aws_s3_public_access_block default {
  name = aws_s3_bucket_policy.default.name
  ...
}

Yeah. I know

Looks almost correct — You need to specify var.expiry_date in the condition for the regex to be run on that input variable.

slack needs a line-by-line review and suggestion feature

@Florian SILVA thanks, your contributions are very welcome

Thank you @Andriy Knysh (Cloud Posse). The current issue that I opened is the following: https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/issues/174 If I could have a feedback when someone have time. I’m not sure yet on how to resolve it the best. :)

Nice!

This is something have wanted to do more of, however, have trouble reconciling it with gitops patterns we follow

The end result needs to be it’s opening a PR against the repo with the changes

(it’s that last part haven’t looked into)

In case anyone has this question. The answer is no, it will not delete the certificate.

You can look at the way that Cloud Posse does module version tagging + testing for any of their modules as a good way to accomplish this.

tl;dr for that topic is:

1. We use the release-drafter GH action to automate tagging / releases: https://github.com/cloudposse/terraform-datadog-monitor/blob/master/.github/workflows/auto-release.yml
2. And we use terratest to accomplish module tests: https://github.com/cloudposse/terraform-datadog-monitor/blob/master/test/src/examples_complete_test.go

The test-harness setup can be easily reused to make that repeatable across many module repos: https://github.com/cloudposse/test-harness

oh this test harness looks interesting. How do you use it though?

There is no good docs to follow, but if you look at any cloud posse module then you can reverse engineer the setup and go from there.

what is the version of TFE?

how do I find the version? We do have several versions of terraform available

TFE v201910-1

should be similar to this ^^^

thats required for a module?

oh it should be in the root folder

I’m confused - this is a standalone terraform module

1. how come I have to add the tfe provider

and it is in the ‘root’ folder aleady

tfe not seen in the code you posted

or I missed something

yeah i’m trying to publish a module to terraform enterprise

what is in [version.tf](http://version.tf)?

oh ic, tfe is not needed

right..

sorry, we should look at version.tf

can I run the codes on my local laptop?

did you enable DEBUG?

https://www.terraform.io/docs/cloud/api/admin/terraform-versions.html

is this something I can accomplish with the terraform state command?

terraform import

oh nice, thanks!

Terraform data structures are immutable so you need to create a new local using the previous object. For example:

new_data = merge(var.old_data, {
  one = {
    two = {
      biz = "baz"
    }
  } 
})

I didn’t test the above so I might be off in some syntax, but you get the idea.

ok, yeah, that is what I am trying. Thanks.

Also if you need true deep merging you cannot do it in terraform core with merge

For that use https://github.com/cloudposse/terraform-provider-utils

lack of data sources to lookup values from existing resources

no locals or other method for easily generating/reusing intermediate values

very limited functions and expressions available in the DSL

it’s yaml, but not really, with syntax that isn’t really yaml, and syntax that IS yaml but isn’t supported (anchors)

parameter validation is downright annoying when using the AWS types and trying to make a variable optional (try using the type AWS::EC2::KeyPair::KeyName and making the keypair an optional parameter…)

terraform isn’t just for aws! IaC for github, or ldap, or dns, or or or or…

lack of data sources to lookup values from existing resources this one blows my mind every time
terraform isn’t just for aws! and this is pretty key. terraform brings together or covers most/all your sources not just the AWS pieces.

All good ones!

• exported outputs of stacks are not allowed to change

• lacks a central place where templates can be shared

No escape hatches to import/export/manually edit resources / statefiles

oh that’s a good one @Alex Jurkiewicz! state manipulation commands are huge. so many times i’ve wished i could just move stuff around in the cfn stack to avoid resource cycles or get myself out of a broken stack launch/update

We have some redshift clusters managed as nested stacks in cloudformation, and they are wedged in a completely inoperable state. It’s impossible to make ANY change to the resources within. We’ve had to fall back to importing them into a new Terraform configuration and setting a “deny all” stack update policy in cloudformation. But those stacks will live forever as some busted thing

These are great

I feel like we should to make a cheap website called tf-vs-cf.com that just lists these things for easily sending to clients, managers, new engineers, etc.

@Matt Gowie let’s do it! i’ve created an org and a repo to host the static site and picked up the domain tfvscf.com (no dashes ).

we can do a static site hosted on github pages.

for now maybe add these topics as issues? and then figure out the site design as we go.

all contributors are welcome!

https://github.com/tfvscf/tfvscf.com

Hahah you went and did it @managedkaos — cool. I’ll try to contribute over the weekend

let me know your github ID and i’ll send an invite

Would be happy to host on Amplify unless someone has a better option. I know GH pages is a good option… but I have a whole simple Amplify setup that I like and use for customer sites. That is how I cheaply host mattgowie.com + masterpoint.io:

module "masterpoint_site" {
  source = "git::<https://github.com/masterpointio/terraform-aws-amplify-app.git?ref=master>"

  namespace                    = var.namespace
  stage                        = var.stage
  name                         = "masterpointio"
  organization                 = "masterpointio"
  repo                         = "masterpoint.io"
  gh_access_token              = local.secrets.gh_access_token
  domain_name                  = "masterpoint.io"
  description                  = "The simple HTML website for Masterpoint Consulting (@masterpointio)."
  build_spec_content           = data.local_file.masterpoint_build_spec.content
  enable_basic_auth_on_master  = false
  enable_basic_auth_on_develop = true
  basic_auth_username          = "masterpoint"
  basic_auth_password          = local.secrets.basic_auth_password
  develop_pull_request_preview = true

  custom_rules = [{
    source    = "<https://www.masterpoint.io>"
    target    = "<https://masterpoint.io>"
    status    = "301"
    condition = null
  }]
}

@managedkaos GH is @Gowiem

yeah that would be cool! i picked this up as a project to do some static site work (which i have been meaning to ramp up on). I’ve also been wanting to try amplify so yep, I’m open!

Matt that’s awesome. Is it free? Netlify is pretty much my go to for static site hosting and has cicd and is free. I thought amplify with aws resources would cost?

Thank you guys for this, I was looking into make it based on @Matt Gowie suggestion.

@sheldonh it’s basically free. With Amplify, you pay per build I’m pretty sure, but the hosting is free? Don’t quote me. I do know that I pay somewhere between nothing to less than 50 cents a month for my two static sites that I manage on Amplify.

Got you. So it’s free because within free tier on AWS then, but not free as it free service level, right? Asking as netlify is pretty much the gold standard for static website ease of usage when I looked in the past, but am open to trying something new for future projects if it makes sense, esp integrated with AWS.

I think its free/next-to-free because its cheap.
Static Web Hosting - Pay as you go

Build & Deploy
$0.01 per build minute

HOSTING
$0.023 per GB stored per month
$0.15 per GB served

https://aws.amazon.com/amplify/pricing/

I don’t even know if it’s within free tier… I just think they charge you for each build minute and for the large majority of FE sites build minutes are extremely low.

they have some good examples on the pricing page

This example:
A startup team with 5 developers have an app that has 300 daily active users. The team commits code 2 times per day. comes out to $8/mo.

so a site that is waaaaay less than even that will be crazy cheap

i think the benefit to Netlify is you are not tied to AWS. i used it a while back and it was easy to onboard from GitHub. I can see the AWS onboarding being overwhelming if you’re not already using it.

For an everyday dev that just wants to deploy a static site, yeah use Netlify. Easy decision.

If you are already on AWS or at least know how to set up an account and maybe already have some other workloads there, perhaps maybe consider Amplify?

Yeah, I fully agree with that decision tree.

Update your version of terraform

i did it with tfenv. But, still not work at all

FYI this just looks like a KMS issue that can be easily replicated in the UI. KMS key policies are saved in a random order no matter how they are applied/saved

nvm, done some digging and added a comment to that ticket.

Also opened an AWS support request, will see what they say.

note it’s public preview and currently limited to aws_subnet and aws_vpc…

provider: New default_tags argument as a public preview for applying tags across all resources under a provider. Support for the functionality must be added to individual resources in the codebase and is only implemented for the aws_subnet and aws_vpc resources at this time. Until a general availability announcement, no compatibility promises are made with these provider arguments and their functionality.

Wow. That’s huge. I love it.

Poor @Imanuel Mizrachi just implemented a Cloudrail rule to alert if no tags were added to a resource, now he’ll need to see if default_tags is set and know which resources are impacted

Just missing “advanced_tags” now…

What are the rules in the default SG?

I do not know. Actually, I do not need default SG.