Depends on your company and how you use domains and Route53 :slightly_smiling_face:

You could have [example.com](http://example.com) in the main account be an alias for [prod.example.com](http://prod.example.com) which is in prod account if you really want to keep that.

You could have a static-only Amplify Console for [example.com](http://example.com) in the main account, and then have your actual app at [app.example.com](http://app.example.com) or [live.example.com](http://live.example.com) in the prod account.

If you’re doing SaaS is gets even wilder Global Accelerator, WAFs, multi-region with multi-domain and so on gets even weirder

Loop in Marketing/Sales cause they’ll definitely have opinions on this and they might make the decision for you

We’re doing something very similar; multiple AWS accounts based on environment (prod, staging, dev, etc) using AWS Organizations from the prod account to manage the others.

We have Terraform creating the top level Route 53 public hosted zone (eg. [company.com](http://company.com)) and we manually add the NS records to our registrar. We experimented with a Terraform provider to manage the records with the registrar, but it created too much risk with the limited API interface that the registrar provided.

Within our non-prod AWS accounts, we have Terraform creating public hosted zones like [staging.company.com](http://staging.company.com), [dev.company.com](http://dev.company.com), etc. and we use the outputs of those zones to create NS records in the prod public hosted zone for those subdomains.

As @Vlad Ionescu (he/him) mentioned, things start to get complicated when you start getting into multi-region, but we’re handing this with private hosted zones for each VPC. As an example, within us-east-1, hosts within the VPC and users connected via VPN are able to resolve things like [hostname.use1.dev.company.net](http://hostname.use1.dev.company.net).

@Drew Davies I see, very similar, I added the top level domain to master account not to prod account. so for master account >> example.com and for prod account prod.example.com what I need to do is create cname from example.com to prod.example.com and then from prod.example.com to ALB/EC2/Cloudfront ….

I think CNAME on Apex is not possible

If you need the apex redirect in AWS, you could utilize s3 (+cloudfront) to redirct http(s):<//example.com> to www.example.com or prod.example.com

I want to send to ALB,

I end up moving top-level domain to prod account. and use dev.X in dev account and delegate dev.x in prod top level domain

That is also an option

If for whatever reason this would not be possible, you could still use the apex redirect trick via s3 - then it redirects to the hosted zone in the sub-account where you can then alias the alb

good to know, thanks. I will test it later

it creates the resources via terraform without an issue but when trying to run the AssertS3BucketPolicyExists function it cant create a session correctly

i’ve had to export AWS_REGION for terratest to work. i don’t know why, i just do it

my env:

AWS_PROFILE=<profile>
AWS_DEFAULT_REGION=us-east-1
AWS_SDK_LOAD_CONFIG=1
AWS_REGION=us-east-1

Do you have a default profile @loren ?

I’m looking at https://github.com/gruntwork-io/terragrunt/issues/671

Adding the additional env vars helped but now I am getting this …

TestAssertBucketPolicyExists 2021-01-04T1914Z logger.go TestAssertBucketPolicyExists 2021-01-04T1914Z logger.go Destroy complete! Resources: 3 destroyed. — PASS: TestAssertBucketPolicyExists (22.53s) FAIL FAIL tf-modules/modules/s3-logs 49.090s FAIL make: *** [terratest-local] Error 1

that doesn’t include the error, so….

i don’t use a default profile, no

Yeh for some reason the error just gets swallowed

note this is for a different test

somehow you have more than one thing trying to operate on the bucket at a time. it’s a race condition in your tf code

for example, if you are using the bucket_policy resource and the public_access_block resource and the bucket_notification resource, only one can hit the api action at a time

you have to thread the bucket name/id/arn from resource to resource so it happens in order

they are completely different buckets though

the are two completely different tests

terratest runs tests in parallel also, by default. the error seems pretty clear. it’s definitely a race condition on the bucket operations. just gotta trace through how you’re doing things to figure out what is causing the conflicting operations

the two tests create two seperate buckets, it tries to get bucket X after terraform has successfully destroyed bucket Y

keep going

now i am getting this …

google “golang use of vendored package not allowed”

that was my made was a mismatch

i am wondering if its a race condition in the terraform the requires a depends_on

because if i comment out the other test in the module it works perfectly fine

@loren adding that depends_on fixed it

depends_on itself?

appears to be what i was describing here: https://sweetops.slack.com/archives/CB6GHNLG0/p1609791301011500?thread_ts=1609782809.000200&cid=CB6GHNLG0

yeh adding that fixed concurrent test execution for the module

you’ve maybe found an odd way of fixing it. or maybe just the race condition not manifesting for a run or two

the new version?

Yeah, I think this is my issue though unfortunately for me. Looks like the state was saved locally instead of in S3 and now no longer exists

did you have the module pinned at a version

Yeah, v0.11.0

hmm let me take a look, i tested this last night and was fine

what aws version are you using, im guessing it must be 3.x

also just for reference what terraform version are you on

I’m 99% sure the problem is the deleted .tfstate file.

gotcha, is there a backup in your directory

In the .terraform directory?

no your apply directory

Ahh, no, doesn’t appear to be. I’m looking into importing the state from existing resources.

gotcha yea you won’t be able to import aws_acm_certificate_validation as its not an actual aws resource not sure if that will cause a problem or not

I know there are ways to import modules, terraform import module.aws_acm_certificate_validation["blahblah"] <resource-id>

Haven’t gotten to that step though, honestly, the Route53 Zones are the only thing I’m worried about losing, as then I’ll have to reset the DNS NS Records

Totally different issue, but having problems importing the zone when the zone has a period in the name. >_>

oh

crucial issue ugh, forgot to push a local commit

ok yea this fixes that oops

https://github.com/cloudposse/terraform-aws-acm-request-certificate/pull/38

@pjaudiomv How would I use the pull request in my local modules? And what’s the general merge time like for these?

I recovered my state, but have the same issue as initially described (which would be fixed by your PR)

once Patric is done I will be merging right away

Thanks PePe

im almost done, I was trying to do too many. things at once. I stepped away from day job and am finishing testing now

ok @David Napier i believe you should be able to use 0.12.0 now of module

Yesss, it worked, thanks!

You made my day.

cool, no prob. sorry about that

thank you PePe for the patience and quick reviews

no problem

thanks for your contributions

More details….

Terraform initially complained that I need to supply providers argument along with the module

Error: Module does not support for_each

  on acm_us-east-1.tf line 14, in module "acm_us-east-1":
  14:   for_each = { for config_key, config in var.site_configs : config_key => config if config.ssl_config.aws.create_ssl_cert == true }

Module "acm_us-east-1" cannot be used with for_each because it contains a
nested provider configuration for "cloudflare", at acm\versions.tf:13,10-22.

This module can be made compatible with for_each by changing it to receive all
of its provider configurations from the calling module, by using the
"providers" argument in the calling module block.

So I updated the code to include

providers = {
    aws = aws.us-east-1
    cloudflare = cloudflare
  }

but now I’m stuck in an endless loop of terraform complaining I need to re initialise.

Error: Could not load plugin



Plugin reinitialization required. Please run "terraform init".

Plugins are external binaries that Terraform uses to access and manipulate
resources. The configuration provided requires plugins which can't be located,
don't satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your
configuration, including providers used in child modules. To see the
requirements and constraints, run "terraform providers".

Failed to instantiate provider "registry.terraform.io/cloudflare/cloudflare"
to obtain schema: Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

Can anybody make a suggestion?

oh and if it helps my providers look like this currently:

.
├── provider[registry.terraform.io/cloudflare/cloudflare] ~> 2.0
├── provider[registry.terraform.io/hashicorp/aws]
├── provider[registry.terraform.io/hashicorp/random]
├── module.acm_us-east-1
│   ├── provider[registry.terraform.io/hashicorp/aws] >= 2.53.*
│   ├── provider[registry.terraform.io/hashicorp/tls]
│   └── provider[registry.terraform.io/cloudflare/cloudflare]
├── module.ses_user_enable_send
│   ├── provider[registry.terraform.io/hashicorp/aws]
│   └── provider[registry.terraform.io/hashicorp/null]
└── module.acm_eu-west-1
    ├── provider[registry.terraform.io/hashicorp/aws] >= 2.53.*
    ├── provider[registry.terraform.io/cloudflare/cloudflare]
    └── provider[registry.terraform.io/hashicorp/tls]

Providers required by state:

    provider[registry.terraform.io/hashicorp/aws]

    provider[registry.terraform.io/hashicorp/null]

    provider[registry.terraform.io/hashicorp/random]

did you try to delete the cache in the folder .terraform.d

Hello Andriy, yes; many times

also, since you are using just one provider of each type, you can use Implicit Provider Inheritance

https://www.terraform.io/docs/modules/providers.html#implicit-provider-inheritance

w/o providing the providers to the module

in short, I assume you can’t have a AWS resource and a Cloudflare resource in the same .tf file?

you can have any resources in a file

from many providers

I only specified them to the module as it fails to initialise unless I do so but I’ll rip it out again and retry. I might have missed something. Thanks for the advice

in your case, I guess TF just stuck at something when you changed the provider config

did you try to destroy using the old code, and plan again?

I haven’t as I didn’t really want to destroy anything in my test environment, If everything works as it should it shouldn’t see any changes in state. Other than the new AWS based certificate I’m trying to create with cloudflare validated dns records.

try to use https://www.terraform.io/docs/modules/providers.html#implicit-provider-inheritance w/o destroying the SSL cert

I’ll try starting again with a bigger hammer. As you say, It might be confused. Thanks for the advice.

specify dthe AWS and CloudFlare providers in the top-level module

don’t send any provider to the low-level module

Okay. Would I still specify required providers within the lower module?

e.g. 
terraform {
  required_version = ">= 0.13.5"

  required_providers {
    aws = ">= 2.53"
	
	  cloudflare = {
      source = "cloudflare/cloudflare"
    }
  }
} 

or completely ditch those?

You can specify those

apologies, on second thoughts won’t I still need to specify the provider in the top level calling module?

module "acm_us-east-1" {
  source = "./acm"
  providers = {
    aws = aws.us-east-1
    cloudflare = cloudflare
  }

*** rest snipped ***

Otherwise, I wont be able to target the us-east-1 region to create the certificate? as I normally run out of eu-west-1

Those are version constraints

if you run from a diff region, you need to specify the correct aws provider to the module

(i’m not sure if you have to specify cloudflare in this case, or Implicit inheritance will work for it. Try not to specify it)

only provider blocks are now at the top level

If this is something that should be possible then I’ll start again, with a basic config block and work backwards from there.

did you delete the cached files in ~/.terraform.d ?

i don’t have a .terraform.d folder only a .terraform folder which I have deleted. Out of total paranoia I also blasted the code away, rebooted and then checked it out again but no difference.

I’ll try a single high level simplified version

resource "aws_acm_certificate" "cert" {
  domain_name       = var.domain
  validation_method = "DNS"
  lifecycle {
    create_before_destroy = true
  }
}

data "cloudflare_zones" "find_zone_ids" {
  for_each = {for ssl_key, ssl_value in var.ssl_root_domain_names : ssl_key => trimsuffix(replace(ssl_value, "*.", ""), ".") if var.ssl_validation_method == "DNS_Cloudflare" && var.ssl_perform_validation == true}
    filter {
        name = each.value
    }
}

resource "cloudflare_record" "cloudflare_cert_validation_records" {
  for_each = {
    for dvo in aws_acm_certificate.this.domain_validation_options : dvo.domain_name => {
      name    = dvo.resource_record_name
      record  = dvo.resource_record_value
      type    = dvo.resource_record_type
      zone_id = data.cloudflare_zones.find_zone_ids[dvo.domain_name].zone_id
    } if var.ssl_validation_method == "DNS_Cloudflare" && var.ssl_perform_validation == true
  }

  allow_overwrite = var.ssl_allow_overwrite_dns
  
  zone_id  = each.value.zone_id
  name     = each.value.name
  value    = [each.value.record]
  type     = each.value.type
  proxied  = true
}

I assume something as simple as that should work and just keep the providers at the parent level without the need to send them on to a module.

I’ll need to clean the “var.” names up as I’ve only just quickly pulled that together. Main point is that the mixed up resources types should be fine. Based on your initial statement?

check for .terraform.d folder in your $HOME directory (not in the project)

Okay, looks like I owe you a BIG thanks and a even bigger SORRY for not clarifying about the .terraform.d directory earlier!

I stupidly thought, it was a difference between Window or Unix terraform or terraform grunt or whatever other people use. I unfortunately didn’t know it was a separate directory altogether. I’ve mistakenly thought everything was under the .terraform directory.

Feel a right Fool now.

While the code still isn’t working its no longer complaining about needing to re init.

Thank you and sorry for wasting more of your time than was needed.

no problem at all

(that folder always the culprit if you have issues with provider initialization)

Internal Wiki updated and lesson truly learnt. So thanks! I’d be lost without all the support you and others have given me here. I still suggest Cloud Posse publish a charity payme type of thing, As I’d happily donate each time somebody helped me.

if you can compile the provider and place the binary in the ~/.terraform.d folder, you can test it

or, you can publish it in your own registry (never did it before) and then load it from there https://www.terraform.io/docs/internals/provider-registry-protocol.html#provider-addresses

can you paste local.files

it’s a map with servers option. If linux_init.sh is updated after resource creation, then it is not uploaded again

{
server1 = {
"StartupScript"            = "linux_init.sh"
...
}
server2 = {
"StartupScript"            = "linux_init.sh"
...
}
}

this worked with me on Terraform 0.14.3

Oh I see, you mean updating the file it self

yes

hmm, try to use template with path

https://registry.terraform.io/providers/hashicorp/template/latest/docs/data-sources/file

Thanks ! I ‘ll check tomorrow but that’s look to do the trick by googling on template I’ve found someone who did it with your point

https://github.com/hashicorp/terraform/issues/4274

what are this certificates the same? are this certificates use in like a alb? one on each region?

The certificates are the same but one is used on a ALB in eu-west-1 and the other is used on Cloudfront and needs to be created in eu-east-1 because they are the same, the verification records are identical.

so you need 2 acm certificates one in the eu-west-1 region and another in us-east-1 (this one is for CloudFront since it requieres you to have it on this region) then aws acm will generate diferent Name and Value so you can add those dns records to your cloudflare

I have done this before and I have not get the same acm certificate, another things is that if you will have one certificate for your alb and another for your cloudfront they should have different subdomains on leas you will route your alb throw cloudfront in this case you will only need 1 certificate witch is the cloudfront certificate

cheers, Miguel. I’ll have a look at my design again.

yo can ignore maybe do this with ignore_changes: https://www.terraform.io/docs/configuration/meta-arguments/lifecycle.html

thank you

@Alex Jurkiewicz that’s a very good proposal, thank you

we can add those diff length IDs (w/o changing the current functionality)

Yup, I like it!

great! I’ll code it up… soon

https://github.com/cloudposse/terraform-null-label/pull/118

AFAIU, you’re thinking of Packer wrong. Packer is for building AMIs. When you’re dealing with Docker, images, and ECR then you’re dealing with a different toolset then what packer is used for.

You use-case looks like you could look into Hashi’s new OOS project Waypoint though — https://www.waypointproject.io/

@Matt Gowie Packer can definitely build docker images https://www.packer.io/docs/builders/docker it can also push them to ecr as well https://www.packer.io/docs/post-processors/docker-push the missing piece for me is how to make terraform aware that it did this and what the ecr url is

using packer to build docker seems odd since there’s already a ‘native’ build tool for that

at any rate, packer doesn’t really provide any outputs except a manifest.json if you enable it (and I’m not actually sure if that’s universal to every builder) and then you an add k/v pairs to it with additional data.

you can say ami also have a native tool too I guess https://docs.aws.amazon.com/cli/latest/reference/imagebuilder/index.html but be cool to just work in hcl2 for both docker and ami images. Thanks for the info

I try to keep these in separate repos and stick to conventions for where the AMIs and Docker Images end up.

@Jeremy G (Cloud Posse)

That is a big relief.

If dns_zone_id will not be used then you can just pass it as dns_zone_id = null to the module and that should do the trick.

This likely falls into the fun data tier updates you can do with providers like postgres, elasticsearch, rabbit, etc.

There is Kafka provider that will do this for you (https://github.com/Mongey/terraform-provider-kafka) but you will need to have an accessible route to hit Kafka outside your cluster.

https://www.terraform.io/docs/configuration/expressions/splat.html

you can use `

[*]

Interesting behaviour - I’ll give it a shot. Thanks!

And after all that, it was my mock data that was the problem rather than the source data.

You can just add the mixed instances policy on your ASG: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group#instances_distribution

@Miguel Zablah thanks for sharing it. However, how can we add a conditional statement here to allow a few ec2 to use spot and a few to use on-demand within the same ASG?

You will add it like this:

 mixed_instances_policy = {
    instances_distribution = {
      on_demand_base_capacity = 1
      on_demand_percentage_above_base_capacity = 50
    }

   ...
  }

on_demand_base_capacity this will set how many of on demand instances you will like as basis (default) on_demand_percentage_above_base_capacity will tell how many of the instances that you tell the ASG to have will be on demand (the other % will be spot)

In this example you will have 1 instance on demand always and 50% will be spot this means that if you have set your ASG to have 4 instances running 2 will be on demand and 2 will be spot

ffs

Really? SSO? - i’ve just written some CFN terraform wrapper stuff and was going mad

me too, I’m releasing the module ASAP

I created CFN template and call it from Terraform, now what I need to do is, create the module and import the resources there

I found that this missing account assignments, so partial usage for now

@Jeremy G (Cloud Posse)

@Ayman

if you change the name it will be recreated each time

it will only be recreated if there is a change in an attribute that forces replacement

look at the plan and see what is the red - sign and see which one is it

I use a var for the name attribute, it should be the same

will try a plan

any name change will force replacement, you can’t change it every time

it uses a fixed value from variables.tf file

for example, var.name

I’m using terraform 0.13.5

if it is a fixed value then this should not happen

so I think it is different

did you changed var.attributes , var.stage ?

no, the attribute is ['rds'] which is fixed

the database was created from snapshot

manually or using TF?

using TF

But using tf from the snapshot, correct?

yes

terraform state show module.rds.xxxxxx

use the resource name that shows in the plan

and see what the name is

and compare it

it doesn’t have name from the outputs

list of the rds attributes from state command

https://github.com/hashicorp/terraform-provider-aws/issues/5385

what is terraform trying to replace? the rds instance or the rds cluster?

rds instance

look then at the id, identifier and cluster_identifier

the id is the name basically

yeah

is that being changed?

identifier_prefix is added

but the one is complaining that forces replacement is the name only?

is this a cluster or a rds instance only?

are you using cloudposse modules for this?

yes, I am using cloudposse rds module

here are the outputs of plan command

but are you creating a cluster or just an rds instance?

just rds instance

ok

the “name” sttribute value in the state is test or teststage

teststage

it is database name passed into terraform rds resource

so you created the instance from snapshot using the module

and now you are running tf again without changing anything and is trying to recreate it?

yes

if that is the case remove the snapshot_identifier

do not pass it after is created

I think is trying to recreated again from the snapshot

there is one more replacement after removing snapshot from tf file

Ok but is still complains about the name

Mmmmm

What happened if do not pass a name?

Have you try without using a snapshot? And see if keep trying to replace the instance?

name variable should be required from database_name

I personally never had this problem

I think it is caused by snapshot

I have not use snapshot from an instance but I have done it for a cluster

But I didn’t have this problem

ok, thanks, I’ll give it a try tomorrow

Can’t change snapshot identifier without rolling database :) Also instances module doesn’t have apply immediately false and ignore lifecycle for some things so it can roll too.

I tried not to create RDS from snapshot and it worked

so I think this should be a terraform bug

created an issue to follow up, https://github.com/hashicorp/terraform-provider-aws/issues/17037

Referring to this
Terraform will now support reading and writing all compatible state files, even from future versions of Terraform. This means that users of Terraform 0.14.0 will be able to share state files with future Terraform versions until a new state file format version is needed. We have no plans to change the state file format at this time. (#26752)

As i understand it, i can apply a module with tf 0.14.6 and then try to apply it with 0.14.3 without an error thrown

if you bump to 0.12.30, you’ll even keep you backwards compatibility, i think. so you can easily test 0.14, and revert to 0.12.30

https://sweetops.slack.com/archives/CB6GHNLG0/p1609959289075600

or maybe not, that might only be relevant for the remote state data source, my bad

Ya i thought the same at first until i realized it was just the data source

I managed to make it work with what seem to be minor changes. Here are the modifications I made: • terraform-aws-lb-s3-bucket

# <https://github.com/cloudposse/terraform-aws-lb-s3-bucket/blob/master/main.tf#L29>

# terraform-aws-s3-log-storage 0.15.0+ supports TF 0.14.0+
source                             = "git::<https://github.com/cloudposse/terraform-aws-s3-log-storage.git?ref=tags/0.15.0>"

# <https://github.com/cloudposse/terraform-aws-lb-s3-bucket/blob/master/context.tf#L22>

source  = "cloudposse/label/null"
version = "0.22.1"

• terraform-aws-alb

# <https://github.com/cloudposse/terraform-aws-alb/blob/master/main.tf#L43>
# Update the version to the latest one after applying the changes listed above

I did not run any deep tests tho, but now I can successfully run a plan

Those PR are related

• https://github.com/cloudposse/terraform-aws-lb-s3-bucket/pull/21

• https://github.com/cloudposse/terraform-aws-lb-s3-bucket/pull/27

Hey Hao, are you running your tasks on EC2 instances or Fargate?

If you’re on Fargate, no choice than to define the CPU/RAM for your task.

If your tasks are backed by EC2 instances, keep in mind that you could starve your OS if your tasks use all the resources.

Hey @Coco, yeah, I know this

now the container can run without memory setting

so I’d like to pass null to cpu/memory

also tried with cpu/ram but failed with 137 error, which should be some issues with resource constraints

you can pass soft and hard memory limits to ECS+EC2

can I pass hard memory only since soft one is optional? I tried both memories but still got 137 error.

it worked if I only set hard memory manually for a testing container definition

task_cpu,  task_memory

are not required to be passed when ECS+EC2 is used but in the container definition it is required and then you can just set it as 0 if you want

got it, let me give it a try

got this error now:

Error: ClientException: Invalid 'memory' setting for container 'test'.

with:

  container_memory             = 0
  container_memory_reservation = var.container_memory_reservation

without task’s memory and cpu used

do not define any cpu or memory in the task def

sorry that is what I mean t to say

that is how we set up our instances so it uses the available memory and cput of the host

yes, I commented out them:

#  task_memory                        = var.task_memory
#  task_cpu                           = var.task_cpu

+ content     = jsonencode(
            {
              + Statement = []
              + Version   = null
            }
        )

Looks like you’re having an issue passing the policy to the aws_organizations_policy resource

Can you paste your TF code here?

Did you implemented based on the example & using our catalog for testing:

https://github.com/cloudposse/terraform-aws-service-control-policies/blob/master/examples/complete/main.tf#L1-L9

https://github.com/cloudposse/terraform-aws-service-control-policies/tree/master/catalog

Here is all of the relevant code. I didn’t try it specifically with one of the example configs, but tried creating my own with a very simple change from one of the examples I appreciate the replies and any help you can offer.

Here is the actual yaml I’m using

I figured it out. While going through the code I just pasted, I noticed that I used “map_config_local_base_path” instead of “list_config_local_base_path”. I’d changed those while trying to get 0.3.0 to work and forgot to change them back. The actual error was due to it not being able to find the yaml file at all. Thanks for the replies!

The actual error was due to it not being able to find the yaml file at all. @Andriy Knysh (Cloud Posse) looks like we need a check for this?

…in the yaml config module

https://www.youtube.com/watch?v=V2b5F6jt6tQ

@Maxim Mironenko (Cloud Posse) can you help with this

@Ankit Rathi please, try this new releases: <https://github.com/cloudposse/terraform-aws-s3-bucket/releases/tag/0.28.0> and <https://github.com/cloudposse/terraform-aws-iam-s3-user/releases/tag/0.13.0>

@Maxim Mironenko (Cloud Posse) @Erik Osterman (Cloud Posse) yeah its working now for us … thanks a lot for the amazing work

Hello @Ofir Rabanian, on my side I have divided the code in smaller project , and have one terraform state for each. Of course each project can read other project resources.

Don’t do too small project as you have to maintain and update terraform version for each

Yup, this is a problem.

#atlantis solves it by locking the workspaces (not the same as terraform locks)

will bring up in #office-hours

I made a video about it and have presented it , to the previous HashiConf in October 2020 as a 15min quick talk. I didn’t cut / encode the video yet . but if the topic interest some people I can move forward to do it.

@Pierre-Yves very much so! would love to get a link.

Here’s a link to the recording: https://sweetops.slack.com/archives/CHDR1EWNA/p1610574702013100

We’ll post a “Cloud Posse Explains” video clip once it’s available (@Andy Miguel)

https://youtu.be/R8PkldStQKw

you can also find the office hours where hashicorp, spacelift, scalr, and env0 demoed their products on this episode: https://www.youtube.com/watch?v=4MLBpBqZmpM

and those presentations are also available here: https://www.youtube.com/playlist?list=PLhRztDM6UvndoGRk0h1L_4M9xjtjbu0Jb

not_actions should work the same way as actions

https://github.com/cloudposse/terraform-aws-service-control-policies/blob/master/main.tf#L22

Awesome. Thanks!

note that the YAML config in the catalog https://github.com/cloudposse/terraform-aws-service-control-policies/tree/master/catalog

is Terraform-style YAML, not CloudFormation-style YAML

also keep in mind that NotAction can be used only with effect Deny

looks neat

there was someone asking a similar question the other day

the answer is no but it seems to be that there might be a bug on the TF provider

in a RDS cluster you can define the snapshot_identifier as null after the creation

but now you can clone databases so maybe you can clone db instances?

If you provision with a snapshot id in the terraform then you need to keep that id in the terraform else you’ll trigger a roll back or wipe. At least for us when I tried changing id it was going to recreate which I think would result in data change.

it was me lol

I believe it is a bug, so created an issue to follow up, https://github.com/hashicorp/terraform-provider-aws/issues/17037

no, it does not support cross-region peering (it was created before AWS had cross-region support and was not updated)

but you can try https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account

you can use the requster and accepter in the same account, but in different regions

https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/master/examples/complete/main.tf#L9

If you’re already using Terraform then you should use https://github.com/cloudposse/terraform-aws-eks-node-group

Or https://github.com/cloudposse/terraform-aws-eks-workers (haven’t used myself but similar to node-group)

right. thanks

will discuss in #office-hours

is this session done?

@Erik Osterman (Cloud Posse)

https://www.aaron-powell.com/posts/2020-03-23-approval-workflows-with-github-actions/

Thanks for sharing - taking a deeper look. Another thing that strikes me hard is the fact of sharing the plan artifact between plan and apply if its not done in the same run. Using artifacts I’m a bit concerned of sensitive data leaking - was thinking of encryption

Will discuss in #office-hours

Thanks for the insightful remarks during yesterdays office hours. Will take a look again at the TACOS recording

use for_each

https://www.terraform.io/docs/configuration/meta-arguments/for_each.html

example: https://github.com/cloudposse/terraform-aws-components/blob/a29b69a02bb21abe4c0a844d93398bc2cd00ef59/modules/dns-delegated/main.tf#L30

yes Andriy pointed out the best way to accomplish this ex.

resource "aws_s3_bucket_object" "create_folder" {
  for_each = {
    data = "data"
    url  = "url"
  }
  bucket = "my_bucket"
  acl    = var.acl
  key    = each.value
}

Thank you very much!

the type of resource with for_each is a map

Either just output the whole module object, module.s3_bucket_for_emr_logs, or use for to construct an object (list/map) of the attribute/s you want

will return a list of resources, from which you can get the attributes you need

outputting the whole module like module.s3_bucket_for_emr_logs will output a map of objects

the map’s keys will be values from toset(["${var.prefix}-emr-logs", "${var.prefix}-emr-logparser-logs"])

e.g. [ for item in module.s3_bucket_for_emr_logs : item.this_s3_bucket_id ]

which should be similar to values(module.s3_bucket_for_emr_logs)[*].this_s3_bucket_id

yeah, i hadn’t seen the pattern like values(module.s3_bucket_for_emr_logs)[*].this_s3_bucket_id before… i’ve just gotten in the habit of outputting the module and letting the user pick what they want

Got it. Thank you very much for your help.

I think it’s done by setting the launch type, eg:

resource "aws_ecs_service" "service" {
  name             = "myservice"
  cluster          = var.ecs_cluster_id
  task_definition  = aws_ecs_task_definition.my_task.arn
  
  launch_type      = "FARGATE"
  platform_version = "1.4.0"
  ...

Hmm but the eksctl crate-cluster command immediately creates these nodes, even before the first workload is deployed

oh sorry, didnt see you’re using eks.

i use ecs directly

I got in the same issue. you need the namespace and other labels to match fargate profile, let me pull examples files

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "13.2.1"

  tags            = var.tags
  cluster_name    = var.cluster_name
  cluster_version = var.cluster_version
  map_roles       = var.map_roles
  map_users       = var.map_users
  map_accounts    = var.map_accounts
  subnets         = data.aws_subnet_ids.private.ids
  vpc_id          = data.aws_vpc.this.id

  fargate_profiles = {
    default = {
      namespace = var.namespace
    },
    kube-system = {
      namespace = "kube-system"
    },
    kubernetes-dashboard = {
      namespace = "kubernetes-dashboard"
    }
  }
}

I’m using the terraform-aws-eks module, notice how each profile point to a nampespace, if you want more than one namespace in a profile, I think you need to use the console to do it, not yet test that use case

  fargate_profiles = {
    example = {
      namespace = "default"

      # Kubernetes labels for selection
      # labels = {
      #   Environment = "test"
      #   GithubRepo  = "terraform-aws-eks"
      #   GithubOrg   = "terraform-aws-modules"
      # }

      tags = {
        Owner = "test"
      }
    }
  }

that is the first part, 2nd part in yaml files, let me pull something

example for k8s-dashboard.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard

notice the k8s-app label match the profile name in .tf file, but where I add this k8s-app ? I can’t remember

so back to you question, you don’t create Fargate nodes, because it’s a managed service, you only create Fargate pods inside these nodes.

https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html

I hope this give you some light

Thanks Mohammed for your input.. I manage running fargate pods on the cluster I created with eksctl –fargate .. but the thing I do not get is: when I run the cluster creation with eksctl it DOES create nodes of type “Fargate” (see screenshot I posted at the beginning of the thread) - but when I use the terraform eks module, it does not create nodes of this type.. and if I create a cluster without node_groups and a fargate profile, I have no nodes to run the pods on. So I am trying to figure out how I can get the same result with terraform as with eksctl, this being a running cluster with let’s say 2 nodes of type Fargate…

I see, no nodes will be available unless you deploy some yaml files

I saw the same thing, when the pods start creation, fargate will create the underlaying nodes for you.

think of it same as ecs tasks, you can create ecs service with zero tasks. same here apply you can create eks with fargate compute backend with zero nodes, since no pods are running.

I see.. but if I just use TF to create the cluster and the profile, then I would see this after creation completes:

which I thought was kinda odd

Yes totally fine for the last two

the coredns needs to be patched to work with fargate profiles

do onething: deploy sample k8s yaml file there ( match the ^^ notes)

and you will see nodes

ok will do that - and how do I patch coredns as you mentioned?

sorry and what do you mean by match the ^^ notes

https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.html check section (Optional) Update CoreDNS

^^ means upper messages

here a sample dashboard

Ok great.. thanks - so that would mean, with TF: I create a fargate profile for my namespace but also another one for coredns and then use kubectl to patch coredns. Then once I spin up the first pods I will see the actual fargate nodes appearing

exactly

cool thanks a lot - I will give that a go

I created the cluster like this with TF:

fargate_profiles = { default = { namespace = local.namespace }, kube-system = { namespace = “kube-system” }, python-web ={ namespace = “python-web” } kubernetes-dashboard = { namespace = “kubernetes-dashboard” } }

Then I ran the patch coredns command

and then I deployed a pod

it works - the node is now visible too of type Fargate

the only thing I see is that 0/2 pods is still mentioned for coredns .. guess that is wrong right?

@Mohammed Yahya

let me pull something

but awesome already to see the node appearing!

I feel you

### Patch coredns - DONE

kubectl patch deployment coredns -n kube-system --type json \
-p='[{"op": "remove", "path": "/spec/template/metadata/annotations/eks.amazonaws.com~1compute-type"}]'
# OR 
kubectl rollout restart deployment coredns -n kube-system  

use export KUBECONFIG="${PWD}/kubeconfig_XXX"

hmm the first command I had already executed, didn’t work, second one is having no effect either

I see I remember something missing from my documentation, let me pull something else

can you screenshot the labels of the coredns pods

sure

also run kubectl logs coredns-xxxx

the last 2 ones are the ones which are up since I executed the restart command you proposed

oh

they are suddenly ready…

yes restart take time

fantastic

awesome, thanks so much for your help

anytime

remember to update your yaml files with labels and namespaces before deployment or use default namespace

that’s the trick

yes, the namespace must match a fargate_profile namespace

and is the label important too?

Yes, don’t ask me why , did not work for me without label

this is fairly new, so expect some odd behaviors

but much cheaper than ec2 slave nodes

and you can scale up and down on the salve node level

Looks like you’re correct to me. Feel free to put up a PR and mention it in #pr-reviews — simple things like that are quick to get merged if you post about your PR there.

Ya, we want to make sure we tag all the resources. Sometimes it slips through.

Adding tags should be a one line fix like this: https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/iam.tf#L19

I’d create the PR if I understood the mechanism you’re intending with [context.tf](http://context.tf)

which exists in the eks-repo, but not in the one i was referring to

Hrmm, looks like terraform-aws-ecs-cloudwatch-sns-alarms was updated to use [context.tf](http://context.tf) pattern

I don’t understand why something so popular like Terraform moving kind of slow on these PR

They do it with odd things and it’s very frustrating. I’m approaching waiting 1 year on Amplify support.

https://github.com/hashicorp/terraform-provider-aws/issues/6917 https://github.com/hashicorp/terraform-provider-aws/pull/15966

Discovered this week that AWS Macie support is also behind. They only have classic, and classic doesn’t any more in new accounts. So they are waiting for more interest.

I can see how they struggle - bet a team of 30 engineers wouldn’t be enough to manage just the AWS provider, let alone any others.

i don’t see how they keep up without switching to code generation to create basic resources and data sources that map 1:1 with the golang sdk

Yea, I think that’s the only way. I think pulumi has started doing that

There is a terraform-aws-modules one — https://github.com/terraform-aws-modules/terraform-aws-security-group

Ahh, thanks for that! I’m not sure why that didn’t come up in my search.

@David Napier https://github.com/cloudposse/terraform-aws-alb/releases/tag/0.27.0

Thank you!!

1. cluster_master_host is empty b/c it’s the output of this https://github.com/cloudposse/terraform-aws-emr-cluster/blob/master/main.tf#L499

you need to specify var.zone_id in Route53 so the module would create a record in the DNS zone and point it to cluster_master_public_dns

1. You created the cluster in a private subnet

you can’t access it from the outside of the VPC

you can use a bastion host to SSH to it, or a VPN

Thank you.

yes

True, no multiple attachment, it’s one-one mapping, I guess it make sense to them

“The security token included in the request is invalid” prob means that you don’t have permissions to do that. When you do it from the console, you are logged in. When you do it from the command line, you use some AWS profile with some config (access keys or roles). Check the AWS profiles

i am using the same user in terraform, cli and web console

keeping it simple since it’s the first time i use the china partition

web console login is using user/password (w/o mfa), while cli/terraform are using key you created under IAM user tab. put the key into your aws profile or env variable.

i did that. I was able to create a vpc and other things with the same user/profile

it’s just the creation of this role that doesn’t succeed

are you able to create IAM role via web console using the same user ?

yes

try to attach AdministratorAccess policy directly your terraform user

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/470a2237a6b5678fbfdbd5173e4f29db4d2396da/iam.tf#L15. this is failing

@Qing Jiang did you manage to have the eks module working in the chinese partition?

oh i have no experience on the chinese parition part of aws. my working region is mostly us-west-2

I solved it. I was using aws-vault, I needed to use the --no-session option to make it work. In the “regular” AWS it was not needed so that created a bit of confusion

Ideally you’d delete the helm chart/app/yaml that defines the Ingress. The ALB Ingress Controller would see that it was deleted and it would delete the ALB for you.

The workflow for an ideal destroy should look like this: • delete all apps, except alb-ingress-controller, external-dns, and maybe cluster-autoscaler • Wait a while for the ALBs, Route53 records, and some EC2s to be deleted • Delete the remaining apps • Run terraform destory

Agree with @Vlad Ionescu (he/him) - that destroying the resources in the cluster before destroying the cluster is the best way.

https://github.com/rebuy-de/aws-nuke ?

Exactly what I was about to suggest. I used it, works well. Has a few kinks to figure out.

cool app. saved for later thanks

@Yoni Leitersdorf (Indeni Cloudrail) anything specifically i should know?

i basically want to leave the account there as well as the default admin role and policy

cloud-nuke https://github.com/gruntwork-io/cloud-nuke

Here is what I posted in our internal slack:

So notice that it’s important to exclude certain roles. For example, if I nuked the SSO, I would locked myself out of the account.

Also, using aws-nuke with aws-vault is a bit odd, took some time to figure out. See my message screenshot above.

@Yoni Leitersdorf (Indeni Cloudrail) thank for sharing, looks like a perfect tool. about to use it to nuke every asset not created by terraform in our env ..

By default, aws-nuke will use dry-run. Watch its results carefully to see if you missed something in the filters.

If you’re looking to run this as part of a schedule / CI workflow:

1. Cloud Posse way is via GH Actions: https://github.com/cloudposse/testing.cloudposse.co/blob/master/.github/aws-nuke.yaml
2. My solution (probably wouldn’t recommend): https://github.com/masterpointio/terraform-aws-nuke-bomber

Cloud nuke as @Mohammed Yahya says very good

Assuming you’ve plus one’d it? Reviewing the code and then commenting with how it would work for your? There are a lot of open PR’s in that repo including one about rabbit which we want to use. I guess it’s either merge locally into a fork or maybe try your TAM even? I don’t know if anyone else has any ideas.

Yeah I it haha Someone I work with mentioned pulling it in locally but it would be pulling it into terraform managed by gitlab which could be a bit dodgy. This resources is a very minor piece at least, and could even be replaced with an external_data source object that using an sdk to deploy the resource.

Yeah tough isn’t it. Big job for maintainers too Good luck :)

This got merged finally . I also posted on the terraform community forum, perhaps a project manager saw it? Could also be coincidence

name as Name tag?

no as in the name of the VPC itself

I have vpc with the same name in different regions

The name of a vpc is the Name tag

Is it not?

we are saying the same thing

yeh we are

can you have the same name for a VPC in different regions in the same account?

but yes in different region is fine

perfect thanks

for the second, have you tried override files? https://www.terraform.io/docs/configuration/override.html

Would that work with a specific modules resource tho?

i don’t understand the question

I see so similar to this

https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/blob/custom_alarms/autoscaling.tf#L59

Maybe?

I copied that from the datadog module

Those are Interesting propositions but I wonder about the outside module resource management, I think that will be a core change in terraform

Ya it would be a core change. I spoke to one of the tf devs and they said those two feature requests would defeat the purpose of a module being an isolated item

It’s too bad, because at times i find a module is like 90% covering my use case and just 10% is wrong which renders that module unusable

yep

I also struggled quite a bit with the .terraform.lock.hcl - somehow for us it also had sometimes changes after a colleague pushed (after init+apply) and I was trying to developer further - not sure what that related to.

Also wonder how far along renovate / dependabot are in terms of also re-creating the file on a module bump for a root module

good question on dependency update tooling. they are zero far along, i expect… though looks like renovate is at least talking about it… https://github.com/renovatebot/renovate/issues/7895

dependabot still lacks hcl2 support, so, probably much further off for that one

like this one : https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn

You can test == {}

depends on how you define it… if it’s an object with required attributes, might need to be more pedantic:

yellowfin.instances == {} && yellowfin.schedule.down == {} && yellowfin.schedule.up == {}

you could also check against null instead…

var.worker_groups.yellowfin == null ? 0 : 1

the null check works even for objects with required attrs…

i got the order wrong

that did not seem to work

@loren

Yellowfin is a var or a local?

Don’t you have to use var.Yellowfin or local.yellowfin?

i have a map of maps

so the yellowfin = {} and have inside worker_grpups

it species the nodes for each type of worker node as we have different ones

like instance type, min and max numbers

but the yellowfin map can be empty as we only need these nodes in certain situations

Ahhhh ok

before i used to do this with an enabled map where yellowfin = 1 or 0

And if you test for {} instead of null?

module "yellowfin_worker_node_group" {
  source = "git::[email protected]:redacted/tf-modules.git//modules/eks-node-group?ref=bf46d92"

  count = var.worker_groups["yellowfin"] == {} ? 0 : 1

this does not work

the module still gets executed based on the output of the plan

== null does that same thing

Mmmm

yeh its weird

i am not sure if its because i am doing a map inside of map or something

any ideas as i am running on empty

i worked it out

yellowfin is not {}

but yellowfin.instances == {} will work

I was dealing with something similar to this yesterday

is there a len function for a map?

Right, so like I first said… https://sweetops.slack.com/archives/CB6GHNLG0/p1610915155075300?thread_ts=1610914335.074800&cid=CB6GHNLG0

You basically have to define what “empty” means for your object

You can null an object and test for that, but it means you need to set the var accordingly, e.g. worker_groups = { yellowfin = null }

Hello @Steve Wade (swade1987), here is the code I use to clean a config map from its empty value where I create a new map without the empty item:

  cleaned_config_map = {
    for srv, cfg in local.config_map: srv => cfg
    if cfg != {}
  }

then parse cleaned_config_map with for_each and you’ll only have the map entries who have non empty values. I didn’t find a way to do it without a temporary map.

use this https://terragrunt.gruntwork.io/docs/features/before-and-after-hooks/ if not cover your use case, use makefile and then call terragrunt within

You can put the before_hook in your parent config, and all the terragrunt configs inheriting from that parent will pick it up

But if you want a before_hook across completely separate sets of terragrunt configs, i.e. no shared parent config, you might have to get creative with the function read_terragrunt_config()….

use the lifecycle block to ignore changes to the task revision

Thanks, I’m already using those. I’ll try without creation of task-definition, so it will use latest revision for that family.

So:

task_definition = try(aws_ecs_task_definition.task.*.arn[count.index], var.family)

if task definition is defined in terraform it will use it when creating service, if not it will use latest revision in that family

the error

This is a known issue with for_each where if the items on which you are iterating doesn’t yet exist, it won’t work

that sucks

is there a workaround?

first thing i see is that the “false” condition is invalid… can’t for_each over null. change that to {}

that error generally occurs only when the for_each keys are unknown. it is fine if the values of the maps are unknown, that’s generally the whole point after all

the keys i see are cpu_high and cpu_low, which are certainly known, so i think i need to look at the test config…

though the test config hasn’t been updated, so it’s just using the default value for custom_alarms… hmm…

it could be the way module.this.enabled is used… need to trace through how that is set, to see if it relies exclusively on user input or static locals or data sources that themselves do not depend on resources…

i would say first try just:

for_each                  = local.all_alarms

and see if that works

If I try this in tf 0.13.5 on my local ( same code as the repo) it works

but on the tests it does not

easy then, bump the requirement to tf 0.13

HAHAHAH

there were tons of changes in for_each between 0.12 and 0.13, it got a lot more flexible

mmmmmm that could be then

personally i’d say the null label and context stuff is trying to be a bit to smart/magic, and it’s overwhelming the ability for 0.12 to process for for_each

give this a try to confirm it is the enabled part of the logic:

for_each                  = local.all_alarms

alternatively, another way to construct your for_each:

for_each                  = { for k, v in local.all_alarms : k => v if module.this.enabled }

i prefer your first approach as it’s less loopy, but this might give 0.12 enough of a clue about how to construct the map

testing

something else to think about , is what if the user doesn’t want the default_alarms at all?

welllllllllllllll they can……..

the module already had some ( I’m keeping backwards compatibility) but I could default that too

so that failed too… looks like you’re bumping into some limitation in tf 0.12 and how it evaluates for_each expressions, that is fixed in a later version :(

yep