#terraform (2019-11)

i hope this link will help you https://aws.amazon.com/blogs/security/how-to-use-bucket-policies-and-apply-defense-in-depth-to-help-secure-your-amazon-s3-data/

here’s what I’m trying to do: create a cert in dev account with a subject alternative name (SAN) in “vanity” domains account

and validate both with DNS validation

here’s teh problem I’m getting: Access Denied!

I do tf apply from my ops profile. It is a trusted entity on both my dev and domains accounts

You need two separate aws providers

Is that how you have it setup?

I straight up copy / pasted the provider blocks used in the https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account/blob/master/requester.tf#L29

yes! thanks @Igor

here’s a gist with the least possible stuff; the acm parts are straight from tf docs

https://gist.github.com/jafow/48422a081fb35b939f1156da47e6afc5

on looking at the vpc-peering module above, I notice there’s no profile attr, just the assume_role - I wonder if that’s my issue.

the Acces Denied error indicates that dev account assumed role does not have permissions to access the domains account route53 record.

but what i’d expect is that it should be assumed under the domain profile to validate the record, because that 2nd provider alias is used.

aws_route53_record.cert_validation_alt1 doesn’t have provider attribute set

I assume you want that to be aws.route53 as well

I can’t tell from the gist what your code is trying to do

Oh, I see, you’re using a SAN. I have never tried that before – does AWS allow multiple cert-validations on the same cert?

I also don’t see you using the aws_acm_certificate_validation resource

aws_route53_record.cert_validation_alt1 doesn’t have provider… yep @Igor that could be it!

you’re right I left that off of the gist; copy paste fail. updating now to better show the intent.

oh good thinking, sorry I didn’t look at other data sources.

https://github.com/cloudposse/terraform-aws-multi-az-subnets

This one looks interesting

Yes, we use that one all the time

@Callum Robertson look what @Andriy Knysh (Cloud Posse) just did

https://github.com/cloudposse/terraform-aws-eks-cluster/pull/33/files#diff-2ed9e16ccfbe630166af4f665095a460R124-R134

for getting assumed roles to work in a local-exec

we’ve been testing the EKS modules on Terraform Cloud, and needed to add two things:

1. Install external packages e.g. AWS CLI and kubectl since those are not present on TF Cloud Ubuntu workers

1. If we have multi-account structure and have users in the identity account, but want to deploy to e.g. staging account, we provide the access key of an IAM user from the identity account and the role ARN to assume to be able to access the staging account. TF itself assumes that role. But also AWS CLI assumes that role (see above) to get kubeconfig from the EKS cluster and then run kubectl to provision the auth configmap

epic work @Andriy Knysh (Cloud Posse)!

I’ll test this out over the weekend and report

the PR has been merged to master

https://github.com/cloudposse/terraform-aws-eks-cluster#works-with-terraform-cloud

See credential_source = Environment… https://docs.aws.amazon.com/cli/latest/topic/config-vars.html#using-aws-iam-roles

@Julio Tain Sueiras

thanks @loren!

Hello, do you know any vscode extension that work well with v0.12 syntax?

Oh, see this: https://sweetops.slack.com/archives/CB6GHNLG0/p1573061306076800

Is this the same provider Microsoft is working on? https://github.com/microsoft/terraform-provider-azuredevops

I used to. No complaints.

I use it as well and have been pretty happy with it

by list of labels you mean tagging?

I meant this https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/#add-a-label-to-a-node

tags attached only to ec2 instances

Good timing, we tried doing this but nodes wouldn’t join cluster.

  # FIXME # 
bootstrap_extra_args = "--node-labels=example.com/worker-pool=01,example.com/worker-pool-colour=blue"

Don’t have SSH yet so commented out until can debug

it could be done using the three variables https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/userdata.tpl

https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/variables.tf#L374

https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/variables.tf#L380

https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/variables.tf#L386

https://kubedex.com/90-days-of-aws-eks-in-production/

@vitaly.markov @kskewes thanks for testing the modules

it would be super nice if we could add your changes to kubelet params to the test https://github.com/cloudposse/terraform-aws-eks-cluster/tree/master/examples/complete

and here in the automatic Terratest we have Go code to wait for all worker nodes to join the cluster https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/test/src/examples_complete_test.go#L81

which was tested many times

but we did not test it with node labels

would be nice to add it to the test and test automatically

@Andriy Knysh (Cloud Posse) I’ve found another bug (I think), by default terraform create a control plane with a specified kubernetes version from tf vars, but the node group will be provisioned with default version provided by ami, currently 1.11

I specify another variable eks_worker_ami_name_filter = "amazon-eks-node-${var.kubernetes_version}*" to use correct ami

I think EKS currently supports 1.14 only

and you can set it here https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/main.tf#L101

oh you are talking about worker nodes?

your changes would be good

but still I think EKS support 1.14 only (last time I checked)

is most_resent not working to select the latest version? https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/main.tf#L141

and you can always use your own AMI https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/main.tf#L182

is most_resent not working to select the latest version?

yeap, it doesn’t work correctly

Correct. It does not. I haven’t tested suggested change above, we pin AMI.

Just utilised custom security groups, so glad this is available in workers module.

@Andriy Knysh (Cloud Posse) I’ve opened a PR https://github.com/cloudposse/terraform-aws-eks-workers/pull/27 that solves the issue with extra kubelet args

thanks will check

$ tree
. (production)
├── ap-southeast-2
│   ├── eks
│   │   ├── eks.tf
│   │   ├── manifests
│   │   │   └── example-stg-ap-southeast-2-eks-01-cluster-kubeconfig
│       ├── provider.tf -> ../provider.tf
│   │   ├── variables-env.tf -> ../../variables-env.tf
│   │   └── variables.tf -> ../variables.tf
│   ├── gpu.tf
│   ├── managed_services
│   │   ├── aurora.tf
│   │   ├── elasticache.tf
│   │   ├── mq.tf
│       ├── provider.tf -> ../provider.tf
│   │   ├── s3.tf
│   │   ├── variables-env.tf -> ../../variables-env.tf
│   │   └── variables.tf -> ../variables.tf
│   ├── provider.tf
│   ├── variables.tf
│   └── vpc
│       ├── provider.tf -> ../provider.tf
│       ├── subnets.tf
│       ├── variables-env.tf -> ../../variables-env.tf
│       ├── variables.tf -> ../variables.tf
│       └── vpc.tf
├── us-west-2
│   ├── eks
│   │   ├── eks.tf
<snip>
└── variables-env.tf

Note we are a linux + macOS shop and use symlinks in terraform and ansible already.

we do it this way

var.environment = var.stage

a region = a variable

modules are created by functional pieces/aws products or product groups

ECS task module, ECS cluster module, ALB module, Beanstalk module

and every module takes var.environment, var.region as an argument from a .tfvars file

sp no matter what region the module will just work

if you are doing mutiregion it is IMPERATIVE that you use a var.name that contains the region ( this is always thinking from the point of view of using cloudposse module structure)

otherwise things like IAM roles will collide because it is a globalservice

use Regional s3 state buckets if you are going multiregion

Cheers, I forgot to say we have modules in a separate git repo. This structure is for within a region and done too minimise blast radius when running terraform apply. The individual files line [mq.tf](http://mq.tf) is instantiating the mq module plus whatever else it needs.

ok , cool yes so similar of what we have

We overload stage with region to achieve that. I guess name would work as well.

we like to have a flat structure module where there is no subfolders and many TF files floating around

Eg. stage = "stg-ap-southeast-2"

same idea

cool

Flat structure is nice for referencing everything line global variables hah.

Have you ever merged two state files? If we wanted to collapse again that could be an option. Splitting out same I guess.

never

I will never try to do that

I think at that point I will prepare to do a destroy and add the resources to the other file and do an apply

Yeah. Pretty gnarly. Redeploy better if possible.

Think I’ll continue splitting environment into multiple directories to limit blast. Symlink around some vars.

Does it make sense to put the whole stack into a module and then use that module in test/uat/prod environments?

tt depends

I guess if you are comfortable with having dbs, albs, instances , containers and a bunch of other stuff in one single terraform if fine

you kinda need to think in terms of how your team will interact with that piece of code too

I will still separate deployment from infra but I’ll have db, alb and RDS in one module

what is required ?

ECS cluster I mean

yes

sorry for clumsy grammar

so let say

ecs cluster, service, task, task definition

we provision ecs cluster with terraform

in that order

is there any specific option for ecs cluster ?

as we plan only to use as fargate option for ecs service later on

when I see in ecs cluster, they say it has ecs cluster as instances

is not on the cluster level

here : https://github.com/cloudposse/terraform-aws-ecs-alb-service-task

we don’t want to follow create instances and use them as cluster

https://github.com/cloudposse/terraform-aws-ecs-alb-service-task/blob/master/main.tf#L243

Fargate can only run with 16 gb of memory and 10 gb of disk

no EBS mounts

if you need more then you need to use EC2 type an manage your instances

ya

but for easy scalling and handle traffic up and down, is fargate enough ?

depend of your needs but yes it is easier

1 important thing need to ask

and also 3x more expensive than EC2 type

is like a slim down version of k8 running pods

is there anyway we can use terraform to provision ecs service without specific image ? It’s a mandatory to have image at first in ECR right ?

no is not

you can deploy a service that does not work

then deploy a new task def and it will work

oh I see

or a service with a correct task def but empty ECR repo

the reason why I ask those question

is because

we planing to do Ci/CD step for infras.

flow is . vpc -> all thing include ecs cluster -> ecs service

you think it’s can be achieved ?

yes

almost forgot

have you ever been in issue with update the ecs service itself ?

my issue is the name of ecs service, the container name sometimes make terraform stuck as well to reapply. Something relate with lambda_live_look_up on old task definition

no , I have not seen that

we do not use tf to deploy a new task def

we use ecs-deploy python script (google it)

ya

we follow that tool to

but you know what is big issue about ecs-deploy ?

it based the URL container image url. for ex: 1233452.aws.dk..zzz/test:aaa

so what happen when we want to apply new image: 1233452.aws.dk..zzz/exam1:aaa

it stuck forever.

The issue here is how to accommodate different inputs (having different column types or numbers, etc)

use dynamic blocks https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/blob/master/main.tf#L17

@Saichovsky ^

what module exactly ?

Nice find @vitaly.markov thanks for testing again. It’s not a misconfiguration, we just didn’t think about that case when created the modules

but maybe we can use the additional SGs to connect the two worker groups together https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/variables.tf#L410

by adding the SG from group 1 to the additional SGs of group 2, and vice versa

or we can even place all of the groups into just one SG https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/variables.tf#L398

Mm. I didn’t think about the example. For blue green node pools we created our own security group and put all pools in it and then specified in master. This way there was no chicken egg and reliance on pool module output to construct sg members.

Happy to share this if useful? Would be much nicer with for each module support.

yea that’s a good solution - create a separate SG, put all worker pools into it, and then connect to the master cluster

@kskewes please share, it would be worth adding to the examples

Sure. Reminder set to post gist.

@kskewes I wasted time before I found a root of cause, I used to create EKS using eksctl, I didn’t expect this behavior

I think would be better to add an example to repo

Me too mate. Spent day or so tracking down. When back at computer will post. We can slim it down for repo.

Our [eks.tf](http://eks.tf) file with:

1. vpc and subnets from remote state (we tend to use lots of state files to limit blast radius).
2. single security group for all nodes so nodes and their pods can talk to each other and no chicken/egg with blue/green/etc. Note we left out a couple rules from module’s SG.
3. blue/green worker pools.
4. worker pool per AZ (no rebalancing and we need to manage AZ fault domain).
5. static AMI selection (deterministic). Note, we are dangerously close to character limits (<64 or 63?) for things like role arn’s, etc.

Workflow is to:

1. terraform apply (blue pool)
2. …. update locals (ie: green pool ami and enable)
3. terraform apply (deploys green pool alongside blue pool)
4. … drain.
5. …. update locals (ie: blue pool ami and disable)
6. terraform apply (deletes blue pool, leaving green only)

Please sing out if any questions or suggestions. Will PR custom configmap location soon (haven’t written but we want).

I think the evolution of this could be even simpler. Always keep blue and green “online”. Only change the properties of blue and green with terraform. Automate the rest entirely using a combination of taints and cluster autoscaler. The cluster autoscaler supports scaling down to 0.

So basically you can taint all the nodes in blue or green and that would trigger a graceful migration of pods to one or the other

Then Kubernetes will take care of scaling down the inactive pool to zero.

kubectl taint node -l flavor=blue dedicated=PreferNoSchedule

This would then scale down the blue pool

@oscar you might dig this

That’s a great idea. My colleague has done an internal PR to start cluster auto scalar and we’ll discuss this. New to EKS so bit of a lift and shift.

Thanks!

For ref, we do this:

module "eks_workers_1" {
  source = "git::<https://github.com/cloudposse/terraform-aws-eks-workers.git?ref=tags/0.10.0>"

  use_custom_image_id = true
  image_id            = data.aws_ssm_parameter.eks_ami.value

  namespace                          = var.namespace
  stage                              = var.stage
  name                               = "1"
...
  allowed_security_groups            = [module.eks_workers_2.security_group_id, module.eks_workers_3.security_group_id]
...

@oscar I tried to use allowed_security_groups and it broke all SG setup, I’ll create an issue later

I’ve not verified it ‘works’, but it certainly doesn’t ‘break’ on terraform apply

It’s been like this for some time

Thanks Oscar. Problem for us with security groups is if the color is disabled then no security group exists and terraform complains. This way we create a single group up front.

Team mate has done a great job with cluster auto scalar so we’ll end up doing Erik’s suggestion soon. Thanks for that!

Dude check out what was just announced! https://github.com/aws/aws-node-termination-handler

This might make things even easier

NOTE:
If a termination notice is received for an instance that’s running on the cluster, the termination handler begins a multi-step cordon and drain process for the node.

it’s not spot specific

Yeah, we want to roll out PDB’s everywhere so we don’t just take down our apps because we drained too fast

thanks for sharing though, we’ve been chatting in house when we saw

Not seeing any search - am I blind? haha

Oh, I thought something looked different

Awesome, thank you a lot

Yep. No workaround I know of in tf 0.11. Upgrade to tf 0.12

I tried setting acl = ""

but I got

Error: Error putting S3 ACL: MissingSecurityHeader: Your request was missing a required header
	status code: 400, request id: 594BF7171170FCA0, host id: N+nVSWtyZ4S9+XMfo4d57W1LuuyPZWSI3y357Li0hUD46eBFRmEUt9c0z8iGssCE3iVsVbs11BU=

I guess the only thing that can write this flow logs

the policy that Tf is applying :

policy                      = jsonencode(
            {
                Statement = [
                    {
                        Action    = "s3:PutObject"
                        Condition = {
                            StringEquals = {
                                s3:x-amz-acl = "bucket-owner-full-control"
                            }
                        }
                        Effect    = "Allow"
                        Principal = {
                            AWS = "arn:aws:iam::127311923021:root"
                        }
                        Resource  = "arn:aws:s3:::globalaccelerator-logs/*"
                        Sid       = "AWSLogDeliveryWrite"
                    },
                ]
                Version   = "2012-10-17"
            }
        )
        region                      = "us-east-1"
        request_payer               = "BucketOwner"
        tags                        = {
            "Attributes" = "logs"
            "Name"       = "globalaccelerator-logs"
            "Namespace"  = "hds"
            "Stage"      = "stage"
        }

        lifecycle_rule {
            abort_incomplete_multipart_upload_days = 0
            enabled                                = false
            id                                     = "globalaccelerator-logs"
            tags                                   = {}

            expiration {
                days                         = 90
                expired_object_delete_marker = false
            }

            noncurrent_version_expiration {
                days = 90
            }

            noncurrent_version_transition {
                days          = 30
                storage_class = "GLACIER"
            }

            transition {
                days          = 30
                storage_class = "STANDARD_IA"
            }
            transition {
                days          = 60
                storage_class = "GLACIER"
            }
        }

        server_side_encryption_configuration {
            rule {
                apply_server_side_encryption_by_default {
                    sse_algorithm = "aws:kms"
                }
            }
        }

        versioning {
            enabled    = true
            mfa_delete = false
        }
    }

the resource part

I can’t even modify the ACL

anyone have any ideas ?

@Igor Rodionov

Reading

it is driving me nuts…

ACL :

to create the bucket I used :

# Policy for S3 bucket for Global accelerator flow logs
data "aws_iam_policy_document" "default" {
  statement {
    sid = "AWSLogDeliveryWrite"

    principals {
      type        = "AWS"
      identifiers = [
        data.aws_elb_service_account.default.arn, 
        "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
        ]
    }

    effect = "Allow"

    actions = [
      "s3:PutObject",
      "s3:Get*"
    ]

    resources = [
      "arn:aws:s3:::${module.s3_log_label.id}/*",
    ]

    condition {
      test     = "StringEquals"
      variable = "s3:x-amz-acl"

      values = [
        "bucket-owner-full-control",
      ]
    }
  }
}

# S3 bucket for Global accelerator flow logs
module "s3_bucket" {
  source                 = "git::<https://github.com/cloudposse/terraform-aws-s3-log-storage.git?ref=tags/0.6.0>"
  namespace              = var.namespace
  stage                  = var.stage
  name                   = var.name
  delimiter              = var.delimiter
  attributes             = [module.s3_log_label.attributes]
  tags                   = var.tags
  region                 = var.region
  policy                 = data.aws_iam_policy_document.default.json
  versioning_enabled     = "true"
  lifecycle_rule_enabled = "false"
  sse_algorithm          = "aws:kms"
  #acl = ""
}

one very weird thing is that the KMS key used to encrypt the objects is unknown to me

maybe since I do not have access to the key I can unencrypt the objects

We do not have any account with ID

399586xxxxx

I guess that ID comes from GlobalAccelerator or something like that

@jose.amengual here is an example of using the module in another module https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/blob/master/main.tf#L80

usage example https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/blob/master/examples/complete/main.tf

maybe ^ will help

what value did you set here https://github.com/cloudposse/terraform-aws-s3-log-storage/blob/master/variables.tf#L115 ?

So after some research :

I never had a problem creating the bucket

but I did one mistake

I used :

sse_algorithm          = "aws:kms"

without passing a key

and that somehow chose a kms from who knows where and my objects are all encrypted with a key that I don’t own

that is not the module doing that, that is the API or terraform doing it

my guess is that the objects are encrypted using the Origins KMS key ?

like the aws log delivery service ?

I have no clue

I wan to to test creating buckets this way

by hand and see what happens

the flow logs were successfully written in the bucket so that is why I though it was weir that it can encrypt but not decrypt the objects

I created another bucket using default aes-256 server side encryption using the manged aws kms key and the flow logs were correctly written to the bucket and I was able to download them without any issued and all this using the same module but just using the default encryption

so I tried earlier a new bucket same module using sse_algorithm = "aws:kms"

not passing a custom key and it used the default kms key

so I have no clue WTH

the other thing I could try is to use terraform 0.11 use the old version of the module and try again

maybe is a provider bug

I think it could be related to this :

After you enable default encryption for a bucket, the following encryption behavior applies:

    There is no change to the encryption of the objects that existed in the bucket before default encryption was enabled.

    When you upload objects after enabling default encryption:

        If your PUT request headers don't include encryption information, Amazon S3 uses the bucket's default encryption settings to encrypt the objects.

        If your PUT request headers include encryption information, Amazon S3 uses the encryption information from the PUT request to encrypt objects before storing them in Amazon S3.

    If you use the SSE-KMS option for your default encryption configuration, you are subject to the RPS (requests per second) limits of AWS KMS. For more information about AWS KMS limits and how to request a limit increase, see AWS KMS limits.

#terragrunt

there’s no aws api for it, i don’t think. you might need to roll your own thing on lambda or ec2 that leverages an ldap library

That was my other thought. To do it with LDAP or some Microsoft tool.

We’ve had some luck with the openldap package and python-ldap

Yeah, using conftest to validate outputs from terraform plan is slick.

@kskewes are you using conftest at work?

Not yet but it’d be good.

are you talking about TF Cloud or TF Enterprise (on-prem installation)? https://www.terraform.io/docs/cloud/index.html#note-about-product-names

TF Cloud - to start with.. thats what Hashicorp is pushing lately

we see these features in TF Cloud that are very useful:

1. Automatic state management - you don’t have to setup S3 bucket and DynamoDB for TF state backend

1. atlantis-like functionality included (run plan on pull request) - you don’t have to provision atlantis if you want GitOps

1. Security and access control management - you create teams and assign permissions to the teams (which can run plan, apply, destroy, etc.), and then add users to the teams

and you can use workspaces

and they are the recommended way to use Cloud

not that you can’t use it in OSS but they are not recommended for multi-team repos etc

We have an official Demo on Friday,

The bigger issue I have with TF Cloud is it requires me to give them Access Keys with essentially admin access. Which means I need to trust them to keep those keys secure… Whereas with Atlantis I can run it in an account and the security of it is my responsibility.

Thanks for the inputs @Andriy Knysh (Cloud Posse) @jose.amengual.. Appreciate that! I am using terragrunt for state-s3 management , tied up terragrunt plan/apply on Jenkins.. but only the Security and access management is thru iam-roles.. I will look forward the Demo on Friday to see further on TF cloud

then you buy TF enterprise, that is self hosted, if you do not trust them

Yes giving them the access keys is the biggest concern

All the rest is ok

Not sure if buying tf enterprise is worth it

You have to pay for it plus for its hosting

exactly

In that case, might be better to use open source terraform and atlantis

For cost and security reasons

yes

or use GithubAction hosted-runner

if you use github

I guess we just have to wait for the news that tf cloud was hacked and access keys compromised :)

Internal players are the biggest concern

that is a good point

Environment: Terraform: v0.12.13 local.zone is: nonprod.mydomain.com local.origin_dns is: origin.nonprod.mydomain.com local.origin_san is: [“main.dev.mydomain.com”, “origin.dev.mydomain.com”, “main.qa.mydomain.com”, “origin.qa.mydomain.com”, “main.uat.mydomain.com”, “origin.uat.mydomain.com”]

Issue 1: First run: everything is created when I look in the portal, but the TF output says:

Error: 6 errors occurred:
    * missing main.dev.mydomain.com DNS validation record: _654c27fb72c292a9967470a6a4c10216.main.dev.mydomain.com
    * missing origin.qa.mydomain.com DNS validation record: _88c3a0c95150482d39451905c3703883.origin.qa.mydomain.com
    * missing main.uat.mydomain.com DNS validation record: _f5a5dd6ef183a5104c5e65cc301a27d0.main.uat.mydomain.com
    * missing origin.uat.mydomain.com DNS validation record: _2527e37bcae7b44a3d28db696f258b84.origin.uat.mydomain.com
    * missing main.qa.mydomain.com DNS validation record: _c1f13019befd352bcee1787e92f43a0b.main.qa.mydomain.com
    * missing origin.dev.mydomain.com DNS validation record: _b2eda108e906975e5775231a8b92dcbf.origin.dev.mydomain.com

Issue 2: Second+ run: I am getting a Cycle error and all of the records:

Error: Cycle: module.cert.aws_route53_record.default (destroy deposed 0aea8e5c), module.cert.aws_route53_record.default (destroy deposed 0eee4568), module.cert.aws_route53_record.default (destroy deposed 8317334a), module.cert.aws_route53_record.default (destroy deposed cfd8b435), module.cert.aws_route53_record.default (destroy deposed 8741e753), module.cert.aws_route53_record.default (destroy deposed 44a3ff11), module.cert.aws_acm_certificate.default (destroy deposed 312e6bad), aws_lb_listener.front_end_https, module.cert.aws_route53_record.default (destroy deposed bad2ac93)

anyone have any ideas? thanks in advance!

Hrmmmm! so looks like you’re using the HCL2 version of the module

that one i know fixed a bunch of problems we had with tf-0.11

yeah, i’ve tried many adjustments to it, and all roads lead back to the above “errors”

I haven’t seen this one before…

have you tried (for debugging) to reduce the configuration to the smallest working one?

then go up from there?

resources are created on run 1, but it errors out, Cycle error with destroy deposed from the lifecycle on the cert

yeah, i’ve done a debug TF with just the module and variables involved.. no joy

i’m going to see if i can increase the create timeout on validation by adding this to the aws_acm_certificate_validation resource:

  timeouts {
    create = "60m"
  }

looks like that’s an option, not sure if it’ll work.. i already had to increase retries and decrease parallelism.. this thing makes a slew of route53 API calls

moment of truth…

getting all the Creation complete after 42s and watnot… waiting for the error at the end.

no joy:

Error: 6 errors occurred:
	* missing origin.uat.mydomain.com DNS validation record: _2527e37bcae7b44a3d28db696f258b84.origin.uat.mydomain.com
	* missing main.qa.mydomain.com DNS validation record: _c1f13019befd352bcee1787e92f43a0b.main.qa.mydomain.com
	* missing origin.dev.mydomain.com DNS validation record: _b2eda108e906975e5775231a8b92dcbf.origin.dev.mydomain.com
	* missing main.uat.mydomain.com DNS validation record: _f5a5dd6ef183a5104c5e65cc301a27d0.main.uat.mydomain.com
	* missing origin.qa.mydomain.com DNS validation record: _88c3a0c95150482d39451905c3703883.origin.qa.mydomain.com
	* missing main.dev.mydomain.com DNS validation record: _654c27fb72c292a9967470a6a4c10216.main.dev.mydomain.com

all the records were successfully created, tho

@John H Patton did you loom at the example https://github.com/cloudposse/terraform-aws-acm-request-certificate/blob/master/examples/complete

it gets provisioned on AWS and tested with terratest https://github.com/cloudposse/terraform-aws-acm-request-certificate/blob/master/test/src/examples_complete_test.go

yeah, i have looked at so much terraform all over the place, those examples included… i’ll revisit that, however… perhaps i’ve missed something

yeah, i have… the equivalent for me would be the following tfvars:

region = "us-east-2"

namespace = "na"

stage = "nonprod"

name = "nonprod-zone"

parent_zone_name = "nonprod.mydomain.com"

validation_method = "DNS"

ttl = "300"

process_domain_validation_options = true

wait_for_certificate_issued = true

mydomain is a placeholder, of course

and my san is different

let me try using wildcards… haven’t tried that yet

oh, so the module works only for star subdomains in SAN

[domain.com](http://domain.com) and *.[domain.com](http://domain.com) will have exactly the same DNS validation records generated by TF

not-star subdomain will have its own different DNS validation record

the module does not apply many DNS records

for many different issues we had with that before

[domain.com](http://domain.com) and *.[domain.com](http://domain.com) will have exactly the same DNS validation records generated - this is how AWS works

so, how would i create a cert with the following: CN: origin.nonprod.mydomain.com SAN: origin.dev.mydomain.com origin.qa.mydomain.com origin.uat.mydomain.com

i mean, i see the validation records being created correctly, and in the portal the cert is in the issued state

but, i can’t seem to use the cert

as for the many DNS records… yeah, i’ve run into that issue already… seems to be a rate limit on route53 that causes issues, i needed to reduce the parallelism to 2

i think i’m going to need to create the cert separate from the rest of this infra

multi-level subdomains are not supported in a wild-card cert

you have to do something like https://www.sslshopper.com/article-ssl-certificates-for-multi-level-subdomains.html

if you use the module, you’ll have to request many wildcard certificates, one for each subdomain DNS zone

well, i’m not sure how to make the hardcoded domains in the SAN work right… it’s driving me nuts.. been fighting this for over a week

I figured it out, finally! This is an array of an array:

  distinct_domain_names = distinct(concat([var.domain_name], [for s in var.subject_alternative_names : replace(s, "*.", "")]))

It outputs:

distinct_domain_names = [
  [
    "main.nonprod.mydomain.com",
    "origin.nonprod.mydomain.com",
  ],
]

This needed to be flattened, then I can work with it in other variables/data sources correctly

  distinct_domain_names = flatten(distinct(concat([var.domain_name], [for s in var.subject_alternative_names : replace(s, "*.", "")])))

that took entirely way too long to figure out

I don’t think so

there

so I need two filter blocks?

no, that depends on your tags

if with one tag you can find what you need

then you just need only one

So, how do I fetch or specify the value of the “Name” tag?

Like so:

  filter {
    name   = "tag:Name"
    values = ["thevalue"]
  }

https://www.terraform.io/docs/providers/aws/d/autoscaling_groups.html#name

name - (Required) The name of the filter. The valid values are: auto-scaling-group, key, value, and propagate-at-launch.

You can’t use name = "tag:Name"

the tag Name has a capital N

by the way

ah, i see.. this is a slightly different mechanism.. one moment…

does it have a tag name ?

some resources allow for tag{} lookup filters

not all

even when I use this as a filter, I get nothing returned:

  filter {
    name   = "key"
    values = ["tag:Name"]
  }

  filter {
    name   = "value"
    values = ["*"]
  }

ah-hah.. here’s the thing… ASGs have funky tags:

  tag {
    key                 = "Name"
    value               = "somename"
    propagate_at_launch = true
  }

that;s what you’re matching against

if the tag:Name doesn’t work for the key, try:

 filter {
    name   = "key"
    values = ["Name"]
  }

  filter {
    name   = "value"
    values = ["thevalue"]
  }

can anyone test this on their own ASGs?

I’m getting nothing back on all these variations

what version of TF are you using? what version of the aws provider?

I found this from back in 2017: https://github.com/terraform-providers/terraform-provider-aws/issues/1909

TF 0.11

AWS provider ~> 1.0

that might have been it

I’ll try with latest AWS provider now

yeah, i’m using aws -> 2.35

give it a shot

Alright, I’m on 2.35 now

but im still not getting anything returned from the data lookup

are you sure you are running against the right AWS account and region ?

https://github.com/terraform-providers/terraform-provider-aws/issues/3534

account and region are fine, my state is finding everything else

this is also my first time using this data lookup

I got it to work, using only one filter

nice!!!

congrats

thanks for taking a look

Thanks… I blew both cache dirs away ~/.terraform.d and .terraform Same result.

Adding debug to the terraform Run I see this:

2019/11/13 14:23:05 [TRACE] HTTP client GET request to <https://releases.hashicorp.com/terraform-provider-okta/3.0.0/terraform-provider-okta_3.0.0_SHA256SUMS>
2019/11/13 14:23:05 [ERROR] error fetching checksums from "<https://releases.hashicorp.com/terraform-provider-okta/3.0.0/terraform-provider-okta_3.0.0_SHA256SUMS>": 403 Forbidden

possibly an upstream problem?

maybe try pinning to an earlier release of the provider

perhaps I need to harass hashicorp

https://releases.hashicorp.com/ This shows no sign of okta.

https://github.com/terraform-providers/terraform-provider-okta/issues/4

So I cloned the repo and built it and installed locally and got it working.

I know there’s also #terragrunt and #geodesic but I’m not at that point yet. Just trying to build a solid foundation in terraform for right now.

This is one way. https://charity.wtf/2016/03/30/terraform-vpc-and-why-you-want-a-tfstate-file-per-env/ We do something similar but split an env into multiple directories. Vpc, eks, services, etc.

Have you read Terraform: Up and Running 2nd Edition? It’s pretty much the book on how to organize your Terraform code.

Not yet but it’s on my to do list. I’ll move it higher up on the priority list

A wild PR appeared! https://github.com/cloudposse/terraform-aws-eks-cluster/pull/34

Need more coffee…

Great to hear, thanks

Nice!

working example here https://github.com/cloudposse/terraform-aws-ecs-web-app/tree/master/examples/complete

Terratest for the example (it provisions the example on real AWS account) https://github.com/cloudposse/terraform-aws-ecs-web-app/blob/master/test/src/examples_complete_test.go

We use ECS for 90% of our apps (Fargate specifically) and it works great.

and make var.create_custom_domain a bool so you don’t need to compare to a string var.create_custom_domain == "true"

Thanks @Bruce & @Andy - looking into it today, will update here.

So it turns out the KMS encryption key was causing the failure, once I disabled that it ran just fine. Thanks again! off to other errors!

Sort of. I found that it tries to exit safely and this issue would normally happen when I try to exit with CTRL+C and don’t allow it to safely terminate

also, if there is any issue with a provider (bad config, etc.), TF (which written in Go) will just panic and exit immediately leaving you with some resources created on AWS but not in the state

there are no templates , but you can just copy the module (including the examples if you need it) and update in place, including README.yaml and all TF code (that’s what we usually do)

aws_api_gateway_domain_name.domain[*].domain_name is a list of strings

you have to iterate it to get each item as string

so, using [count.index] ?

I think I’ve almost got it

resource "aws_api_gateway_domain_name" "domain" {
  count           = var.create_custom_domain ? length(var.domain_names) : 0
  certificate_arn = var.aws_acm_certificate_arn
  domain_name     = var.domain_names[count.index]
}

resource "aws_route53_record" "domain" {
  count   = var.create_custom_domain ? length(var.route53_record_types) : 0
  name    = aws_api_gateway_domain_name.domain[count.index].domain_name
  type    = var.route53_record_types[count.index]
  zone_id = "xxxx"

  alias {
    evaluate_target_health = false
    name                   = aws_api_gateway_domain_name.domain[count.index].cloudfront_domain_name
    zone_id                = aws_api_gateway_domain_name.domain[count.index].cloudfront_zone_id
  }
}

this works, but the r53 domain only creates an AAAA record, not the A record for one of the gateway domains. What am I missing?

what’s in var.route53_record_types?

any ideas

Does var.domain_names have two items?

By the way they are written, those two resources will create only A record for the first domain, and only AAA for the second domain

oh, thats not what I want at all, I want an A and AAAA for each domain