#terraform (2021-10)

automatically generated …. as long as the team wrote their cloudformation support

Oh, I guess something not clear here

any resource type published to the CloudFormation Public Registry exposes a standard JSON schema and can be acted upon by this interface

the funny bit is that there are also resources there for running terraform…

this sounds amazing

this is pretty much how ARM has worked on Azure for ages - kind of funny that AWS finally went there

basically it will be APIs that are well structured and defined generically enough that you can pretty much generate the required bits for cloudformation off the top of them

i was chatting with Glenn Gore 3+ years ago about this (and also the ui standardisation across services) so it’s been a long time coming

I’m not very good at WAF rules, but it seems like WAF can not inspect the URL itself, only the UriPath. (I may be very wrong here though, but I was unable to find it possible to inspect the server name itself).

So, in case you use the same load balancer / cloudfront distribution / or whatever it is, for multiple domain names, I’d suggest finding a way to pass the server name as a header through to WAF, and then use the Single Header inspection statement to match for the server name. Then you combine and_statement, or_statement to get what you want.

Possible Rule priority 0 (if the Server-Name header is present in the request): • this rule will ALLOW request if server name is xx.production.url1.com or yy.production.url1.com and IP is from the allowed list

action {
   allow {}
}

statement {
   and_statement {
      or_statement {
         byte_match_statement {
           positional_constraint = "CONTAINS"
           search_string         = "xx.production.url1.com"

           field_to_match {
             single_header {
                name = "Server-Name"
             }
           }

           text_transformation {
              priority = 0
              type     = "NONE"
           }
         }      

         byte_match_statement {
            positional_constraint = "CONTAINS"
            search_string         = "yy.production.url1.com"

            field_to_match {
               single_header {
                  name = "Server-Name"
               }
            }

            text_transformation {
                priority = 0
                type     = "NONE"
            }
         }      
     }

     ip_set_reference_statement {
        arn = ""
     }
  }
}

the Rule priority 1 • will BLOCK all requests to xx.production.url1.com and yy.production.url1.com

action {
   block {}
}

or_statement {
         byte_match_statement {
           positional_constraint = "CONTAINS"
           search_string         = "xx.production.url1.com"

           field_to_match {
             single_header {
                name = "Server-Name"
             }
           }

           text_transformation {
              priority = 0
              type     = "NONE"
           }
         }      

         byte_match_statement {
            positional_constraint = "CONTAINS"
            search_string         = "yy.production.url1.com"

            field_to_match {
               single_header {
                  name = "Server-Name"
               }
            }

            text_transformation {
                priority = 0
                type     = "NONE"
            }
         }      
}

Rule priority 2 • ALLOW requests with production in the UriPath (this rule will be executed after the first 2, so your conditions are satisfied)

action {
   allow {}
}

         byte_match_statement {
           positional_constraint = "CONTAINS"
           search_string         = "production"

           field_to_match {
             uri_path {}
           }

           text_transformation {
              priority = 0
              type     = "NONE"
           }
         }   

Hope this helps even if just a little

Again, I’m not very good at WAF rules and if any of the nested statements don’t work, then a different strategy should be used. But basically, try breaking all rules into the smaller ones, and come up with an efficient strategy of prioritizing Allow / Block rules to get what you need.

@Almondovar

I have no words to thank you enough, tomorrow morning i am going to try that out and let you know

Happy to help! Please let me know if you will have more questions, we’ll try to find a solution together.

Hey @Almondovar - to make things simple, the server name will be in the Host header, so this is the one you will want to match with the server names.

Unless HTTP 1.0 (which I would suggest to avoid), the Host header is a must for HTTP 1.1 and HTTP 2, so you should always have it in the request.

1. use a variable to hold a list of the repos you want to create; it would be a list of string like ["repo1","repo2"]
2. Create your repo resource with a for_each to loop over the set of the repo list like

   for_each = toset(var.repolist)
   

3. In the policy resource, use the same loop over the repo resources for the repository assignment.

i will try to get demo code in a bit.

Tqs

Be helpful to see the inputs to resolve the first error. On the second, when using bridge mode you’re using the ecs host’s IP for the task and differentiating via port. The TG cannot register multiple tasks via IP because in bridge mode they share the same IP as the host. You thus need to change the TG type to instance.

The second is definitely more obvious than the first. Let me get the inputs I am using. I am using a yaml decode to a map and then iterating through the values in the map. Let me get the layers here.

My guess is the container port or port mappings variable is incorrect. Do either of those get used to build the security group rule?

I don’t believe so. In the terraform-aws-ecs-alb-service-task this is what it looks like ^

If I set the network_mode to awsvpc it will deploy the service but then it just doesn’t want to allocate to the ecs instance that is available on the cluster.

it seems to me that the following code: https://github.com/cloudposse/terraform-aws-code-deploy/blob/master/main.tf#L177-L184 has the ec2_tag_filter and ec2_tag_set the wrong way around - the lookup should be for ec2_tag_filter object and our content needs to be the ec2_tag_set. If you can confirm this, then I’ll create a PR or pehaps you can make this change directly for me?

it’s a really nice and complete module - thank you - I was going to write my own until I looked at the Registry and saw this lurking.

@Stephen Tan I admit it’s been a while since using this module personally

I can’t look into the problem right now that your facing

it’s ok - I’ve got something working. If you can accept a PR then I’ll create one

@RB can you review https://github.com/cloudposse/terraform-aws-code-deploy/issues/6

I’ve created a PR to fix this issue here: https://github.com/cloudposse/terraform-aws-code-deploy/pull/7

Do I get the job?

oh lol i just created a pr too

haha

take your pick!

let’s go with yours

did you try yours out locally ?

yes, it works, but yours is simpler. I didn’t spot the list of list until later

I think I have some dyslexia occaisionally

lol, no worries

not sure why we use the lookup or even how it works tbh which is why I removed it

lookup will look up a key and resolve it to a null if the key isnt there

unfortunately since terraform doesnt allow optional keys (yet), the code is not super useful

but at some point, the optional input object keys will be available and then the lookups will be more useful

happy to use anything that works tbh

the code is merged and released as 0.1.2. let us know if this works

thanks again for the contribution!

thanks!

yes, that works now, although when applying, roles won’t attach, but I’ll debug that now

@RB - I’ve created a new PR here: https://github.com/cloudposse/terraform-aws-code-deploy/pull/9/files - not sure why my old commits are included in the PR though. I did a merge from upstream so I don’t get it.

is that required in all cases? what if you didn’t want to use the service role?

hi - sorry for being so late to reply. Yes, I see, so the structure of the ARN will vary depending on your platform - ie ECS, EC2 or Lambda. Your ARN is constructed with just the ECS case in mind and this role assignment will fail if you try to use with EC2 or Lambda. https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-service-role.html “The managed policy you use depends on the compute platform” the possible policy ARNs are:

aws iam attach-role-policy --role-name CodeDeployServiceRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole

,

aws iam attach-role-policy --role-name CodeDeployServiceRole --policy-arn arn:aws:iam::aws:policy/service-role/AWSCodeDeployRoleForLambda

,

aws iam attach-role-policy --role-name CodeDeployServiceRole --policy-arn arn:aws:iam::aws:policy/AWSCodeDeployRoleForECS

I’ve updated the PR - sorry it’s so messy. I’ve also made another EC2 related change to do with making tagging work properly https://github.com/cloudposse/terraform-aws-code-deploy/pull/9

I’m not sure how this would work tbh, and I’m wondering whether it’s above and beyond what the module is intended for.

I guess I’d have to add additional CIDRs to the VPC and use another CIDR the second time I call the dynamic subnet module?

(i have not used the module…just brainstorming…)

is it possible to call the module once, and then reference those subnets as data where you need them again?

In my own use (again, without the module) I create a VPC in its on state and then use data resources to pull in the subnets. so only one module (the VPC module) is really trying to create subnets.

In the end I got around the problem by adding an additional cidr and then passed that into the dynamic subnet module. But, I suspect I’ll also need to do something to let both CIDRs talk to each other? Networking is not by bag.

Maybe some package manager like https://gofi.sh/ and food https://github.com/fishworks/fish-food/blob/main/Food/helmfile.lua ( just an example )

This one is also interesting https://github.com/mumoshu/shoal#history-and-context

Or can https://registry.terraform.io/providers/tenstad/remotefile/latest/docs work ?

hrmmm ya we had this problem with terraform-provider-helmfile

which depends on a lot of binaries

#office-hours

@Erik Osterman (Cloud Posse) thanks just saw the officehours video

last update was in August so not dead yet

which PRs are you trying to get reviewed ? 55 and 56 ?

ipv6 one really, it needs eyes on it.

It works. I’m just not sure if there was a philosophy there with how private vs public work. Also considering how ipv6 works in general.

I applied it against public only, but technically if we pass a igw variable it could be applied to private as well (though that would make it not private anymore).

That’s #55

left some comments on #55

Thanks! I’ll review

that error is coming from the aws api itself

see this https://docs.aws.amazon.com/AmazonECS/latest/userguide/fargate-task-defs.html

cc: @Gerald

you are going to have to be a bit more specific

what are you trying to do?

are you using TF to hit the github api?

i am trying to use terraform to manage my github repository creation and i am using codebuild and codepipeline as my cicd tools

so its throwing this in codebuild

when i run this locally it does not throw this

but recently i decided to set the terraform plan –refresh=false

to reduce the amount of time that it tries to refresh github to reduce the api calls. this worked by the way but its also bad should incase someone makes a change on the aws console and i am not aware

yes, dealing with their rate limits is hard

effects of rate limits could also be exasperated during development and not an issue during normal usage

so i was able to by pass this by running. This kind of will reduce the amount of API calls made to git.

terraform plan --refresh=false 

@Almondovar unfortunately, the statement part is required as per Terraform documentation:

statement - (Required) The AWS WAF processing statement for the rule, for example byte_match_statement or geo_match_statement. See Statement below for details.

What’s good though, is something to consider: • the WAF will allow the request if it is not caught by any blocking rules This means that you do not need an “allow-all” rule with empty statement if you design your blocking rules so that they catch all wrong requests; everything else will be auto-allowed.

Good morning @Constantine Kurianoff thank you once more time! indeed it works different than the classic cisco firewall rules i was used to, have a nice day

The most helpful output now would be the raw error message

This code working fine, I have the issue when I change it to s3 backend

I have the issue when I change it to s3 backend so thats not working fine - that’s what should have an error message = )

the error message is pretty self-explanatory

what are you having trouble with?

The question is: How do I create the read replica? I’ve already changed the cluster size to 2, and enabled automatic_failover…

And i do not want to enable cluster_mode… basically how do I create read replicas in the other AZ’s so that the failover works?

are you sure you set cluster size to 2? It’s set to 1 in dev

you’re right… that is my “final” code but on my tests I had 2 for both dev and prod

so you’re saying that it should work right? I can try to test again so tee if it works… maybe I did something wrong. Will also try using 0.40 just to see if there is any difference.

this will be another major differentiator from cloudformation

will bring this up next office hours

You attach launch templates to autoscaling groups, not loadbalancers

i am wondering if i need to do something clever with https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group#resources

for #2, check out https://www.youtube.com/watch?v=V2b5F6jt6tQ

for #1

Security scanning is graciously provided by Bridgecrew. Bridgecrew is the leading fully hosted, cloud-native solution providing continuous Terraform security and compliance.

#3 what is SO? (Stackoverflow? )

in any case, all Cloud Posse modules are used by many companies on TFC and Spacelift (https://spacelift.io/)

the modules are combined into components, which are deployed by terraform/TFC/Spacelift etc.

https://github.com/cloudposse/terraform-aws-components

https://github.com/cloudposse/terraform-aws-components/tree/master/modules

components are combined from many modules (and/or plain terraform resources)

Thanks @Andriy Knysh (Cloud Posse)

SO = SweetOps

Regarding using Spacelift / TFC…

Some context to begin with… Each business unit has an environment in AWS. There are 4 environments (sandbox, dev, qa and prod). Today, we are not using workspaces but plan to use them soon… In Scalr (TFC alternative) each “environment” is tied to a single AWS account. We are starting to adopt OPA…

I’ve been considering pushing all Security Groups into a single workspace. The SGS would be stored in git within a YAML multi document file, to list SGs per application or service that needs one, along with the SG rules. This file would list all SGs for the entire organization.

Why do it this way?

(a) to create transparency in git for security and devops teams (b) allow appdevs to create a PR but not allow them to merge. The devops and security teams would manage the PRs and merging of release branches. (c) upon approval of PR, automation will run and convert the YAML to a single TF file, which will reference the SG module. Gitops will then push the changes up via TF, into the single BU specific workspace (d) prevent appdev teams from adding custom SGs in custom TF module code (OPA will automatically block any code that references an SG outside of a data lookup). In this single workspace model, they’ll have to subscribe to their custom SG via the workspace within their organization (TFC) / environment (Scalr). (e) We are starting to deal with network micro segmentation. We are going to tackle WAF and SGs to start with. We need a layer of separation and control prior to TF being deployed.

I know that is a mouthful. But, I’m bringing this up because I noticed that SO also has a module for SGs. I’m wondering if this concept and the reasoning behind it is a common pattern now in devops circles?

Thanks in advance for even reading this. A double thanks for adding any tips, advice or corrections

@Andriy Knysh (Cloud Posse) ^^ Do you have any thoughts on this?

@Elvis McNeely I like that idea to have SGs in YAML files

in fact, we’ve been moving in that direction on many of our modules

we call it the catalog pattern

https://github.com/cloudposse/terraform-aws-service-control-policies/tree/master/catalog

https://github.com/cloudposse/terraform-datadog-platform/tree/master/catalog

we define all the resources as YAML files, and then use them in other modules as remote sources

using this module to read the YAML files and deep-merge them (if needed deep-merging) https://github.com/cloudposse/terraform-datadog-platform/blob/master/examples/complete/main.tf#L1

or just simple code like this https://github.com/cloudposse/terraform-datadog-platform/blob/master/examples/synthetics/main.tf#L2 if no deep-merging is needed

“we define all the resources as YAML files, and then use them in other modules as remote sources”

What do you mean by “other” modules? I assume some of those configs in the yaml files are defaults? Remote sources are referencing them?

other modules load the YAML as remote sources and convert them to terraform and apply

Thanks for sharing @Andriy Knysh (Cloud Posse). I know this is a very basic question, but I want to understand a bit more. I see in your TF modules you are using:

source = “../../modules/synthetics”

before TF runs, are you pulling in all modules to the local directory? I would like to simplify this process on my end, a bit more. I wish the provider block would allow us to list a manifest of “modules”, within a TF template, something like this:

From:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 2.7.0"
      configuration_aliases = [ aws.alternate ]
    }
  }
}

To:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 2.7.0"
      configuration_aliases = [ aws.alternate ]
    }
  },
  required_modules {
    synthetics = {
      source  = "<https://github.com/cloudposse/terraform-datadog-platform.git>"
      version = ">= 2.7.0"
    }
  }
}

Ok

What do you use to convert the yaml to TF? I’m wondering if you use an OSS package or have you created something internally?

the examples show just the local path loading

but the YAML files can be loaded as remote files

https://sweetops.slack.com/archives/CB6GHNLG0/p1634573772461000?thread_ts=1634341554.443400&cid=CB6GHNLG0

What do you use to convert the yaml to TF? I’m wondering if you use an OSS package or have you created something internally?

Ok, I’ve done something similar in the past. But, I ran into issues. Like, DR. When setting up DR for two different regions, TF loops don’t operate in provider blocks (as of late) which forced me to write provider + module blocks from yaml. Have you hit that kind of issue? If so, how did you solve it?

yes, to deploy into many regions using one module with one terraform apply, the provider blocks need to be created manually or generated from templates - this is an issue with TF providers

That’s good to know. I was frustrated when I discovered that issue, but I will just make a plan to handle provider blocks in the yaml conversion.

Can you explain why / how you guys came to the conclusion to use this.context? It’s an interesting concept, I’m wondering what brought you to the point of doing this for all of your modules?

to not repeat the same common inputs in all modules (and we have more than 130 of them)

the inputs were defined somehow differently in each modules

now they are all the same

like region, name, business_unit etc? Var names that all modules need?

reuse the common code to make fewer mistakes

we don’t have all of those as common inputs (no business_unit, we have tenant for that), we have namespace, stage, tenant, environment, name, attributes`

see the null-label module

but you can use the same pattern and use your own inputs as common, e.g. business_unit

Right, but those inputs are common across all of your modules? In our case, we require a set of 12 vars that also translate to tags

common across all our modules

I really appreciate your time answering my questions

they also used by null-label to generate unique and consistent resource names

so the label and context solve two issues:

Have you used Scalr? I know you guys did a great videocast comparing TF statefile tools. I’m wondering if anyone (or you guys) have tested using your modules in Scalr? They have a slightly different model compared to spacelift

1. Unique and consistent resource names

1. Consistent common inputs across all modules and components

Have you used Scalr?

1. is the label module? 2. Is for context?

no

I landed on the SweetOps docs https://docs.cloudposse.com/fundamentals/concepts/ Which cover a lot of what we discussed today. Thanks again for your time

@Andriy Knysh (Cloud Posse) I’ve spent some time reviewing everything you shared. The concept of components, forces all of the IaC to be in a single state file, meaning you aren’t using TF workspaces for major pieces of infra. Is that a deliberate choice? We are attempting to move away from have such large stacks in a single statefile as we found it to be difficult to manage within operations.

each component state is in a separtae state file

Oh, I think I missed that. So a single TF execution loops over components and does a TF plan / apply?

if we use S3 backend, we use the same bucket, but the state files for each component for each environment/stack (e.g. prod in us-west-2, dev in us-west-2, etc.) are in separate state files in separate bucket folders (TF workspace prefixes)

So a single TF execution loops over components and does a TF plan / apply?

we deploy each component separately, either manually or via automation like Spaxelift

Hmmm, In the example found here: https://docs.cloudposse.com/fundamentals/concepts/

A single deployment is deploying a single component found in that example yaml file?

deploying a single component, yes

I guess spacelift has a way of specifying which component to deploy…

a component is a high-level concept combining many modules and resources

Right

for eample, we have eks, vpc, aurora-postgres components, each consisting of many of our modules (and plain TF resources)

I guess spacelift has a way of specifying which component to deploy…

for each component in each stack (by stack I mean environment/stage/region), we generate Spacelift stacks

so we have Spacelift stacks like uw2-prod-vpc and uw2-dev-vpc and ue2-staging-eks etc.

Ah, so there is a process that does that outside of spacelift?

which deploy the corrsponding component into the corresponding environment/stage/region

Ah, so there is a process that does that outside of spacelift?

yes

https://github.com/cloudposse/terraform-spacelift-cloud-infrastructure-automation - module to deploy Spacelift stacks (via terraform)

https://github.com/cloudposse/terraform-provider-utils - TF provider that converts our infra stacks into Spacelift stacks

https://github.com/cloudposse/terraform-provider-utils/tree/main/examples/data-sources/utils_spacelift_stack_config

so

1. We define all our infra stack configs (vars, etc.) in YAML config files for each env/stage/region, separating the logic in components Terraform from the configuration (vars for each env/stage) in YAML config files, so the components themselves don’t know and don’t care where they will get deployed)

1. The the provider iterates over the YAML configs and creates Spacelift stack config from it

1. Then the Spacelift module deploys the Spacelift stacks into Spacelift

1. Each Spacelift stack is responsible to deploying a particular component into particular env/stage/region, e.g. uw2-prod-eks Spacelift stack will deploy the EKS component into us-west-2 region into prod environment

In step 2, so TF run is pushing a “stack” over to spacelift per iteration. That stack may just sit there until someone is ready to use it

Step 3, comes at a later date, as need by a consumer?

we deploy Spacelift stacks for each combination of component/env/stage/region first

Step 4, so spacelift is really dealing with pushing different components into different workspaces

then those Spacelift stacks deploy the components into the corresponding env

Ah

Ok, I see

So spacelift stacks has a configuration of what component goes where

yea, we provision the Spacelift stacks with terraform, then those Spacelift stacks provision the infra stacks with terraform

same applies to TFE

So spacelift stacks has a configuration of what component goes where

I wasn’t aware that TFE supported this concept.

they have, after the module and provider parce our YAML config files and generate Spacelift stacks from them

I wasn’t aware that TFE supported this concept.

I know you can create a single workspace and push a TF apply to that workspace, but, I didn’t think TFE couple push multiple TF components to different workspaces

TFE resources can be deployes with terraform as well

But a single TF run, in TFE, can’t push different portions of the infra to different workspaces, can it?

(I was working with TFE a year ago and don’t remember all the concepts now, and sure they have updated a lot). We are working with Spacelift now, and generate all the stacks/configs by our provider and module

Ok. We are moving into Scalr, I would like to use some of these SweetOps concepts into this next gen TF process we are building

yea, just remember that we do these main steps in general:

1. Create terraform components (they don’t know in what env/stage/region) they will be deployed

1. Create YAML config files for each component with vars for each env/stage/region

1. Our provider and module parse the YAML configs and generate Spacelift stacks for them - Spacelift stacks are separate for each combination of component/region/env/stage etc.

1. Provision the generated Spacelift stacks with terraform

1. Each Spacelift stack now can plan/apply the corresponding infra stack

Thank you, that’s really helpful.

use for_each on the module to create multiple instances of it? then module.test will contain the outputs from all of them

woahh that’s a good plan. i’ll test that out.

oh actually i forgot.. another wrench in things… they are using different AWS providers

oh, then i’d combine them using a local or whatever other expression

nice! i’ll try that

e.g.

output "my_outputs" {
  value = {
    test1 = module.test1.my_output
    test2 = module.test2.my_output
    test3 = module.test3.my_output
  }
}

I was hoping to do something cheeky like

output "test" {
  value = module.["test1*"].my_output
}

unfortunately, no, you can use an expression in the static part of a label, e.g. module.<label>

sad

closest you can get is maintaining a list of the modules…

locals {
  modules = {
    test1 = module.test1
    test2 = module.test2
    test3 = module.test3
  }
}

output "my_outputs" {
  value = { for label, module in local.modules : label => module.my_output }
}

In case anyone in the future stumbles upon this in an archive… :sweat_smile: I achieved a sub-optimal solution to address my root issue.

I wrote a quick bash script in my Makefile to do some regex magic and make sure all the modules are present in my outputs.tf file.

check_outputs:
	@for i in `grep "^module " iam-delegate-roles.tf | cut -d '"' -f 2 | sed -e 's/^/module./g' -e 's/$$/.iam_role_arn,/g'`; do \
 		grep -q $$i outputs.tf || { echo -e "****FAIL**** $$i is not present in outputs.tf\n****Add this output before committing.****"; exit 1; }; \
 	done

Then loaded that script into a pre-commit hook:

  - repo: local
    hooks:
      - id: check-outputs
        name: All Role ARNs in outputs.tf
        language: system
        entry: make check_outputs
        files: outputs.tf
        always_run: true

https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ac.html

@idan levi If you’d like to authenticate using only the Internal User Database of Elastic(Open)Search, then you could use the following block inside aws_elasticsearch_domain resource:

  advanced_security_options {
    enabled                        = true
    internal_user_database_enabled = true

    master_user_options {
      master_user_name     = <MASTER_USERNAME>
      master_user_password = <MASTER_PASSWORD>
    }
  }

This will create a main (“master”) user for you inside ElasticSearch domain and you’ll be able to authenticate with just username and password when accessing Kibana (or when accessing ElasticSearch APIs)

This seems to work, but feels hacky and not an intended method. It at least lets me move forward though!

name = "results-test"
delimiter = "."
attributes = ["fifo"]

Deleted everything by hand including state and the issue is resolved

In my case it’s usually an auth error to a provider or something. I used TF_LOG=DEBUG to figure those out

Replace name prefix with a hardcoded name

Or lifecycle ignore_changes

Thanks @Alex Jurkiewicz. This would mean that name_prefix’d resources can’t really be imported properly?

I guess lifecycle ignore_changes suits me the best atm.

btw, what resource is it?

here’s a related open aws provider issue https://github.com/hashicorp/terraform-provider-aws/issues/9574 but it’s possible you tripped over a new bug

looks like only the importation of the aws security group resource works but not for any other resource, according to the checklist in the ticket

it was an rds parameters group in my case today

ah ok makes sense. at least that resource is on bflads list. hopefully they complete it soon

can anyone help me in this

Think of it like this: nodes will be randomly killed. With a termination handler, there will be a clean termination process (move all the pods off the node, respect Pod Disruption Budgets, and so on)

I think if we use the min_healthy_percentage = 90 if we have 10 nodes, it will just bring down only 1 node at the time, correct?

instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 90
    }
    triggers = ["tag"]
  }

i mean, sure, ths 1 node will be randomly selected, but when we got timewindow for outage anyway, it wont create any issues right?

I think it would be 1 node at a time, yes.

Also, those are nodes defined as “healthy” from the ASG standpoint. That means they may still be starting your pods, for example. It depends on how interruption-friendly your apps are.

This question is more about what happens to your workload if they’re not getting gracefully terminated. Also note that for most eg. most of the ingress controllers (working on k8s endpoints directly) it’s important that you’re somehow doing graceful shutdowns of your backend services including a preStop hook to wait a few seconds to ensure the k8s endpoint controller is able to adjust the k8s endpoints before the backend has been shutdown. So you’ll most likely see some hickups, like failing inflight requests if you don’t handle the shutdown.

Some more background can also be found here https://kubernetes.io/blog/2021/04/21/graceful-node-shutdown-beta/

unfortunately dynamic lifecycle blocks are not permitted in terraform as of yet

Yeah, so there is basically no way to properly use this module without the autoscaler, right? Maybe a disclaimer would help then

have you tried setting enable_autoscaler to false?

yeah I mean, that’s exactly the point :smile: I’ve turned off enable_autoscaler which also is disabled by default. the table get’s created using the initial values for read/write due to https://github.com/cloudposse/terraform-aws-dynamodb/blob/master/main.tf#L50 but after that you cannot change them anymore using terraform as they’re ignored by lifecycle . The only idea I had to fix this is having 2 more or less identical aws_dynamodb_table . One with count = local.enabled && var.enable_autoscaler ? 1 : 0 which includes the lifecycle and one with count = local.enabled && !var.enable_autoscaler ? 1 : 0 which runs without the lifecycle

ooof I see what you mean.

We’ve done this in https://github.com/cloudposse/terraform-aws-ecs-alb-service-task but it can be a real problem as someone will always come up with a new lifecycle ignore iteration to create yet another resource. There is a PR being worked on that will output the logic of the module to build your own custom resource. But… there must be a better way to do this while still taking advantage of what the module has to offer

@Thomas Eck could you try simply tainting the dynamodb table resource for now anytime you want to update the capacities ?

then run another plan ?

ehm I mean, using terraform taint would force terraform to destroy and recreate the table which is definitely not what we want here of course this would work as it would get recreated with ne settings put as you cam imagine that’s not feasible. I see the point regarding the lifecycle iterations but tbh. I don’t think it would be that much of an issue for the dynamodb repo as the lifecycle there is clearly set to make autoscaling work

ah ok so taint wouldnt simply allow you to modify it in-line

for now, perhaps you could update the table manually since the lifecycle is ignored. not my favorite workaround

yeah I mean, that’s what I did but I’ll get a drift to my IaC and also have some manual interventions which cannot be reviewed and so on So if you guys are not up for a workaround like the one I ’ve proposed I would simply move away from the module as I can’t use it without the autoscaler which of course is not an issue, so don’t get this wrong Don’t want to blame or something like this but it would make sense to add a disclaimer then, highlighting this issue

I agree. That would be a good disclaimer in the interim.

Try searching the archive, http://archive.sweetops.com

Good idea, strangely, not there (perhaps this only archives the office hours?)

i’m sure it’s in there somewhere

I tried to search for some stuff in the archive some days ago. I definitely knew that the thing I was looking for had been discussed. No luck however:( And I gave up

@Erik Osterman (Cloud Posse) the archive is a daily sync?

that’s a great question!

this module was written before dynamics could be used on blocks

you’re right. we should update this so the first dynamo db resource supports both options without having to create a separate resource

created a PR for Fix/dynamodb encryption logic and got one issue from bridgecrew bot :

LOW   Ensure DynamoDB tables are encrypted
    Resource: aws_dynamodb_table.with_server_side_encryption | ID: BC_AWS_GENERAL_52

if you could take a look:

https://github.com/cloudposse/terraform-aws-tfstate-backend/pull/107#discussion_r754267296

It’s very much a matter of taste:

• CDK type technologies (AWS CDK, Terraform CDK, Pulumi…) are geared towards application developers wanting to build infrastructure. For example, if your company is big on Typescript, then Pulumi is great.

• My experience with CloudFormation is that it’s actually quite a bit behind Terraform in many ways, and the gap just keeps getting bigger (odd, right?). I would not recommend picking up CloudFormation these days if you’re starting fresh.

Thank you! I’ll keep these in mind when I make the decision

I had the assumption that Cloudformation would be doing better

You’d think, right?

I’m not a fan of cloudformation but AWS recently fixed a major limitation: cloudformation deployment can now resume from the last error point, instead of rolling back. This has always been a major pain in the r.e. for me.

CDK does get around the cloudformation limitation of re-use: nested cfn stacks were a pain, but the CDK uses inheritance (which is a bit of a hack IMO but it’s not bad one) to support breakdown a stack in to multiple smaller stacks.

But even then, in general terraform is still much better than CDK:

• has much better plans (tells you which parameter change is causing a resource to be replaced rather than just updated, this is huge)

• the concept of module is simple and efficient (I just wish that variable definitions did not have to be repeated up a module hierarchy)

• the ability to import is much more powerful than that of the CDK AFAICT (although I will be digging deeper into this shortly so this may be incorrect)

• tag propagation from provider default_tags is awesome, esp in conjunction with ignore_tags CDK has user-defined functions, which is the biggest thing I miss in terraform HCL2. Sometimes I have a small bit of logic that I want to use in several places but on different objects…, there is no way to refactor this code into a function that I could call like the builtin functions (and the workaround of defining a module requires way too much code and work IMO).

Terraform also has providers for pretty much anything that has an API, you can even use it to order pizza from Domino’s

You can get very creative with it

thank you all!

Yes although the gain re provides is in access to huge lib of existing providers; not in writing your own.

Because if you have to write your own, you’ll have to do it in Go if you use terraform, within a very specific framework for plugins (you’ll have to define schema for plugin and each resource etc - this can be a plus in some cases but not always).

Whereas if you use CDK or CDKtf, you can write any code you want, in any language supported by CDK/CDKtf, and using any SDK available for your provider in that language (so if the sdk for your service is not available in Go, you can still use it in your infra). In some cases this will be a big plus.

But CDK and CDKtf are on par on that front.

One place where CDK is currently better than CDKtf is in its library of higher level constructs. These are the programming language equivalent of tf modules for the big pieces like a vpc, and eks cluster (just look at any of the main eks modules in terraform registry, creating the aws_eks_cluster is the tip of the iceberg) etc.

Also CDKtf is not really considered production ready. So unless you’re just playing around with IAC, I’d say go with terraform, with some third-party modules for the big pieces. Eventually, those bigger pieces might be better supported in cdktf but this remains to be seen.