#terraform (2020-03)

@rms1000watt lolol

yessss, famous

#famous

just for the record, I’d recommend using serverless.com

way more features

and developers working on it

As bad ass as RMS’s tf stuff is I would +1 that.

Terraform && Lambda don’t play well together imo.

i think it’s the api gw that is hard. lambda+tf is easy and beautiful

serverless is alright as long as your api gw usage fits in their selection of defaults. it starts getting really restrictive and messy if their selections don’t fit your use case

i found the old code with the issues i ran into. they’re still not fixed unfortunately.

Terraform does not automatically update deployments after making changes to its resources: https://github.com/terraform-providers/terraform-provider-aws/issues/162

Terraform loses the usage plan configuration after updating the deployment resource: https://github.com/terraform-providers/terraform-provider-aws/issues/714

So, I use both serverless + terraform, I basically let serverless handle the Lambda and the API Gateway in front of it, everything else is handled by terraform. I use SSM as a way to pass ARNs and other needed values for terraform-managed resources to the serverless plan, as it can read from SSM when doing it’s deployments.

It does do things like circumvent any policy-as-code and other requirements you may have already built in to your terraform pipelines. You’d need to enforce those sorts of things through maybe config rules that examine the deployed resources or something.

Not a perfect solution, but it’s the best way I’ve found to marry the two concepts together and not block my devs working on serverless projects while still having a reasonable separation of responsibilities.

here’s the hack i used to get auto deployments working. i think it works but this is quite old so there may be better hacks now.

resource "aws_api_gateway_deployment" "thing" {
  lifecycle {
    create_before_destroy = true
  }

  depends_on = [
    aws_api_gateway_integration_response.x, # make sure these are chained in a way so that all resources get created before this deployment resource, otherwise you can get errors
    aws_api_gateway_integration_response.y,
    aws_api_gateway_integration_response.z,
  ]

  rest_api_id = "${aws_api_gateway_rest_api.thing.id}"
  stage_name  = "thing"
}

module "path_hash" {
  source = "github.com/claranet/terraform-path-hash?ref=v0.1.2"
  path   = path.module
}

resource "null_resource" "auto-deploy-thing" {
  count = var.auto_deploy ? 1 : 0

  depends_on = [
    aws_api_gateway_integration_response.x, # i think this is necessary too, so that it auto deploys at the end
    aws_api_gateway_integration_response.y,
    aws_api_gateway_integration_response.z,
  ]

  triggers {
    hash  = module.path_hash.result # ensures any code changes trigger a deployment
    one   = var.one # ensures any variable changes trigger a deployment
    two   = var.two
    three = var.three
  }

  provisioner "local-exec" {
    # AWS limits these requests to 3 per minute per account, sleep/retry when throttled
    command = "for attempt in 1 2 3 4 5 6; do aws apigateway create-deployment --rest-api-id=${aws_api_gateway_deployment.thing.rest_api_id} --stage-name ${aws_api_gateway_deployment.thing.stage_name} --description 'auto deploy ${aws_api_gateway_deployment.thing.stage_name}' && exit 0 || ex=$?; echo Failed attempt $attempt; sleep 15; done; echo Failed too many times; exit $ex"
  }
}

@Alex Siegman’s approach seems probably best

I used https://github.com/aws/chalice + Terraform but just hardcoding some of the references across the 2 projects, and I preferred that over defining API Gateway in Terraform.

Same concept, only I didn’t want to hardcode. There’s still problems with both approaches though, in that you can run in to issues changing dependent resources. Beyond that though, it’s worked well for us on a small scale so far

Yep it’s a bit of a house of cards solution

Yeah. . . mixing Serverless (CloudFormation) and Terraform. . .

It’s so ugly

But that’s where I am mentally right now

How about API Gateway resources in a CloudFormation stack managed by Terraform

You can pass in references from Terraform into the CFN template

I don’t know how painful API Gateway in CFN is though

CFN in Terraform is surprisingly good

heh, serverless is basically a generator for CFN, so use that to get the CFN template with api gateway resources, then parameterize the template for use with terraform!

API Gateway is painful period, thus the popularity of Serverless, Zappa and friends

you’re welcome to use this as a reference also… we’re using it as an internal module in the project, it’s probably not generic enough for a general purpose api gateway module… but, maybe it helps as an example of all the pieces! https://github.com/plus3it/terraform-aws-ldap-maintainer/tree/master/modules/api_gateway

nice

Is that your company/project?

plus3it is where i work, yes been working with the same core folks for a good 10 years, through different companies, but this one is basically ours

excellent, thanks

@randomy the source looks promising but i get this error

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

data.external.example: Refreshing state...

Error: command "python" produced invalid JSON: invalid character 'e' in literal true (expecting 'r')

  on main.tf line 1, in data "external" "example":
   1: data "external" "example" {

let me find an example

https://github.com/raymondbutcher/terraform-aws-lambda-builder/blob/master/main.tf#L5-L19

https://github.com/raymondbutcher/terraform-aws-lambda-builder/blob/master/validate.py

oh wait, that one doesn’t return anything useful

look at main.tf and zip.py in https://github.com/raymondbutcher/terraform-archive-stable

interesting! so the command just needs to return json and the result can be shown

brilliant

✗ cat main.tf               
data "external" "example" {
  program = ["echo", "{\"a\": \"b\"}"]
}

output "value" {
  value = data.external.example.result
}
✗ tf apply
data.external.example: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

value = {
  "a" = "b"
}

yep, cool just be mindful that whatever you run there will become a requirement for using your terraform project. so if you use a ruby script, for example, then people may get annoyed that they have to install ruby

we’re an osx company so everyone has sh so ill use something that will run a local shell command

thanks a lot raymond for unblocking me

you the man

you’re welcome

i understand that. the remote state is getting in the way then. the module is probably storing some data in .terraform itself.

i went hunting for it but had trouble finding the outputs. the only place i see the outputs is using tf show | less

it doesn’t show up in tf show -json | less

yeah, you need to look in the module to see what it is doing. if my intuition is correct, it is not strictly a tf thing

i have a meeting, will be gone an hour+

trying to create a minimal working example to try later today

ok, so yeah, this is a bug in the module. it is writing the stdout/stderr to path.module which is in .terraform. in the contents null_resource, it is then using fileexists to check if the file exists and setting stdout to null if it does not. setting stdout to null has the effect of removing the key from the map

oh riiiight it’s a file based approach.

outputs.tf is then checking if the triggers dictionary is null, but it is not, it still has the id key. so the false condition is triggering and it is trying to assign the stdout key, which does not exist cuz null

ive made the switch to the external data source. thanks for investigating that @loren. so many tf wizards here!

i think this could be fixed either by changing the expression in outputs.tf or changing the expression in the contents null_resource…

the module should definitely be resilient to deletion of .terraform or it won’t work with remote state across teams/systems

i’d love to see a pr but part of me is thinking, what’s the point, if the external data source works so much better

im surprised he’s using the hack instead of the data source. perhaps the module can be updated to use the data source itself

but then whats the point of the module lol

the local_exec approach lets you see the output of the command in the tf log

the external data source masks all output

i think im okay with that, but that is a good limitation to know

a data source also executes always, where a resource has a CRUD lifecycle

so the idea of the null_resource approach is that it gets you stateful changes

ah I see! you have a great grasp on this. would you have any cycles to submit a PR? im sure a lot of ppl would love that

(including myself xD)

eh, probably not in a timely manner…

there is a tf module that spins up vault pretty easily. if you don’t want that, you can stick secrets in manually into s3 kms encrypted, and then decrypt the key using terraform before passing it in

for instance, the password for your rds db would still be stored in your tfstate but it wont be in your code

the cloudposse guy, erik, says this

https://devops.stackexchange.com/a/4628

s3 encrypted was a route we were seriously considering as well

Thanks I will check that out

we do it here but now we’re considering migrating to vault or what erik suggested. probably gonna go with the suggestion

Yeah chamber looks great… going to look into that some more for sure!

Also, we love using SSM parameter store directly from terraform to read and write secrets.

This works well with chamber too, since chamber works as a cli for SSM parameter store.

https://www.terraform.io/docs/providers/aws/d/ssm_parameter.html

Here’s an example of writing an SSH key to SSM parameter store https://github.com/cloudposse/terraform-aws-ssm-tls-ssh-key-pair

Interesting topic for a new video.

I believe it’s because it’s based off of TF 0.11 support -> 0.11/master

0.4.0 of this module begins TF 0.12 support the latest version of TF 0.11 support was 0.3.1

gotcha, thanks for clarifying!

np!

can you use a separate state and module for cloudfront?

i think it can only do one version at a time per .terraform directory. are you sure you can’t fix the issue in v2 where it re-applies every time? i would focus my efforts on fixing that.

https://aws.amazon.com/about-aws/whats-new/2020/01/announcing-amazon-relational-database-service-snapshot-export-to-s3/

i guess because it’s a fairly new feature it’s not available in an automatic fashion

Personally I would go with cron triggered Lambda to export those

we have a similar solution to copy snapshots across regions

works quite nicely

thanks

preference imho

i can’t think of anything besides preference. i find the 2nd easier to read.

haven’t had any personal experience with these providers but I think the expectation is that you’d be consuming the secrets stored in these provider targets as opposed to populating them.

the only thing you’d be expected store in vcs in relation to your secret(s) would be a pointer or tag reference

if you want to store passwords in git then I would use KMS which is supported by terraform natively

ah ok, gotcha!

Yup, basically what everyone else already said. Another place to store the lastpass/1password API secret would be with SSM Parameter Store, which can then be accessed by terraform.

have you tried it?

no not yet

try it, let’s see if it works

My guess is it does not support wildcards. But didn’t you get this working? https://sweetops.slack.com/archives/CB6GHNLG0/p1582677639079700

I’ve made a Terraform wrapper that can generate resources in Python. This recursively uploads files to S3. https://github.com/raymondbutcher/pretf/blob/master/examples/aws/s3.tf.py

I’d vote against using s3 bucket objects in tf without super valid reason. IMO tf is good for infrastructure where s3 objects are outside of it’s scope.

So for example keeping those files in repository with CI trigger to push when they change. Disable delete permissions, enable versioning, etc

I initially agreed but started to wonder if that kind of thinking is informed by Terraform traditionally being “bad” at certain things. What if the CI system dynamically creates a TF config containing S3 objects and uses TF to manage the life cycle of those objects? In some cases you might prefer these objects to be strictly managed. In other cases you might prefer a versioned bucket with old versions deleted after so many days.

Alternative: write your own Terraform provider to fill in the gap. The API for authoring plugins is quite developer-friendly.

Basically you can encapsulate any logic in a data resource (in read only cases) or even a regular resource when you need to handle the entire lifecycle.

In fact you may be able to even use this existing data source - https://www.terraform.io/docs/providers/external/data_source.html

No need for a custom provider, just do ls with the right params, split the output accordingly and use as a regular list to create your s3 resources. As to whether it’s a good idea or not,

What if the CI system dynamically creates a TF config containing S3 objects and uses TF to manage the life cycle of those objects Then you increase complexity IMO. You have repo with files and tf generated files and tf state. Where with files in repo you just have files in repo and cicd pipeline.
In fact you may be able to even use this existing data source. I think this is the answer Brij was looking for. I was just adding my 2 cents :D

What is the pattern that you use?

(also, I think the point of the talk was to emphasize the natural evolution an organization takes in adopting terraform. the “hard lessons” and mistakeds along the way. e.g. the “terralith”. if I recall correctly, towards the end it’s more about what the current best practices are, but even those evolve.)

Good point Erik - that should be the takeaway, maybe I missed that skimming it.

do you do it like this with separate repos? https://terragrunt.gruntwork.io/docs/features/keep-your-terraform-code-dry/#remote-terraform-configurations

we don’t entirely follow their layout, no. but it’s close-ish…

• we have a repo per tf “capability” module, e.g. cloudtrail, guardduty, config, etc. these are pure tf and know nothing of terragrunt. these may be our own modules or a community module

• we have one repo that includes at least one tf “root” module, which implements “capability” modules, passes attributes from one module to another, and includes locals/vars and customer/implementation-specific logic

• the same repo with the “root” module(s) contains a terragrunt config per account that references the root module as the source

Ok thanks. That sounds similar to what they describe except that you have the root modules and tfvars (part of the terragrunt config?) in the same repo.

So if you change a root module, it affects all environments/accounts using it because they’re in same same repo as opposed to pointing to a separate repo with a git ref.

Is that right?

yes, mostly. you can set the source in the terragrunt.hcl to point at the remote ref, per terragrunt config. so you can version that and still use a single repo. but it also means you may have a lot of places to update that source ref as you roll out changes to the root.

we prefer to version the repo level, using local relative paths for the source arg, and use pull requests to review expected changes, and let the build system apply it on merge/tag

tfvars (part of the terragrunt config?) yes, the tfvars are effectively part of the terragrunt config… the terragrunt config specifies which tfvar files to pass when running terraform. you can also use the terragrunt inputs block to set vars as TF_VAR_<var> environment variables, instead

here’s a pretty simple example… this same terragrunt.hcl config goes in every account’s base directory. the only differences from one account’s base config to the next is the values in the base.tfvars file

# Include all settings from the parent terragrunt.hcl
include {
  path = find_in_parent_folders()
}

terraform {
  source = "../../..//roots/aws/base"

  extra_arguments "base" {
    commands = get_terraform_commands_that_need_vars()

    required_var_files = [
      "${get_terragrunt_dir()}/base.tfvars",
    ]
  }
}

inputs = yamldecode(file("${get_parent_terragrunt_dir()}/base.common.tfvars.yaml"))

the parent terragrunt.hcl sets the backend remote state config, establishes the provider blocks, and maybe sets some other inputs (which get merged with the child block, child values override)

That makes sense, thanks. I’m wondering whether we should be using remote repos or not. We currently have everything in 1 repo like you’ve described but not with Terragrunt. Having a change to the root config affect multiple environments at once is a bit annoying sometimes.

i like the combination of “capability” modules vs “root” modules. it’s a reasonable tradeoff in managing the ref updates. if we have an account that really needs to be different, we set that up as either a separate “root” module, or as a custom tf config within the account directory. or we use vars in a “root” module as feature flags, to turn something on/off per account.

separate “root” module meaning this line changes in the terragrunt.hcl:

  source = "../../..//roots/aws/base"

pointing to a different root config:

  source = "../../..//roots/aws/snowflake"

or if it’s truly a one-off, you can put the .tf directly in the account directory:

  source = "."

Our projects tend to be more like 1 config has multiple environments (dev, uat, load, prod, etc) and we often want them to be the same, but rolled out slowly as the change gets tested in each environment. We use feature flags or if it’s tricky just not applying to an environment until it’s been tested in the previous ones.

So for us, versioning seems like it’d be quite helpful. I can’t tell if you’re doing similar kinds of environments.

ahh, so i might have a meta root config per env, so your dev accounts would call the dev root config, and that dev root would point to a specific version of the underlying root module…?

I don’t quite get it. That sounds like an extra hop when you could point directly to the specific version

just trying to manage where the version is set, to minimize the places you need to touch when it needs to be updated. if you only have one config referencing the version, then it’s no benefit. when you have 50, it gets annoying and error-prone to change it 50 places

Oh, you’re talking about updating 50 dev environments at once? We’d generally only want to update 1.

updating 1 at a time when there are 50+ to touch takes way too long. the dev/user can do what they want in the account. we’re just going to manage the baseline resources the customer requires for networking, security, audit, and compliance. those are supposed to be the same across all accounts for a given environment-type…

if an account gets an approved deviation, we expose that via a var and include it in their tfvars

That makes sense. We’re talking about very different types of projects. Mine are typically projects for 1 customer’s full AWS infrastructure and they retain ownership of their TF code and state files. This is everything all the way down to web servers, databases, lambdas, specific to their needs. We can’t have one repo with resources for multiple customers in this case.

that makes sense too! technically we do this per each customer, where the customers just tend to have a lot of teams working on their own projects/apps, and the customer wants some central visibility and control

do you use terragrunt apply-all? i’ve never felt the need for something like that, but it sounds good for what you’re doing

it would actually be the only good use case i can think of for apply-all, also… but no, we’ve been using terragrunt a long time now and earlier versions were not handling async output very well. literally interleaved individual characters from the stdout/stderr streams. totally useless. so we wrote our own async subprocess executor/logger in python, and it calls terragrunt for us. i like it better this way, mostly, since we can prefix all the log messages with some context, so we can then filter on which config the message is associated to

I did exactly the same. Terragrunt apply-all is a mess. I wrote wrapper around terragrunt to mange changes. Updating tens of modules in dozens of accounts can quickly get out of out of sync.

an async command runner with options for streaming stderr/stdout with context (and without deadlocks), and that handles errexit options well, would be a cool python library. didn’t see anything 2 years ago, but maybe something exists now.

It sounds like Pretf will need an apply-all feature, or some way to manage multiple TF stacks. If/when I get around to it, I’ll try to make it generic.

Thanks for all the details, it’s good to see how other people use TF - quite differently sometimes.

My mad scientist idea is to use terraform to manage terraform projects in other directories but I’m not sure there’s a way to bubble up the plan outputs without the use of a custom provider.

If a customer provider is needed, it could be a generic external process CRUD provider

hmm, there is already a generic REST provider, for creating tf providers for arbitrary REST APIs…

I was just thinking that but I’m in the kitchen so didn’t check

Pretf could run a http server before running terraform

https://github.com/Mastercard/terraform-provider-restapi

https://github.com/dikhan/terraform-provider-openapi

Hmm would prefer a built in provider but can’t see anything obvious. Could maybe hijack some other protocol but that’s not ideal.

yeah, neither has been accepted into terraform-providers yet. definitely annoying.

i don’t see issues for that on either yet… maybe open an issue to see if they have interest in getting it into the upstream terraform provider registry?

looks like hashicorp is trying to make it easier to get community providers into the registry… https://www.hashicorp.com/blog/announcing-providers-in-the-new-terraform-registry/

For now, you will only be able to find providers that we here at HashiCorp maintain. Today you can still find excellent providers from our partners and community at terraform.io. Eventually, all of these will be available on the Registry.

hmm, hopefully soon

i could prototype with the mastercard one to see if it’s viable

i have a feeling that the plans will look awful

unfortunately (but it makes sense) it doesn’t actually talk to the rest API during the plan phase, so there’s no way to show nested plans in the plan. it only shows the arguments passed into the resource, which will just be the path to the nested tf project

well that seems… un-useful…

I’ve given up on this for now. I don’t have a personal need for it, and it’s difficult. I think a good implementation for running it locally would have an interface kind of like tig. You’d use arrow keys to browse through the different plans. Lots of details to figure out, it would take a lot of work to make it good.

makes sense. focus on doing one thing good. if someone sees a use case for the -all piece, let them figure it out

we spent some time on implementing the config map uisng kubernetes provider, but had some issues with that. But that was some time ago. Yes, we need (and are planning) to implement it with kubernetes provider . No ETA, if you’d like to help, it would be appreciated

Definitely open to help . Are there any github issues, branches, or forks with previous attempts or documenting the issues you guys ran into?

Ya, so this has been fixed in the upstream terraform provider so it’s now possible to use the configmap resource

We haven’t had a chance to implement it. It’s not been scheduled to be fixed yet (by us) but we’ll try to get to it in an upcoming customer engagement.

when we tried to do it, it was not any issues with the upstream provider that we faced. It was that we could not (correctly) dynamically construct the entires in the config map from additional_users and additional_roles

although we did not spend much time on that

The gist of it is in here: https://github.com/terraform-aws-modules/terraform-aws-eks/pull/355 (the other popular EKS module)

EKS cluster is not ready before applying the config map with kubernetes provider is another issue

we implemented a few attempts in our current script

something similar needs to be implemented for k8s provider

ya, they have a workaround for that

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/7dc56e976bf0f8c48820085a4cdee37e676ca2b8/variables.tf#L204

https://github.com/terraform-aws-modules/terraform-aws-eks/blob/23c005e5ef6107a52af37e6c17e9d572ebc3d81d/cluster.tf#L48-L61

I’m guessing that wait_for_cluster workaround wouldn’t work if public access if turned off and you are outside the network (tf cloud)?

another thing we need is to be able to assume roles before applying the config map

that’s also currently implemented in the bash script

at least for us, it’s important and needs to be accomodated in the k8s provider

I’m guessing that wait_for_cluster workaround wouldn’t work if public access if turned off and you are outside the network (tf cloud)?

this was working for us on TF Cloud https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf#L140

For a cluster without public access?

He means where terraform cannot access the kubernetes API. (but our current solution has the same limitation, but works because we typically run it with atlantis which has VPC connectivity)

yeah thats a good point… doesn’t matter if I can identify when the cluster is available if I cant connect to do stuff on it anyways lol

so we can assume roles with the aws provider directly:

provider "aws" {
  assume_role {
    role_arn     = "arn:aws:iam::ACCOUNT_ID:role/ROLE_NAME"
    session_name = "SESSION_NAME"
    external_id  = "EXTERNAL_ID"
  }
}

I havent dealt with multiple aws providers in the same module, but it seems like it would be preferred to have the module accept an aliased aws provider that will be used to authenticate the k8s provider, instead of passing aws_cli_assume_role_arn etc and assuming the role directly in the module?

and I guess to clarify… was the intent of the origiinal implementation to effectively allow 1 method to authenticate for creating the cluster (through the provider) and another method to authenticate to configure it? (cli commands and aws_cli* args?

or was that a workaround, similar to how I used the profile in my aws provider, but then had to pass it to aws_eks_update_kubeconfig_additional_arguments ?

yes, in some cases we create a cluster using one IAM role, but then auth to kubernetes using another IAM role (that might be not for eveybody, but we have a lot of use-cases for it and using it now)

ok good to know

I know terraform has it’s flaws but it’s really good for things that it’s designed to do. With tf11 I had many problems but tf12 solved many of them. Maybe you just trying to do something that terraform shouldn’t do?

Nope, I’m doing things it was exactly designed to do. It just cannot handle anything beyond the most basic of tasks without needing to do asinine things.

Additionally, terraform is only as good as the providers you use. If you are using the AWS provider only then you are blessed with a mature implementation that can usually get done all that you need to do

if you are stuck on Azure and need to do anything ‘new’ you will always be riding the bleeding edge and all the issues that entails.

Additionally, 2+ year old bugs and issues abound: This one for instance: https://github.com/hashicorp/terraform/issues/17034

right, that’s how it works. there is no api for accepting it. you have to use the console to accept the license. then you can use the ami via any api/console interface

also, the acceptance is good for 1 year

oh… got it.. Thanks!

Hm, replicate_source_db isn’t provided as a variable on terraform-aws-rds so I’m assuming not. Seems like the replica repo hasn’t been maintained in a while, which is reasonable since Aurora…

Anyway, I believe I got my answer. I’ll have to dig into the replica code and see if that is reusable.

To go full circle on this… I ended up using a fork (https://github.com/techfishio/terraform-aws-rds-replica) which is updated for tf v0.12 and that worked out great.

Yep, as you discovered this is a terraform / AWS API limitation.

@endofcake has a write up on an alternative https://medium.com/@endofcake/using-terraform-for-zero-downtime-updates-of-an-auto-scaling-group-in-aws-60faca582664

(but we haven’t implemented it in our modules)

i’ll second terraform+cloudformation for ec2 instances and ASGs…

thanks

@loren Did you come across any working examples for CF+TF especially with launch templates?

we define the launch template in CF also. TF just manages the lifecycle of the CF stack

plus this way we can, if we want, just use CF directly…

here’s a real module we manage this way, probably too complicated as an example, but it’s all i have handy… https://github.com/plus3it/terraform-aws-watchmaker/tree/master/modules/lx-autoscale

perfect

thanks

I will look into that

one thing you’ll see are the “toggle” variables… one just changes the cfn init metadata, which can be used to modify a running instance. i don’t recommend this, generally, as it does not integrate with the CF signaling, so errors are not caught well.

the other changes the userdata in the launchconfig, which forces a new launchconfig, which triggers the CF ASG update policy (launching a new ASG+LC, waiting for success, then destroying the old ASG/LC)

https://github.com/claranet/terraform-aws-asg-instance-replacement does this

personally i like blue/green approaches with 2 ASGs better than rolling updates with 1 ASG, but happy to see any advancement in pure TF here…

It is not pure, it relies on a lambda func

I’d probably try the CFN approach if starting from scratch today. The instance replacement module works pretty well though. The instance replacement happens outside of the Terraform process which may be good or bad depending on the situation.

ahh i see now. the TF+CFN approach does work pretty well

Another approach, if you are using the ELB health checks, is to tie the name of the ASG to AMI, or launch config, or launch template version, etc. We use the build date of the AMI and insert it into the name of the ASG; whenever a new AMI is part of the terraform apply our ASG module creates a new ASG, waits for the instances to be ‘in service’, puts them in the existing Target Group and then destroys the old ASG

That’s interesting. Do you know what exactly is terminating the instances?

yeah, as discussed here i think https://groups.google.com/forum/#!msg/terraform-tool/7Gdhv1OAc80/iNQ93riiLwAJ

as long as ASG name changes everytime when new version of launch template gets created TF creates the new ASG using the lifecycle { create_before_destroy = true }

the downside is you need enough capacity available in your account since all the old instances will be deleted only when all the new instances are online.

True, we’re a fairly small platform so that’s not an impact to us for now. Cross that bridge when we come to it.

https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html#force_delete
Normally, Terraform drains all the instances before deleting the group. That’s the part i was missing. TF terminates them.

nice, thanks @Zach and @Karthik Sadhasivam!

Looks like it was well covered, but for some anecdotal observations: I scripted in python/boto3 a way to rolling update instances in an ASG using a strategy where I double the number of instances, set the termination policy to oldest, and then halve it again. It is reliable, but it is also pretty slow though. I would recommend not going that route. A blue/green style has a definite cutover and removal of old instances. ASGs can take a while to terminate old instances. All solvable problems, of course. Mostly depends what you’re optimizing for.

As for doing this in terraform, others have covered it better. The infrastructures I last did the rolling updates in were both CloudFormation ones.

If you’re doing the actual ‘rolling’ in the same ASG, we found that you need to do 2n + 1 (where n is desired ending capacity) instance launches, otherwise AWS will sometimes leave up an older instance in an effort to ‘balance’ the AZs

you can also disable az balancing…

https://docs.amazonaws.cn/en_us/autoscaling/ec2/userguide/as-suspend-resume-processes.html#process-types

azrebalance is nasty feature

My approach with boto/py doing ECS/K8S rollout was to detach instances, drain ECS, wait for ASG to spin new ones, wait for draining to complete, terminate detached nodes. This way old nodes were safe from azrebalance termination.

Termination policy oldestinstance will have desired effect only when instances across AZs are balanced.
When you customize the termination policy, if one Availability Zone has more instances than the other Availability Zones that are used by the group, your termination policy is applied to the instances from the imbalanced Availability Zone. If the Availability Zones used by the group are balanced, the termination policy is applied across all of the Availability Zones for the group.

interesting. we haven’t actually noticed that behavior, but probably because that particular group scales all the way down to 1 overnight and back up in the morning. iirc, without oldestinstance it would keep the oldest one, rather than rolling it over the way we expected

so with that note, i’d expect what is actually happening is the unbalanced AZs each terminate the oldest node, until the AZs are balanced, and then it proceeds to terminate the oldest node across AZs until only one node is left

What if unbalanced AZ doesnt have old instances? I’ll try to better explain possible issue. If AZs are unbalanced then termination policy is applied to AZ with most instances running

Hypothetical situation but happened way too often for me in the past: AZa: 1 old instance AZb: 1 old instance scale out x2 AZa: 1 old instance AZb: 1 old instance AZc: 2 new instances scale in x2 One instance will be terminated from AZc and then default termination policy will terminate one old instance. Not a desirable outcome if you wanted to get rid of all old instances.

azrebalance sometimes killed node that haven’t yet drained that’s why I used detaching approach. Different but also effective would be using scalein protection but I think it’s more complicated.

well, looks like scale out got dorked first in that scenario… the first scale out should have put something in AZc

Not sure if I got it right but if you have DR infrastructure it should have it’s own tf state.

got it, that makes sense

So now I can YAML my HCL!!!

nope

Cool. Thanks!

Did you see the example here? https://github.com/cloudposse/terraform-aws-elasticache-redis/tree/master/examples/complete

see the fixtures for example settings

it should work with any TF 0.12 version

When I do a terraform init, I get the error: Unsupported Teffaform Core version with an explanation of: This configuration does not support Terraform version 0.12.9.

Looking at the versions.tf, could I be bumping into a required_providers issue or would that be a different error message?

It says that all 0.12 versions are allowed

So not sure what’s the issue is

You can copy the module locally and update the provider version or remove it completely and test

We tried to do it using terraform and it was painful and we gave up on it - I’d also be curious to hear what others do. It seems like the preferred method is to use one of the AWS or other framework CLIs to do the actual code deploy…

On the current project I’m doing, we’re deploying shared/backing infrastructure manually (not via CI/CD) with Terraform and then deploying Lambda projects using the Serverless framework via CI/CD.

I’ve moved to building the lambdas through a normal pipeline in concourse, and let it upload the function zipfile, and deploy it.

We’re trying to lift the application code out of the TF dirs, since we don’t have a high level of automation there yet. I think it surfaces it better for other devs as well. I’ve found the lambda deploy process through TF to be spotty (at least with Python)

I’ve mention this previously but since you asked… I made https://github.com/raymondbutcher/terraform-aws-lambda-builder and before that https://github.com/claranet/terraform-aws-lambda which should make it easy enough. Slightly different approaches in them. They are good for infrastructure Lambda functions. They don’t yet have a good story for promoting a built Lambda package from one env to the next, although I’m not sure if Serverless etc do that either. That Concourse approach is probably on the right track if you’re building a zip once and pushing it to multiple environments.

@randomy yes, you already mentioned about it. Thanks !!

i don’t think it globs… it’s a literal prefix. try removing the *?

was just about to try that

hah, that was it

nice!

cool feature! exact same use case

i just found out about it and didnt have a clever way to solve it ..until now !

It’s a weirdly specific generic feature

Another question:

 requester_aws_assume_role_arn             = "arn:aws:iam::XXXXXXXX:role/cross-account-vpc-peering-test"

So XXXXXXXX is AWS account id of accepter?

I haven’t used this module but my initial impression is no, that would be the account of the requester

There are policies shown in https://github.com/cloudposse/terraform-aws-vpc-peering-multi-account#usage but they’re a bit hidden, you need to click to expand them

, but do u know how to create a role for cross-peering? I checked their github, dont’ see tf-module for create this role.

it might be worth creating a module for it, as it’s always going to be the same if you’re creating roles just for this, but i think it’s exactly like this example but with different content for the policies https://www.terraform.io/docs/providers/aws/r/iam_role_policy.html#example-usage

populate assume_role_policy with the trust policy from the module populate policy with the iam policy from the module

i think so anyway. like i said, i’ve never used this module :)

got it, thanks @randomy

There are no hard and fast rules but I think this fits the case of when to not use a module. https://www.terraform.io/docs/modules/index.html#when-to-write-a-module

@Perry Hoekstra Yeah, agreed with @randomy. I would create your own internal (inside your repo) module for lambda which includes creating the lambda function (one resource) and then also includes the lambda_permission resource role for you specific use-case.

Gotcha, thanks for the hint.

Let me thread this to to avoid noise…

Define enmasse?

Ie, I’d be perfectly fine having EVERYTHING Atlantis if I had, say 30 identical “things of isolated infrastructure” (whatever that may be!) and I could easily do my testing in “development” and once that is done I can do the pull request for rolling it out for real.

The reality though of many things I do (ie, I am NOT a provider for other multitenant “stuff”) - I manage FEW, but complicated infrastructures.

That means I traverse one, possibly two “almost identical” environments for almost everything I do.

I get alot of return on investment in modules of course, for things which are repeated in each of these few environments I do have.

For example “users”, “services” etc. Of course I always do my best to keep things “DRY”.

But when it comes to repeating full terraform Stacks - that just doesn’t happen often enough for Atlantis to actually be beneficial outside of it’s possible ability to create a good view of things that have or are about to happen.

Nice. Yeah I do mostly the deployment in two places, qa - prod seperate accounts. I just am trying to approach ways to simplify even though I’d be fine coding everything up through powershell/python etc, I’m trying to set the team up for consistency and for least amount of setup to get CICD, which seems like Terraform cloud to me

Cloud seems to have benefits to get going…

But I’m since day 1 way way back on proper remote state and dynamodb for locking.

I separate things in many many different “stacks”, all in the same repo.

I make cross-references to “parent” or “neighbour” remote states here and there.

Obviously, I did the old “huge stack” at first. Then started splitting out as that really doesn’t work very far.

I can’t have 15 minute terraform plan stacks…

nice

I’m pondering adding Atlantis and having it running in Autoplan mode…

to detect “missed” apply’s mostly I think

but it could turn out that I could actually do the “trivial” updates via that, so if I start that way it might turn out I can have it help me rather than work against me while keeping the most critical updates done on my very own command line.

I have quite a complicated Makefile which manages everything in my structure so every single folder gets its unique remote state path etc.. so have to make all this work in Atlantis instead. Soon

If I can somehow make it detect module updates “globally”, then it will be awesome.

Ie, it finds an update in modules/fridh/bar and it can detect every “stack” which uses this bar module and run plan on all of them…

I’m guessing this could be complicated with several levels of inheritence… but it ought to be doable!

Otherwise I’d have to have it run plan on every stack, regardless of it having updated. That won’t work… there’s too many of them .

It will be easier if you remove the modules directory from that repo, put modules in separate repo(s) and pin the versions.

Then make everything hierarchical. Any change to a file in a directory affects any stacks/environments below it.

did you implement an equivalent of terraform cloud’s run triggers?

where one pipeline can basically trigger dependent pipelines in a DAG-like behavior?

Exactly. Did you do something like run triggers, or did you make it apply every stack (or whatever you call it) every time, or have you left these situations to be handled manually? (or some 4th option I haven’t thought of)

I’m trying to come up with the “ideal” Terraform project structure and am now thinking about the CI/CD side of it. It does seem hard.

We did not solve it. It would look something like this:

  reapply_eks:
    title: Apply EKS
    image: 'codefresh/cli:latest'
    commands:
      - codefresh run ${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}/reapply-eks -d -b=${{CF_BRANCH}} -v CF_RELEASE_TAG=${{CF_RELEASE_TAG}} -v CF_PRERELEASE_FLAG=${{CF_PRERELEASE_FLAG}}
    

basically, just add a few more steps at the end of the pipeline to trigger the projects that could be affected. it would trigger automatically, but the DAG is “human generated”

care would have to be taken to ensure that cycles aren’t created between the various pipelines.

that seems close enough to run triggers, having used neither terraform cloud nor codefresh

Hope you don’t mind the questions. How did you fit pull requests into it? I guess the ideal is to create pull requests, plans appear based on which files changed, you approve/merge it, it gets applied. But then you have these dependant projects that may have changed that need to be approved before applying. And there is config drift where changes could appear at any time. Ideally they would appear as pull requests for consistency? Atlantis doesn’t seem to deal with this, just saying it’s out of scope.

But then you have these dependant projects that may have changed that need to be approved before applying Dependent projects fall into 2 classes. (a) dependent projects with open PRs (b) dependent projects that read from some shared storage like SSM.

In the case of (a) not thinking about any automation. This would be process driven. As for (b) this is pretty easy. See below.
Ideally they would appear as pull requests for consistency? Yes/no, I agree PRs are nice, but we got into trouble for enforcing a PR workflow too rigorously (see our reference-architectures). In practice, I much rather more things be automatable with simply pipelines than PRs. So in the case where project A has some state changes that project B depends on, then project A writes those to SSM. Then triggers a pipeline run of project B. Since project B has no changes, no PR approvals are required. In codefresh, however, we can add an “approval step” which executes prior to executing the step. This is also possible with terraform cloud.
Atlantis doesn’t seem to deal with this, just saying it’s out of scope. this requires computing a DAG across projects and it’s pretty much out of scope for everything that exists today for terraform. Also, the more interconnectedness the less we have in terms of “microservices” and the more we have of tightly coupled interconnected services resembling more and more of a terralith.

Thanks. It’s getting late here so I’ll have to think about this more tomorrow. Regarding the DAG idea, could that be solved naturally with drift detection that also runs immediately after any changes are made? I’m thinking it could run on a schedule + straight after applies and it would handle both scenarios (maybe not as quickly as a DAG or run triggers, but still automatic).

but at what point have we just recreated a monolith? basically if every change causes every project to be planned (and possibly applied), it’s basically a monolith - even if it’s on a cron

I don’t know if this is pushing it towards being a monolith. It’s surfacing configuration drift and making it easy to resolve it. The status quo is to expect people to remember (or follow instructions) that there are other projects they need to check and update afterwards. I do get that it’d be annoying to make a change, then wait ages for it to go and check every project. Depending on the project size, that could be fine or too slow.

Yes, don’t necessarily have a hard rule on this, but I agree with you on this one. The nuance here is if you can add business logic to it.

An example of combining business logic with security groups is this @antonbabenko’s module: https://github.com/terraform-aws-modules/terraform-aws-security-group

at cloudposse, we’ve not used this approach for security groups. logic of security groups is tied to the the resource being secured (E.g. redis would handle it’s own security groups and expose the group name as an output)

Speaking generally and not security groups exclusively - If your module helps you enforce a standard, add extra information, tags (billing tags maybe), definitely a module per resource could be motivated I feel.

Only few rules of mine are of this “loose” form. Most are like Erik says - tied to a larger resource itself.

So, having a few rules then come from a module, why not?

I’ve personally flip-flopped back and forth on making modules in tf. It is too easy to ‘think’ you are being clever with them when in fact you are making an inflexible mess with them instead.

I never go module first.

Me either, I learned my lesson the hard way…

I strongly lean towards moving to a module the moment I see “oh, I can (or need to) re-use this”.

And actually… I never modularize it before the point where I actually need it again.

flat first, then if the project firms up into a structure that becomes repeatable on a per-team/client boundary I ‘may’ cut out modules for it.

I’m quite fluent with terraform state mv by now. So no sweat moving things after the fact.

It seems like that should not be a statement one should have to throw out there though right?

I think it’s tough to grasp unless you’ve spent alot of time in TF.

I mean, compare the manual gymnastics you need to do today visavi a Midnight Commander interface for moving state and resources - imagine that?

plus dependencies pretty much enter a realm of some psychedelic nightmare when you start using modules.

or I’m just needing more time to drink more of the kool-aide…

depends_on = [null_resource ...

Hahah

Ha, I am mercilessly taunted at work for a module I made that creates a single resource, and has more parameters than the resource it manages..

Ok - thanks. I tried that and couldn’t attach a load balancer as no ports could be exposed in the task definition. (Apparently rejected by the underlying aws cli.) I believe this is simply a missing enum value in the aws provider.

I was not aware the value was not available from the cli. From my understanding of the link I assumed that providing ‘default’ would cause it to be set to NAT.

well, network_mode is an optional field in the task definition. perhaps default just means it is left unspecified?

seems it causes it to be null - I will write here if I reach any conclusions

if the varaible you’re using is a list type, you need to first cast it with toset() with for_each otherwise it will reject the variable

It accepts it just fine!

It successfully plans

Weird. Its only supposed to operate on Map and Set types

Yeah the problem I have is that the null on the for_each statement doesn’t treat itself as a condition to not execute the dynamic block

Do you even need the null condition there? If the set passed to for_each is empty it should skip the block I thought

for_each on resources must be set or map. for_each on dynamic blocks can also be a list

ahhh thanks

that is because the resource id must be unique, and lists allow duplicate values

try changing that null to []?

tried that =[

No go!

Same result

what exactly is the error?

Terraform interpolates it all successfully in the plan, however, it executes against the dynamic block, so the source_id which is a security group id in this cause is [” “], so the API call fails

on an apply

right, i was thinking it’s not the ingress for_each that’s the problem, it’s actually the expression on the security_groups attribute…

correct

because the logic is executing against the dynamic block even though the false on the condition is null

maybe that expression needs to be a for loop instead of the conditional?

Curious to know what you’re thinking @loren?

@loren

Much respect!

If anyone is curious

worked magic!

Something like that, but maybe also question why is there an empty string in your list in the first place…

This is the current variable

ah, nvm @loren, you’re bang on! I found that the variable I was feeding it had an empty string >.<

Thank you so much for your help mate, legend

Am I understanding it correctly that the list immutable beside appending new items?

correct

I haven’t written any provider so I can’t say about ability to do that, any language can save data, but I would be vary to cache data in provider. This could lead to some inconsistency issues and I would favour more time for computation of those list then implement additional functionality like cache validation.

Not that I know of. This is not a module limitation. This is a terraform (aka AWS API) limitation.

Note:

so basically, if you restore from a snapshot you might be able to “game it”

https://www.terraform.io/docs/providers/aws/r/rds_cluster.html

also, “plain text” is a relative term

you can have “plain text” in an encrypted value. e.g. if you have the terraform state on S3 encrypted.

Is it TF 11 or TF 12? - I guess you should be able to use the dynamic blocks for your purpose if it is TF 12. https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

this is TF12, haven’t heard of dynamic blocks, will read up on that

Beautiful!

dynamic "condition" {
      for_each = var.limit_tags
      content {
        test     = "StringEquals"
        variable = "aws:RequestTag/${condition.key}"
        values   = condition.value
      }
}

Thanks @Morten Hjorth Fæster

You are most welcome

The last post in 1 day ago. I could use some insight on that if any cares to. As a quick fix I just tried using a local.tf file and pulling content in but I don’t think that will work without using symlinks. I’m really adverse to using symlinks for this… seems to add some accidental complexity

I’m working on this example which is aiming to be the “ideal” project structure: https://github.com/raymondbutcher/pretf/tree/mirror-module/examples/enterprise It symlinks files from parent directories on-the-fly. One of the stacks gets the root module remotely (like Terragrunt). You can have *.auto.tfvars at different levels and they’ll get symlinked into the working directory automatically. This is still in a branch, I want to keep refining it and flesh it out some more.

It would be relatively easy to create a directory structure with 100 environment directories each with a tfvars in them.

I don’t know enough about Terraform Cloud to know if it would work there.

I wasn’t certain symlink was going to be a cross platform and reliable approach. If that’s the only way to do this then I guess that makes sense.

Mac and Linux are fine. Windows does support symlinks (junctions) but I’ve never tried it. You could easily copy them as normal files instead but figuring out which ones to delete when cleaning up is a bit trickier.

I think you can try using HEREDOC syntax

Ah yep. Thanks

(and for others sake, here’s a link to what that looks like https://stackoverflow.com/a/57380137)

pro tip: use heredoc with a dash like the example

aws_key_path = <<-EOF
               #{martinTestPrivateKey}
               EOF

Yeah it really does make all the difference

the dash means it truncates trailing spaces from the first line thereby allowing you to indent your content for readability without having to worry about whitespace

Buried in the docs. I always use that makes things much more readable

self-plug… here’s a blog post about getting started…. look at the code snippet called iam.tf https://www.sheldonhull.com/blog/getting-started-with-terraform/

This wasn’t indented, but because it has the dash instead of <<EOF i could have indented it and no issues. Without the dash you have to watch out for whitespace…. end random info

well, wrong trust policy for YYYYYY account

local seems easier or update the module with a flag to use it inside of itself somehow

Ok, i expose my context, It will be clearer, im using this module -> https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment?ref=tags/0.18.0, this module bring me hostname as output variable, so, i want to save this hostname value into the additional_settings input variable as a environment application variable, so, locals is the best way to do this? I have a lot of modules with this issue. If i use locals variables i have to create a local for each module.. Thanks for your help

@ByronHome that creates a circular dependency issue. All the module input is needed before creating the module resources. You can’t take output from the module and have it update itself after-the-fact in a normal “TF” way.

what you’d want to do is fork the module, update it to add that host as additional properties as part of the module

and if you can find a sane and modular way to do that, feel free to submit a PR back into the original module

Okay thank you so much, i will rethink my context

AFAIK there’s no easy way to do that. Waiting for tf13 https://github.com/hashicorp/terraform/issues/10462#issuecomment-604107965

I do but calling it ‘good’ is not really the right way to think of it…

add a variable like so:

variable module_depends_on { type = any default = null }

ha ha

then in the module use it in your resources like so:

depends_on = [var.module_depends_on]

I hate that anytime you go to use modules or get more complex than a single provider you are actively punished it seems.

oh, when calling the module use something like this:

module_depends_on = [ azurerm_postgresql_firewall_rule.cicd, module.psql_demo, ]

yeah

honestly, that’s where the declarative aspect of Terraform drives me crazy

a decent scripting interface would make all of this easy

it sort of works for me to get around some issues with firewall rules being required to be in place for hosted postgres before running some psql commands

I’m not sure that would work for what I’m doing

I couldn’t use the psql provider either as providers cannot have dependencies either

so I had to custom compile a usql go client and include it in my module

craptastic

(not still feeling a bit burned by that hack, not at all… :))

this is what I have

in the module input, I want certificate_arn to reference the resource I’m creating

depends_on would be perfect there

Sry, actual work I had to do there. Why don’t you just pass it in as a var with the domain name and pull the arn from the data source using that domain name? https://www.terraform.io/docs/providers/aws/d/acm_certificate.html

or just pass in the arn directly as a var from your created aws_acm_certificate?

(sry, not in the aws provider every day)

Yeah, I can do that but I need to create a separate state

for the cert

which is what I’m going to do

Not clever but it’s clear

I don’t really like the workarounds I’ve seen for depends_on + I don’t think they’ll work for my use case

you could use remote state as a work around?

https://www.terraform.io/docs/providers/terraform/d/remote_state.html

yes, I’m using remote states

I’m going to handle it is two separate remote states

it’s clean

curious, how many remote states become too much for a single deployment?

we have a bunch of remote state files

yeah, too much for a single deployment

I’ve gone down that path, I’d rather have a large number of independent states than one monolithic one

I’ve been able to manage with just a few per environment but it seems almost excessive to have to break out another state just to do work arounds for dependency management

any rule of thumb wisdom on when you break out your states?

I’m having a hard time reconciling (mentally) the fact that I have to cut apart my remote states and, in turn, my pipeline code, just to keep a sane dependency graphs

I’m just ruminating now, no need to answer questions concerning my mental problems

honestly, this is my biggest challenge with Terraform

it’s difficult to come up with a sane layout/strategy with large systems

ok, well it isn’t just me then, that’s oddly comforting

just to clarify, I think you mean remotely when the resources are not exposed externally and sitting on a private VPC

this use-case is not yet supported, but hashicorp is probably working on something to support this

for now, it means using the self-hosted terraform enterprise

I used the cli via local exec in a makefile, but I wanted to switch it to the provider.

+1 on @Erik Osterman (Cloud Posse)’s comment

yeah okay i thought so. for things like the helm provider it sounds like I will have to continue deploying via my local machine (but will host statefile in tf cloud)

You can use venona with Codefresh

don’t have enterprise codefresh

nice, yeah, we’ve been using codebuild as an easy command runner. great for scheduled and event-driven tasks of all kinds

Yeah, it’s a decent service. I’ve mostly used it for running Packer to create AMIs.

That’s slick! I like that design…

Thanks!

Similar to this too, https://github.com/Mastercard/terraform-provider-restapi

this has a data provider to issue GET requests.

I was thinking on that one when you were talking about the webhook provider, you can definitly use it to trigger the webhook

@Joe Niland sorry for late response. will check for it tomorrow. I am on a short vacation right now