#terraform (2020-04)

You mean chaining it ? What I used was to store certain results in SSM, everything else can read from that.

I know of SSM but haven’t used it, my assumption is that consul would be the non-cloud specific alternative to it right?

correct

Thanks @maarten, I appreciate the quick feedback

out of curiosity, are you looking up an EIP you generated and assigned to the NLB or is the the managed IP’s that AWS assigns to the NLB?

If the later, I wonder if the provider can distinguish the difference for that resource and assumes that the IP can change (similar to how it does in ALB’s) and just doesn’t keep the IP static

since the resource itself is known to have changing IP’s out of band of the provision process

I let AWS assign those IPs in this case.

The problem is the resource (aws_lb) doesn’t supply the data at all - so I have to go via the data source.

But you are on to something.

or hmm maybe not.

yeah if you’re letting AWS assign them, there’s no gauntree they’ll remain that IP I don’t think

The problem is the internal IPs…

I’m not sure I can explicitly decide them?

maybe I can create two elastic interfaces. let me check

no, they can only be auto-assigned it seems.

Amazon creates those elastic network interfaces, and “owns” them, but they are created “in” my account

yeah I think you’re right on NLB, no ability in API to specify anything beyond the subnet

I can live with this… but anyone else is going to freak out when things show up as changes.

I wonder if you can use lifecycle ignore_changes on that scope

I wonder if maybe it’s mostly depends_on which makes this an issue…

Currently depends_on for data resources forces the read to always be deferred until apply time, meaning the results are always unknown during planning. depends_on for data resources is therefore useful only in some very specialized situations where that doesn't cause a problem, as discussed in the documentation.

Yeah, it’s that.

And I use depends_on here for a good reason…

Because a data source cannot be “optional” I have to make sure the aws_lb resource gets created first. So data source is used second. Otherwise, the first terraform run won’t work fully. dangit.

https://github.com/hashicorp/terraform/issues/17034

ah boo

well, I normally wouldn’t suggest this but in this kind of scenario you could replace your eni/eip datasource with an external datasource. Write a botoscript or something to lookup the IP and handle it gracefully.

because it seems more of an issue with the provider and resource than with the API itself

Amazon needs to add these IPs in their API responses in order for Terraform to be able to fix it properly I guess.

Because going via the network interface list is a bit of a kludge.

I’ll see if I can do some coalesce trick with an external program like you said.

I have the same pain with ecs task definitions… because they can be updated from terraform - or from CodeDeploy …

hope this data source issue on plan can be fixed soon .

yeah, for that it looks like airship.tf has a nice approach with drift management via lambdas

And my current way of running terraform doesn’t even support executing an external programs right off the bat … since I use AWS profiles.

But I tested it, and it works.

airlift yeah this one had some nice things. “cron” jobs etc…

# lookup_type sets the type of lookup, either
# * lambda - works during bootstrap and after bootstrap
# * datasource - uses terraform datasources ( aws_ecs_service ) which won't work during bootstrap
variable "lookup_type" {
  default = "lambda"
}

clever indeed.

CFN stacks with custom resources can help sometimes. I used one here to do ECS deployments that don’t interfere with Terraform. It’s just using ECS rolling updates, not CodeDeploy though, but we might extend it to support CodeDeploy. https://github.com/claranet/terraform-aws-fargate-service (it’s in early stages of being taken from a production system and turned into generic module)

Also you can use 2 separate stacks and output data source values from one, use it in the other, if you want to “save” values. Not always ideal of course.

Yep, considered the two stacks..

I adapted my module to use the tricks from the airlift.tf modules now and have solved the task definition issues Now I’m building a similar lambda to solve the NLB lookup .

does airlift require setting the lookup variable to one value during bootstrap, and then another afterwards?

i mean airship (guess you did too?)

https://github.com/blinkist/terraform-aws-airship-ecs-service/blob/master/variables.tf#L221-L230

If it’s non technical-users does it make sense for them to do the deployment themselves ? From Administrator POV Terraform would be perfect to create 500 predefined instances for example.

We wouldn’t want them to do the deployment, but they would sign up for the package that would trigger a deployment. So think of a case where a member of the public signs up for a service that deploys a server for their use. The end user would be a data scientist. So it may be 200. It may be 500. It may be a 1000.

I’m asking for a general pattern, but the couple of use cases I’ve seen is with an organization that wants to allow teams of data scientists from the public to use a common dataset with jupyterlab instances stood up, etc.

Ah ok, in that case Terraform could for sure work out. In AWS Case I think Cloudformation would maybe be more straight forward as no tooling would be necessary.

What’s the pattern you would use for Terraform? Would it be something like a webhook trigger on Jenkins to run a Terraform module that accepts different inputs? Is that a better option than say use a language SDK to great a gui?

Scalr is moving in that direction

I was on a demo call with Scalr a while back for a different use case. I understand they use Terraform on the backend, but I don’t see how Terraform can be used in a normal GitOps way for them. Do you know what pattern they use?

@Joe Presley I’d leave Terraform out of it as it’s not an API and with many automated deployments error handling of Terraform will be an issue. Maybe you can look into using the provided IaaS for the Cloud Provider and it’s native deployment mechanisms like Cloudformation.

That makes sense. Thanks for the feedback.

oh sorry, I didn’t see the terraform-aws-ecs-container-definition module part

nm that

@sweetops I’m using v0.23.0 and the following works for me:

locals {

  log_config = {
    logDriver = "awslogs"
    options = {
      awslogs-create-group  = true
      awslogs-group         = "${module.label.id}-logs",
      awslogs-region        = var.region,
      awslogs-stream-prefix = module.label.id
    }
    secretOptions = null
  }

  # ... 
}

# ...

log_configuration            = local.log_config

You having trouble beyond that?

nevermind, got it

https://github.com/scottwinkler/terraform-provider-shell ?

thanks!

github has fallen over, in case that’s not responding for you… https://www.githubstatus.com/

I just noticed the same, thanks for confirming I’m not the only one…

that link is for an actual custom provider… there is also a module that abuses null resources terribly to get the outputs into state… but can’t find it right now cuz it’s on github

Yeah, I know they are all dirty hacks and all. I’m not super proud to have to be using it

or you can use the external provider, which is easy too, and pretty clean. only extra downside to me is that the stdout is not output which can make it hard to troubleshoot. i compensate by using the python logger to write to a file

I just need one dumb airflow fernet key

its such a small python script you can oneline the thing

What! That’s so cool. Going to check it out for sure. And I can figure out a way to shoehorn some power shell in there

https://www.terraform.io/docs/configuration/functions/merge.html

negatory. terraform loads from all .tf files in the directory. only option is to create separate modules

okay, thanks!

kubeconfig path was used only for the module’s internal use, to get kubeconfig from the cluster and apply the auth config map

you can use the same command aws eks update-kubeconfig as the prev version of the module was using

to get kubeconfig from the cluster

but outside of the module since it does not need it now, and using CLI commands in the module is not a good idea anyway - not portable across platforms, does not work on TF Cloud (you have to install kubectl and AWS CLI)

Ta its all in geodesic so its been fine. I just tripped up leaving it on “rel=master” .

The .11 version had a kubectl file which applied the configmap auth; we were hooking on to that thread to bootstrap rancher.

yes, if you want to continue using the prev functionality, don’t pin to master

in fact, never pin to master, always pin to a release

(fewer surprises since any module will evolve and introduce changes)

yep yep; Totally down to me rushing to test migrating to the v.12 version. I’ll file it along side my decision to upgrade EKS to 1.15 on a friday and it murdering the cluster in the eyes of rancher (but running fine in EKS)

Break them out!

Agreed - now how do I defend it…

Tell them it’s how terraform module versioning and integration works.

You increase the reliability by seperating as you tag and release versions. This powers the terraform modules. You can’t git tag folders

I’ll probably write up a blog post on this soon. I ended up building an Azure DevOps pipeline that I just copy and pasted into different terraform module projects and it automatically versions them using Gitversion.

I showed this logic and my team is starting to do the same thing now not a monolith repo. The very nature of terraform modules and being able to version breaking changes and so on would get requires single repost per terraform module.

I’m sure you can get some work around but it’s definitely an anti pattern in my opinion to try and do a monolith repo for Terraform modules. And let me clarify I mean this for terraform cloud. I’m doing all of my pipelines and telephone cloud I don’t even know how it could even work reasonably with their terraform module registry if you are trying to use a monolith repo. It’s not really even a question of an uphill battle, in my opinion, it’s a question of if they’re trying to force an anti-pattern based on how they prefer to do it rather than how terraform cloud is structured to do it

Lastly I’ll mention that you should create your get repos for the simple yaml file and a Terraform plan. Then you won’t have any problems setting up your and managing your repos. I have 90. I think the closest team has like 10 because they’re working on more traditional applications. small purpose repos just work better in my opinion with terraform as well as CICD tools

Hope that helps

Terraform docs back this up
you can publish a new module by specifying a properly formatted VCS repository (one module per repo, with an expected name and tag format; see below for details). The registry automatically detects the rest of the information it needs, including the module’s name and its available versions.

https://www.terraform.io/docs/cloud/registry/publish.html

Cheers

Since we are using GitHub, they are arguing you could just pin it to a commit hash…

Of course, ie tags

I really appreciate the input! I’ll give you a summary in a bit of what happens…

Ask them how they plan on ensuring

• one version of a module in one folder is easily versioned against another that has pinned changes from a different time.
• not download the entire repo for each module use… Which it would do by default

To me it seems they need to get comfortable with the terraform recommendation and not try think more repos more problems. It makes it easier.

And solve the management of them by in and hour or so write a yaml file that created all your repos and even adds GitHub hooks for slack and more. It’s actually not much extra effort at all then and your actually improving your source control system manangement too

https://www.hashicorp.com/resources/how-decompose-monolithic-terraform-configurations

And ig you choose to not use terraform registry then you have to start managing each jobs GitHub authorization instead of having that handled by the built by in registry oath connection. It’s really to me a lot more work to try and not use their recommendation IF you are using terraform cloud.

for some background, @Mike Martin are you practicing monorepos for other things you’re doing?

(i find the monorepo / polyrepo argument to be more of an engineering cultural holy war with everyone on both sides wounded and fighting on principle)

also, one repo per module is not necessarily required either. we do that as large distributors of open source. there’s also a hybrid mode.

https://www.terraform.io/docs/cloud/workspaces/repo-structure.html

This talks about doing something like creating a terraform-networking repo and then sticking modules for that in there. Since this is for your organization it can be highly opinionated and grouped accordingly.

100% agree. The folks I’m working with don’t do everything through code and so using one get repo for everything miscellaneous has been typically the approach. they get nervous about new repos cuz they’re not used to it. Making it easier to manage and create them consistently I think is a big big win that helps that move forward

@Patrick M. Slattery You need to create a condition and provide an empty value and a non-empty value. Here’s an example:

    dynamic "action" {
      for_each = var.require_deploy_approval ? [1] : []

      content {
        name      = "ProductionDeployApproval"
        category  = "Approval"
        owner     = "AWS"
        provider  = "Manual"
        version   = "1"
        run_order = 1
      }
    }

I know a lot of folks use IDEA, which is just a more generic PyCharm. I use VS Code, but as I move to TF12 it doesn’t handle the mix of 11 and 12 very well, and the experimental language server is not the best, but it’s been improving. I think the IntelliJ plugins are more featureful and supporting of 12. I could just have my stuff set up poorly, I haven’t spent much time on it.

Yes I used VS Code originally but TF12 broke it, and they never really got a good plugin after that

Yeah. Besides lack of time and just having everything already built in TF11 so path of least resistance, that’s probably the biggest thing holding me back is that it’s not well supported in my dev environment

hmm so IDEA is just the java IDE from JetBrains, they make PyCharm too. I’d have to assume its the same plugin in fact.

My first introduction to finalizers were with installing rancher

OMG what a PIA to uninstall an app with dozens of finalizers

I am so glad to get validation from someone like yourself on those stupid things

@Maxim Mironenko (Cloud Posse) is back, so we can take a look at some things

let’s move PR requests to #pr-reviews

ohh cool, sorry I did not that channel existed

it’s pretty new

I don’t know if its considered best practice but I use one bucket per account and use workspaces for environments. I switch between accounts with -backend-config

I’ve got a shell function to switch back and forth:

tfswitch() {
  export AWS_PROFILE=$1
  terraform init -reconfigure -backend-config=backends/$1.tf
}

I guess I’d also be hesitant to have the prod s3 state bucket share a space with the dev s3 state bucket and the IAM policies for both state writing and cross-account access get complex to manage

I agree… those are my concerns too.

additionally if you’re operating as a team you’ll still probably need separate dynamodb lock tables for each env

ok sure.. I haven’t used them thus far but I can take a look to see how that would work out.

I’m using one single s3 bucket, one single dynamo table for maybe 30 or so state files (30 “stacks”)

how do you use 1 dynamo table for multiple states?

is it just an item per state in the same table and you specify it explicitly in the backend config?

my backend.config is the same in ALL stack folders:

terraform {
  backend "s3" {
    profile        = "aws-profile"
    region         = "aws-region"
    bucket         = "s3-bucket"
    dynamodb_table = "my-infra-terraform-locks"
  }
}

then person B making a change to state B has to wait until person A making a change to state A is finished?

if they’re all using the same lockfile?

then I have a little Makefile by which I do everything … make plan for example.

That sets the key for each folder based on the current path I’m in:

so if I’m in repo/terraform/infra/battlefield/bl-prod-mesos:

it will do:

echo key = "infra/terraform/b9d-infra/infra/battlefield/bl-prod-mesos/terraform.tfstate" > .terraform/backend.conf

I could also have hardcoded the above in all individual folders… but so far I never did. Makefile does everything the same every time

They just want to be able to see all state files in one bucket. I question that.. why do you need to see the state files? Smells fishy.

We have one account per environment generally speaking, and one state bucket per account. Sometimes we end up with a second or third state bucket in account for a different client project, that’s usually just for small projects and only in the dev/test account.

regular string interpolation:

• 0.12: value = var.one != "" ? "${var.one}:${var.env}" : "${var.two}:${var.env}"

ah perfect. thanks aton!

0.11 looks off to me on first sight. I would create a local first with : and then reference it for value here

something like this:

locals {
   trueValue = "${var.one}:${var.env}"
   falseValue ...
}
value = "${var.one != "" ? local.trueValue : local.falseValue}"

I simply forgot 0.11 tricks altogether by now

thanks guys, i used v0.12

^bump

can you post the specific error

looking at what you’ve supplied it doesn’t look like there’s anything for terraform to init

do you have a main or any other kind of .tf with provider resources defined?

@androogle i updated the original post

The behavior of terraform changed in 0.12

0.11 used to allow initialization of a directory with dot files

We have a long thread discussing this here: https://sweetops.slack.com/archives/CB84E9V54/p1582930086018800

@Erik Osterman (Cloud Posse) aww, damn. i was looking for anything in the changelogs yest about this specific behavior. maybe just init into a new dir, and then copy the tfvars/envrc into the copied-module dir?

This is a design decision that comes down to what you want to achieve with multi-region support

One consideration is to move your S3 state out of AWS so that it’s decoupled from AWS failures.

Otherwise, if you want to use something like the terraform-aws-s3-statebackend module, best-practice would be to have one state bucket per region to decouple state operations so they aren’t affected by regional failures.

Note that terraform cloud is not hosted on AWS, so using terraform cloud as the statebackend is an alternative.

I don’t know in CCi however I’d use terraform-docs json & jq to prepare it.

I don’t know for certain, but I guess likely not since most CP modules are AWS focused and tags in AWS are exclusively (I think) key value pairs. This would probably be a good contribution to that module though.

I definitely do not have a well oiled setup yet, but what I do like is that Okta lets me add MFA to things that otherwise don’t really support it. And have different MFA policies depending on what application they’re accessing.

The other great advantage of SAML is a single point to de-provision access quickly. Auditing login events also becomes a lot easier.

I’d look at the decision by prioritized pragmatism - are things that Okta is going to do part of the current major priority of the business? Usually I see this kind of priority come around during a compliance event, like targeting ISO 27001 or PCI DSS compliance or an IPO.

Yah ~ansible~scode is pretty bad for terrraform now due to the lack of support on the plugin. JetBrains (IDEA or PyCharm) has a better plugin although its suffering from the same sort of lack of updates and doesn’t recognize some of the newer features in 0.12

ansible ?

err sorry, vscode I was editing ansible IN vscode when I wrote that

i’ve been using the vscode with the language server for tf 0.12 support. it’s still a work in progress, but works for a good 90% of my use cases

I kept trying the language server and it never seemed to do anything (at the time) so I eventually gave up

been using it for months now, no problem

Ok I can give it a shot again.

The nice thing with Jet Brains IDE plugin is you get full featured ‘where is this used’ and refactoring, and you can cmd/ctrl-click to jump to the definition of a var/local or into a module

i never use that kind of feature anyway can’t keep track of the shortcuts

same usually, but since this is mostly ‘click and go’ its the one I find helpful

I have switched to IntelliJ because of this, and I am a big fan of their plugin

yes, you can use null for that

Ah, of course. Fantastic, thanks.

My colleague helped get this added to the AWS provider but I’m not very familiar with it myself. Did you see that it was only just released, and check the relevant PRs? https://github.com/terraform-providers/terraform-provider-aws/blob/master/CHANGELOG.md#2560-april-03-2020

Thanks man, yeah I saw it was only recently released. I’ve updated to latest AWS provider but still doesn’t seem to work. The documentation just isn’t clear enough, or else it’s still broken?

I wish there was an example in the PR with it working

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-codepipeline-pipeline.html

See how there is ArtifactStore and ArtifactStores - the latter has regions

Oh, it looks like Terraform combines them into a single concept, somehow.

yeah, i tried using the plural ArtifactStores instead of ArtifactStore, but it threw another error saying: Unsupported block type

if you have 1 artifact_store block then it maps to ArtifactStore but if you have multiple it will map to ArtifactStores

have you got multiple blocks and did you set region in them all?

my goal is to run the pipeline with a single artifact in one region (CodeBuild) and deploy to multi regions (ECS)

so i only need the single S3 bucket artifact in one region

Let me see if I can convince my colleague to join this channel. It sounds like what he is currently working on (which is why he worked on the region feature).

cool, great

I don’t think he’s online at the moment.

ok

it might be that it doesn’t support my use case yet

the alternative seems to build in each region, then deploy from there, but that just seems redundant

I tried to figure it out myself but I can’t tell why multiple artifact stores are necessary. Will wait and see what he says. It might be a while though because it’s the Easter weekend.

sure, no worries. I really appreciate you jumping in to try and help though. thanks

I might be on to something. I created a single region pipeline using TF, and then manually added the other Deploy regions, and ran terraform plan again to see the differences. It looks like it might need multiple artifact_store blocks for each region, but the location isn’t the S3 bucket, it’s a codepipeline ID for that region, which I’m trying to figure out…. The diff shows this:

  - artifact_store {
      - location = "codepipeline-eu-west-1-221120437065" -> null
      - region   = "eu-west-1" -> null
      - type     = "S3" -> null
    }
  - artifact_store {
      - location = "codepipeline-us-west-2-423457879940" -> null
      - region   = "us-west-2" -> null
      - type     = "S3" -> null
    }

are they your account ids? maybe they are real buckets

yeah those aren’t my AWS account numbers, and those aren’t S3 buckets in my account

looks like AWS-owned IDs?

https://stelligent.com/2018/09/06/troubleshooting-aws-codepipeline-artifacts/
When you first use the CodePipeline console in a region to create a pipeline, CodePipeline automatically generates this S3 bucket in the AWS region.

oh wait, you are right.. it did create those S3 buckets for me

this is starting to make more sense. It might be required to have an artifact_store (S3 bucket) in each region I plan to deploy to

I have enough to move forward. Thanks again for your help Raymond!

Specifically I’m looking for a way to provision a Launch Template + ASG and then get the resulting instance IDs that are spun up from those resources.

@Matt Gowie not sure if TF lets you this , Im think you could you use terraform local-exec and run an aws cli script which will query the ASG created and print the instance ids.

Yeah @msharma24, was afraid of that. Will have to look into that and see if I can make something like that work.

my gut reaction is

Same, but then I thought about it and the alternatives seem worse.

Does the apply happen in a separate CI/CD system? Do you have Slack notifications to bring you into the approval page? Then another Slack notification if the apply fails? This seems messy.

it’s messy either way i suppose. we consider pr approval as “go” and the merge then just is the ci trigger. the ci job status needs to be monitored in either case.

I’m trying to build something similar to Atlantis using GitHub Actions, and having all of the human interaction in the pull request page is really tempting.

apply before merge makes it so much simpler

really? github-actions works fine on branches, right?

yes they do

we typically operate around several events… prs, branches, and tags. the ci does different things in all cases

external CI system?

usually all the same CI system, but it can be several. depends on the project

i mean do you have a separate CI system or just github?

we haven’t done much with github-actions ourselves. we’ve toyed with it here and there, but haven’t had a reason to move things actively

we’re mostly still on travis-ci, but also codecommit, appveyor, azure pipelines, and now a little github actions

i’m trying to build a terraform workflow that’s all in github

also toyed with a github repo linked to gitlab-ci…

so no logging into separate CI systems

yeah, if you have private github access, and github actions logs are private as well, i can see that

and no extra infrastructure to run

i’m always super cautious with lettings prs have access to credentials

yeah, it checks the user’s identity before running a plan, and pull requests from forks won’t work

are the logs private? i mean, what if an authorized user is debugging something and changes the pr to log sensitive values?

i may move the actual terraform apply part into codebuild to reduce the AWS access that github needs

i think the logs will be as private as the repo itself, so a user could potentially reveal sensitive values that way

if the terraform apply part is moved to codebuild, access to logs could be restricted to

but i’m more stuck on the user interface and workflow side of it, hence the question about applying before merging

heh

if you want something else to compare to, here’s one of our public projects that implements the pr/branch/tag workflow using travis-ci…. https://github.com/plus3it/wrangler-watchmaker/blob/master/.travis.yml#L31-L91

i’d like to get a plan in the pr, but exposing the credentials is a no go on the travis-ci side…

would could use a github webhook to trigger codebuild, run the plan, and post the status check back to the commit/pr with a link to the job. we’ve done that also, for other projects

what is watchmaker?

here’s what i’m working on https://github.com/pretf/example-project/pull/1#issuecomment-611670654

the travis stuff looks probably good but i want to keep the entire workflow within github, and preferably within pull requests

and i want to avoid: plan, approve, merge, apply fails, confusion

it’s very early stages. still trying to figure out how i want it to work.

Yeah, totally, was just providing another reference, maybe spark some ideas around the interface/config…

Thanks, it is helpful

I’ll do this iteratively. Start entirely in GitHub, then look at moving pieces into external systems (like CodeBuild). But first I need to nail the interface.

Atlantis seems to have it about right, but apply-before-merge is radical.

It’s explained here https://www.terraform.io/docs/commands/plan.html#detailed-exitcode

Thanks @randomy I just RTD

Im just working on making our CI build fail and send Slack notification when the terraform plan stage for X env fails. Using azpipelines slack not and AzDO YAML pipeline to build CICD.

Like the error says, you can’t use for_each in some blocks yet. Hashicorps is expanding the use of for_each soon, but you can’t use it everywhere.

is there another workaround to do achieve variable-defined providers?

I’m not sure its capable of it yet

i don’t think so… https://github.com/hashicorp/terraform/issues/19932

You can do it with https://github.com/raymondbutcher/pretf

Or manually define every possible region provider, then for each region call a module and pass the region-specific provider in, along with an “enabled” flag for that module based on whether that region was included in your list.

I’m interested as well. I’ve heard some say for serverless just user serverless cli or Sam as terraform overcomplicates this one aspect and all toolchains are driven for cli for this. I’d like to also know if anyone has used that or equivalent and prefer it over the other methods.

yeah i would like to hear about it too

You would like to look into: https://docs.aws.amazon.com/batch/latest/userguide/launch-templates.html and Cloud-Init section of this you can provide this also via the terraform resource afaik.

I would use suggest looking TF data templates to pass to pass custom scripts as user data to the launch templates

@curious deviant If the TF modules are hosted in private repo, your CI agent will need clone access to your repository ,You may create a bot user in ur github org with read-only access to terraform modules repo and configure the access token in the CI pipeline with some sort of encryption , so when the pipeline executed the CI agent authenticates into your github org to clone TF repos

thanks !

yes, you instantiate a module and provide the variables for it. The variables’ values can come from [variables.tf](http://variables.tf) or from any .tfvars files

see example https://github.com/cloudposse/terraform-aws-eks-cluster/tree/master/examples/complete

OK cool so I just put all the variables I want defined in the variables.tf and go from there. Cool thank you

It’s been a while since I had to manage lambdas but my go to was the Serverless Framework. Terraform isn’t good at managing the lifecycle of a lambda function.

terraform does suck for lambdas.

one of the best modules (or so I’ve heard) is https://github.com/claranet/terraform-aws-lambda if you do want to go this route.

Im using SAM from AWS

or create the ebs volume separately and use userdata to attach it on boot? This is correct

the issue happens even I run the example from the examples/complete directory verbatim. I would expect that I could run terraform apply seemingly over and over if there are no changes to state, but the 2nd run always results in the error.

can’t confirm right now the problem, but the gist of the issue is a fundamental problem with how the auth config map is managed. basically, managed eks node groups create this automatically if it doesn’t exist, and update it if it does exist. the hack is to create an interstitial dependency in terraform using a null resource so we first create the map with terraform (so terraform is aware of the resource) before the node groups and then create the managed node group.

not ruling out that the behavior might have changed as is sometimes the case.

@johncblandii have you noticed anythign lately?

@Alex Friedrichsen manually delete the config map from the cluster

It was created before by something I don’t know what

And now it’s not in the terraform state for some reason

Or you can import it

Thanks @Erik Osterman (Cloud Posse) and @Andriy Knysh (Cloud Posse).

Seems like the issue was in fact what was described about the configmap not being in state.

Equal parts working with an existing cluster (provisioned previously with an older version of the same module) and equal parts network timeout on my end when trying before with new clusters. Thanks again.

@Alex Friedrichsen if you don’t do it already, you should pin the module to a release (either the last one or the prev one if you used it before). Don’t use master b/c it could introduce breaking changes.

Ahhh cool, glad you got to the bottom of it. Yes, makes sense that it would have been a migration problem from the old version of the module to the new one.

The terraform remote s3 backend only supports server side encryption (using AWS KMS keys).

Noted thanks for the help

That’s for the storage of remote state.

However we use client side encryption for the storage of input variables. We do this with mozilla’s sops tool.

So our inputs to terraform are committed to git, encrypted.

There is a terraform sops provider that makes loading encrypted secrets from a sops file easy

We find it much easier to handle then a full HA vault deployment (We’re a small team, so vault is overkill).

Got it , I was confused too whether we should use vault or no , it’s definitely an overkill for small teams , That was some good insight

Thank you

You’ve seen #variant (variant2)

comes with Slackbot built-in

Terraform v0.11.14

not yet, check this issue, and the prs linked to it… https://github.com/terraform-providers/terraform-provider-aws/issues/9330

Oops oky I’ve found a place where the “required” are marked as such (but not in the main page, only when I click inputs)

And none of the examples basic/complete validate, missing required arguments or having unsupported arguments… (tf 0.12.24)

I’m surprised this works. Looking at it makes think it would produce a list like ['key1=key2', 'key3=key4', 'value1=value2', 'value3=value4']

you made me doubt myself so i went back to check it worked right:

Outputs:

trevor = bob=fred,dan=true,default_tag=default_value

and can confirm they go through my annotations of my ingress controller and then back as tags to the elb it sets up

Thanks for confirming! Interesting behavior of formatlist

enterprise or cloud?

oh, i suppose enterprise was renamed to cloud

Is the answer to create a new route table with the same routes and then do a aws_main_route_table_association?

Did you try importing the route table into your state then editing via specific TF code?

I’m writing an open source module, so I don’t think would work cause it needs to be easily repeatable across projects.

I would normally recommend avoiding the default route table and leaving it unused, but you should be able to use this resource in tf to adopt it and then tag it.

https://www.terraform.io/docs/providers/aws/r/default_route_table.html

If there are just two options then maybe make the incoming variable a binary “linux” or “windows” and do a map lookup on that within the module?

Or maybe use an ami data source to lookup the platform value and similarly map it out?

So. There was one container and the alarm does not disappear.

thanks I will try it

basically, pass through an enabled (or similar) variable and use that for count. there isn’t a terraform feature for enabling/disabling a whole module.

HashiCorp says they’re going to add for_each to the module level soon

thanks for you input, it did worked, but then I had to handle many array. so I have remove the exception that needs this customization

Ya, programatic creation of workspaces

Consider defining them all in a .yaml config and then terraforming it out.

But you can’t do a for_each style approach on providers so what would yaml give you?

creating the workspaces uses a single provider with terraform cloud, no?

the provider config can reference variables, right? so per workspace, toggle the region variable, reference it in the provider config?

then each workspaces is associated with a region which terraform runs against

Right, that’s what I said in post, workspace per region. I was wondering if there was anything more elegant or that’s what I’d have to do.

Not that I am aware of

@johncblandii

well, it’s a workspace per deployment, so it could be a matrix of region+whatever-else. half of why i avoided workspaces entirely, just too inflexible

Nice. I’ve actually been wanting to pilot doing this in Azure DevOps since this would be a very very simple thing to just flip into a matrix build and then have it run plans for every region and even account all in one plan.

It’s not prebuilt, have plumbing to do, but if I can convince folks to give me a chance to do it, i think its more scalable

yeah, we basically use terragrunt to give us a lot of that flexibility, while retaining deployments in independent remote-states

also allows us to test locally more easily

but if you don’t want terragrunt or some other wrapper, there’s a lot more legwork IMO

So it comes back to if I’m NOT using terraform cloud that terragrunt provides the most flexibility to rapidly deploy compared to doing all that in the Azure DevOps pipeline.

I’m still mixed on approach. Trying to keep it simple, but investing in a little more complication with something like terragrunt might provide initial complications while simplifying cross region/account deployments a lot

Decisions decisions

you can still use terragrunt in the pipeline

To a team early in any automated pipeline adoption do you feel terragrunt will over complicate or will be reasonable to adopt as a team

i think if you’re looking at multi-account, multi-region then the complexity will be there with any solution, and everyone will need to learn something new and dig pretty deep to understand the abstractions

And terragrunt is established enough that it’s a solid approach vs trying to do it all in pipelines etc. Leverage terragrunt to simplify while using pipelines to run what already was simplified

Makes sense. I tried it last year but still wasn’t 100% on cloud pipelines with terraform so it over complicated it for me. I’m pretty solid now so probably would make more sense

latest terragrunt has gotten pretty powerful with it’s dependency and file/code generation features, and access in terragrunt.hcl to tf 0.12 features and functions (e.g. for loops and templatefile(), etc)

sounds good. I’ll review for a new “root-module” repo and see how it goes then

#terragrunt has obviously proven very effective for this. Also, our latest project is using #variant and it’s hands down the coolest cli approach I’ve seen and it’s all written in pure HCL and compiled down to a single binary. We can define our cli exactly the way we want it to work and it’s not limited to terraform.

we’re using it with helmfile as well.

i’d love to see an implementation using variant to compare to terragrunt

I’m in a windows environment so didn’t look at variant more. I’ll have to rexamine.

I work with PowerShell and it seemed bash focused

ya, i want to discuss on office hours soon. just need to get approval from client.

just use wsl+bash instead of powershell, problem solved!

variant cross compiles (it’s in go)

variant gives you a modern cli interface to wrap any and all commands, workflows, pipelines, etc you need.

That way you define it in variant, run it locally for development, then stick those commands in your Ci/CD pipelines for automation.

this gives you the best of both worlds. simpler CI/CD pipelines and local execution for development.

Ok, so basically an alternative to like act to run your github actions locally.

Instead now we remove build system and have the commands run by this executable and pipelines just run that and don’t do more logic/processing

ya, sorta. however, with variant, there’s a focus on the cli interface

e.g. this can be done with make, ansible, etc.

I’m already doing that with InvokeBuild in powershell for most things, but would be interested in learning more on this. Trying to learn Golang a bit too so i like cross platform/single exe concept

but you end up with ugly as sin commands

in our case, we literally run:

mycli deploy stack

Got you. I love trying new things. Just trying to see what value it brings for my use case.

In my case, I just run Invoke-Build deploy -Configuration qa and it would do the same thing. it’s just PowerShell Core tasks that can be defined

I’ll have to experiment, sounds pretty cool

with variant, you have fully control over the positional arguments (e.g. mycli a b c) and options (e.g. mycli a b -foo=2)

it compiles down to a single binary that is easily shared

And as a plus i can easily run in vscode the same commands for example

built-in testing framework

built-in slack bot

That’s a plus for sure. Dependencies.

Nice! So I’m just chatting, not saying my solution is better. All of those things except dependencies are easy in PowerShell Core, as I have a function for New-BuildSlackMessage and I’ve designed these functions to update not post another completion message.

Argument parsing in powershell is arguably better than any cli, but definitely isn’t a single binary

After build completes it posts with the humanized timing. I hate build notification defaults as so noisy.

Definitely will look at variant when I get some time as I’m loving a single binary, just have to figure out the flexibility compared to what I’ve got right now

https://github.com/raymondbutcher/pretf has a lot of feature overlap with Terragrunt but with other features like support for multiple AWS credentials, and writing dynamic resources in Python if you feel the need.

In particular it’s good for structuring projects with tfvars in environment/region-specific subdirectories, then you run pretf from that there without cli arguments and it all works.

I used terraform CLI workspaces + terraform cloud worksapces to achieve that. my terraform cli workspaces follow this format {environment}_{region} i.e. dev_us-west-2 . Creating a new terraform CLI workspace auto-creates the TF Cloud workspace with the correct naming scheme using the name prefix in the remote backend config. The one caveat is that TF Cloud doesn’t understand TF CLI workspaces (terraform.workspace is “default” to it currently). There has been a bunch of complaints about that so I’m assuming in the future it will be fixed. The workaround I have is adding a workspace variable in TF_CLOUD and in my terraform code i just have this ternary operation that allows me to do local tf applies as well as remote tf applies:

var.workspace != "" ? var.workspace : terraform.workspace

How I get my region from my TF CLI workspace i.e. prod_us-east-1:

output "region" {
  description = "the environment provided in the Terraform CLI workspace (`<environment>_<region>[_<unique-name>]`)"
  value = element(
    split(
      "_",
      var.workspace != "" ? var.workspace : terraform.workspace,
    ),
    1,
  )
}

I honestly do not think TFC using default is a bug they’ll fix. It is just an overuse of the word workspace where anyone using the CLI thinks of workspaces differently.

In this case, there is a single place (tf workspace) being used so it is default

but…to your point, yes…just set a var for the region. AWS_REGION (if you’re using AWS) to simply get it

This issue comment shows promise but i havent verified https://github.com/hashicorp/terraform/issues/22802#issuecomment-617642088

I’m doing something very similar. I don’t do name parsing, and everything is remote. I just use 2 workspace variables in remote

• region

• stage The stage let’s me get the remote credentials appropriate for the service account rather than including in the workspace. The region allows also setting. I am trying to setup workspaces with code as much as I can but only a few projects have that fully done. If i could link the github repo when doing tf init that would be great!

Thanks for sharing this info! Cool to see different ways people are doing it. Sounds like while I’m on terraform cloud I should use just one workspace with region name like I was approaching.

In the meantime, evaluating terragrunt is promising, albeit it might be hard to build the entire workflow of plan/wait for approval and all that terraform cloud offers already. Makes me think I’ll have a harder time selling that

Just learned…. New variable: TFC_WORKSPACE_NAME

@sheldonh in TFC, that is unfortunately going to be the TFC workspace (default) as opposed to the TF CLI workspace name. Im personally curious if it is the full TFC workspace name <prefix><tf_cli_workspace> and you could prob cut the prefix out in your TF code. I haven’t tested it but this HashiCorp member said as such: https://github.com/hashicorp/terraform/issues/22802#issuecomment-618583576

I just stopped a while back even trying to use it and instead configure based on a var if required. Thanks for clarifying that variable doesn’t do what expected

will check for it right now

Thanks @Maxim Mironenko (Cloud Posse)! Sorry for commenting on the closed PR - there didn’t look to be an issue template I could use to open one for that.

here it is: <https://github.com/cloudposse/terraform-github-repository-webhooks/releases/tag/0.6.0>

thanks for your contribution

Thanks for taking care of this, appreciate it!

Just haven’t gotten around to it.

PR’s welcome

oh great, i’ll take a look

Are you working towards CIS compliance?

no, we are in the middle of a cloud security audit and some things came up which made me think about current resources im deploying and making sure they dont show up in the audit in the future

I realized that the vpce is created with a public policy so I wanted the option to tighten it down a bit

Ok, makes sense

I can’t seem to get it to work on redacting the output body. I defined an output with the name secret and the value still shows.

it doesn’t seem it works with map/object output. For example, if you take the lambda environment variables config. and pass in a secret to the map, the entry in the map is not masked. Looking at the source code it seems it may need to look for <string> = <string> and replace the right hand side.

I’ll see if I can add support for that

@zeid.derhally is just a very crude regex on the body and looks for the obvious ways they are expressed. If you get into objects I think it gets hairy quickly.

We could possibly create a new env variable called SECRET_STRINGS and anything that is in SECRET_STRING gets obfuscated.

but then you still need to write the secrets to that ENV… might be a deal breaker

the code does checks line by line, so having a regex that looks for a pattern of key/value would work, it wouldn’t be perfect but i think it would catch the majority of cases. Unfortunately it has been a while since i’ve written anything in golang

https://github.com/gruntwork-io/terratest

Most of the recently updated CloudPosse modules have tests which are good examples.

thank you..I’ll take a look

i always highly recommend these slides, also from the gruntwork team… https://www.infoq.com/presentations/automated-testing-terraform-docker-packer/

Yeah, Loren’s link is a good one. Here is the talk: https://www.youtube.com/watch?v=xhHOW0EF5u8&feature=emb_title

I’ve been testing with Python. It’s not documented yet but here are some examples: https://github.com/raymondbutcher/terraform-aws-lambda-builder/tree/master/tests https://github.com/claranet/terraform-aws-fargate-service/tree/master/test https://github.com/raymondbutcher/terraform-archive-stable/tree/master/tests

Thank you everyone for your help . Really appreciate it.

why not run terraform plan/apply again and see what it will try to create?

did you deploy anything to EKS/Kubernetes like cert-manager or externa-dns?`

before you destroy the cluster?

for example, if you deploy nginx-ingress to Kubernetes, and then will try to destroy the EKS module and the VPC, it will error out, because nginx-ingress deploys a load balancer into the public subnets

I just did that @Andriy Knysh (Cloud Posse). First I needed to get access back into my EKS cluster by using the user that provisioned the cluster (in my case, my terraform bot account)

if you don’t delete the nginx-ingress release and try to destroy EKS/VPC, it will be stuck at deleting the subnets b/c the load balancer and ENIs will be still attached to them

@Andriy Knysh (Cloud Posse) that was it

I feel like the helm release was deleted first, but the load balancer that got provisioned might not be deleted w/it

If anything like that happens, we just manually delete the load balancer using AWS console, and then destroy/plan/apply work again

that’s what I did which irritates me. But good to know that’s the best we can get sometimes.

if you forget to delete Kubernetes releases and destroy the EKS cluster, then yes, you can delete the leftovers manually. It applies to nginx-ingress and a buch of other releases like cert-manager and external-dns

did you try to create 3? what is the value ofmaster_public_dns output in this case? It might be an array if you create 3 masters, so you can add a DNS record for all of them. (I don’t remeber if it was a single value or an array, although we did not test with 3 masters)

also, what AWS console shows for the master DNS if you create 3? Does it show all three domains, or just 1?

Just 1.

master_public_dns is a CNAME pointing to a single compute.internal ip address/record

using a dynamic block

https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

looks great! thanks

well …not great, it looks ugly, but works

hah, that is a verbose dummy value. that is how i’d do it except the for_each would fit on line line.

thanks again! multi line is because of pycharm auto-format

pets vs cattle

Continuous Integration of IaC is just as valuable as CI of app dev work

automated testing

how do you do automated testing with terraform? we recently started terratest for internal modules coming from devops team but havent pipelined it

terratest

we’re a few degrees behind testing but ill definitely add more information for testing, ty

code quality/validation. Sonarqube isn’t much help but terraform has lots of linting/etc

I’d make some kind of joke about ClickOps to start. Then I’d roll right into ‘this craze of DevOps’ and how that has taken over our industry, then I’d bounce right into the fact that infrastructure was jealous so now we have Infra As Code.

Then I’d go over how it speeds up deployments, allows for closer integration into the same tools that the developers use for version control, and why it rocks

or something like that

I also generally am not allowed to talk too much for a reason so take what I say with several grains of caution…

I love the term ClickOps. I’ll definitely use that. I have some deployment speed slides, version control, and other bennies

lol wuttt what did you do Zachary

I’m way too into this stuff, it can be great for the right crowd, but scary to others

Sounds ylike you have a fun task regardless.

If you get it recorded or something you should share with the rest of us to live vicariously through ya

haha yea definitely fun! i’m trying to make it less company specific so in the future the deck can at least be released

If you want to be a total nutcase about it you can create your deck with hugo, markdown, and reveal.js (my own example here: https://github.com/zloeber/deck.loeber.live)

(rendered here: https://deck.loeber.live/#/home)

not quite the same as a pptx file…

you should add that link to the repo description at the top

thats beautiful looking! i will definitely have to convert this over and put in my own repo called presentations or similar great portfolio booster too

ha, I just haven’t created a professional powerpoint in like 20 years and found this as a geeky alternative

Nice!

It needs to follow the import example

terraform import module.my.terraform.reference.aws_db_instance.this my-rds-name-in-aws

so maybe something like this

terraform import module.db.module.db_instance.aws_db_instance.this my-rds-name-in-aws

where my-rds-name-in-aws is your RDS instance name

ouch !

thanks !

just to confirm this worked!

@RB I have not, but obviously the ordering problem in Terraform sucks. I did just notice yesterday that CP calls this out in their kms-key module:

https://github.com/cloudposse/terraform-aws-kms-key/blob/master/variables.tf#L76

A valid KMS policy JSON document. Note that if the policy document is not specific enough (but still valid), Terraform may view the policy as constantly changing in a terraform plan. In this case, please make sure you use the verbose/specific version of the policy. So maybe you need to provide all policy key / values?

In this case, please make sure you use the verbose/specific version of the policy

i will try this!

It looks like you are doing that though after reading your issue more closely… so I’m not sure what is up there.

I supply a policy like so in my module and I don’t have that issue:

https://github.com/masterpointio/terraform-aws-ssm-agent/blob/master/main.tf#L248

Maybe you need the top level version and id? That would be odd though..

the https://www.terraform.io/docs/providers/aws/d/iam_policy_document.html data source provides the version argument for you and I’ve already set the policy_id which maps to Id in JSON

you make a good point tho, perhaps I should try it with the raw JSON to see if it still has the issue… i did see this reordering happen on another usage of this data source for an iam policy but once applied, and modified, i no longer saw the reordering in the plan

¯_(ツ)_/¯

Haha yeah — I dunno. The ordering issue is a PITA wherever it rears its ugly head.

i have a gross python script for it now as a workaround… trying to get this put into our atlantis builds so we can use it in custom workflows

twitter

or, ask on twitter for a poc. hashicorp seems pretty responsive on twitter…

You can also use the HashiCorp community forum on Discuss: https://discuss.hashicorp.com/c/terraform-core/27

For terraform cloud ? I’m specifically referring to the new website service. Some usability issues I’ve wanted to give feedback on and see if it helps.

Yes,
Terraform Cloud & Enterprise questions can be categorized under the “Terraform Cloud & Enterprise” subcategory.

Cool. That makes it easy. I’m so used to forums having low priority by staff and them wanting github issues or the like. I put some initial feedback and keep up to date over there then

Yeah, this is kind of a grey area though as the Enterprise platform isn’t open source, so it’s hard to reach via Github. Try there first.

I’m trying to make it so that I created the webhooks after the fact

terraform apply -var-file=“us-east-2.tfvars” -var=‘webhook_enabled=false’ even if I do that it just keep trying to enable it

I must be getting old

Is it just the github provider being invoked/initialized? Did you provide correct creds to that provider?

Well the module is being invoked even when the enabled flag is not true

I think that happens with providers unfortunately. There is no count param so you can’t turn them off.

At least I think I remember running into that and it being an issue.

Ohhh so they always get evaluated

Yeah. So maybe passing a valid github_organization and github_token will do the trick? Regardless, that’s a pain.

I will

Pass it and see

yep, that worked

is not creating any webhooks

so it is the provider that gets evaluated always first

annoying

the bot needs to be an admin to create webhooks

after that, we usually switch it to Read

tested the module many months ago, did not see the issues with webhooks at the time

did you look at the tests for the atlantis module?

but then if I switch the bot user to read for that repo, then it tries to create the webhook repo again

is this maybe different in github enterprise ?

after changing to read

this will happen

but I can pull with that user no problem

the bot/atlantis user I have is a Member

in the org

not sure, never tested with GH enterprise

but yes, the bot needs to be an Admin at the time of provisioning

there is no way around that since to work with wenhooks you have to be admin

I understand, the bot user needs Admin for provisioning and I do not have a problem on the provisioning stage, webhooks get created and all but after is provision and I go and change the permissions of the user as a Member( can't create webhooks but can read the repo)

that is when if I run terraform plan, it fails because it gets a 404 because it can’t access the /hooks endpoint

I do not have Github enterprise, just Organizations (I think that is how is called now)

the bot needs to be Admin for terraform plan as well

(that not good, but we did not find a way out)

ok, lets set this straight

for terraform plan, bot user needs admin

after webhooks are created IF I need to run plan again, it needs AGAIN admin

looks like yes

(but we test it many months ago, things could change, and GitHub made a lot of changes in the last few months, especially for security and permissions)

so we were saying the same thing in our different levels of ESL

are you by any chance using both inline SG rules in your aws_security_group resource, and aws_security_group_rule to add additional rules to that SG?

@loren Im using aws_security_group and adding a list of values to security_groups in ingress {} block

can you share the .tf and the exact error output?

The SG config is in the main.tf of the module and when I create the module templaete, i just pas the var.redshift_ingress_security_group_ids as a list from tfvars

nothing really jumps out at me there… are you sure this is cross-account?

seems its fixed in todays provider release , gonna test out https://github.com/terraform-providers/terraform-provider-aws/pull/11809

Are you using AWS CLI creds from enviroment variables, configuration, or aws-vault?