#terraform (2020-09)

Not sure if this from the logs helps…sure doesn’t help me:

2020-09-01T10:21:55.404-0400 [DEBUG] plugin: starting plugin: path=.terraform/plugins/registry.terraform.io/opsgenie/opsgenie/0.4.7/darwin_amd64/terraform-provider-opsgenie_v0.4.7 args=[.terraform/plugins/registry.terraform.io/op
sgenie/opsgenie/0.4.7/darwin_amd64/terraform-provider-opsgenie_v0.4.7]
2020-09-01T10:21:55.424-0400 [DEBUG] plugin: plugin started: path=.terraform/plugins/registry.terraform.io/opsgenie/opsgenie/0.4.7/darwin_amd64/terraform-provider-opsgenie_v0.4.7 pid=15505
2020-09-01T10:21:55.424-0400 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/opsgenie/opsgenie/0.4.7/darwin_amd64/terraform-provider-opsgenie_v0.4.7
2020-09-01T10:21:55.435-0400 [INFO]  plugin.terraform-provider-opsgenie_v0.4.7: configuring server automatic mTLS: timestamp=2020-09-01T10:21:55.434-0400
2020-09-01T10:21:55.465-0400 [DEBUG] plugin.terraform-provider-opsgenie_v0.4.7: plugin address: address=/var/folders/r1/2sj8z7xn12s5j5729_ll_s7w0000gn/T/plugin781003244 network=unix timestamp=2020-09-01T10:21:55.465-0400
2020-09-01T10:21:55.465-0400 [DEBUG] plugin: using plugin: version=5
2020/09/01 10:21:55 [TRACE] BuiltinEvalContext: Initialized "provider[\"registry.terraform.io/opsgenie/opsgenie\>"]" provider for provider["<http://registry.terraform.io/opsgenie/opsgenie|registry.terraform.io/opsgenie/opsgenie"]
2020/09/01 10:21:55 [TRACE] eval: *terraform.EvalOpFilter
2020/09/01 10:21:55 [TRACE] eval: *terraform.EvalSequence
2020/09/01 10:21:55 [TRACE] eval: *terraform.EvalGetProvider
2020-09-01T10:21:55.524-0400 [TRACE] plugin.stdio: waiting for stdio data
2020/09/01 10:21:55 [TRACE] eval: *terraform.EvalValidateProvider
2020/09/01 10:21:55 [TRACE] buildProviderConfig for provider["registry.terraform.io/opsgenie/opsgenie"]: no configuration at all
2020/09/01 10:21:55 [TRACE] GRPCProvider: GetSchema
2020/09/01 10:21:55 [TRACE] No provider meta schema returned
2020/09/01 10:21:55 [WARN] eval: *terraform.EvalValidateProvider, non-fatal err: Missing required argument: The argument "api_key" is required, but was not set.
2020/09/01 10:21:55 [ERROR] eval: *terraform.EvalSequence, err: Missing required argument: The argument "api_key" is required, but was not set.
2020/09/01 10:21:55 [ERROR] eval: *terraform.EvalOpFilter, err: Missing required argument: The argument "api_key" is required, but was not set.
2020/09/01 10:21:55 [ERROR] eval: *terraform.EvalSequence, err: Missing required argument: The argument "api_key" is required, but was not set.
2020/09/01 10:21:55 [TRACE] [walkValidate] Exiting eval tree: provider["registry.terraform.io/opsgenie/opsgenie"]
2020/09/01 10:21:55 [TRACE] vertex "provider[\"<http://registry.terraform.io/opsgenie/opsgenie\|registry.terraform.io/opsgenie/opsgenie\>"]": visit complete

it must be opsgenie, because supplying the OPSGENIE_API_KEY env var stops that error, even though the one provider I do declare has app_key defined.

@Dan Meyers

(our tests are using OPSGENIE_API_KEY)

and I think we removed api_key from the modules because providers in 0.13 should be passed by reference rather than invoked inside the module

also, if you haven’t yet had a chance, check out our new config submodule of the opsgenie stuff - it supports YAML configuration for your desired opsgenie state

just a point of clarification, the note above says app_key is defined but the error references api_key – just want to make sure thats a typo

Also, here’s a clip from #office-hours where we talk about the new config module: https://www.youtube.com/watch?v=fXNajuC4L1o

Thanks, @Erik Osterman (Cloud Posse). I was on the office hours call, when this was discussed. Very exciting and a really cool abstraction. I had already set up our basic config and just wanted to throw a few minutes at it to bring it up to 0.13 and see some of those changes reflected in Opsgenie (once our trial has been extended) so i’ve just been working on my original implementation. Once I get a clean plan from that and a few spare moments, i’ll probably convert to the config module.

let me know if you don’t get it working

@Jeremy G (Cloud Posse) could this be related to null label changes?

@sahil kamboj Which modules of ours are you calling? What versions of our modules are you using? As we say in all our documentation, you should be using a specific version, not pinning to master.

@Erik Osterman (Cloud Posse) Unlikely to be due to label change as neither terraform-aws-rds nor terraform-aws-rds-replica use [context.tf](http://context.tf) or the new label version.

@Jeremy G (Cloud Posse) sry was disconnected for month due to COVID it was a silly mistake in name parameter , its the db name not rds and should be same as master.

one con: you’re limited to running their version of terraform and cannot BYOC (bring your on container). the good thing is then your limited to running vanilla terraform, the bad thing is you cannot use any wrappers or run alpha versions of terraform.

fortunately, they’ve just released runners for TFC. this was a huge con before that, since it wasn’t possible to use things like the postgres provider to manage a database in a VPC.

related to this, providers are now easily downloaded at runtime. also was a limitation, but no longer is.

the biggest compliant I hear is the cost of TFC enterprise & business.

edit: Disregard - see below

another con is that you can’t run your own workers (in your own accounts) without shelling out for a $$$ enterprise contract

so any moderately regulated workload becomes difficult to deal with from a compliance standpoint, because you’re basically giving a 3rd party highly privileged access to your accounts

you’re probably only going to get Cons in this thread which doesn’t really reflect on the product itself at all. it’s a pretty great solution for anyone who’s tried to automate terraform on their own and felt the pain. for most of us who have kicked the tires it’s frustration that we can’t use it because of tick boxes rather than technical deficiencies.

@Chris Fowles can you clarify? I thought runners are now supported with the business account.

Thank you

Yea, so agree with @Chris Fowles - I would pick terraform cloud over all the alternatives (e.g. Atlantis, Jenkins, or custom workflow in some other CI/CD platform). What it does, it does very well and better than the alternatives.

ok, but there’s now a “hybrid” mode where the runners can be a “private installation” but the dashboard is SaaS.

any doco on that? I’ve not seen that yet

ahhh ok found it

https://www.terraform.io/docs/cloud/workspaces/agent.html

https://www.hashicorp.com/blog/announcing-hashicorp-terraform-cloud-business/

pro: also supports SSO now

con: (okta only)

con: Business pricing is “Contact us because we don’t know how to bill this yet”

Ya, hate that.

Sounds like after first creating the bucket and table with the module, the step of reimporting the local state was not performed.

Have you followed these steps: https://github.com/cloudposse/terraform-aws-tfstate-backend#create

I see a PR here ^^

^^

oh jeez. ya, all of that should be changed to using the try() function instead. more reason to do that now.

@antonbabenko thought you might also want to be aware, in case you get reports on your modules (i hit it on your vpc module)

@RB unfortunately, try() doesn’t fix it… in the repro case in the issue, this still generates a persistent diff: empty = try(random_pet.empty[0].id, "")

thanks @loren

interesting so it affects both cases.

it should affect our modules when enabled=false I suppose

both cases, since both cases return “” (empty string)

and TF 0.13 just can’t compare empty strings correctly

pretty much, yeah. one workaround i’ve found is to use a ternary with the same condition that you use for the resource, so this does work: empty = false ? random_pet.empty[0].id : ""

if TF 0.13 can evaluate the expression all up front, then it works

isn’t better to way for them to fix it?

before changing every module?

we don’t need to change every module since it could affect it only when enabled=false

but yes, it’s better for them to fix it

they get pay for it

they responded and explained why it is happening. it makes sense, though i don’t know what edge cases led them to make the change so that resources with 0 instances are not stored in the state. i’d expect this issue will not be solved quickly, if at all

personally, i’ll be switching to that workaround wherever i can, which i think is a more stable solution anyway

maybe if we all upvote the issue, they will resolve it sooner

please do!

Thanks a lot, @loren!

what about using https://github.com/scottwinkler/terraform-provider-shell (and have all of that in TF state)

thanks, I’ll have a look into that.

Shouldn’t. Terraform plan can be run to preview before you do anything.

Terraform apply also asks for confirm.

Always test/learn in safe environment :-)

Lastly RESOURCES = STUFF YOU CREATE data sources read …. If you didn’t create with terrafl use data and it won’t try to create it.

Ie…vpc subnets etc use data unless you are creating.

Thanks for the reply. Appreciate the information

I was looking for something similar as well. The suggested solution was https://sweetops.slack.com/archives/CB6GHNLG0/p1599083030235300?thread_ts=1599082729.235200&cid=CB6GHNLG0

I don’t think that’s what I’m looking since I’m talking about totally outside of the context of a Terraform project.

We’re in the process of doing the same thing, and we’ve settled on AWS’ SSM Parameter Store, to store key/values as opposed to Terraform outputs, as a source of truth for both Terraform and other tooling (eg. Ansible, GitHub Actions, etc.)

aws_ssm_parameter resources to write data to SSM, and aws_ssm_parameter data sources to read data from SSM.

that’s a nice idea.

It does mean the scripts will need to reach into the parameter store for values tho.

For sure, but there are lots of SDK’s available for AWS API endpoints.

One thing I am considering is the dotenv file and the tfvars file is the same format

Annoyingly Param Store still isn’t supported by RAM, so you’ll need well defined roles, prefixes and encryption keys tho’ https://docs.aws.amazon.com/ram/latest/userguide/shareable.html

.. or … https://www.num.uk/

I had a little bit of a hard time convincing my architect to throw stuff into SSM, but it’s gone very well and he’s really embraced the idea.

Ah yeah — I was not thinking yesterday. This is a perfect usecase for SSM PStore + Chamber. Appreciate the reminder @Drew Davies!

iterator, as in count or for_each? if so, sure, certainly

cool. never had the need so just making sure before I wasted more time

there is a bit of a chicken/egg thing going on. cloudposse has a pretty good writeup of how to keep it all in terraform… https://github.com/cloudposse/terraform-aws-tfstate-backend#create

Also fyi terraform cloud can be used for free for state file management and has versioning, locking and more. Might be worth considering. I have been using it exclusively for almost the past year instead of any s3 buckets.

I found a github action that creates the backend in terraform cloud fyi

make sure you have the aws provider version et in your provider resource and try running terraform 0.13upgrade

or similar, I think it just cant be null anymore

Is it an aws provider issue? It looks like it’s a null provider issue

I am generating this provider block now:

terraform {
  required_providers {
    aws = {
      source = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}
provider "aws" {
    region = "us-east-1"
}

oh I see, yea i just skimmed over. I hit this exact issue with aws

I would try to explicitly set null provider too maybe

hmm ok, I’ll give that a shot, thanks

or try terraform init -reconfigure

No luck, it looks like I might need to remove it from state manually?

@Andriy Knysh (Cloud Posse)

It looks like I was able to replace the state references manually, and that worked

For example, every line had:

"provider": "provider[\"<http://registry.terraform.io/-/null\|registry.terraform.io/-/null\>"]",

just replaced it with

"provider": "provider[\"<http://registry.terraform.io/hashicorp/null\|registry.terraform.io/hashicorp/null\>"]",

(then state pushed)

intersting..good to know

There is a terraform to perform the find-replace automatically:

terraform state replace-provider -auto-approve -- -/null registry.terraform.io/hashicorp/null

I’ve seen this a few times with 0.12 -> 0.13 conversions that don’t use terraform 0.13upgrade and get messed up state files. Bit of a pothole IMO.

Good tip @Alex Jurkiewicz

Just ran into this issue too. Perhaps an idea to place this info somewhere?

Error: Provider configuration not present

To work with
module.subnets.module.nat_label.data.null_data_source.tags_as_list_of_maps[3]
its original provider configuration at
provider["registry.terraform.io/-/null"] is required, but it has been removed.
This occurs when a provider configuration is removed while objects created by
that provider still exist in the state. Re-add the provider configuration to
destroy
module.subnets.module.nat_label.data.null_data_source.tags_as_list_of_maps[3],
after which you can remove the provider configuration again.

You should submit a PR to add this to the 0.13 migration page in the official Terraform docs

Hrm… is that convention enforced or just an example?

The reason we did this is so could correlate teams with repositories with stakeholders (e.g. it’s 3am, you get an alert for some service, but the error is not obvious and requires some domain expertise, you don’t know what to do, so who should you escalate to?)

Right. Figured, but that’s just text as far as this exercize is concerned, right? That info is consumed by (tired) humans or …something else?

Yup, just consumed by humans. Not used programatically.

Got it. Thanks.

This is in response to this bug: https://github.com/terraform-providers/terraform-provider-aws/issues/5011

I understand you can turn off refresh but that would also inhibit the plan

I’m thinking perhaps we can output the current state to a file, do a terraform plan, and push the outputted state back up ?

what are your thoughts?

i do a terraform plan with a new version all the time. it’s never affected the tfstate

Keep in mind a plan does a state refresh, so it must update the state file.

here’s an old issue recommending the use of -refresh=false

https://github.com/hashicorp/terraform/issues/3631

perhaps it is updating the local copy. that makes sense. it is certainly not updating the remote state, or at least, not in a way that impacts the ability to run applies with the older version

interesting. i’ll try this out locally and see. this was a concern with my uppers regarding atlantis doing plans across the org so i thought id ask

we had the problem a couple times where someone “accidentally” updated the remote state on a dev environment by using a newer version of terraform than the rest of the team. we implemented strict pins of the required_version in the terraform block, and it’s never been a problem since. upgrades are now very deliberate.

terraform {
  required_version = "0.13.1"
}

yep, i think this is what i’ll have to do as well across a repo before i can add atlantis bot to access the repo

Refreshing Terraform state in-memory prior to plan…
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage. I would assume that means it isn’t updating the state, but perhaps it behaves differently if you save the plan to a file?

I guess it does say “the refreshed state” but that doesn’t necessarily mean it isn’t updating the state file with the latest version…

Here’s some stuff to get you started

https://github.com/cloudposse/terraform-example-module

https://docs.cloudposse.com/community/faq/

https://docs.cloudposse.com/community/code-reviews/

https://docs.cloudposse.com/community/automated-testing/

https://docs.cloudposse.com/terraform/terraform-best-practices/

If you want to participate in review Pull Requests, send me a DM. We get dozens of PRs and can use all the help we can get.

So far so good with the config mod. Our rotations are pretty simple for starters, but we do need things like ignore_members and delete_default_resources in teams, so i’ll submit a PR for those changes in a little while.

Ping @Dan Meyers when you need a review since he’s currently the “owner” of that.

Will do. Thanks, Erik!

IMO, if it’s the same region between your environments, then that’s fine.

I was thinking that if you had a service that was fundamentally cross-region, for example a jitsi deployment where you want RTC video bridges in regions close to the users.. then you would want to place the stage at a higher level than the region

yeah, you can have it inside each stage.

There is also other scenarios like AWS ACM or Lambda@Edge, where you need to have them on us-east-1 regardless of where your main region is supposed to be.

personally, I’ve recently moved away from terragrunt, due to the additional (cognitive) overheads it presented with regards to the additional wiring.

I have this swapped around

terraform/$ACCOUNT/$ENVIRONMENT/$REGION/$THING

and using Atlantis/OPA integration

also consider cross posting in #terragrunt for more feedback

personally, I don’t like imputing the state by filesystem organization (the “terragrunt” way).

This year, we’ve moved to defining the entire state of an environment (e.g. prod us-west-2) in a single YAML file used by all terraform workspaces for that environment. That way our project folder hierarchy is flat (e.g. projects/eks, or projects/vpc). In one of the project folders is where you have all the business logic in terraform. Now the relationship between eks and vpc and region, environment etc, will all be defined in a single configuration file called uw2-prod.yaml (for example). We make heavy use of terraform remote state so pass state information between project workspaces. Best of all, the strategy works natively with terraform cloud, but terragrunt tooling does not. Using #terragrunt with terraform cloud means terragrunt needs to be triggered by some other CI/CD system.

Here’s a sneakpeak of what that configuration looks like. No copying dozens of files around to create a new environment.

Do you have some type of hierarchy of variables that get loaded in and if so, how does something like Atlantis make sure it triggers for the right thing e.g. if eu-west-1 vars change and it is a single file, is that for prod or dev? Personally I don’t mind the opinionated hierarchy. Isn’t too much boiler plate code so relatively DRY, is pretty obvious what is going on, can’t shoot yourself in the foot easily. Folks who are new to Terraform get it

Are ya’ll using Terraform cloud now rather than e.g. Atlantis?

Yes, so basically managing terraform cloud is done using terraform cloud. So the workspaces are defined by the configurations. When the config/uw2-prod.yaml file changes, terraform cloud picks up those changes and redeploys the configuration (e.g. tfvars) for all workspaces. And using triggers, we can depend on the the workspace configuration.

The problem with atlantis is it cannot be triggered from other pipelines (not elegantly at least. e.g. I don’t consider a bot running /atlantis plan a solution). But with terraform cloud, it can be triggered from other pipelines. This is the main reason we’ve reduced usage of atlantis in new engagements.

hierarchy of variables that get loaded yes, so each project can still define defaults.auto.tfvars for setting shared across all projects.

Then there’s the concept of globals for an environment:

projects:
  globals:
    stage: prod
    account_number: "xxxxxx"
  terraform:
    dns-delegated:
      vars:
        zone_config:
        - subdomain: prod
          zone_name: uw2.ourcompany.net

Those globals get merged with vars

So changes to projects/ causes a plan for all environments (since it affects all environments)

Changes to .yaml causes plan for the yaml modified. Once this is modified, it cascades and triggers all dependent workspaces to plan.

The import aspects to note:

1. state stored in git via yaml configurations for each environment
2. terraform cloud manages terraform cloud workspaces defined in the yaml
3. terraform projects use settings defined in workspace (that were set by the terraform cloud configuration). this is why it’s all “native” terraform.
4. triggers are used to auto plan/apply dependencies.

you know, in our original approach we started with our little project called tfenv to terraform init -from-module=... everywhere. We outgrew that because there were just too many envs. We were fighting terraform. Terraform 0.12 came out and changed the way init -from-module worked. The lesson taught us to stick to vanilla terraform and find out an elegant way to work with it. The problem with terragrunt in my book is that is diverging from what terraform does natively. It provided a lot more value before 0.12 and 0.13, but that value is diminishing.

https://sandimetz.com/blog/2016/1/20/the-wrong-abstraction

the other thing we outgrew was the one-repo per account. it was impossible (very tedious) for aws account automation and general git automation.

Woah, ok, many questions here.

We’re having trouble managing our terraform structure. We aren’t a typical sass that has one product with dev/qa/prod stages.. rather we operate kind of like a consultancy (though we’re not) and operate the same solution multiple times for different clients. So we have heavy heavy re-use of all our infra code.

We are using vanilla terraform now with a structure like so:

but with more prod, more test, more clients, more modules, more everything. it is gnarly.

Also.. every one of those client roots is its own AWS account.

each of the ops/*-client/terraform/$module/ root modules uses relative paths to ../../../../modules/$module

not shown of course are the vars and other data specific to each deployment

there is so much duplication… and things start to diverge over time.. one client doesnt use cloudflare so we wire up a special dns provider in one of their root modules

it’s a massive headache.. hence why we are looking at terragrunt

We seem to be a bit behind the status quo, as I’ve noticed more people being vocal about moving away from terragrunt as terraform improves

The single project yaml definition looks great, but is only available on terraform cloud? What other options do we have?

Creating a new deployment when using Terragrunt will still require copying a bunch of files and a directory tree, but at least that content is very thin.. it’s all just vars, not actual terraform code.

Now I’m unsure.. Is that project config (config/uw2-prod.yam ) a terraform cloud feature or tooling from cloudposse?

We also make heavy use of remote state, and any attempt to generalize the config is intractable as you cannot use vars in tf backend configs.

Yes, so basically managing terraform cloud is done using terraform
cloud. So the workspaces are defined by the configurations. When the config/uw2-prod.yaml file changes, terraform cloud picks up those changes and redeploys the configuration (e.g. tfvars) for all workspaces. And using triggers, we can depend on the the workspace configuration.

Lot to unpack there. Sounds like cloudposse tooling?

It’s our own strategy, but terraform cloud doesn’t allow you to bring your own tools. Therefore the solution is technically vanilla terraform.

For coldstart though, we use a tool written in variant2 to bring up the environment, but for day-2 operations everything is with terraform cloud

Do you have a count set for the module?

@Alex Jurkiewicz yes..

Your reference is wrong

it needs a [0] probably

The error is saying “the thing you are trying to read the attribute security_group_id on is not a map/object”

thanks.. I guess it has to be module.app_beanstalk_environment.0.security_group_id

I wish that would have been the error, it would be easier to find the issue

no, it needs to be module.app_beanstalk_environment[0].security_group_id

I think both work.. thanks!!

I just want to be able to set it with an =

I really, really hate block syntax. I don’t think I’ve seen a single case where I prefer it over just a list and a for loop.

I’ve just hit a wall similar to this. I’m trying to define a number of s3 buckets. Some of them have lifecycle_rules, one uses encryption, and one is publicly exposed. This translates to about 2-3 different blocks for each use-case. It would be great if I could either assign a block directly, pass a block, or have it work with empty blocks pre-defined. I’m wondering if there’s something fundamental that I missed with TF 12 syntax.

This looks like a ‘no, not ever’: https://github.com/hashicorp/terraform/issues/21458#issuecomment-496022674

That is absolutely and utterly infuriating.

prioritizing the reader over the writer grumble

I would like to start a petition to fork Terraform for the express purpose of removing attribute blocks because they’re dumb.

yes, they are a poor hack for the lack of more expressive variable typing

What really annoys me about them is it just renders down to a json array anyway, yet they stop you from just specifying an array to replace them for “readability” (see: verbosity)

Then they had to add a whole bunch of extra syntactic rubbish to make up for it, like the ‘dynamic’ keyword

yeah. The purpose of HCL is to be more declarative than full code. dynamic is extra unnecessary code

I’m so glad to hear others express this. I’d been thinking I was alone, and now I feel validated.

I’m also glad I’m not alone. Has this been brought up on github anywhere? I’d even really like to see just an optional flag that says “–allow-assigning-to-blocks” or something, so the user has to acknowledge it’s not best practice but let us do it anyway

You used to be able to do that, there was a hacky way to do so. They patched it out several versions ago (around 0.9/0.10 IIRC).

I recall, it was “accidentally allowed” from what I read

yes, unintentionally

Hashicorp aren’t interested in giving you multiple ways to declare the same graph. Which IMO is a good thing. I don’t want two ways to define these sub-blocks. That’s a price too high

That’s fair, but I think the way that they decided on is far too limited and doesn’t mesh with the rest of the functionality they’ve given the language. Unfortunately I can’t think of a reasonable suggestion to fix the current system other than “just let us do it the natural way”

I agree. I think they’ve painted themselves into a corner but I hope not

var.name is passed into terraform-null-label for processing into the prefix. As far as I can tell it shouldn’t replace slashes though. It might be easier to pass it through separately and let null-label do the work of creating the prefix for you

name = each.value
namespace = var.orgPrefix
stage = var.system

yes, then it works, but still - why mangle the name

I just blew all my repos, which is of course a rookie mistake since I did not check the plan properly 1st

I promise I spent a few hours on this :smile: just came across try(...) which seems to solve my problem

in modules with count, the output is list, and you need to use something like join("", xxxxx.*.name) to get a single item

in modules with for_each, the output is a map

this is a map that I’m working on

I get an error about it being an empty tuple

changing to this seems promising

if it’s empty, try will not help

it just hides issues

hmm

how does module.alb look like?

pretty standard

https://gist.github.com/cwar/1feb175e6505e393e391d4e866279672

this var.create_alb ? module.alb[0].alb_data : var.params.alb

is the same as var.create_alb ? join("", module.alb.*.alb_data) : var.params.alb

and same as try(module.alb[0].alb_data, var.params.alb)

all should work, but they have a slightly diff behaviors

I see.. is the empty tuple not considered an error?

show how you invoke the module?

do you correctly set the variable var.create_alb in both the module and the invocation of the module?

it sounds like it’s false here

module "alb" {
  source = "./modules/ALB"
  count = var.create_alb ? 1 : 0

variable "create_alb" {
  description = "Set to true to create ALB for this service, leave false to remain on shared cluster ALB"
  default     = false
}

is in my [variables.tf](http://variables.tf) file on the module… most of the invocations here will have this set to false

and true here

alb                          = var.create_alb ? module.alb[0].alb_data : var.params.alb

this is in the same module though how can they have different values?

alb                          = var.create_alb ? module.alb[0].alb_data : var.params.alb

and

module "alb" {
  source = "./modules/ALB"
  count = var.create_alb ? 1 : 0

are in the same module

the first is a local

that i use for setting listener rules and such

if I try the join function w/ empty string I get this:

Error: Invalid function argument

  on ../../../modules/services/app/main.tf line 8, in locals:
   8:   alb                          = var.create_alb == true ? join("",module.alb[0].alb_data) : var.params.alb
    |----------------
    | module.alb[0].alb_data is object with 2 attributes

Invalid value for "lists" parameter: list of string required.

I feel like perhaps I’m making a silly mistake somewhere..

this module.alb[0] gets the first item from the list

should be join("",module.alb.*.alb_data)

ah I see

where module.alb.*. is list

you know the try(...) actually is working very well here

yes try does the same

hmmm

@Andriy Knysh (Cloud Posse) - thank you so much for helping me talk through this issue, I really appreciate it and everything else CP has provided the community! I am happy w/ current solution but still interested in talking this through… but if you are busy no need to continue here!

if you share the entire code, we can run a plan and see what happens (it’s difficult to understand anything looking at snippets of code)

module.alb.*.alb_data

and

var.params.alb

should be the same type

and since alb_data is not a string, join(""…) will not work here (sorry, did not notice that)

so try is the best in this case

(but still does not explain the original issue you were seeing)

hmm this confuses me more or maybe sheds some light…

Error: Inconsistent conditional result types

  on ../../../modules/services/app/main.tf line 8, in locals:
   8:   alb                          = var.create_alb ? module.alb.*.alb_data : var.params.alb
    |----------------
    | module.alb is tuple with 1 element
    | var.create_alb is true
    | var.params.alb is object with 2 attributes

just playing around w/ it trying to understand

it says module.alb is a tuple with 1 element but I’m trying to refer to the alb_data output.. not module.alb

the 2nd output I understand… module.alb is empty because we don’t create the module since it has count set to 0

I try to avoid using * since 0.12 came out

try(module.alb[0].alb_data, var.params.alb) should work for you

no, outputs can only be created explicitly

Maybe if a tool was written to convert the tf to hcl, read all the modules and outputs, then check if all modules were outputted. If not, flag it

Or better yet, make the tool dump the module outputs if they are missing

I thought you could export the entire module output as a full object. Is that only for resources?

you can export the entire module reference as an output of the current module

We need a way to splat an output onto the module

output * {
value = module.bananas
} 

I version the resource which forces a recreation

I think tainting the resource also does this, but is something you’d do on demand. Anyone?

yea Tainting will totally work too. I have a pipeline that pgp encrypts the secret and then decrypts it and stores the creds in vault. any consumer of the creds uses vault to retrieve them. this allows for seamless rotation.

@Luke Maslany can you add some more details about what technology you’re using for your pipeline? Perhaps there are alternatives to using/rotating IAM access keys and instead using runners with service accounts.

Thanks both.

I might be overthinking this then. We do have pipelines that store IAM keys in the AWS SSM parameter store.

In my mind I was looking to identify a way to have terraform toggle between the two access keys associated with an account.

I was thinking to have terraform recreate the key that wasn’t in active use, then use the new key to complete the terraform execution.

If the new deployment failed, the old key would be unaffected and would continue to work.

If the deployment succeeded, the production instances would now be running with the new key, and the next time a deployment was run it’d replace the old key as part of the new execution.

I’m currently looking at a deployment pipeline that uses a Jenkins job, to execute terraform, which creates a new ASG using the most recent AMI of an immutable image and user data created through the interpolation of a file template. The interpolated file template contains the access key. The user data then performs a transform on the config file for the application on the instances at runtime.

I am not thrilled about the current solution as the keys are visible in the user data for the instance.

However I am not sure how quickly I will be able to unpick the current implementation as it uses multiple nested modules, running some pretty old versions of terraform.

With that in mind, I’ve been looking to see if there was a quick win that would allow me to cycle the keys on each deployment, by adding some logic into the module that currently provisions the aws_iam_access_key to swap between key1 and key2 depending on which key was already in active use.

I have a feeling though it is going to be quicker to just fork the current module and update the user data script to pull the key direct from the SSM Parameter Store at runtime using an IAM role on the instance.

this is better in the #atlantis channel

look at the Cloudposse atlantis module and see how they use Parameter Store

you can do somemething similar

You could probably use dynamic blocks to do what you want https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

I did make that attempt. I wasn’t able to get it to work.

The way that S3 is configured in terraform doesn’t really lend itself to making a generic module that covers all possibilities since there are so many options that aren’t separate resources.

.. and since those options are implemented as blocks. ( with all the restrictions of block syntax )

Sometimes blocks in blocks

yes. blocks in blocks. sometimes nesting to a stupid degree.

Cloudwatch is similarly annoying

I think that feeling comes from the overhead necessary to implement a declarative language. At least with CF, I never find myself going over a single blog post trying to figure out how for / for_each / objects work for a new use-case, only to find out that the way I wanted to do something was decided by the TF team to be not-a-best-practice. With CF, I more often find myself just realizing that the thing I want to do is missing.

With that said, though, the TF team does seem to be pretty responsive, always asking about the exact use-case the user is trying to accomplish, and often offering a work-around.

… I just feel like I rarely have the time to pull away from getting something working to create such posts.

Yea, totally agree

(unfortunately, we still have some places we added >= 0.12, < 0.14 #FML

Anyways, we have some better tooling to handle this and hopefully will be less painful with every iteration

@Erik Osterman (Cloud Posse) Is the new standard to only pin >= 0.12?

yes, >= 0.x where x is the minimum supported version

so maybe >= 0.13 if it uses for_each on modules

but never < y

Let’s DO IT!!!!!!

lol

fyi, just saw that alpha releases are the new norm… i don’t think they are about to drop 0.14 so soon after 0.13 though… https://discuss.hashicorp.com/t/terraform-0-14-0-alpha-releases/14003

and here are details on the feature in this alpha release… https://discuss.hashicorp.com/t/terraform-0-14-concise-plan-diff-feedback-thread/14004

More Info: 1: I found https://github.com/cloudposse/terraform-aws-rds-cluster/issues/26 from 2018, but thought there might be new information elsewhere. 2: We use Jenkins and Atlantis, so we could have Jenkins call the Ansible playbook and put our root passwords in Vault, but I’d like to make things simpler.

I use Parameter store

in jenkins there is a Parameter Store plugin for jenkins

you can create the PS with terraform with a lifecycle rule to ignore value changes

that is what Terraform knows

but the value is injected by hand, jenkins, another tool

chamber etc

Can you paste what you’re seeing here?

Error: Could not load plugin

Plugin reinitialization required. Please run “terraform init”.

Plugins are external binaries that Terraform uses to access and manipulate resources. The configuration provided requires plugins which can’t be located, don’t satisfy the version constraints, or are otherwise incompatible.

Terraform automatically discovers provider requirements from your configuration, including providers used in child modules. To see the requirements and constraints, run “terraform providers”.

Failed to instantiate provider “registry.terraform.io/-/aws” to obtain schema: unknown provider “registry.terraform.io/-/aws”

You try TF_LOG=DEBUG terraform plan to see verbose output

Wait, what’s that “/-/aws” there? How are you declaring the plugin?

I am not declaring the plugin I just have my provider as aws

Can you look for any other references to aws in your code? Obviously exclude things like ‘resource “aws..“’

` 2020/09/11 12:28:45 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/09/11 12:28:45 [INFO] backend/local: starting Plan operation
2020-09-11T12:28:45.887-0400 [INFO]  plugin: configuring client automatic mTLS
2020-09-11T12:28:45.910-0400 [DEBUG] plugin: starting plugin: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.6.0/darwin_amd64/terraform-provider-aws_v3.6.0_x5 args=[.terraform/plugins/registry.terraform.io/hashicorp/aws/3.6.0/darwin_amd64/terraform-provider-aws_v3.6.0_x5]
2020-09-11T12:28:45.922-0400 [DEBUG] plugin: plugin started: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.6.0/darwin_amd64/terraform-provider-aws_v3.6.0_x5 pid=30307
2020-09-11T12:28:45.922-0400 [DEBUG] plugin: waiting for RPC address: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.6.0/darwin_amd64/terraform-provider-aws_v3.6.0_x5
2020-09-11T12:28:45.957-0400 [INFO]  plugin.terraform-provider-aws_v3.6.0_x5: configuring server automatic mTLS: timestamp=2020-09-11T12:28:45.957-0400
2020-09-11T12:28:45.988-0400 [DEBUG] plugin: using plugin: version=5
2020-09-11T12:28:45.989-0400 [DEBUG] plugin.terraform-provider-aws_v3.6.0_x5: plugin address: address=/var/folders/cs/fpp3k3zj61q0hd1pn41nmf9xl5ttrf/T/plugin474919102 network=unix timestamp=2020-09-11T12:28:45.988-0400
2020-09-11T12:28:46.212-0400 [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2020-09-11T12:28:46.215-0400 [DEBUG] plugin: plugin process exited: path=.terraform/plugins/registry.terraform.io/hashicorp/aws/3.6.0/darwin_amd64/terraform-provider-aws_v3.6.0_x5 pid=30307
2020-09-11T12:28:46.215-0400 [DEBUG] plugin: plugin exited 

`

I couldnt find any … this worked like 10 mins back .. I first had a conflict of tf version so I upgraded to tf 13.2 and ran the plan it worked… made some changes and ran consecutive plans which lead to this issue

Can you try this in a clean environment without state, without .terraform folder, etc.?

Yea that helped… my state file was corrupted

Makes sense

Can you share your code for using the module? Free free to replace any ids etc.

Is there a chance that the terraform state didn’t get persisted? … so it’s trying to recreate it

@Andriy Knysh (Cloud Posse)

specifically I’m looking to access the token attribute from the aws_eks_cluster_auth data resource here https://github.com/cloudposse/terraform-aws-eks-cluster/blob/df8b991bef53fcab8f01c542cd1c3ccc6242b61c/auth.tf#L72

I think we should not do it, it’s not a concern of the module

we can get EKS kubeconfig from the cluster anytime

@Chien Huey what is your use-case?

ah sorry,

yes, we have the token already

we can add it to the outputs

I am trying to use the helm provider functionality along with the CloudPosse EKS modules to bootstrap a cluster. As part of that bootstrap, I want to use the helm provider to install fluxcd

I thought about kubeconfig, which we can get from the cluster anytime w/o including it into the module

@Erik Osterman (Cloud Posse), this touches on your recent statement – or maybe it’s in your module dev docs – that you do not expose secrets via TF module outputs. I get that, but there are plenty of use cases where it is really helpful.

if we expose that, we can add a variable to show it or not

default to not

What if the module has a feature flag for the output? If the flag is set to true, the output contains that setting. If its set to false, then it’s null.

yes that’s what I wanted to do

the flag should default to false to not show the token in the output

@Chien Huey if you want to open a PR and add a feature flag called aws_eks_cluster_auth_token_output_enabled

We’ll approve that

ok great, I’ll do that

@Andriy Knysh (Cloud Posse) any opinion on the variable name?

aws_eks_cluster_auth_token_output_enabled or auth_token_output_enabled

maybe the second

ok, let’s do auth_token_output_enabled

heads up @Chien Huey

got it

thanks everyone

I would not use our terraform-root-modules or the current state of reference-architectures. The root modules are on life-support for existing customers. Most of them are tf 0.11. Our reference architectures for 0.12+ are not yet public, but should be by the end of the year.

They’re entirely redesigned from the ground up to work with terraform cloud

no, I want to create “my own” root module that is ultimately just using a bunch of public modules

but was using those links as a reference point on how to start

Aha, yes, that may be helpful then as an idea for how to organize things.

Also, happy to jump on a call with you anytime and get you unblocked. calendly.com/cloudposse

I do this on pipelines by adding a private key to the pipeline container and then adding the repo url to known hosts. then i can reference the module using ssh

do you modules start start with git://<fixed-pipeline-user>@<your private git>/…?

If so, in my case I would like to still let users still need to run geodesic locally

never mind… I was thrown off by an error with the module url: all I needed was to reference the module correctly:

export TF_CLI_INIT_FROM_MODULE="git::<ssh://git@<private> git repo>/..."

yea I use mostly gitlab ci and my before_script on yamls end up looking like this

before_script:
  - mkdir ~/.ssh
  - chmod 700 ~/.ssh
  - cat "${GIT_PRIVATE_KEY}" >> ~/.ssh/id_rsa
  - ssh-keyscan git.domain.com >> ~/.ssh/known_hosts
  - chmod 600 ~/.ssh/*
  - terraform init

Great! That works!

Yes check out the module inputs, you add them as a list using allowed_security_groups

@pjaudiomv thanks for your reply but it seems like allow the security group you provide to access the created default security group…?

ah ok so it creates a default security group but the ingress is set to the provided security groups. this is not a configurable option to not create that group

cool thanks

my gitlab yaml ewnds up looking like this

---
image:
  name: hashicorp/terraform:0.13.1
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'


before_script:
  - mkdir ~/.ssh
  - chmod 700 ~/.ssh
  - cat "${GITLAB_PRIVATE_KEY}" >> ~/.ssh/id_rsa
  - ssh-keyscan git.domain.com >> ~/.ssh/known_hosts
  - chmod 600 ~/.ssh/*

stages:
  - plan
  - apply

terraform_plan_only:
  stage: plan
  resource_group: terraform-lock
  script:
    - |
      for ACCT in $AWS_ACCT_LIST
      do
        rm -rf .terraform
        echo $ACCT
        eval "ACCT_FILE=\$$ACCT"
        cat "${ACCT_FILE}" > "$(pwd)/.env"
        source "$(pwd)/.env"
        export $(cat $(pwd)/.env | xargs)

        terraform init \
          -backend-config=bucket="$S3_BUCKET" \
          -backend-config=dynamodb_table="$LOCK_TABLE" \
          -backend-config=region="$AWS_REGION"

        terraform plan
      done

  only:
    - merge_requests

terraform_apply:
  stage: apply
  retry: 1
  resource_group: terraform-lock
  script:
    - |
      for ACCT in $AWS_ACCT_LIST
      do
        rm -rf .terraform
        echo $ACCT
        eval "ACCT_FILE=\$$ACCT"
        cat "${ACCT_FILE}" > "$(pwd)/.env"
        source "$(pwd)/.env"
        export $(cat $(pwd)/.env | xargs)

        terraform init \
          -backend-config=bucket="$S3_BUCKET" \
          -backend-config=dynamodb_table="$LOCK_TABLE" \
          -backend-config=region="$AWS_REGION"

        terraform plan -out=plan.plan
        terraform apply plan.plan
      done
  only:
    - master

I can paste what those env vars look like in a min

AWS_ACCT_LIST space seperated list of account aliases staging-2343525 prod-25345634 dev-234346

each one of those has a corresponding env var thats a file

ex staging-2343525

  AWS_ACCESS_KEY_ID=AKGFHGCHGKAIZEXBLEG5
  AWS_SECRET_ACCESS_KEY=SECRET
  AWS_REGION=us-east-1
  LOCK_TABLE=tfstate-lock-2343525
  S3_BUCKET=tfstate-2343525

thus could be a horrible example, but is my experience and works for my use case

I get what you’re trying to show me. That stuff I fully understand. The more interesting part I’m looking to understand from folks is the IAM assumable roles / user delegation setup through Terrafrom. So maybe my question is not clear enough

ah ok yes that is a much larger and different discussion

Thank you though dude — sorry, I didn’t express that I do appreciate the help!

yea I use a bunch of cross account roles and a seperate identity provider

np

Are you using AWS organization?

It’s a lot easier to create accounts under AWS Organization because it will automatically create the role OrganizationAccountAccessRole. You can assume that role and finish configuring the account

@zeid.derhally I am using Organizations and I do create the account through that process. Good to know that is the smartest path. I will add that to my list of “Many steps to accomplish adding a new account / environment”.

you should probably post the messages in the thread to keep the chat a bit cleaner..

Looks to me you are looking for a ami id from var.ami_rhel77 but have defined a ami id for var.ami_omserver40

sorry, I pasted the wrong variable but I do have

variable "ami_rhel77" {
  default = "ami-0170fc126935d44c3"
}

and it’s in the correct AWS region

assuming the ami ID and the owner accounts ID is correct, I would double check that you have shared the ami to the account where you execute the terraform code

the “ami-0170fc126935d44c3” is a RHEL 7.7 public image available to all and shared by Red Hat

oh, in that case I’m not sure what might be wrong.. Maybe the owner account id is not their accounts id? I don’t use rhel but quick google suggests that they share ami’s from 309956199498

That was it! I had the wrong owner_id . Thanks a lot!

I still have the same error but for the eks modure which has this local variable:

eks_worker_ami_name_filter = "amazon-eks-node-${var.kubernetes_version}*"

and I have this

variable "kubernetes_version" {
  default = "1.18"
}

I think latest EKS version is 1.17

let me try it

awesome! that worked. Thanks again! I didn’t know eks is a little behind the versions on kubernetes.io site.

AMis are region specific. Are you using the same region?

Thanks Andriy for replying. Yes, I’m using the AMI for the same region. This already has been resolved. The owner_id was incorrect.

i started out using workspaces, but felt they were too implicit/invisible. the explicitness of a directory hierarchy made more sense for our team/usage

sure, but isnt that duplicating whole bunch of iac just for the sake of it? or do u have the “core” modules under ./module and reference it under production/staging/test/whatever?

exactly, we use a core module over and over across multiple accounts and envs

modules in a different repo and pinned to your compositions works great

technically, we use terragrunt to manage the workflow, but that’s not strictly necessary

we just use 1 mono-repo for the modules rather than 1 repo per module (small team, too much to deal with if we broke it further)

so if i understand the modules correctly, the “referencee” still have the [vars.tf](http://vars.tf) duplicated for every module u want to utilize?

staging/main.tf -- reference ../modules/whatever.tf
staging/vars.tf -- vars for ../modules/whatever.tf
test/main.tf ... same as above
test/vars.tf
production/...

you have options… you can expose them on the cli, or code the values in [main.tf](http://main.tf), or use a wrapper like terragrunt to use core module directly (something like terraform’s -from-module

Will answer this in next week’s #office-hours

I am not a big fan of terraform workspaces due to the fact we can not keep the state files in different s3 buckets. I prefer terragrunt over terraform workspaces

I think I was lucky and only started with TF 12 +, workspaces are the best.. both in s3 and TF Cloud. I just went for parameterised stacks and depending on the workspace select key (using the built in ‘terraform.workspace’ variable) in a map and that is it. Later on I have extended this out to every env having its own file. I saw a lot of examples like this:

https://take.ms/M7qwC

and just didn’t get it, this is not DRY. I don’t care about ‘stage’ or ‘prod’. I just have environments with different settings.

Similar issue from the project issues itself, but back in 2018: https://github.com/cloudposse/terraform-aws-elasticsearch/issues/13

I was previously solving this via a two phase apply, but I was really hoping the upgrade to 0.13 would allow me to get around that hack.

if the zone specified to var.dns_zone_id is being created in the same apply, then this will happen

there is no way around that limitation in terraform. just have to remove that condition from the expression

Yeah, that sounds like my sticking point… now is there no way to get around that dependency snag even with the new module depends_on?

depends_on - Creates explicit dependencies between the entire module and the listed targets. This will delay the final evaluation of the module, and any sub-modules, until after the dependencies have been applied. Modules have the same dependency resolution behavior as defined for managed resources.

Like why does that not solve the problem. If I’m getting the dns_zone_id value from module.subdomain and I specify “Hey wait for subdomain to be applied” via depends_on… I was assuming that was the whole point of depends_on. But maybe I’m misunderstanding that.

it shifts the problem to the user calling the module, but no it does not remove the limitation

oh, depends_on is not count

Argh. This is frustrating.

@Matt Gowie the depends_on in 0.13 lets you have modules depend on other things, but it doesn’t fix the count issue. Count needs to be calculated before terraform starts applying anything, which is why the issue is still appearing

i haven’t tried using the two together, but i don’t think it will matter… terraform understands the directed graph, so because of the var reference, it already knows that it needs to resolve one before the other

I wonder if I instead update the module to accept dns_zone_name and then use data.aws_route53_zone to look up the zone to find the zone_id then that might do it? I can statically pass the dns_zone_name.

the count and for_each expressions all must resolve right at the beginning of the plan. they can depend on a data source, but only as long as that data source does not depend on a resource

Okay — Take away for myself is that count is always calculated during the plan and needs to resolve. Regardless of depends_on. TIL.

yeah, i’d still recommend removing that condition from the expression and rethinking the approach

Maybe I’ll update the module to add an explicit flag for the hostname resources instead of having it rely on a calculated enabled + dns_zone_id != "" condition

that’s been my workaround also

I’ll shoot for that for now. Thanks for the insight gents!

add a version constraint to your aws provider block

I added the version constraint into my aws provider block, deleted the .terraform directory and run terraform init again. I still get the same errors.

Is it because the vpc module has this versions.tf file:

terraform {
  required_version = ">= 0.12.0, < 0.14.0"

  required_providers {
    aws      = ">= 2.0, < 4.0"
    template = "~> 2.0"
    local    = "~> 1.2"
    null     = "~> 2.0"
  }
}

does this help? https://www.terraform.io/upgrade-guides/0-13.html#why-do-i-see-provider-during-init-

in particular, the commands using state replace-provider

so, I had already read that documentation and also run the state replace-provider. See the output of my terraform providers :

I also tried to rule out the state files by copying all my terraform code to a new directory and run terraform init. Still got the same error.

yeah, i think you have some kind of irreconcilable version constraint?

- Finding hashicorp/aws versions matching ">= 3.0.*, >= 2.0.*, >= 2.0.*, >= 2.0.*, >= 2.0.*, >= 2.0.*, < 4.0.*, ~> 2.0, ~> 2.0, ~> 2.0, ~> 2.0, ~> 2.0, ~> 2.0, >= 2.0.*, < 4.0.*, >= 3.0.*, >= 2.0.*, >= 2.0.*, >= 2.0.*"...

in particular:

~> 2.0

in ├── module.alb_om40

and module.access_logs

and module.s3_bucket

those are all Cloud Posse modules

Stat version of them are you targeting, it may be that you want to target a newer version

I point the modules source to the Github repo of Cloud Posse using the latest tag

Got it

i believe they’ve become amenable to reducing the restriction to >=… open a pr

let me show you one example from the ALB module tag 0.17.0

Ah it may be that you need to pin your provider to 2 then

or fork the module and change the version constraint

(and open a pr )

Yeah, if you’re trying to upgrade to AWS provider 3 then you’ll need to deal with those ~> 2.0 blocks.

If you submit PRs for using >= 2.0 and post them in #pr-reviews then we can check em out and try to get them merged if they pass tests. The new version did introduce a bunch of small changes that have broken tests though so it’s not a totally pain free module upgrade all of the time.

sure, give me some time as I’m pretty busy with work but will try to create this PR today. Thanks for all the help

Well, I finished work and started looking in to this PR. I don’t know how would I go about this. because module terraform-aws-alb calls the terraform-aws-lb-s3-bucket module which calls the terraform-aws-s3-log-storage module. They all use ~> 2.0

update the deepest module first, publish new version, then move up the stack

Done. PRs created.

Is it the part about using terraform against localstack that stands out? (which is pretty interesting)

for me, the ability to define state moves in code, and plan/apply them is really slick. i’m always renaming and rethinking things, and that often translates to a lot of state manipulation

you can not do it in one go

you will have to create a TF project import the resource with terraform import and represent that resource in code by looking at the state file

BUT

you can use this

https://github.com/GoogleCloudPlatform/terraformer#use-with-aws

which will do it for you

it is a pretty good tool

and you need to RTFM a bit too

Personally, terraform is a fast moving river and it changes rapidly. This is one of the reasons why I am staying away from 3rd party modules for now. (Again, just iterating that this is my personal opinion only).

But if there is a 3rd party module that does what I needed, then I’d rather prefer to adopt that module from a well established source like CloudPosse over a random community module in the registry.

The guys here are very knowledgeable in terraform, and so I’d trust them to keep things up-to-date, well tested and compatible with the recent releases of terraform.

thanks Peter for your inputs

Speaking from personal experience, the other terraform module (terraform-aws-modules/terraform-aws-eks) is really unstable. They commit breaking changes all the time. It was a nightmare to use. The CloudPosse one has been WAY more stable

thanks Andrew

https://github.com/terraform-aws-modules/terraform-aws-eks/issues/635

@roth.andy thanks for the pointers. I think if you refer to a tag rather than master you will always get a stable module

even in cloud posse tf module recommendation is to use a tag rather than master

hi guys

so CloudPosse modules are also community-driven and open-sourced, and we accept/review PRs all the time

and for the EKS modules, we support all of them and update all the time

and they are used in production for many clients

thanks @Andriy Knysh (Cloud Posse) for the inputs. Does cloudposse module give us the ability to use the tf deployment mechanism ?

deployment mechanism = kubernetes_deployment

so basically if I use cloud posse eks module to provision eks cluster can we then somehow deploy helm and flux on the eks cluster

we deploy EKS clusters with terraform, and all Kubernetes releases (system and apps) using helmfiles

works great

thanks Andriy I will look into helmfile.

can it also be used to deploy flux ?

sure

flux has helm charts (and operator)

you can always create a helmfile to use the chart

ok so I just need flux then and no helmfile as flux can do what helmfile will do

butt how do I install flux with cloud posse ?

helmfile is to provision flux itself into a k8s cluster

https://docs.fluxcd.io/en/1.17.1/tutorials/get-started-helm.html

ahh cool got you

you can use helm for that

thanks a lott Andriy

but helmfile adds many useful features so it’s easier to provision than using just helm

https://sweetops.slack.com/archives/CE5NGCB9Q/p1600313811153900

(one other HUGE differentiator is :100: of our our terraform 0.12+ modules have integration tests with terratest which means we don’t merge a PR until it passes at least some minimal smoke tests)

yes. and the EKS modules have not just a smoke test, in the tests we actually waiting for the worker nodes to join the cluster

and for the Fargate Profile module, we even deploy a Kubernetes release so EKS would actually create a Fargate profile for it https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/test/src/examples_complete_test.go#L173

thanks for your inputs guys

I had one question regarding terraform helmfile

https://registry.terraform.io/providers/mumoshu/helmfile/latest

can this provider be used in production ?

or is there any other way to deploy charts using terraform + helmfile ?

Appreciate all your inputs

We’re not yet using it in production, but have plans to do that probably in the next quarter.

@roth.andy has been using it lately, not sure how far it went

@mumoshu any one you know using it in production?

not great

I’m currently using local-exec

I’m going to take another stab at it later though, I just couldn’t spend any more time on it

@Erik Osterman (Cloud Posse) they’re not in the sweetops slack, but a company partnering with me is testing it towards going production.

their goal is to complete a “cluster canary deployment” in a single terraform apply run. they seem to have managed it :)

andrew, @Andrew Nazarov, and many others have contributed to test the provider(thanks!). and i believe all the fundamental issues are already fixed.

i have only two TODOs at this point. it requires helmfile v0.128.1 or greater so i would like to add a version check so that the user is notified to upgrade helmfile binary if necessary.

also importing existing helmfile-managed releases are not straight-forward. i’m going to implement terraform import and add some guidance for that. https://github.com/mumoshu/terraform-provider-helmfile/issues/33

Sweeet

also importing existing helmfile-managed releases are not straight-forward. i’m going to implement terraform import and add some guidance for that. Ahh good to know! that will be important when we get to implementing it.

thanks guys that really helps us. Will let you know once I present the findings to our team

fyi the eksctl provider has support for terraform import now

First I run into this bug https://github.com/cloudposse/terraform-aws-ecs-web-app/issues/63

And after specifying repo_owner I run into

Error: If `anonymous` is false, `token` is required.

  on .terraform/modules/athys.ecs_codepipeline.github_webhooks/main.tf line 1, in provider "github":
   1: provider "github" {

I switched to using alb service task and the alb ingress modules instead of that one

Will give that a shot, thanks

@RB Any idea what would cause

Error: InvalidParameterException: The new ARN and resource ID format must be enabled to add tags to the service. Opt in to the new format and try again. "athys"

  on .terraform/modules/ecs_alb_service_task/main.tf line 355, in resource "aws_ecs_service" "default":
 355: resource "aws_ecs_service" "default" {

Actually think I found a var to fix that issue. Still don’t really understand the cause, as all resources are brand new (so why use old arns?)

did you tick the boxes in your aws account to opt into the long arn format ?

if so, did you rebuild your ecs cluster ?

Rebuilt ecs cluster, but didn’t realize there’s a setting on the was account. That’s probably it

Thanks

i added the use old arn variable to make sure to not tag the task/service (i forget w hich one) to allow using that module for the not-long arn formats

nice! np

The new ARN and resource ID format must be enabled to add tags to the service

this must be done manually in the AS console

for each region separately

ah it’s per region. interesting.

mine are set to “undefined” which means

An undefined setting means your IAM user or role uses the account’s default setting.

you need to be an admin to do so

no idea, but great question. following

i also don’t know, but sometimes can decode why terraform wants to do something from the config and the plan output. if you can share those, maybe someone will be able to help

It’s pretty much just a rule that Terraform mentions is forcing a replacement.

which often flows from something in the config. but can’t help if we can’t see it

I’ve co-authored https://github.com/Flaconi/terraform-aws-waf-acl-rules/ and have not seen any of those issues. Maybe you can share your code ?

That’s wafv1 ^ my Terraform is for wafv2

@sweetops This might help you:

<source>
  @type forward
  bind 0.0.0.0
  port 24224
</source>

<filter *firelens*>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

<match *firelens*>
  @type datadog
  api_key "#{ENV['DD_API_KEY']}"
  service "#{ENV['DEPLOYMENT']}"
  dd_source "ruby"
</match>

I believe to ship to two locations you would just need two <match> sections that ship to your respective destinations.

Haha btw — That is a fluentd configuration. We have the AWS fluentbit / firelens configuration as a sidecar on each of our ECS containers and then host a single fluentd container in our cluster for shipping externally.

Here is the Dockerfile for that container:

FROM fluent/fluentd:v1.7.4-debian-1.0

USER root


RUN buildDeps="sudo make gcc" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \
 && sudo gem install fluent-plugin-datadog \
 && sudo gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
	&& rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem

COPY fluent.conf /fluentd/etc

Terraform now has an official stance on how to pin as pointed out to me by @Jeff Wozniak

https://www.terraform.io/docs/configuration/provider-requirements.html#best-practices-for-provider-versions

this bit us hard so we no longer use it since all of our modules are intended to be composed in other modules

we use a strict pin only in root modules, and any composable modules use only a min version

and the min version is usually optional. we only add the restriction if we know we’re using a feature that depends on a min version of something (e.g. module-level count requires tf 0.13)

Hmmm interesting. So at a minimum, a min version. Not leaving it empty.

Also tried:

subnets            = [aws_subnet.adv2_public_subnets.*.id]

where I tried to implement what I saw in this example: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lb#application-load-balancer

the object is a map, so you need to use for to access the attribute you want:

[ for subnet in aws_subnet.adv2_public_subnets : subnet.id ]

ah, that’s how you do that. It wasn’t clear to me how to apply that to a value.

it’s not intuitive. would love if we could index into the map for an attribute, the way we can with a list/tuple, a la aws_subnet.adv2_public_subnets[*].id

I had no idea that could be done (for list/tuple)

Sweet. That does exactly what I want. Thanks! For reference,

resource "aws_lb" "aws_adv2_public_gateway_alb" {
  name               = "services-${var.environment}-public"
  internal           = false
  load_balancer_type = "application"
  
  subnets            = [ for subnet in aws_subnet.adv2_public_subnets : subnet.id ]

It’s already fixed guys

https://stackoverflow.com/review/suggested-edits/27209940

Not that I know of. But if you want to roll that into your projects workflow then the common approach for this type of thing is to script around it using Make or bash.

Use the extension .auto.tfvars?

Terraform also automatically loads a number of variable definitions files if they are present:

Files named exactly terraform.tfvars or terraform.tfvars.json.

Any files with names ending in .auto.tfvars or .auto.tfvars.json.

https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files

We heavily use symlinks in Linux to hydrate global, environment, region variables and auto tfvars

yeah, right… ok

all very interestings ideas

Your variables could be outputs in nodule A, then module B could use a remote state data source to retrieve module As outputs which can then be used

(or use YAML configuration files)

More and more we’re taking the approach that HCL is reponsible for business logic and YAML is responsible for configuration.

How do you load the YAML into TF vars?

So our modules always operate on inputs (variables), but at the root-level module, they operate on declarative configurations in YAML.

Ah. And you load as locals?

Ya, here’s an example: https://github.com/cloudposse/terraform-opsgenie-incident-management/blob/master/examples/config/main.tf

Ah then you load into a module similar to the context.tf pattern and that can provide the variable interpolation bit. Cool, good stuff

yeah, nice! https://github.com/cloudposse/terraform-opsgenie-incident-management/blob/master/examples/config/main.tf

This is totally what I was looking for.. amazing. I like it @Erik Osterman (Cloud Posse)

nevermind……

being dyslexic is not cool some times

Your default should be an empty list, [] . You can then use toset() for the for_each

or: var.geo_restriction == null ? [] : var.geo_restriction