#terraform (2020-05)
Discussions related to Terraform or Terraform Modules
Archive: https://archive.sweetops.com/terraform/
2020-05-01

Anyone take the HashiCorp Certified Terraform Associate exam yet?

It may have been asked before, but I’m new to checking out these awesome modules.
I have an environment/account already setup (out of my control) - ie: vpc, ig, subnets, etc.. already created.
Is it possible to use a module, like: terraform-aws-elastic-beanstalk-environment
and not create all the extras, or would I have to go through an import everything ?

no need to create those resources again

you can do Data lookups

base on tags


that sounds like it could be easier.

like that

I’m going to have to lookup a lot of tags..

thanks

you need to define enough to find it

not all

if it has a tag that is unique, then that is it

Hi everyone A team at work has been developing their entire infrastructure in one single Cloudformation Template file which now over 4000 lines of spaghetti I helped them fix a bunch of cyclic dependencies and now I want to re write the CFT => TF module’s which mainly has alot of I inline lambda Glue jobs, catalog, tables CW alarms and events A many S3 buckets
This infrastructure is live in production in five environments
Im seeking advise on what will the best approach to deploy the infrastructure I have developed into Terraform?
What about the existing S3 buckets which has TB of data? Should do a TF import on S3 bucket? Deploy a parallel env with TF and delete CFT elements manually as I can’t delete the CFT stack?

I will use terraformer

CLI tool to generate terraform files from existing infrastructure (reverse Terraform). Infrastructure to Code - GoogleCloudPlatform/terraformer

we recently used to import some stuff

worked very well

we just changed the resources names a bit

Noice

I will take a look at this.

I will say, creating each resource bit by bit will be better, using modules reduces de time for implementation

but it is possible to do it with this

@jose.amengual thanks for u advise

np

+1 for Terraformer, we’re using for moving parts of the current infrastructure that was manually deployed by the previous lead to Terraform Modules.

what about cdk? seems neat, gives you full benefit of a real programming language

it is going to take you a REALLY long time tod o

It’s a mountain of a task to climb :)


anyone have seen this ?????
security_groups = [
+ "sg-015133333333d473b",
+ "sg-05294444444432970",
+ "sg-0a35555555553ea35",
+ "sg-0c8a33333dbca389b",
+ "sg-022222229ccf66e24",
+ "terraform-20200502011517105000000003",
]

Data lookup where that comes from is this :
data "aws_security_groups" "atlantis" {
filter {
name = "group-name"
values = ["hds-${var.environment}-atlantis-service"]
}
provider = aws.primary
}

Weird looks like a timestamp to me, what happens if u apply it

Do not know why it is under =[]

it complain that is expecting sg-
as a value

never seen such thing before

Taint the resource?

If you you can’t cure it, kill it lol

lol

it was created by another TF

is there, it works

I can add it manually in the UI


data.aws_security_groups.atlantis.id

returns the terraform-2222

data.aws_security_groups.atlantis.ids.* returns the id

stupid

it should not return anything

is there, it works

you mean you could see the id
in the AWS UI??

I mean it should give you some error since the data resource is aws_security_groups and now a not a resource “aws_security_group”

very subtle difference

# atlantis
data "aws_security_groups" "atlantis" {
filter {
name = "group-name"
values = ["hds-${var.environment}-atlantis-service"]
}
provider = aws.primary
}
resource "aws_security_group" "ods-purge-lambda-us-east-2" {
name = "hds-${var.environment}-ods-purge-us-east-2"
description = "Used in lambda script"
vpc_id = local.vpc_id
tags = local.complete_tags
provider = aws.primary
}
locals {
sg-atlantis = join("", data.aws_security_groups.atlantis.ids.*)
sg-ods-purge-us-east-2 = join("", aws_security_group.ods-purge-lambda-us-east-2.id.*)
}

aha! now i get it

you see how easy is to make a mistake ?

WTH is that terraform-XXXX???
2020-05-02
2020-05-03

Hello, I have a newbie-type question. Is there a way to force rebuilding a specific resource? Basically, using “terraform taint” in other resources than ec2 instances such as aws kinesis stream or dynamodb table, etc (My current workaround is changing the resource name which usually forces the rebuild, however it not very practical.)

well is you remove the resource and add it again it does it too

you can do a count argument and enable and disable it

yeah that’s a better option than renaming on one hand. On the other hand it requires more steps. In the setup I have in mind (automated testing), it actual might be better .

nvm, got it working with terraform taint actually

I’m not so familiar with taint
2020-05-04


You can also use ‘untaint’ for times when terraform gives up on a resource or gets confused and wants to destroy it, even though it turned out ok

If a terraform module (module UP) is updated, is there a way for atlantis or something else to rerun a terraform plan
for a module (module DOWN) that depends on UP ?

personally, we version all our modules, and use dependabot to update the version refs with a pr…
If a terraform module (module UP) is updated, is there a way for atlantis or something else to rerun a terraform plan
for a module (module DOWN) that depends on UP ?

interesting so dependabot will notice that the module UP has changed and somehow find all modules that depend on UP and submit a PR to them to update the git
reference tag in the source
which will trigger atlantis’ terraform plan
comment ?

if i understand that correctly… how do you configure dependabot to do this ?

if it’s a public project, and tf 0.11, then the free dependabot service works great

if you need tf 0.12, we’ve built a fork and a github-action

could you link this ?

im surprised you folks havent’ blogged about this.

Terraform module that deploys a lambda function which will publish open pull requests to Slack - plus3it/terraform-aws-codecommit-pr-reminders

(note that’s a read-only github token, please don’t flay me)

haha i was just wondering about that. might be good to comment that inline

here’s the repo for the gh-action, https://github.com/plus3it/dependabot-terraform-action/
Github action for running dependabot on terraform repositories with HCL 2.0 - plus3it/dependabot-terraform-action


and our fork of dependabot-core, which the gh-action uses, https://github.com/plus3it/dependabot-core
The core logic behind Dependabot’s update PR creation - plus3it/dependabot-core


hopefully dependabot will eventually get around to merging tf012 support upstream… https://github.com/dependabot/dependabot-core/pull/1388
Fixes #1176 I opted for both hcl2json and terraform-config-inspect. hcl2json for terragrunt and terraform-config-inspect for tf 0.12 I wanted to go with terraform-config-inspect for both, but it di…


:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform
and get to know others in the community on a more personal level. Next one is May 13, 2020 11:30AM.
Register for Webinar
#office-hours (our channel)

Hoping someone can help - I’m curious how would you use the import
command to import an existing resource within TFC if i’m using TFC to trigger plan/apply and not via cli?

@Mr.Devops You can still use import
to update your TFC state from your local environment AFAIK. Once you update it once from your local environment then it will be in your state and TFC should just know it’s there.

ah ok i will look into this thx @Matt Gowie
2020-05-05

HashiCorp Consul Service for Azure (HCS) Affected May 5, 16:23 UTC Investigating - We are currently experiencing a potential disruption of service to the HashiCorp Consul Service for Azure (HCS). HCS Cluster creations are currently failing. Teams are currently working to identify a solution and will update as soon as information is available.
HashiCorp Services’s Status Page - HashiCorp Consul Service for Azure (HCS) Affected.

Hi I’m using CloudPosse’s terraform-aws-eks-cluster module
, how do I decide whether to instantiate the worker nodes with their terraform-aws-eks-node-group
module vs their terraform-aws-eks-workers modules
? The node-group approach seems to use what is intended for EKS, whereas workers module uses autoscaling group. Am I correct that node-group module is the way to go?

there are pros and cons with managed node groups

also, the workers module was created long, long before managed node groups were supported

I would start with managed node groups until you find a use-case that doesn’t work for you

for example, if you need to run docker-in-docker for Jenkins, then the managed node groups won’t work well for that and you should use the workers module.

Keep in mind, that a kubernetes cluster can have any number of node pools

and you can mix and match node pools from different modules concurrently in the same cluster

you can have a fargate node pool along side a managed node pool, along side a ASG workers node pool


@Erik Osterman (Cloud Posse) Would you mind elaborating why dind won’t work well with managed node groups? I use dind and was thinking of moving my nodes to managed.

@Jeremy G (Cloud Posse)

@s_slack EKS disables the Docker bridge network by default, and Managed Node Groups to not allow you to turn it back on. This is a problem if your Dockerfile includes a RUN
command that installs a package, for example, as it will not have internet connectivity. Another issue for us is that you cannot configure Managed Node Groups to launch nodes with a Kubernetes taint on startup, which we wanted to keep our spot instances from having non-whitelisted applications running on them.
What happened: We utilize docker-in-docker configuration to build our images from CI within our dev cluster. Because docker0 has been disabled, internal routing from the inner container can no long…

Perfect. Much appreciated. @Jeremy G (Cloud Posse) @Erik Osterman (Cloud Posse)

Hi. I am hoping someone can help me with an interesting problem. I am trying to use the data lifecycle manager to create snapshots of the root volume of my instance but the root ebs volume needs to be tagged. Terraform doesn’t have a way to add tags to the root_block_device of the aws_instance. I tried to use the data.aws_ebs_volume to find the ebs volume that is created but I can’t figure out how to use that to tag it. The resource.aws_ebs_volume doesn’t seem to have a way to reference the id from the data.aws_ebs_volume which means that I can’t import the volume either. Hope that makes sense.

@breanngielissen Can you tag via the AWS CLI? If you can then you can use the local-exec
provisioner to tag it.
resource "null_resource" "tag_aws_ebs_volume" {
provisioner "local-exec" {
command = <<EOF
$YOUR_AWS_CLI_CODE_TO_TAG
EOF
}
}

@Matt Gowie That worked perfectly. Thank you!

@breanngielissen Glad to hear it!
2020-05-06

anyone know of a terraform module for opengrok or a similar code search app that can be run in aws ?

their docker container description says to use the standalone one …

i was looking into ECS using its docker, but it might be better to use an EC2 instance with userdata to setup opengrok, and keep code on an EFS

or just use an EBS

so then the setup would be…
• EC2
• userdata to retrieve ssh key, configure opengrok, clone all github org’s repos
• EBS / EFS for storage of github org repos
• ALB with a target group
• route53 record
• acm cert curious to hear other peoples’ thoughts

Can Sentinel be used to enforce policies on IAM roles? For example, don’t grant IAM roles related to networking to a user? I searched through the documentation but couldn’t find any examples related to IAM.

It’d be nice if this module allowed additional IAM permissions: https://github.com/cloudposse/terraform-aws-emr-cluster
Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS - cloudposse/terraform-aws-emr-cluster

ooo let me take a look at this
Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS - cloudposse/terraform-aws-emr-cluster

it looks like it creates an IAM role https://github.com/cloudposse/terraform-aws-emr-cluster/blob/6444ee4f2a48d3d41c2499914ab8b882e454b11c/main.tf#L280
Terraform module to provision an Elastic MapReduce (EMR) cluster on AWS - cloudposse/terraform-aws-emr-cluster

you can use a data source to grab the IAM role that it creates and then do a aws_iam_role_policy_attachment
to attach a new policy to it, outside of its module reference.


Yea, this is our typical pattern - provision the role, expose it, then let the caller attach policies

it would be nice if those modules that are already creating the iam role(s) would also output their respective arns. then you wouldn’t need to use a data source to get the arn

@RB ya I’d consider it a “bug” if they don’t

Using data provider shouldn’t be needed.

because in order to use bootstrap actions, the instance needs permissions to download the file

Hi Guys, Is there any example that creates global accelerator with alb as endpoint?

I think i find it
2020-05-07

Hi All - I would like to setup a simple AWS EKS cluster with 1 master and 2 worker nodes..which set of terraform files I need to use. Please guide..I have been trying to figure out and end up bringing up something I could not access from local machine

Have you seen our EKS cluster module? It has a good example to get you started

yes but it has lot of things in root folder there are some .tf files and also in example I could see ‘complete’ folder has lot of files..Not sure from which folder i need to trigger “terraform apply” and also which variable.tf file I need to feed my details

Hey, I’m thinking about how to properly manage resources created by AWS with Terraform. One example is the S3 bucket which is created for Elastic Beanstalk (elasticbeanstalk-<region>-<account_id>). I would like to add cross-region replication and encryption for this bucket due to compliancy reasons. Any ideas?

here better

this is for Security Hub PCI standard, which requires for S3 buckets to have cross-region replica and encryption

cross account or cross region ?

cross region

on one account

so then you do not need the account id in the name in that case ?

we did team-nameofthing-region

we added the regions to everything

we even duplicated similar IAM roles for that

because IAM is a global service

but resources are not

which makes things very difficult to troubleshoot in case of access issues and such

and you need to keep in mind that some resources like ALBs can’t have names longer than 32 characters

so is better to keep it short

the bucket is created by AWS automatically like that

and I haven’t found a place where could I specify the bucket for Beanstalk myself

mmm I do not know much about EB

is this for a multi region setup ?

Hi. I’m using TF modules and wondering if anyone has a hack to conditionally enable/disable one based on a variable.

currently, the module has to support it, by accepting a variable that is used in the count/for_each expression on every resource in the module

Terraform module which creates VPC resources on AWS - terraform-aws-modules/terraform-aws-vpc

follow the create_vpc
value through the module .tf files…

thanks. i’ll take a look.

terraform core is working on module-level count/for_each support. it is in-scope for tf 0.13… https://github.com/hashicorp/terraform/issues/17519#issuecomment-605003408
Is it possible to dynamically select map variable, e.g? Currently I am doing this: vars.tf locals { map1 = { name1 = "foo" name2 = "bar" } } main.tf module "x1" { sour…

Managed to deploy EKS cluster using Terraform..with basic thing running. However when I try to connect from my local machine from where I ran terraform. I get below error message. An error occurred (AccessDenied) when calling the AssumeRole operation: User: arnxxxxxxx:user/xxxx is not authorized to perform: sts:AssumeRole on resource: arnxxxxxxx:user/xxxx


I am using same aws access key and secret key to deploy the EKS cluster using terraform but when I try to connect it from the same machine with same AWS config i get this error message..Any help would be really helpful?

Is that error from running kubectl
?
What does your kubeconfig look like? It’s probably in ~/.kube/config
. (Don’t paste the whole config in here; we don’t need to see your cluster address or certificate)
If the access keys on your local system already have permissions to connect to EKS, make sure your kubeconfig doesn’t have a redundant role assumption in the args:
args:
...
- "--role"
- "<role-arn>"

Actually yes I have added my role again using command.same role which I am using to connect to EKS. This I have done using command
aws eks --region us-east-1 update-kubeconfig --name test-xxx --role-arn arn:awsiam:xxxx:user/xxx

this is because when I checked in kubeconfig file it had new iam role created by Terraform but not mine which i connected in my local machine..only due to this i assumed I couldnt connect so I ve added it. So you mean if we dont add it should be able to connect?

It works when if I dont add the redundant roles. Thanks @David Scott for your help. I really saved lot of time

Glad I could help!

One silly question: I use the terraform_state_backend
, I was on version 0.16.0
and when I try to upgrade and use version 0.17.0
it’s telling me something around Provider configuration not present
because of null_data_source
. I don’t know how to solve that conflict and I can’t run destroy
:rolling_on_the_floor_laughing: …so is it safe to just remove all the modules with type "null_data_source"
from terraform.tfstate
? That’s the only way I’ve found to make it work, or to better say: to not show that error message, I don’t know if it works, I’m just running until terraform plan

All right! I found that going back to 0.16.0
and running terraform destroy -target=the_troublemaker
, then terraform plan
will do it…after that, upgrading to 0.17.0
didn’t give me any problems

yea, we’ve seen that before as well. The problem is the explicitly defined providers in modules, e.g. https://github.com/cloudposse/terraform-null-label/blob/0.13.0/versions.tf#L5
Terraform Module to define a consistent naming convention by (namespace, stage, name, [attributes]) - cloudposse/terraform-null-label

if you define providers in modules, then you can’t just remove it from the code or rename it

TF will not be able to destroy it w/o the original provider present

label module 0.16.0
does not have it defined anymore

I’m refactoring some Terraform modules and am getting: Error: Provider configuration not present To work with module.my_module.some_resource.resource_name its original provider configuration at m…

(the moral of the store: try not to define providers in sub-modules, provide the providers from top-level modules. In most cases, they are inherited from top-level modules automatically)

this is very true….I just hit this issue with the github webhooks module

I’m dealing with the following; different customers, different terraform projects across different github repositories, some on the customer github org some on mine. The different projects can have information in common like my ssh-keys, e-mail addresses, whitelisted ip’s. An idea which was opted was, why not put those semi-static vars in a private git terraform module and distribute it like that. I’m personally afraid this module will end up like the windows registry, but I have no valid alternative either. I’m curious to know what everyones take is on this.

I heard of that solution from a colleague who worked on a project with a customer where they did that. I didn’t go down that route, because the things that were shared were things that didn’t change much, like project ids (this is for GCP). I simply hardcoded the outputs from various runs into the vars as I ran the environment from start to finish.

But I can see the central terraform module as either a God Class or as a glorified parameter store.

@Andriy Knysh (Cloud Posse) Does that mean you folks at CP will eventually be removing your usage of providers (like GH provider for GH webhooks) in your modules?

if they are still in some modules, yes, need to be removed. We tried to clean it up in all modules converted to TF 0.12

if you see providers in modules, let us know

I think this is the only one that I know of. I hit issues with it with the codepipeline module on a project. And I know @jose.amengual had troubles with it a week or two back.

Terraform module to provision webhooks on a set of GitHub repositories - cloudposse/terraform-github-repository-webhooks

yea, that one will trigger changes across many modules

yes that is the one, I had that problem

@Andriy Knysh (Cloud Posse) Yeah… I could see that being a PITA. Ya’ll doing a good job with versioning won’t make it too bad I would hope?

yes versioning will help not to break everything (if people don’t pin to master )

I bet there is a few millions doing that lol

Ah yes, hopefully people listen to you folks and don’t pin master
. But if they do then they’re outta luck.

¯_(ツ)_/¯

I think the examples in the README should be pinned to the latest version

not to master

yes they should (the latest version always changes though)

Sounds like a good automation task for release:
- Build README with new tag as input, README example is updated to use new tag
- Tag repo
- Push repo

but codefresh will have to inject that, right ?

Does anyone know if its possible to create an AWS ClientVPN completely with Terraform? It seems like there are some resources missing from the provider such as Route Tables and authorizations

correct

I just went trough this a few months back

Route Tables and authorizations need to be done manually

or over CLI

ok thank you, ive been going crazy wondering why I stopped where I did on writing Terraform to create this a few weeks back

now it makes sense

I was surprised too since without those, nothing works!!!!!

lol right? lets just give them half of what they need! One more quick question, do you have to associate the security groups with the networks you associate with the VPN manually as well?

i dont see a way to do that with that resource either


We are working internally to update the community VS Code extension to fully support Terraform 0.12 syntax and use our Language Server by default. A new version will be shipping later this year with the updates.

Question for VSCode folks — Is it that much better than Atom?
I’ve switched text editors a few times over the years from emacs => Sublime => Atom. Now I’m finally considering another switch to VSCode cause people love it so much, but I’m not sure if I get the hype.

We are working internally to update the community VS Code extension to fully support Terraform 0.12 syntax and use our Language Server by default. A new version will be shipping later this year with the updates.

its pretty good. I made the switch and love it. I’ve even slowly been forgetting about pycharm when working on Python and just defaulting to vscode

id recomend just installing it and checking it out. I can give you a list of good starting extensions to grab if you want too

and my settings.json file with a bunch of customizatiohn, just so you can see how it all works

Those are some nice offers — makes this more attractive… Yeah, I’ll happily take you up on that.

sure give me about 20 mins, on a call until about 5pm cst

Thanks man!

i find vscode significantly faster than atom. or did when i switched away from atom several years ago…

@Matt Gowie sorry about that, completely forgot about you. I forgot I went through and commented my settings file as well the other day, so that may help you understand some of it too. https://gist.github.com/afmsavage/1bf0241472f74fa21112d4d3698bcb80

Awesome — Thanks @Tony

terraform registry now exposes the # of downloads per module. 1 more metric for vetting open source modules.

nice! thanks for sharing

@loren working with the guys for concurrent development with a unified goal, right now the hashicorp one is more stable , alot less features, and mine is less stable but more experimental features (is explained in the repo)

Thank you for all your work! Been using your language server for quite a while. Brilliant work!

@Matt Gowie https://github.com/cloudposse/terraform-aws-ssm-parameter-store/releases/tag/0.2.0 (thanks for your contribution)
Terraform module to populate AWS Systems Manager (SSM) Parameter Store with values from Terraform. Works great with Chamber. - cloudposse/terraform-aws-ssm-parameter-store

Thanks @Matt Gowie and @Andriy Knysh (Cloud Posse) - definitely appreciate the 0.12 upgrade!
Terraform module to populate AWS Systems Manager (SSM) Parameter Store with values from Terraform. Works great with Chamber. - cloudposse/terraform-aws-ssm-parameter-store

any eks + spotinst + terraform integration around here? Is it possible to do everything within terraform? Im POC-ing it right now through the portal (where it basically drains all your nodes to their nodes) but I’m curious how that would work w/ my existing terraform state

I have it all within terraform but not as a single run. Cant figure out how best to reinstate the same number of nodes so i swap out to an ondemand scaling group first, drain to it then swap back in the spot asgs

ugh. not a fan of that.

Yeah :(
2020-05-08


I have a pretty broad and general question about modules and module composition. In a previous role I built out a multi account / multi region AWS architecture very much following the terragrunt methodology. i.e a single repo that defined what modules were live in what accounts/regions based on directory structure and git tagged modules

In a new role I have a clean slate. Introducing Terraform to the organisation and using Terraform Cloud. Terraform Cloud supports private module registry where a repo of the form terraform-<provider>-<name> can be automatically published as a module when a git tag is pushed

Historically I am used to working with a big monorepo where all modules reside. A huge advantage of this is easier module composition. i.e a module called service_iam
could include dozens of other IAM helper modules from the same repo

I suppose I am having a bit of trouble in my head figuring out what my new approach should be. I want to avoid code duplication and also a spaghetti of modules referring to other modules at specific versions

Theres a question in there somewhere…

Has anyone else done this “transition” I dont think I want to end up with a repo per module as that has a management / operational costs

Should I just create registry modules that have many many sub modules nested inside?

@conzymaher If you’re wanting lean away from doing a module per-repo and you’re looking to use the registry then I’d check out of the modules from the terraform-aws-modules
GH org. The terraform-aws-iam
one is great. They do the multiple modules in one repo pattern and works well in my experience.
Terraform module which creates IAM resources on AWS - terraform-aws-modules/terraform-aws-iam

I’m a big fan of those modules. None of them have a dependancy on each other though. But theres nothing to stop that
Terraform module which creates IAM resources on AWS - terraform-aws-modules/terraform-aws-iam

Thanks for the pointer. I hadn’t considered looking at some of the other modules in that repo. The atlantis module is basically the model I want. A “top level” module that uses other modules that are outside of that module, but can also use optional sub modules

Np. Glad you found one that is what you’re looking for!

Cloud Posse folks, do you have any plans to publish a Terraform module for deploying a typical Lambda function? I can’t find anything on your GitHub account

we have a few very specific modules that deploy lambdas for very specific tasks https://github.com/cloudposse?q=lambda-&type=&language=


Yeah, I get it…that’s what I’m doing…I was just thinking that it would be nice to have one module for a generic Lambda; I know it’s (almost) straightforward, but there are a few corner cases that usually modules takes care about it

your contribution to the open-source community would be very welcome @x80486

@randomy has several awesome modules for managing lambdas

(we have no plans right now to publish a module for generic lambda workflows)

I’ve heard good things about https://github.com/claranet/terraform-aws-lambda
Terraform module for AWS Lambda functions. Contribute to claranet/terraform-aws-lambda development by creating an account on GitHub.

but haven’t used it

Thanks. https://github.com/raymondbutcher/terraform-aws-lambda-builder is better than the Claranet one IMO (I made both)
Terraform module to build Lambda functions in Lambda or CodeBuild - raymondbutcher/terraform-aws-lambda-builder

Has anyone used https://github.com/liamg/tfsec and had it actually find legit security vulnerabilities? I’m skeptical.
Static analysis powered security scanner for your terraform code - liamg/tfsec

ya, i use that currently in atlantis
Static analysis powered security scanner for your terraform code - liamg/tfsec

What’re your thoughts? Has it caught any serious gotchas for you?

it’s nice and i believe it has some overlap with checkov and tflint

but it does find certain things the others dont.
we really need a nice comparison betw linters and their rules


So I’m noticing that we’re having issues w/ our child modules when folks remove them from their root module due to having a provider
block in the child module. We currently use the provider
block to setup the following for aws
:
assume_role {
role_arn = var.workspace_iam_roles[terraform.workspace]
}
Is there a way to set up assume_role for the child module so we can test it without the provider
block then as to not have missing provider error messages like the following?:
To work with module.kms.aws_kms_key.this its original provider configuration
at module.kms.provider.aws is required, but it has been removed. This occurs
when a provider configuration is removed while objects created by that
provider still exist in the state. Re-add the provider configuration to
destroy module.kms.aws_kms_key.this, after which you can remove the provider
configuration again.
Or a cleaner pattern to follow?
TIA

we always setup the provider credentials in the root module, using multiple providers with aliases if there are multiple credentials (e.g. different roles). we then pass the provider alias to the module that needs it

Modules allow multiple resources to be grouped together and encapsulated.

We compose our modules as well and run tests against them for that larger piece of infrastructure they build. I’m just wondering how I could continue testing the child module w/o having a way to setup some necessary config. We use kitchen-terraform currently. Maybe it entails working w/ that to test the child module w/o baking in a provider block which causes headaches down the road.

ya, this issue is coming up more an more often. we’ve (@Andriy Knysh (Cloud Posse)) fought a few issues in the past week

yeah, i think you’d have to setup the credential through kitchen-terraform


At least we’re in agreement - cause after re-reading the docs - I see the issue w/ nested providers - but now need to figure out how we can still test independently our child modules w/o those provider blocks.

I’m refactoring some Terraform modules and am getting: Error: Provider configuration not present To work with module.my_module.some_resource.resource_name its original provider configuration at m…

see the thread below that message

we also have multi-provider modules, for cross-account actions (e.g. vpc peering). in that case you must have a provider block in the module. but we only define the alias. the caller then passes the aliased providers they define to each aliased provider in the module

and the setup in the test config (though in this case it is vpc peering in the same account)… https://github.com/plus3it/terraform-aws-tardigrade-pcx/blob/master/tests/create_pcx/main.tf#L38-L41

Cool. I’ll take a look at all these. Hopefully there’s a way forward that reduces the issues we see when our dev teams remove child modules w/ their own provider modules but still allows us to test the child modules and the composed repos independently

Thanks @loren @Erik Osterman (Cloud Posse) !!

You don’t specify providers in child modules. This is logically correct since your submodules don’t need to know or deal with how they are being provisioned.

You specify providers in root modules

Or in the test fixtures

Those could have different providers with different roles or credentials

Or for different regions or accounts

Right. It’s mainly now how to remove the provider blocks (which I see from the other thread that Erik posted - you guys have been removing from all your modules) and continue to have testing ability for these child modules as well.

I’m only going to add that I personally avoid mapping providers into modules as much as humanly possible

Every time you have to do this I’d ask yourself if you are doing the right thing or not.

sometimes is simply unavoidable

sorry, I’m late to the party and likely missed the point but all this self-isolation made me feeling like proclaiming some stuff out loud….


right, in almost all cases you don’t need to specify providers in child moduler, nor map providers from the top-level modules

just define them in root modules or test fixtures, they are inherited automatically

Anyone have a cli or quick way to trigger a terraform cloud run? I’m can cobble together rest call but just checking. Have azure devops pipeline running packer and want to trigger it to run a terraform plan update to sync the SSM parameters for ami images after I’m done.
2020-05-09

I’m sure someone out there may have thought about this, but it would be nice if terraform would have the ability to output its graph to lucid chart(3rd party integration) -feat request for Hasicorp?

I suppose if you can figure out how to transform graphviz language into a csv you can simply import that into lucidchart (https://lucidchart.zendesk.com/hc/en-us/articles/115003866723-Process-Diagram-Import-from-CSV)
Use Lucidchart’s CSV Import for Process Diagrams to create flowcharts, swim lanes, and process diagrams quickly from your data. This tutorial will walk you through the steps of formatting your data…

though it is an interesting challenge I’ll leave that task up to you to figure out

thx @Zachary Loeber
2020-05-11

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform
and get to know others in the community on a more personal level. Next one is May 20, 2020 11:30AM.
Register for Webinar
#office-hours (our channel)

Hello,
Do folks feel workspaces
serve a purpose while using terraform open source? I personally have found the use of any other variable such as environment
sufficient to distinguish between different environments. It may also be so that I haven’t understood fully the purpose of workspaces
in terraform. Any advice/insights appreciated.

i dont find workspaces to be that useful. i’d much rather a module references with an environment argument and an environment directory with those module references

We’ve (cloudposse) switch over to using workspaces. previously we would provision a lot of s3 state backends, but that process is tedious and difficult to automate using pure terraform

I’m not too familiar with workspaces but how does it work with atlantis ? you guys have a repo per environment still or due to using workspaces you are now have one repo and multiple workspaces ? @Erik Osterman (Cloud Posse)

we now use a flat project structure. we define a root module exactly once. we don’t import it anywhere. we define an a terraform varfile for each environment. we don’t use the hierarchical folder structure (e.g. aws/us-east-2/prod/eks/
) and instead just have projects/eks
) with files like projects/eks/conf/prod.tfvars
)

this enforces that we have the identical architecture in every environment and do not need to copy and paste changes across folders. what I don’t like about the multiple folders is that architectures drift between the folders without human oversight.


you guys have a repo per environment still or due to using workspaces you are now have one repo and multiple workspaces ?
@jose.amengual: yes, we used to, but now have abandoned that approach.

cool so we are using the same aproach

we adopted that approach before gitops was a thing. it proved to not be very gitops friendly and changes were left behind leading to drift. we moved to the flat folder architecture this year in all new engagements.

almost, we do not have subfolders for the conf/
we have it all int the root

for us, it has worked well, the main reason we ended up with one repo was due to multiregion deployments, if we decided to split by env we could endup with an exponential number of repos

Terraform state supposed to help you with infrastructure drift but by having so many repos or folders then who solves the problem of configuration drift when using multi-folder or env-repos?

pretty much impossible

I’d like what you talk about @Erik Osterman (Cloud Posse) to be a decent norm but the diminishing returns come fast when you try to do all this rigorously with only one or two envs and requirements popping up all the time where they end up having to differ in the end for various reasons.
For the things which I do repeat in a very identical fashion I could probably use workspaces.
I have (pretty much) a structure such as:
infra/logical_grouping/myprefix-env-stack1
infra/logical_grouping/myprefix-env-stack2
infra/logical_grouping/myprefix-env-stack2-substack
where -extension
in many cases either utilize remote data from the “parent” stacks, sometimes several of them. Sometimes, in the case where it makes more sense to not tie things so closely, the information sharing is done via regular data
sources instead.

I’ve made the mistake of putting too many things in the same “stack” way too many times. I’ve learned to separate early now.

(While also learning to become really proficient in terraform state mv
)

@mfridh just to be clear, what I’m describing still breaks terraform state apart so it’s not a terralith. we also have taken this a step further, so we have a single YAML config that describes the entire environment and all dependencies. so as @joshmyers was pointing out in the #terragrunt channel, we don’t have this problem anymore with hundreds of config fragments all over the place.

the key here is the terraform state is stored in workspaces, the desired state is stored in the YAML configuration, and the declarative configuration of infrastructure is in pure terraform HCL2 + modules.

Here’s a hint as to what that looks like: https://archive.sweetops.com/aws/2020/04/#ebb3d28d-3753-417c-a88a-7775f8df633c
but we’ve progressed beyond this point.
SweetOps Slack archive of #aws for April, 2020. Discussion related to Amazon Web Services (AWS)

All config fragments are in a single repo, but still a lot of them…multi region definitely doesn’t help that

@Erik Osterman (Cloud Posse) not using the https://github.com/cloudposse/testing.cloudposse.co approach still?
Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Does the use of workspaces mean your states are all in the same s3 bucket? Last time I tried it, I found this to be problematic. (Can’t remember the exact issue/limitation)

@joshmyers using geodesic (and still using that repo), but the 1-repo-per-account proved not very friendly for pipeline style automation

@randomy yes, all in one bucket, but with IAM policies restricting access based on roles

note, workspaces get their own path within the bucket so it’s easy to restrict.

are they any different to the usual environment variable? Could do all states in same s3 bucket without workspaces, right?

@joshmyers, yes, you’re correct. Could implement “workspace” like functionality without actually using the workspaces feature just by manipulating the backend settings for S3 during initialization.

I don’t know what the advantage would be to doing so

That’s what my Makefile does. No advantage over workspaces I guess but to be fair, workspaces didn’t exist when I started to do this ;).

I like that yaml defined ordering you linked to Erik. Looks legit. Although I’m not sure how you actually implement it… variant
reads it and runs various targets in order?

That’s one thing to consider…. I’m also considering Atlantis. But not sure if that would actually be a good idea for practical reasons or just “for show”.

has anyone ever created a ClientVPN configuration using Terraform to call CloudFormation templates? Or even if anyone has created a ClientVPN config in Cloudformation you might be able to help. I am getting this error when trying to create routes via cloudformation.
Error: ROLLBACK_COMPLETE: ["The following resource(s) failed to create: [alphaRoute]. . Rollback requested by user." "Property validation failure: [Encountered unsupported properties in {/}: [TargetVPCSubnetId]]"]
Code:
---
Resources:
alphaRoute:
Properties:
ClientVpnEndpointId: "${aws_ec2_client_vpn_endpoint.alpha-clientvpn.id}"
Description: alpha-Route-01
DestinationCidrBlock: 172.31.32.0/20
TargetVPCSubnetId: subnet-5c4a7916
Type: AWS::EC2::ClientVpnRoute

I’m using terraform aws_ec2_client_vpn_endpoint
to create CVPN and aws_cloudformation_stack
to add routes

I can make the exact resource in the console without issue

I have a bash command that outputs a list in yaml format. I use yq
to put that list into a file (each line is a value). There are about 2700 items in the file. How can I get that list from a file into a terraform variable? The only other approach I see is to do some magic to get a plain list into Terraform list variable file. Basically a *.txt -> *.tf transformation.

Generate a variables file. Json works fine.
Terraform module plugin concept for external data integration - frimik/tf_module_plugin_example

That’s from days of yonder before Terraform had proper provider plugin support.

Another alternative in your case, if you don’t want to pass through files on disk is to run that bash command as an external data source script: https://www.terraform.io/docs/providers/external/data_source.html
Executes an external program that implements a data source.

Have you considered using yamldecode(file(...))
with a local?

The yamldecode function decodes a YAML string into a representation of its value.

We use this pattern and it works well.

I’ll look into yamldecode
. I’m also wondering if I could have just used a data
source for the initial load of values.

if you can output to json instead, you can write <foo>.auto.tfvars.json
to your tf directory, and terraform will read it automatically and assign the tfvars

Input variables are parameters for Terraform modules. This page covers configuration syntax for variables.

Terraform also automatically loads a number of variable definitions files if they are present:
* Files named exactly terraform.tfvars
or terraform.tfvars.json
.
- Any files with names ending in
.auto.tfvars
or.auto.tfvars.json
.

i’ve also used terragrunt to do this kind of pre-processing

or maybe try pretf
, https://github.com/raymondbutcher/pretf
Generate Terraform code with Python. Contribute to raymondbutcher/pretf development by creating an account on GitHub.

here’s an example of using terragrunt and hcl string templating to massage some input into a auto.tfvars
file… i think .auto.tfvars.json
is better, but you can pick your own poison… https://github.com/gruntwork-io/terragrunt/issues/1121#issuecomment-610529140
Is there any way to disable the addition of a comment when using generate blocks? I was attempting to write an .auto.tfvars.json file, and it is written fine, but terragrunt injects a comment into …

Thanks @loren Those are awesome options.


Merge Issue With Terraform Maps

locals {
default_settings = {
topics = ["infrastructure-as-code", "terraform"]
gitignore_template = "Terraform"
has_issues = false
has_projects = false
auto_init = true
default_branch = "master"
allow_merge_commit = false
allow_squash_merge = true
allow_rebase_merge = false
archived = false
template = false
enforce_admins = false
dismiss_stale_reviews = true
require_code_owner_reviews = true
required_approving_review_count = 1
is_template = false
delete_branch_on_merge = true
}
}

Ok… this is what I’m trying to do
Default settings above… new item below
repos = {
terraform-aws-foobar = {
description = ""
repo_creator = "willy.wonka"
settings = merge(local.default_settings, {})
}
}

But when I some map key values that I want to override, they don’t seem to get picked up, despite the behavior of merge being the last one should replace for simple (not deep) merges.
repos = {
terraform-aws-foobar = {
description = ""
repo_creator = "willy.wonka"
settings = merge(local.default_settings, {
topics = ["terraform", "productivity", "github-templates"]
},
{
is_template = true
}
)
}
}

Any ideas before I go to devops.stackexchange.com or terraform community?

what is posted should work, i think. do you have an error? or an output displaying exactly what is not working?

It just doesn’t seem to pick up the “override” values from the is_template = true when the default at the top was false. No changes detected. Applying my new topics also doesn’t get picked up. This doesn’t align with my understanding of merge in hcl2

i do exactly this quite a bit, and it does indeed work. hence, we need to see what you are seeing

try creating a reproduction case with your locals and just an output so we can see what the merge produces

Well a good example would be running terraform console on this

I just created a new repo. I added topics to it, like you see above. The new topics I’m overriding don’t even show

terraform-devops-datadog = {
description = "Datadog configuration and setup managed centrally through this"
repo_creator = "me"
settings = merge(local.default_settings, {
topics = ["observability"]
})
}

and when i run doesn’t show topics at all in the properties list

Yeah so I just ran console against on specific on that has the topics override it shows empty on topics, not taking it at all

this item
devops-stream-analytics = {
description = "foobar"
repo_creator = "sheldon.hull"
settings = merge(local.default_settings, {
topics = [
"analytics",
"telemetry",
"azure"
]
})
}

Here
> github_repository.repo["devops-stream-analytics"]
{
"allow_merge_commit" = false
"allow_rebase_merge" = false
"allow_squash_merge" = true
"archived" = false
"auto_init" = true
"default_branch" = "master"
"delete_branch_on_merge" = true
"description" = "foobar"
"etag" = "foobar"
"full_name" = "foobar"
"git_clone_url" = "foobar"
"gitignore_template" = "Terraform"
"has_downloads" = false
"has_issues" = false
"has_projects" = false
"has_wiki" = false
"homepage_url" = ""
"html_url" = "<https://github.com/foobar>"
"http_clone_url" = "<https://github.com/foobar>"
"id" = "devops-stream-analytics"
"is_template" = false
"name" = "devops-stream-analytics"
"node_id" = "MDEwOlJlcG9zaXRvcnkyNTQ3NTEwMzc="
"private" = true
"ssh_clone_url" = "[email protected]:foobar.git"
"svn_url" = "<https://github.com/foobar>"
"template" = []
"topics" = [] <------ this should be overriden by my logic?
}

see the last item. It’s empty. Not sure why my merge syntax would fail as you see above.

looks fine to me?
$ cat main.tf
locals {
default_settings = {
topics = ["infrastructure-as-code", "terraform"]
gitignore_template = "Terraform"
has_issues = false
has_projects = false
auto_init = true
default_branch = "master"
allow_merge_commit = false
allow_squash_merge = true
allow_rebase_merge = false
archived = false
template = false
enforce_admins = false
dismiss_stale_reviews = true
require_code_owner_reviews = true
required_approving_review_count = 1
is_template = false
delete_branch_on_merge = true
}
}
locals {
repos = {
devops-stream-analytics = {
description = "foobar"
repo_creator = "sheldon.hull"
settings = merge(local.default_settings, {
topics = [
"analytics",
"telemetry",
"azure"
]
})
}
}
}
output "repos" {
value = local.repos
}

$ terraform apply
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
repos = {
"devops-stream-analytics" = {
"description" = "foobar"
"repo_creator" = "sheldon.hull"
"settings" = {
"allow_merge_commit" = false
"allow_rebase_merge" = false
"allow_squash_merge" = true
"archived" = false
"auto_init" = true
"default_branch" = "master"
"delete_branch_on_merge" = true
"dismiss_stale_reviews" = true
"enforce_admins" = false
"gitignore_template" = "Terraform"
"has_issues" = false
"has_projects" = false
"is_template" = false
"require_code_owner_reviews" = true
"required_approving_review_count" = 1
"template" = false
"topics" = [
"analytics",
"telemetry",
"azure",
]
}
}
}

hmm. What terraform version are you running

$ terraform -version
Terraform v0.12.24
2020-05-12

More terraform thinking out loud. I’ve been reading some of the cloudposse repos just getting a feel for how other organisations do terraform. The scale at my organization is small (handful of engineers and we will have at most 5 aws accounts across one maybe 2 regions) In the past I have handled all IAM in a single “root” module

This has its pros and cons

I’ve noticed in some cloudposse examples IAM resources are created alongside other resources. e.g an ECS service and the role it uses may be defined together. this is massively convenient

But its easier end up with issues like: Team A create a role called “pipeline_foo” and Team B (in another IAM state) create a role called “pipeline_foo”

They have no indication until the apply phase fails that this is an issue

As I move towards Terraform Cloud I also see the advantage of having a single IAM state / workspace

As just that state can be delegated to a team that manage IAM

Anyone have strong feelings on either approach?

Personally I prefer to have unique IAM resources for each module as it’s easily repeatable and self-contained solution. IMO the label module helps to avoid the naming collision

encourage them to use name_prefix
instead of name
?

ask them what they do when they deploy the same tf module more than once?

This is not an issue with modules having hardcodes names. The “submodules” are very configurable but the root module in this case defines a very specific configuration and should only be deployed once per account

My worry is that if IAM resources are created in dozens of states it could become very hard to reason about the state of IAM in an account

I hope that makes sense

@vFondevilla That is an interesting solution. Although for example if that module defines a simple barebones ecs_task_execution_role
and you deploy 20 copies of that module

well alright, but that was the scenario presented

You end up with 20 ecs_task_execution_role_some_id
with identical policies attached? That always felt a bit wrong to me. If they have different policies great but just having many copies of identical IAM resources seems redundant

iam policies have no costs. multiple instances of a policy with the same permissions is worth it to us, for the flexibility of a self-contained module. i suppose we tend to reason about the service we are creating. not the state of an aws service as a whole

That is interesting. Its great to talk about this stuff. Historically I have had a very “service centric” split of state

Yes, that’s the situation in my accounts. Each ECS service has its own IAM execution role managed by the terraform module. This enables us to repeatedly deploy the module in one of the multiple aws accounts we’re using

While it can be easier to reason about / delegate responsibility it does also lead to a lot of remote state hell and dependancy hell

e.g I need to update the vpc
state to add a security group and then I need to use remote state or data sources in my ecs
state to now refer to that security group

Maybe I should relax my service per state thinking

we do separate IAM for humans into its own module and control it centrally. but IAM for services goes into the service module

and move towards an “application” or “logical grouping of resources” per state

Yes for IAM for humans I tend to just create a module that uses lots of the modules in https://github.com/terraform-aws-modules/terraform-aws-iam
Terraform module which creates IAM resources on AWS - terraform-aws-modules/terraform-aws-iam

And that is deployed in a single account

Thanks for rubber ducking this. Do you ever end up with the opposite problem to me? i.e because IAM roles are defined alongside the service, can it be difficult to for example allow access to another principal? Because its created in another module and you can have chicken egg problems?

Or would a scenario like that be moved out to another state

I’m probably overthinking this a bit. Its a greenfields project so trying not to make any decisions that will shoot me in the foot in 12 months

I’m trying to avoid state dependencies just to simplify my life and reduce the steep learning curve for the rest of the team, so sometimes instead of referencing the remote state managed via another module, I’m using data resources as variables for the modules. In our case is working awesome with a team which before me doesn’t knew anything about terraform.

the thing that shoots us in the foot is when we do use iam resources (or most anything else, really) managed in a different state. when we keep all the resources contained in the services modules, it stays clean

Do you mean you use data sources for most module inputs @vFondevilla something like this?
data "aws_vpc" "default" {
default = true
}
data "aws_subnet_ids" "all" {
vpc_id = data.aws_vpc.default.id
}
module "db" {
vpc_id = data.aws_vpc.default.id
subnets = data.aws_subnet_ids.all.ids
#.......
}

yep

Sounds good @loren I will definitely try out this approach. I think it will save me some pain going forward

Much easier than explain someone with barely 0 experience working with Terraform modules how to use remote state and the possible ramifications. It’s pretty manual for deploying new modules (as you have to change the data and/or variables of the module), but in our case is working well.

Yeah I find data sources a cleaner solution than remote state where possible. Nothing worse than the “output plumbing” required when you need a new output from a nested module for example

Hey Guys, having issue after adding vpc to my terraform(0.12), before i am using default vpc to extract info like subnet ids etc. Error: Invalid for_each argument
on resource.tf line 63, in resource “aws_efs_mount_target” “efs”: 63: for_each = toset(module.vpc-1.private_subnets)
The “for_each” value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on. how can i solve

You can try this
for_each = toset(compact(module.vpc-1.private_subnets))

this are same with that Error: Invalid for_each argument
on resource.tf line 63, in resource “aws_efs_mount_target” “efs”: 63: for_each = toset(compact(module.vpc-1.private_subnets))
The “for_each” value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on.

Hello everyone

I am using cloudpoose module for vpc peering. I am having an issue can any one help me with it. Asap. Thank you

@maharjanaabhusan please post your actual question so that someone can answer

Error: Invalid count argument
on .terraform/modules/vpc_peering-1/main.tf line 62, in resource “aws_route” “requestor”: 62: count = var.enabled ? length(distinct(sort(data.aws_route_tables.requestor.0.ids))) * length(data.aws_vpc.acceptor.0.cidr_block_associations) : 0
The “count” value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the count depends on.

module “vpc_peering-1” { source = “git://github.com/cloudposse/terraform-aws-vpc-peering.git?ref=master)” namespace = “eg” stage = “test-1” name = “peering-1” requestor_vpc_id = module.vpc1.vpc_id acceptor_vpc_id = module.vpc3.vpc_id }
Terraform module to create a peering connection between two VPCs in the same AWS account. - cloudposse/terraform-aws-vpc-peering

Do you already created routes in your vpc? Count depend on output of:
- data.aws_route_tables.requestor.0.ids
- data.aws_vpc.acceptor.0.cidr_block_associations
Terraform module to create a peering connection between two VPCs in the same AWS account. - cloudposse/terraform-aws-vpc-peering

I am using aws vpc module of cloudpoose.

try to use https://github.com/cloudposse/terraform-aws-vpc-peering/blob/master/examples/complete/main.tf
Terraform module to create a peering connection between two VPCs in the same AWS account. - cloudposse/terraform-aws-vpc-peering

they create
• vpc
• subnets
• peering

probably you are missing subnets creation

RTFM

requestor_vpc_id = module.vpc1.vpc_id
acceptor_vpc_id = module.vpc3.vpc_id

are these vpcs created ?

Yes

i think me and @maharjanaabhusan have same problem

Does anyone here have a Terraform example for provisioning a forecast monitor in DataDog? Not sure it’s possible with the current provider, couldn’t find any examples for this.

forecast metric and a monitor on that metric ?

usually that is how it works

yeah, I think I got it

an example would be nice though. . . for that provider

@Matt do you still need help with that one?

@Julio Tain Sueiras think I have it now

need to test tomorrow

it’s basically just a variant of a query alert
2020-05-13

v0.12.25 NOTES: backend/s3: Region validation now automatically supports the new af-south-1 (Africa (Cape Town)) region. For AWS operations to work in the new region, the region must be explicitly enabled as outlined in the AWS Documentation. When the region is not enabled, the Terraform S3 Backend will return errors during credential validation (e.g. error validating provider credentials:…
Learn how to enable and disable AWS Regions.

Does anyone know of a terraform plan review tool? Something like GitHub Pull Requests but for Terraform Plan? I know Atlantis will comment on a PR with the plan and allow review and what not, but I would love a tool that I can push a plan to it and then discuss that plan with my team.

Push plan to Jira?

Hahah that strikes a cord — I do PM / Architecture consulting for one client and I use Jira too much as it is. I think that’d be my nightmare.

I’m going to guess after my quick google search that there is no such tool. Which is interesting to me… Too many ideas, not enough time.


You get that with terraform cloud

Use your ci platform to report the plan as a test result on the pr in Github?

the ability to comment on the plan


Huh I thought and looked at my TF cloud project for one client. It doesn’t go into the discussion level that I wanted, but maybe that solves enough of the problem. I just haven’t used it enough.

I was hoping to see something that has the ability to comment on a line by line change in a plan so I can explain / discuss the specific results of resource change to non-ops team members before we move it forward.


That’s an interesting idea. Atlantis had to work around Terraform plans hitting the max character length in GitHub comments by splitting up the plan into multiple comments. Sticking the plan output into a file in the pull request would get around the comment length limitation, and allow for inline commenting.
I have a to-do item to implement terraform in GitHub Actions in my org. When I get around to that task I’ll see if I can add the plan to the pull request as a file when the PR is opened.
Allowing the CI/CD process to change the contents of a commit or add a commit poses its own challenges, but we already have to deal with it using @lerna/version

I guess you could have a dedicated repo for plan artifacts and commit to that via automation and open pull request and all. That sounds like what you need, but dang the house of cards starts getting higher. How about just the review in page and then fail fast and fix if an issue :-)

you could fork atlantis and add what you need

time to learn golang

Took my first swing at some Go terraform-tfe sdk stuff today to create runs from azure devops/cli. Learned a bunch, including finding a much more mature project with great examples. Might fork and modify a bit. Looks like with this go-tfe project on you can easily run terraform cloud actions from github actions now. Super cool!
I’m going to modify this probably to accept command line args for my first PR on a go project https://github.com/kvrhdn/tfe-run
GitHub Action to create a new run on Terraform Cloud - kvrhdn/tfe-run

slick
GitHub Action to create a new run on Terraform Cloud - kvrhdn/tfe-run
2020-05-14

Hi All, I’m looking for a solution in managing/creating multiple AWS route53 zones/records. Any suggestions

What’s the problem? Terraform’s AWS R53 resources work as advertised.

resource "aws_route53_zone" "main" {
count = length(var.domains) > 0 ? length(var.domains) : 0
name = element(var.domains, count.index)
}

for CNAME record, how can i create for specific domain?

resource “aws_route53_record” “www” { count = length(aws_route53_zone.main) zone_id = element(aws_route53_zone.main.*.zone_id, count.index) name = “www” type = “CNAME” ttl = “300” records = [“test.example.com”] }

don’t know if I understand specifically but: count
and index
are blunt instruments and often will fail you when you try to implement more complex use cases


my use cases is, i’m working on creating a poc with route53 using terraform.
I had to create multiple domains in route53 and take care of operation tasks like adding new/updating/deleting records

i haven’t tried with map and for_each. i will give a try

yeah, for_each
is better. count
would be a disaster there, it would likely want to delete and recreate your zones when you change var.domains

Thank you, i will give a try and come back

Hey y’all, I did something really dumb and I’m still a bit too green at Terraform to understand how to resolve it. I’m keeping multiple state workspaces in S3 with a dynamo lock DB. I wanted to purge one of the workspaces and rebuild things from scratch. I didn’t have any resources outstanding, so I blindly deleted the file directly from S3. Now I can’t rebuild it, I suspect, because the lock db expects it to exist. Is there anyway to get back to a blank slate from here so that I can start over for this particular workspace?

go into dynamodb and clear the lock table…

Okay, thank you. I haven’t worked with dynamo yet and was worried I may break something else by clearing it. I’m especially cautious because I’ve already dug myself in to a deep enough hole.

nah, much more likely to break something on the s3 side. the dynamodb table is very lightweight

if you haven’t already, highly recommend turning on versioning for any s3 bucket you use for tfstate

yeah, as soon as I realized what was going on, I wished that I had.

you might have a copy of the tfstate locally, from your last apply

check the .terraform
directory wherever you ran terraform from

I wish. It was a new terraform script that I had only applied in Jenkins.

ah, well, jenkins. there’s the problem


I think I actually have my script working fine at this point, but I ran taint
locally (which was on .25) and it bumped the version of the state file. I can only automatically install up to .24 on Jenkins at the moment. :x

Oof. Yeah, might also want to add a terraform version constraint…

we do this in all our root modules:
terraform {
required_version = "0.12.24"
}

pin the version, then update it intentionally as a routine task

Does anyone have thoughts on how to scale Terraform when working with thousands of internal customers. The concurrent requests would be about 10 at a time. There’s a talk Uber did where they said DevOps tools don’t scale when you’re dealing with 100k servers, so there’s some upper limit. What would you say the limit is for Terraform? Is there a way to wrap Terraform around an api call so an application platform could enter a few parameters for a Terraform module to render the Terraform?

I don’t know if you could do this but maybe you could fork runatlantis.io and instead of receiving a web hook payload change it to receive a regular Rest request

That’s an idea. I remember there is a company that created a service catalog for Terraform templates. I couldn’t find the company, but I did run into this when I did a Google search for it. https://www.hashicorp.com/products/terraform/self-service-infrastructure/

Enable users to easily provision infrastructure on-demand with a library of approved infrastructure. Increase IT ops productivity and developer speed with self-service provisioning

It doesn’t go into details on how the self-service part works.

Self-service = pay for nomad and terraform enterprise

If Terraform Enterprise can provide self-service, that would be a plus if you can afford it. For larger companies, it would pay for itself in not having engineers reinvent the wheel.

my old team did some work on addressing scaling issues on one of the largest terraform enterprise installations deployed - i’d say if you’re getting to that size you want to divide the scope of what an individual deployment of TFE is addressing - maybe broken down by business units or some other logical boundary

there is an upper limit and it gets ugly when things go wrong at that limit

so i’m a bit confused here… where is the concern about scaling?
when working with thousands of internal customer
or
when you’re dealing with 100k servers, so there’s some upper limit.
And I think they are different problems to optimize for.

If you have thousands of internal customers (e.g. developers), I still would imagine it scales well, if:

• massively bifurcated terraform state across projects and AWS accounts

• teams have their own projects, aws accounts

• teams are < 20 developers on average

then i really don’t see how terraform would have technical scaling challenges e.g. by hitting AWS rate limits.

terraform itself yes - terraform enterprise (as in the server) can hit scaling issues at high usage

aha, gotcha - i could see that with something like terraform enterprise.

and (at least when we were working on it a while back - older versions) did not scale out well

i imagine that they’ve implemented a lot of lessons learned in deploying the new terraform cloud platform

but with standard CI/CD systems that can fan out to workers, I see it less of a problem.

yeh - as long as you can scale your worker fleet then it should be easier

the practical problem I see is enforcing policies at that scale.

i’m in a bit of a quandary there between opa and sentinel - i really really like opa

Ya, same. OPA is where I’d want to put my investment.

but then going against the TFE ecosystem

i mean if you’re not going to use TFE as your runner it makes sense


I was also thinking of scaling in terms of management and implementation. For example if a core team has say 10 developers, writing code for thousands of internal teams, each with their own accounts/projects doesn’t scale well. So the urge is to create some system where the end user has a gui they can go to request their infrastructure while using Terraform. What I don’t get is how to use Terraform when you scale beyond what manually writing code in a GitOps fashion does.

generally i’d say you want to scale out to teams via code rather than a gui

but i realise that that’s not always a possible reality for some enterprises

somewhat embarrassed to admit i built a demo PoC for basically this before it existed: https://www.terraform.io/docs/cloud/integrations/service-now/index.html
ServiceNow integration to enable your users to order Terraform-built infrastructure from ServiceNow

I was reading about that. It looks pretty cool. I think what the company I’m working with now wants to do is integrate Terraform with their own infrastructure request toolchain.

that’s something that i’ve seen asked for a lot - i’ve worked on projects to build it a couple of times. i understand why it’s a reality - i just don’t think i’d ever want to have to request infrastructure in that way

as an engineer on the other side of the gui i’d probably hate it

but that’s possibly because i’ve always had the privilege of not being blocked by systems of approval like that

enterprise realities kind of suck

which is why i don’t work in enterprise

i know realistically that a system to request from a service catalogue like that is 1000x better than what a lot of enterprises are dealing with today

@Erik Osterman (Cloud Posse) can you expand on this : https://sweetops.slack.com/archives/CB6GHNLG0/p1589502664287900?thread_ts=1589496149.282000&cid=CB6GHNLG0
but then going against the TFE ecosystem

I have read about OPA, and every time I go the the website it looks to good to be true

I keep thinking in how I’m to somehow get charged, of screwed for lack of support or something
2020-05-15

find . -name '*.tf' | xargs -n1 sed 's/\(= \+\)"${\([^{}]\+\)}"/\1\2/' -i

has anyone used this and have any opinions about it? https://github.com/liatrio/aws-accounts-terraform
Contribute to liatrio/aws-accounts-terraform development by creating an account on GitHub.

or is control tower to be preferred?

Hi All, I have my kubeconfig as terraform local value after eks deployment.. Now when I try to run a command “echo ${local.kubeconfig} > ~~~/.kube/config” using null_resource command option. I get command not found error guess due to multiple lines get replaced as part of local.kubeconfig..Need help on how to run this command. Right now I am coming out of terraform and doing “terraform output > ~~~kube/config” Any help to achieve it as part of null_resources are any terraform resources?

I ended up outputing my kubeconfig(s) to local_file and storing them back in source control. I reference them in null_resource with --kubeconfig=${local_file.kubeconfig.filename}
. It’s not the most elegant solution, but neither is using null_resource for kubectl apply.

Thanks for the reply. Can this be achieved in single terraform run like generating kubeconfig n copying in config path? Also can we use --kubeconfig=${local_file.kubeconfig.filename}
like this inside null resources?

Please ignore my previous msg.. I understood your answer thanks for the help. Will try it n get back if any issues

Question how am I supposed to be running the tests? in a container or does calling the make file work? I’m trying to upgrade the iam module, but first I want to know how you guys are writing tests for the other modules first.

Keeping my eyes on this. You can see some stuff on this in some repos but I haven’t yet had capacity to dive into tests yet.
2020-05-16

I am trying to get the Kubernetes Pod IP..I have launched a kubernetes deployment…in terraform.tfstate file I dont see the IP address of the POD. I need to use that POD IP address to bring another POD. How to get kubernetes POD IP which is launched by Terraform. Any help would be great!

Really appreciate if any help on this?

I’m not sure you would use the pod IP for that.
What do you mean by:
I need to use that POD IP address to bring another POD

In Kubernetes pods get assigned ephemeral IP addresses that change when the pod is moved, restarted, scaled. Using the IP of the pod to communicate with would not be a good experience. instead, Kubernetes depends on DNS names and Kubernetes Service resources.

A good place to start to understand this is: http://www.youtube.com/watch?v=0Omvgd7Hg1I&t=10m0s

The whole talk is great but the relevant part is at 10 minutes in.

Actually I need that POD to get some information from that POD..anyways let me check on DNS config..that would be appropriate as you say..

I will go over this video..

one basic question..i never tried with dns config..is it default dns config we can give are we need to have dns server configured in K8s cluster then set a dns for a particular pod..

BTW Thank you for the response..really appreciate

A good place to start to understand DNS and services in Kubernetes is: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service

If your deployments (and the pods they create) are in the same namespace, once you’ve created the service, you’ll be able to access the pods with the host part of the DNS name.

hmm I get it..it really makes sense to go with service

Say I have a Ghost blog in namespace my-blog
and a Percona DB server in my-blog
- I’d create a Service
of type ClusterIP
for the database server named db
and my Ghost blog could access it with the host name db

i was trying out raw thing out here..this looks complete solution..

If I were to deploy the DB server into another namespace I could reference it with it’s FQDN: db.other-namespace.svc.cluster.local

Another fantastic talk to watch to understand how all these Kubernetes resources interact is this video: https://www.youtube.com/watch?v=90kZRyPcRZw

Ok that sounds clear.. for now I can go with same namespace that should be fine for my case..so that it would be simple for me to refer that..

Thanks Tim..guess these details would really help me kick start on these areas..which is new to me as a k8s beginner..

anyone have a lambda for provisioning the databases inside of an RDS postgres instance? problem we’re trying to solve is creating the databases (e.g. granana
, keycloak
, and vault
) on the RDS cluster without direct access since it’s in a private VPC. In this case we do not have #atlantis or VPN connectivity to the VPC. Looking for a solution we can use in pure terraform.

mm we do have lambdas that interact with the db but not to create them

usually we create from a snapshot

can you just create a snapshot with the schema and whatever data you need and put it on a s3 bucket ?

or share the snapshot

Ohh that is an interesting idea. The s3 import option.

@Jeremy G (Cloud Posse)

Only downside is that it needs to be in the percona xtradb format, but haven’t looked at it yet.

Amazon Relational Database Service (Amazon RDS) supports importing MySQL databases by using backup files. You can create a backup of your on-premises database, store it on Amazon Simple Storage Service (Amazon S3), and then restore the backup file onto a new Amazon RDS DB instance running MySQL. For more information, see

is just the percona tools, you can use the tool to dump in that format and you are set

it is fairly simple

We need to provision users, too.

Limitations and Recommendations for Importing Backup Files from Amazon S3 to Amazon RDS
The following are some limitations and recommendations for importing backup files from Amazon S3:
You can only import your data to a new DB instance, not an existing DB instance. (emphasis added)
That is a deal-killer

can’t you add the users before you do the dump?

we do that too, we never had to create the users

plus you can use IAM auth and have “shared” users OR use SM autorotation secrets for apps ( requires code changes in app)

IAM auth is not recommended for apps with more than 200 connections per second

you can use the lambda examples for autorotate secrets to add users too since you basically need to setup the lambda access to the DB

Contains Lambda functions to be used for automatic rotation of secrets stored in AWS Secrets Manager - aws-samples/aws-secrets-manager-rotation-lambdas

Automatically rotate your Amazon RDS database secrets by using Lambda functions provided by Secrets Manager invoked automatically on a defined schedule.

I’m working on making all this to work on TF

based from this module : https://github.com/giuseppeborgese/terraform-aws-secret-manager-with-rotation/
This module will create all the resources to store and rotate a MySQL or Aurora password using the AWS Secrets Manager service. - giuseppeborgese/terraform-aws-secret-manager-with-rotation

Once the private database is provisioned how do you apply schema changes, serverless?

https://blog.codecentric.de/en/2019/10/rds-database-migration-with-lambda/ Doesn’t sound too bad…
Lambdas are handy for RDS MySQL database migrations. The referenced Github repo offers forward migration using semantic versioning.

for the original use case I think once is provisioned is done, some one else takes over

we are going to start using liquibase

You can only import your data to a new DB instance, not an existing DB instance.
@Jeremy G (Cloud Posse) this is a cold-start problem for us and it’s a new instance. As a compromise we can provision all 3 databases this way at start. Agree, it would be nicer to decouple each database creation from the RDS instance creation, but it’s less a deal-breaker than it is an inconvenience.

A lambda like that would be very cool, especially when combined with a new postgres / mysql provider

@Erik Osterman (Cloud Posse)
User accounts are not imported automatically. Save your user accounts from your source database and add them to your new DB instance later.
This is also a deal killer.
Remember, all we need to do is run 3 MySQL commands per database: CREATE DATABASE, CREATE USER, and GRANT. This does not need to be overly complicated.
What we need are
• a secure way to get master credentials to the script runner
• a way to get other dynamic parameters to the script runner
• a way for the script runner to have network connectivity to the database cluster
• a way for the script runner to communicate status
• a way to integrate and synchronize running the script with the rest of our Terraform operations

Yea, given these requirements, the lambda will be the most generalizable solution that I can think about. Or a lambda that listens to an SNS topic to run permitted psql commands.

I do not agree with that paragraph about user accounts not imported , we use the snapshots and we never had to recreate the users

but that is from a dump that already had the users in it

Have you thought of launching one off task containers for these types DB administration use cases? Lambda always feels like overkill to me when what you really need is an ECS container with a bash script and a bunch of SQL.

ECS is far more convoluted to setup than lambda

in my opinion

you could run and EC2 instance with session manager and run stuff manually trough the web console which does not require VPC conectivity

actually you can use a SSM document to run a script that will do the mysql stuff for you in an instance controlled by SSM

@Erik Osterman (Cloud Posse) I recently developed such a module. It is a lambda function that can provision a database and optionally a user in mysql and postgres in RDS instances without public access in VPC. Not very polished, but does it job perfectly. I can open source it on github within the next 12 hours or so if that helps.

Terraform module to provision database in AWS RDS instance in a VPC - aleks-fofanov/terraform-aws-rds-lambda-db-provisioner
2020-05-17
2020-05-18

I submitted a bug that appears to break aws-tfstate-backend upgrade from 0.16 to 0.17 of the module, https://github.com/cloudposse/terraform-aws-tfstate-backend/issues/47, meanwhile if anyone can explain what the error means, I am unable to make sense of it.

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform
and get to know others in the community on a more personal level. Next one is May 27, 2020 11:30AM.
Register for Webinar
#office-hours (our channel)

Has anyone worked with the golang hcl2 lib before ? I’m having the issue that the unquoted fields, for example the terraform type field for the variables is not parsing well. Just like if the lib would still be hcl1 which it’s not.
2020-05-19

Hey guys
i formatted my laptop and git pull my terraform script and facing error after terraform init
Error: Failed to instantiate provider “aws” to obtain schema: fork/exec /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.62.0_x4: permission denied
Error: Failed to instantiate provider “kubernetes” to obtain schema: fork/exec /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-kubernetes_v1.11.2_x4: permission denied
Error: Failed to instantiate provider “local” to obtain schema: fork/exec /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4: permission denied
Error: Failed to instantiate provider “null” to obtain schema: fork/exec /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-null_v2.1.2_x4: permission denied
Error: Failed to instantiate provider “random” to obtain schema: fork/exec /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-random_v2.2.1_x4: permission denied

Terraform Version 0.12.6. + provider.aws v2.47.0 (s3 backend) Terraform Configuration Files … Expected Behavior terraform init succeedss. Actual Behavior after a validate: Error: Failed to instan…

done everything in that but no solution

when you run direclty
/home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4
can you provide output?

if the same permission denied, then
chmod +x /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4
and try again

same output

given 777 still same issue

use strace

can you help me regarding strace idk how to use it and what m looking for

use google

i mount another drive to this folder , it may be the case ?

strace terraform init

directly on /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4

strace /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4

i mount another drive to this folder , it may be the case
possibly, try some other writable location

strace permission denied

on that folder

:joy: and maybe
ls /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/

fix your permissions on documents folder

i mount on Document folder

no bad words

use mount properly

THNX MAN

deleted .terraform folder and terraform init but same problem

Guys I can see that one PR is pending on s3 bucket module - https://github.com/cloudposse/terraform-aws-s3-bucket/pull/16
for adding CORS
configuration, any updates when it can be merged?

@Maxim Mironenko (Cloud Posse) @Andriy Knysh (Cloud Posse)

@Erik Osterman (Cloud Posse) do you guys have a delay when publishing modules to the terraform registry?

I was looking at the Terraform modules tutorial and I saw this :
source = "terraform-aws-modules/ec2-instance/aws"
version = "2.12.0"

and I was wondering about the benefit of using the registry over git

Nope, no such concept. The registery is just a proxy for GitHub

ahhhhhh I did not know what

It doesn’t work like other registries like Docker hub or package registries that you upload to

so how do they know what to pull ?

The registry requires a strict repository naming convention

Modules still need to be manually added

But from that point on all “updates” are in real-time

but there is a registration step somewhere I guess ?

I will read the registry docs

Using the terraform registry is not the same as git. It is a proxy, but it only gets releases with the right name format. It cannot get branches, tags, or other git references. Because of this, it is faster than using git. Git does a clone (which is larger) and if used mutiple time will do multipl clones. Where using registry will only download once per versions

Interesting thanks

Yes, good distinctions! Wasn’t thinking about that. @Steven

is there a way to use the terraform 12 for loops to create n number of terraform resources? or do we still do the count thing?

look up for_each

for_each
is if you’re iterating over a map. If you just want to create n number of resources, you would still use count
.

for_each
also works with a unique list of things (cast to a set with toset()
), as well as maps… I’d only recommend using count
if it is literally just creating multiples of the same thing, i.e. it’s original, literal purpose. Otherwise when the the list/map changes, count will want to destroy/create way more than expected since the index changes for all items in the list

I’m trying to think of a workflow where a ci/cd server would pass in parameters to a module. Is that a possibility? Are there any gotchas? I’m thinking the parameters would be passed in as TF_VAR variables. The downside that I see is that it would make it a mess to manage state.

What I’m trying to explore is the boundaries of whether Terraform is ultimately a tool that is dependent on GitOps.

I have no idea what do you mean with this ?

what does it mean to be dependant in gitops ?

Yes, ci/cd can pass parameters to Terraform. This is not uncommon. Terraform state needs to be remote or you risk losing it. Terraform manages state. What you pass to it should be things that change the state. If that is not what you are trying to do, then job either needs to change to different state files or use workspaces.

Terraform doesn’t care about GitOps. You can use with or without

That’s useful information. I’m so used to thinking of GitOps as the goal that I’m trying to wrap my mind around other ways of interacting with Terraform besides through git commits.

gitops is a methodology not a tool

Terraform Cloud is fundamentally “GitOps” driven.
2020-05-20

Hello I’m facing issues while spinning an eks_cluster using the cloudposse module

Need some help with that ..

module.eks_cluster.null_resource.wait_for_cluster[0]: Still creating... [2m10s elapsed]
module.eks_cluster.null_resource.wait_for_cluster[0]: Still creating... [2m20s elapsed]
module.eks_cluster.null_resource.wait_for_cluster[0]: Still creating... [2m30s elapsed]
module.eks_cluster.null_resource.wait_for_cluster[0]: Creation complete after 2m32s [id=2129570020838525894]
module.eks_cluster.kubernetes_config_map.aws_auth_ignore_changes[0]: Creating...
Error: configmaps "aws-auth" already exists
on .terraform/modules/eks_cluster/auth.tf line 84, in resource "kubernetes_config_map" "aws_auth_ignore_changes":
84: resource "kubernetes_config_map" "aws_auth_ignore_changes" {

My tf ends with this message ..

I’ve searched around on the web and there are some threads that discuss this ..but none of those solutions have worked

The config map has probably been created before (e.g. by using older version of the module). You need to manually delete it using kubectl. The module uses kubernetes provider to create the config map, and the provider just can’t update it, only create a new one

@Andriy Knysh (Cloud Posse) I’ve done a terraform run from scratch into a completely clean environment, still I consistently get this error. I’ve cleaned up the state files, .terraform folder .. pretty much everything. And still keep getting this error.

the config map is already provisioned (can’t say how). To fix this, there are only two ways: 1) destroy the EKS cluster and provision again; 2) from the command line, find the config map and delete it using kubectl

Andriy, I’m spinning an EKS cluster from scratch. It wasn’t pre-existing. Still I continue to hit this. There is no other piece of code that is creating aws_auth configmap.

If I set the following flag to false will it hurt? apply_config_map_aws_auth

looking at the code it seems that will stop the code from creating the aws_auth map altogether.

@Andriy Knysh (Cloud Posse), What do you think?

@Harshal Vaidya did you look at this example https://github.com/cloudposse/terraform-aws-eks-cluster/tree/master/examples/complete
Terraform module for provisioning an EKS cluster. Contribute to cloudposse/terraform-aws-eks-cluster development by creating an account on GitHub.

when you create the cluster first time, there should be no config map

did you delete all the resources and start again?

That is correct .. I deleted the entire cluster and then started again ..

I’ll look at the example tomorrow

Checkout terraform version manager written by me, supports pip, docker and homebrew !!
Its already becoming popular with 40 .
Terraform & Terragrunt Version Manager. Contribute to aaratn/terraenv development by creating an account on GitHub.

anyone using terraform cloud? I am running a module that requires running awscli locally and their remote server does not have it installed. anyone have any advice?

I tried the obvious thing of apt-get installing it but that didn’t work

A good question for todays office hours but I’ll say that anything you can shift to pure terraform provider code will make your life much easier in the long run

you have basically 2 options.

One is to add any dependencies to the git repo itself. PIA if you need to copy that 100x over.

the other is to use the exec-provider to install what you need. we’ve done this in the past. @Andriy Knysh (Cloud Posse) probably has a snippet somewhere.

either way, it’s a slippery slope.

What terraform cloud needs (and the competitors like Spacelift and Scalr support) is a model where you bring your own docker image
@Jake Lundberg (HashiCorp) @sarkis

Terraform Enterprise does have this ability. I’ll check if we’ll enable this for TFC.

We may only allow that for paid accounts (in the future).

Right - that’s fair. But supporting it in some capacity I think is needed. Though support for thirdparty providers is improving (i think?) which mitigates it to some degree.

We had a recent use case of using a third-party provider terraform-helmfile-provider
, which depends on helmfile
binary, which depends on the helm
binary and helm-diff
, helm-secrets
, etc plugins. Getting all that to run on terraform cloud was a major PIA.

:smiley: oh and kubectl

@Erik Osterman (Cloud Posse) can you go through the process to get this to work? I’d like to make sure we’re capturing this from the field.

@Jake Lundberg (HashiCorp) Here’s what we had… (not proud of this!)
provider "shell" {}
data "shell_script" "install_external_packages" {
lifecycle_commands {
read = <<EOF
set -e
export PATH=$PATH:${local.external_packages_install_path}:${local.external_packages_install_path}/bin
mkdir -p ${local.external_packages_install_path}
cd ${local.external_packages_install_path}
echo Installing AWS CLI ...
curl -LO <https://s3.amazonaws.com/aws-cli/awscli-bundle.zip>
unzip ./awscli-bundle.zip
./awscli-bundle/install -i ${local.external_packages_install_path}
aws --version
echo Installed AWS CLI
echo Installing kubectl ...
curl -LO <https://storage.googleapis.com/kubernetes-release/release/${local.kubectl_version}/bin/linux/amd64/kubectl>
chmod +x ./kubectl
echo Installed kubectl
echo Installing helm ...
curl -LO <https://get.helm.sh/helm-v${var.helm_version}-linux-amd64.tar.gz>
tar -zxvf ./helm-v${var.helm_version}-linux-amd64.tar.gz
mv ./linux-amd64/helm ./helm
chmod +x ./helm
helm version --client
echo Installed helm
echo Installing helmfile ...
curl -LO <https://github.com/roboll/helmfile/releases/download/v${var.helmfile_version}/helmfile_linux_amd64>
mv ./helmfile_linux_amd64 ./helmfile
chmod +x ./helmfile
which helmfile
helmfile --version
echo Installed helmfile
aws_cli_assume_role_arn="${var.aws_assume_role_arn}"
aws_cli_assume_role_session_name="${module.label.id}"
echo Assuming role "$aws_cli_assume_role_arn" ...
curl -L <https://github.com/stedolan/jq/releases/download/jq-${var.jq_version}/jq-linux64> -o jq
chmod +x ./jq
# source <(aws --output json sts assume-role --role-arn "$aws_cli_assume_role_arn" --role-session-name "$aws_cli_assume_role_session_name" | jq -r '.Credentials | @sh "export AWS_SESSION_TOKEN=\(.SessionToken)\nexport AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) "')
echo Assumed role "$aws_cli_assume_role_arn"
echo Getting kubeconfig from the cluster...
aws eks update-kubeconfig --name=${var.cluster_name} --region=${var.region} --kubeconfig=${var.kubeconfig_path} ${var.aws_eks_update_kubeconfig_additional_arguments}
export KUBECONFIG=${var.kubeconfig_path}
kubectl version --kubeconfig ${var.kubeconfig_path}
kubectl get nodes --kubeconfig ${var.kubeconfig_path}
kubectl get pods --all-namespaces --kubeconfig ${var.kubeconfig_path}
echo Got kubeconfig from the cluster
echo '{"helm_version": "$(helm version --client)", "helmfile_version": "$(helmfile --version)"}' >&3
EOF
}
}
output "helm_version" {
value = data.shell_script.install_external_packages.output["helm_version"]
}
output "helmfile_version" {
value = data.shell_script.install_external_packages.output["helmfile_version"]
}

so, if we just had a docker image with our toolchain inside of it, it would have been a lot easier.

Thanks for this Erik. What’s strange is we have custom workers in TFE. I think there’s some major concerns about security, but that you can install anything you want anyway, just slows things down.
I’ll pass this on to the developers so they see what kind of gymnastics folks are going through.

nvm looks like “Your plans and applies occur on machines you control. Terraform Cloud is only used to store and synchronize state. Save settings” is the way to go

Terraform cloud is more than that. That’s just the state file stuff. I use for running plan and apply in their hosted containers

hello I am having the following code to create a launch configuration which creates one secondary volume but I need to create multiple EBS volumes as per the requirement … can anyone suggest what would be the best approach to have that functionality
resource "aws_launch_configuration" "launch_config_with_secondary_ebs" {
count = var.secondary_ebs_volume_size != "" ? 1 : 0
ebs_optimized = var.enable_ebs_optimization
enable_monitoring = var.detailed_monitoring
image_id = var.image_id != "" ? var.image_id : data.aws_ami.asg_ami.image_id
instance_type = var.instance_type
key_name = var.key_pair
name_prefix = join("-", compact(["LaunchConfigWith2ndEbs", var.name, format("%03d-", count.index + 1)]))
placement_tenancy = var.tenancy
security_groups = var.security_groups
user_data_base64 = base64encode(data.template_file.user_data.rendered)
ebs_block_device {
device_name = local.ebs_device_map[local.ec2_os]
encrypted = var.secondary_ebs_volume_existing_id == "" ? var.encrypt_secondary_ebs_volume : false
iops = var.secondary_ebs_volume_iops
snapshot_id = var.secondary_ebs_volume_existing_id
volume_size = var.secondary_ebs_volume_size
volume_type = var.secondary_ebs_volume_type
}
iam_instance_profile = element(
coalescelist(
aws_iam_instance_profile.instance_role_instance_profile.*.name,
[var.instance_profile_override_name],
),
0,
)
root_block_device {
iops = var.primary_ebs_volume_type == "io1" ? var.primary_ebs_volume_size : 0
volume_size = var.primary_ebs_volume_size
volume_type = var.primary_ebs_volume_type
}
lifecycle {
create_before_destroy = true
}
}

Provides a resource to create a new launch configuration, used for autoscaling groups.

you can have multiple ebs_block_device
blocks

variable "secondary_ebs_volume_size" {
description = "EBS Volume Size in GB"
type = list(string)
default = []
}
ebs_block_device {
count = length(var.secondary_ebs_volume_size) > 0 ? 1 : 0
device_name = local.ebs_device_map[local.ec2_os]
encrypted = var.secondary_ebs_volume_existing_id == "" ? var.encrypt_secondary_ebs_volume : false
iops = var.secondary_ebs_volume_iops
snapshot_id = var.secondary_ebs_volume_existing_id
volume_size = var.secondary_ebs_volume_size[count.index]
volume_type = var.secondary_ebs_volume_type
}

does this work

I mean we need to have the functionality that can have any number of devices added based on the requirement

no count wont work

you’ll need to use dynamic blocks for that and terraform 0.12

The Terraform language allows the use of expressions to access data exported by resources and to transform and combine that data to produce other values.
2020-05-21

Guys, there is PR raised by one of my colleague for fixing the auto-scaling issue in RDS cluster - can you guys review it ? https://github.com/cloudposse/terraform-aws-rds-cluster/pull/67
Fix for #63 The cluster_size or the instance_count should be independent of autoscaling_min_capacity as autoscaling_min_capacity is automatically taken care of by AWS through the autoscaling policy…

@Erik Osterman (Cloud Posse) @Andriy Knysh (Cloud Posse) Could you please review the PR or have someone review it: https://github.com/cloudposse/terraform-aws-rds-cluster/pull/67
Fix for #63 The cluster_size or the instance_count should be independent of autoscaling_min_capacity as autoscaling_min_capacity is automatically taken care of by AWS through the autoscaling policy…

Yep, let’s move to #pr-reviews
Fix for #63 The cluster_size or the instance_count should be independent of autoscaling_min_capacity as autoscaling_min_capacity is automatically taken care of by AWS through the autoscaling policy…

sure

Guys do you have any reference articles/pointers on how to go to a ECS Fargate deployment for my microservices with CI/CD enabled?

Yep, check out https://github.com/cloudposse/terraform-aws-ecs-web-app
Terraform module that implements a web app on ECS and supports autoscaling, CI/CD, monitoring, ALB integration, and much more. - cloudposse/terraform-aws-ecs-web-app

Awesome example. Out of interest in this case what updates / registers the new task definition after the latest image is built in CI? And how do you stop that from conflicting with the initial task definition created by terraform?

Any time I have implemented something like this I need to create the Initial skeleton task definition with Terraform. But after I build / tag the image I need some additional automation to generate the new task definition ( that references the new image tag, typically a git SHA) and then register that task definition and restart the service

I’d be interested to hear your approach to this problem. Or is it an exercise left to the reader

This uses codebuild/codepipeline and the aws
cli to bump the version of the task definition (if I recall correctly)

The problem that I usually encounter though. Is that the next time you plan / apply that terraform you will have a bad time

As it will want to revert those resources which will typically take down the service.

that’s why you have to ignore_changes
on the task definition and explicitly taint it when you need to change it.

are you using terraform cloud by anychance?

@Erik Osterman (Cloud Posse) Awesome Thanks!!

Yup just started with Terraform cloud as I used terragrunt on previous projects

Terraform module which implements an ECS service which exposes a web service via ALB. - cloudposse/terraform-aws-ecs-alb-service-task

So terraform cloud will reduce the complexity of running the terraform as part of your workflow.

@johncblandii can @conzymaher trigger a build in TFC with a webhook (esque) functionaltiy and pass a parameter like an image name/tag?

Terraform by HashiCorp

Ah. So traditionally I have separated infrastructure deployment and “code” deployment. I suppose I could run terraform triggered from VCS with “auto apply” to deploy the code by pushing an updated container definition?

so, for example, maybe you’re using CircleCI/GitHub Actions/etc and you use that for all your integration testing. Then you push an artifact to ECR. Then you trigger a deployment pipeline, and as part of that you trigger TFC.

This is the one fuzzy bit of CI/CD for ECS with Terraform so was just interested to see how you guys have solved it. I have also used the AWS GitHub actions to render and update the task definition

I agree about separating infrastructure from code deployments. This is where AWS has muddled the water for us.

This is also why we almost exclusively focus on Kubernetes.



I did a quick tutorial on it

Thanks John. I’ll take a look

Powerful CLI tool to simplify Amazon ECS deployments, rollbacks & scaling - fabfuel/ecs-deploy

It recreates a task definition allowing changes to docker image, env vars, etc so you can customize it for a specific run, if so desired

All of that is based on the CP module Erik linked above

…and others from CP

Is there any way to lookup names of IAM roles with wildcards? Situation: AWS SSO Federated roles are appended with a random string

Doesn’t look like the iam_role data source supports filters … you could use an external data source and run an AWS API call to get all your roles and then filter it down, and return the one you need. We do that for some similar limits with looking up RDS because ‘describe datatbase instance’ doesn’t take filters

just dump them all to a file
aws iam list-roles --query 'Roles[].RoleName' > all-my-roles.txt

then you can grep them

if you have to use terraform, you can create a null resource that runs the above command, saves to a file, reads the file, creates data sources for each iam role, and then you can use other iam attributes

Thanks for the input, that should work for me…

Is there an issue yet to support filters on the data source? That would be useful

that would be a nice addition. i dont see an issue for it

Closest one i could find regarding filtering on tags https://github.com/terraform-providers/terraform-provider-aws/issues/7419
Community Note Please vote on this issue by adding a reaction to the original issue to help the community and maintainers prioritize this request Please do not leave "+1" or "me to…

The limitation is that GetRole api call doesn’t support anything other than the name supplied. I wonder if tf aws provider would list roles first before applying filtering. I also wonder if they already do that in other data sources

the GetRole API says RoleName accepts regex…. https://docs.aws.amazon.com/IAM/latest/APIReference/API_GetRole.html
Retrieves information about the specified role, including the role’s path, GUID, ARN, and the role’s trust policy that grants permission to assume the role. For more information about roles, see Working with Roles .

oh, no, that’s the regex pattern it accepts. dang.

they’d have to use ListRoles and then client-side filtering

Looks interesting… Anyone use checkcov
?
https://www.checkov.io/1.Introduction/Getting%20Started.html

I had a call with them today

using checkov, tflint, and tfsec in our atlantis for every tf module

product looks nice. we’re going to be doing a poc with the cloud posse terraform modules

it catches some stuff the others dont catch which is nice.

all of the linters have a certain amount of overlap

Cool, was wondering how it would work with modules that are not super opinionated

i like checkov more than tflint because it doesnt require a configuration file to work. it just works out of the box.

Is it pure static analysis, or does it need to run a plan or apply?


tf 0.13.0 betas coming soon… Sure hope this upgrade is less painful than 0.12!
https://discuss.hashicorp.com/t/terraform-v0-13-0-beta-program/9066
I’m very excited to announce that we will start shipping public Terraform 0.13.0 beta releases on June 3rd. The full release announcement is posted as a GitHub issue, and I’m opening up this discuss post as a place to host community discussions so that the GitHub issue can be used just for announcements.

Nice, so 2020 is going to be a good year after all
I’m very excited to announce that we will start shipping public Terraform 0.13.0 beta releases on June 3rd. The full release announcement is posted as a GitHub issue, and I’m opening up this discuss post as a place to host community discussions so that the GitHub issue can be used just for announcements.

module expansion: modules will support count and for_each. We're still working on depends_on, but it's looking good and I think it'll make 0.13.0.


I mean, what is a terraform module, if I no longer have to code an enable variable into every resource!


Omg. That will solve so much . Provider is normally outside a module though so still have one challenge which is I want to loop the AWS accounts too. This is a major plus regardless! Less repeated code and more reason to use modules

It’s in beta….

would rather a long beta than a fast bug release

i started using it with beta1, didn’t hit any issues. beta2 fell over with module nesting more than 1 deep. fixed in master, waiting on the next beta to test again…


oh happiness
cli: Add state replace-provider subcommand to allow changing the provider source for existing resources [GH-24523]

Hi Folks, Is there any command or something if i would like to use specific aws provider. Currently I am using latest + provider.aws v2.62.0 and would like to use v2.59.0 to get rid of below error. I am getting below error in v2.62.0 provider.
1 error occurred:
* aws_autoscaling_group.asg: Error creating AutoScaling Group: ValidationError: You must use a valid fully-formed launch template. You cannot use PartitionNumber with a Placement Group that does not exist. Specify a valid Placement Group and try again.
status code: 400, request id: 7db8c6a4-8061-47bb-9440-ded120c14d03

Providers are responsible in Terraform for managing the lifecycle of a resource: create, read, update, delete.

Awesome. Thankyou @Chris Fowles
Providers are responsible in Terraform for managing the lifecycle of a resource: create, read, update, delete.

use:
terraform {
required_providers {
aws = "= 2.59.0"
}
}
2020-05-22

Hi Guys, I am kind of new to terraform enterprise and we just starting using it.When I initiate queue plan from my workspace though configuring keys and all those, it just showing blank, doesn’t do anything. I just canceled it.Am I missing anything? Appreciate your suggestion.

I have restarted replicatedctl and that fixed the issue

why does terraform init download the repo again from github and then terraform apply ignores the cwd and only executes the modules in .terraform? I have to edit my source files twice every time I do this and it is annoying.

This a generic rant or do you really want an answer? )

2020-05-23
2020-05-24
2020-05-25

I keep hearing lately that ‘terraform should build your infra and not deploy your code’ - but how does that work if you are building immutable AMIs as the ‘code package’? If terraform isn’t the one ‘deploying’ the AMI, what else is being used, and how is the terraform state for the ASG/launch-template/etc being maintained?

that is a good question, I usually say do not use TF to deploy code and I mostly refer to people using a ssh trough TF to ssh a zip file and run a bash script, which is not your case

and since you are changing ASG then I do not see why you should not use TF for that , I think it comes down to trying it and if it fits your needs then you are good

I feel like this would be a good question for the Hashicorp Terraform message board. It’s a pretty interesting question — I’d like to know how others are handling that when AMIs are their method of deployment.

When we started deployed IAM in my old company we did not used TF but we used packer to create the images and we had Spinnaker to switch the AMIs by using new ASGs

that is an example were the deployment tool can handle some infrastructure part of the deployment but now that I’m using TF I could see how this could be implemented differently, in fact Sninnaker supports TF stages now


so i’d still recommend something else to handle that (not sure what the else is)

It’s been mentioned here in another thread, there’s several ways of doing ASG updates. We just tie the name of the ASG to the AMI ID, so new AMI = new ASG, and it has a create_before_destroy lifecycle. Some other folks pointed out a blog describing how you can use some AWS CF templates in your terraform, and that can access some hidden ASG APIs that terraform can’t use.

yeh - you can do all that @Zach, and I do, but it still kind of sucks

it’s a hack - not a feature

:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform
and get to know others in the community on a more personal level. Next one is Jun 03, 2020 11:30AM.
Register for Webinar
#office-hours (our channel)
2020-05-26

Does anyone have recommendations on how to best secure a terraform CI/CD pipeline when using federated logins/SSO? My initial thoughts are to create a IAM user in each account (dev/staging/prod) that would have static credentials, but there has to be a better way…

You could add an Instance Profile to the EC2 instance(s) that the pipeline is running on. That would give the pipeline access to the same role that federated users get without having iam user creds. You just have to make sure only trusted users and/or pipelines run on them since the instance profile is available to any running process. In a Kubernetes world that means using something like kiam to hand out access to selected pods
Integrate AWS IAM with Kubernetes. Contribute to uswitch/kiam development by creating an account on GitHub.

If you are on k8s, you want to add the instance profile to a separate node pool that only kiam runs on. All other workloads should run on a node pool that does not have an instance profile. kiam delegates access to other pods that need it based on rules you set up

Sorry, I forgot an important detail there - a 3rd party cloud based CI/CD pipeline

Ah, then maybe a service account user in your SSO? We have done that as well. It can even work when MFA is required by saving off the MFA token using the text-based version of the QR code that is used to register an MFA app

Ok, yea I think that may be better. We’re using AWS SSO, I don’t think it has a concept of a “service acccount user” - so it would just be a regular user with permissions to do what is needed in the child accounts? I’m still a little confused how that would work since the credentials change

since theres no IAM user, just assumed roles…

It would be a regular user that isn’t associated with an actual person. Save the password off as a secret that will get used by the pipeline.
It would use the same IAM roles that your other users use

Ahh - saving password instead of access/secret key - I guess thats possible too. I wonder which would be better, that or a terraformed IAM user in each account

I prefer to avoid using one-off IAM users, since you have to manage rotation of that cred. If you have a service account user in SSO, the rotation of that password can be managed according to your corporate policy the same as all your other users

Good point. Thanks for the input, Andrew.

anyone running celery in ecs / fargate ? looking to run the worker and celerybeat (scheduler) in ecs if possible
• is it OK to run celery worker and beat in ECS ? is it OK to run in fargate?
• what tf modules, if any, would help ?
• any gotchas ?

we’re already using the terraform-aws-modules/terraform-aws-atlantis which doesn’t use a module but simply uses the ecs service and task definition resources using FARGATE as an example

@RB I’m running these for a client project, just like you’re saying: Beat + Celery both in their own services on Fargate. My setup is pretty stock standard — Nothing fancy.

I actually used https://github.com/masterpointio/terraform-aws-ecs-alb-service-task and then just didn’t front the resulting service with an ALB which works great.
Terraform module which implements an ECS service which exposes a web service via ALB. - masterpointio/terraform-aws-ecs-alb-service-task

Healthcheck was tiny bit weird — Looks like I wrote a start_celery
and start_beat
script which the CMD calls. These start the underlying process and create a PID file. Then I have a healthcheck
script which checks on the known PID file to ensure the task is up and running.
# Start celery worker
celery worker \
-l INFO \
--app=app_name \
--time-limit=300 \
--concurrency=2 \
--workdir=/app/ \
--pidfile=/app/celery_pid
That was my start_celery
script.

oh interesting!

Or actually, looking back at it — I just passed a healthcheck_command to my fargate_background_job
module (wrapper for the CP module) like this:
healthcheck_command = "kill -0 $(cat /app/beat_pid)"
healthcheck_command = "kill -0 $(cat /app/celery_pid)"

so youre running both the tasks in the same fargate cluster too ?

Yeah, all services running in the same cluster.

if you get a chance, would love an open sourced celery specific fargate example using your module

ill work through it and try to figure it out

celery specific fargate example
It’s all super light — Not sure I have much of an example I could show. I just wrapped the CP terraform-aws-ecs-alb-service-task
(seems I linked my fork above by accident since I had to fork and modify for my client) and terraform-aws-ecs-container-definition
, but beyond that there is no secret sauce. I’m sure you’ll get it figured.

hmmm, having an issue with the port mappings. can’t seem to find the celery workers ports online. does this look right ?
module "container_definition" {
source = "git::<https://github.com/cloudposse/terraform-aws-ecs-container-definition.git?ref=tags/0.21.0>"
container_name = local.application
container_image = "${module.ecr.repository_url}/${local.application}:latest"
port_mappings = [
{
containerPort = 8080
hostPort = 8080
protocol = "tcp"
}
]
command = [
<<EOF
celery worker \
-l INFO \
--app=app.tasks \
--time-limit=300 \
--workdir=/opt/celery \
--pidfile=/opt/celery/celery_pid
EOF
]
healthcheck = {
command = ["kill -0 $(cat /opt/celery/celery_pid)"]
interval = 60
retries = 1
startPeriod = 0
timeout = 60
}
}

when i ssh to my container, i can run the same celery worker command without any issues

I didn’t expose any ports for my celery worker AFAIR.

Are you trying to get to the Celery UI @RB?

(Is there a celery UI? I have no idea haha.)

lol there is a flower ui for celery on port 5555 or 8080 i believe

i think im making some noob mistakes here

yes i definitely am because im using celery worker to read off the sqs queue so it doesn’t need any port access

it just needs access to the sqs queue and that should do it

Ah looking at your command — that might be the issue:
command = [
<<EOF
celery worker \
-l INFO \
--app=app.tasks \
--time-limit=300 \
--workdir=/opt/celery \
--pidfile=/opt/celery/celery_pid
EOF
]
Not 100% on this, but that might blow up because it’s not following the comma separated strings command syntax. AFAIK docker / ECS won’t like that.

My similar example that I posted earlier — I had that command in a bash script that I bundled with the container and then my CMD was to invoke that bash script.

ahhh ok so every argument needs to be broken up into a separate value in the array ?

Yes, typically. It’s a PITA, but that’s the way Docker wants it.

oh i see, that makes this a lot easier

i have flask and celery worker tied together

so i wanted to use the same docker file

i suppose i can make a base image and have 2 child images, one for flask api, the other for celery worker

and each child can have its own CMD argument

I did the same. One Dockerfile that deployed a Django, Celery, and Beat application. You should be able to make it happen.


Or keep the one dockerfile / image and then the command dictates what the container is actually going to do.

thx again for all the help. been wacking away at this all day. if i get stuck again tomorrow, i might ping you again in this thread if you dont mind.

@RB No problem — Happy to help!

lots of gotchas… one that i didnt expect was that the container command needs to be an array like this
command = split(
" ",
"celery worker -l INFO --app=app.tasks --time-limit=300 --workdir=/opt/celery --pidfile=/opt/celery/celery_pid",
)
whereas the healthcheck command cannot be split on space and instead should be like this
container_healthcheck = {
command = ["kill -0 $(cat /opt/celery/celery_pid)"]

now celery worker is in a steady state

Huh interesting. Well, glad you got it figured!

nope nvm, had to do the same split. not sure why it wasn’t working before.

anyway thanks again!

ah shit, spoke too soon. getting this into a steady state is a pain. just like you said, its a pain to get it to pass healthchecks

ok so when i changed the healthcheck to command = split(" ", "cat /opt/celery/beat_pid")
thats when i could get it to work correctly

idk how your kill command was a healthcheck. it kept killing the process for me.

Huh… wierd.


cat $PID_FILE
won’t actually do a health check. The underlying beat or celery process could die and that file will still be around.

I recently came across this in a shell script. if ! kill -0 $(cat /path/to/file.pid); then … do something … fi What does kill -0 … do?

Oh are you using the right task def syntax?
# <https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_HealthCheck.html>
healthcheck = {
command = ["CMD-SHELL", "${var.healthcheck_command} || exit 1"]
retries = 3
timeout = var.healthcheck_timeout
interval = var.healthcheck_interval
startPeriod = 30
}


The linked AWS API Ref should give you an idea about that.

brilliant

@RB Ah that ended up doing the trick? Glad to hear it!

Separate from this thread but related to celery. I attempted to use sqs for celery’s queue which seems to work however the flower ui doesn’t support it

Do you use redis / elasticache or rabbitmq as the queue?

I’m considering elasticache redia or amazon mq since it will remove our reliance on our current self hosted rabbit

We used Elasticache. That would worked well. Though it is very likely to be more expensive than SQS.

did you get good results with flower ui ?

Unless you need Rabbit for something else then I would steer clear of that route. More complicated than Redis.

we’re looking at the costs betw amazon mq and elasticache now

We’re not using Flower UI. My client’s celery usage is very minimal. We just let it run.

ya we dont use rabbitmq for anything else

oh i see interesting so you have no use for the UI

Yeah, we just schedule jobs through our Django app and it’s our simple background job runner. We have a few scheduled jobs through beat.

Service degradation for Terraform Cloud May 26, 18:50 UTC Investigating - We have identified service degradation for Terraform Cloud and are investigating.
HashiCorp Services’s Status Page - Service degradation for Terraform Cloud.

Service degradation for Terraform Cloud May 26, 20:02 UTC Update - We are continuing to investigate this issue.May 26, 18:50 UTC Investigating - We have identified service degradation for Terraform Cloud and are investigating.

Service degradation for Terraform Cloud May 26, 21:24 UTC Resolved - We’ve mitigated the issue causing the service degradation and are now monitoring. All services are currently operating normally.May 26, 20:02 UTC Update - We are continuing to investigate this issue.May 26, 18:50 UTC Investigating - We have identified service degradation for Terraform Cloud and are investigating.

https://github.com/hashicorp/terraform/issues/25016
module expansion: modules will support count
and for_each
. We’re still working on depends_on
, but it’s looking good and I think it’ll make 0.13.0.
I'm very excited to announce that we will start shipping public Terraform 0.13.0 beta releases on June 3rd. You will be able to download them on releases.hashicorp.com, like the 0.12.0-beta ser…

We are pleased to announce the alpha release of a new version of the Kubernetes Provider for HashiCorp Terraform. The kubernetes-alpha provider lets you package, deploy, and manage…

If I get this demo working I’ll be using the new Kubernetes provider for Terraform during my keynote at the Crossplane Community Day virtual event. https://www.eventbrite.com/e/crossplane-community-day-tickets-104465284478 https://twitter.com/mitchellh/status/1265414263281029120

Yes! An alpha release of a new Kubernetes provider for Terraform that can represent ANY K8S resource (including any CRDs). You can also run this one-liner (image) to convert any YAML over. https://www.hashicorp.com/blog/deploy-any-resource-with-the-new-kubernetes-provider-for-hashicorp-terraform/ https://pbs.twimg.com/media/EY-nj__U8AAzI4C.jpg

Finally! This is great news.

this is excellent - it actually answers a really big question with how we move forward

Do terraform modules now begin to replace helm charts?

What would be the approach if my infra deployed by Terraform is messed up on (few resources got deleted manually) due to some external factor and I am trying to run terraform destroy but it fails with resources missing? How to get over this issue? I tried with terraform state rm <resource> but this is not viable if we have multiple resources. coz need to identify them one by one and delete? Is there any approach or recommendations?



i can’t believe it took me this long to figure that out

oooh that’s a good one

i honestly feel like the sun just rose
2020-05-27

Terraform AWS data aws_iam_policy_document
Problem: every time I do tf plan/apply when using a aws_iam_policy_document, terraform recomputes the policy document. Thus, it creates changes always in its plan. (See image below)
Question: How do you guys deal with it? Do you simply ignore? Do you avoid using the aws_iam_policy_document?

@Carlos R. do you have a depends_on
defined in the datasource maybe ?

nop

would you mind copy/pasting the code-block ?

not at all

data "aws_iam_policy_document" "gitlab_backup_policy_document" {
statement {
sid = "ListOnlyMyBucketInfo"
effect = "Allow"
actions = [
"s3:List*",
"s3:Get*"
]
resources = [module.gitlab_backup_bucket.bucket_arn]
}
statement {
sid = "ListAllBuckets"
effect = "Allow"
actions = [
"s3:ListAllMyBuckets"
]
resources = [
"*",
]
}
statement {
sid = "WriteObjectActions"
effect = "Allow"
actions = [
"s3:Put*",
# TODO: temporary actions below. To be used during testing only.
"s3:List*",
"s3:Get*"
]
resources = ["${module.gitlab_backup_bucket.bucket_arn}/*"]
}
statement {
sid = "ListKMSAliases"
effect = "Allow"
actions = [
"kms:ListAliases"
]
resources = [
"*",
]
}
}
resource "aws_iam_policy" "gitlab_backup_policy" {
name = "${module.label.id}-gitlab"
policy = data.aws_iam_policy_document.gitlab_backup_policy_document.json
}
resource "aws_iam_policy_attachment" "gitlab_backup_policy_attachment" {
name = module.label.id
users = [aws_iam_user.gitlabl_backup_application.name]
policy_arn = aws_iam_policy.gitlab_backup_policy.arn
}

I don’t see the bucket_policy
in there ?

it is a iam policy, not a bucket policy

I am attaching it to an IAM User

you can see it in the last resource block

ah ok, because the screenshot yous showed was about a a data "aws_iam_policy_document" "bucket_policy" {

it’s the same block, but yeah.. the name can be misleading

i think more what @maarten is getting at is that this is not typical tf behavior, so it is something in your config. without seeing the exact config, and probably more of the plan result, it is very difficult to just intuit a reason…

i like to encourage people to write a second, simpler, minimal config to try reproduce the problem and pinpoint what change causes it

@Andriy Knysh (Cloud Posse) I feel like this is something you dealt with recently. Do you recall what you did to prevent it?

Ah we talked in private, there was a misunderstanding regarding the terraform code snippit earlier.. But most likely it is not related to a faulty terraform configuration as it could not be reproduced.

Should I delete the post? so that it does not generate confusion

On the contrary! I think it would be more helpful to summarize the confusion

Alright, sounds good. Will do.


Indeed, thank you very @maarten

@Erik Osterman (Cloud Posse) @Andriy Knysh (Cloud Posse) The problem is that the moment a change happpened to the S3 bucket, for some reason the bucket policy get’s recalculated because here the splat operator is not used : https://github.com/cloudposse/terraform-aws-s3-bucket/blob/master/main.tf#L133 , on line 115 it is. If they both use the splat operator it will be fine.
Terraform module that creates an S3 bucket with an optional IAM user for external CI/CD systems - cloudposse/terraform-aws-s3-bucket

yea, we missed that

and I don’t like [0]

splat+join
works in all cases

thanks @maarten

what Use splat+ join in https://github.com/cloudposse/terraform-aws-s3-bucket/blob/master/main.tf#L133 why Using [0] causes Terraform to recreate the policy on each terraform plan

v0.12.26 v0.12.26

v0.12.26 ENHANCEMENTS: backend/remote: Can now accept -target options when creating a plan using remote operations, if supported by the target server. (Server-side support for this in Terraform Cloud and Terraform Enterprise will follow in forthcoming releases of each.) (<a class=”issue-link js-issue-link” data-error-text=”Failed to load title” data-id=”611026832” data-permission-text=”Title is private” data-url=”https://github.com/hashicorp/terraform/issues/24834” data-hovercard-type=”pull_request”…
This is the local-CLI portion of allowing -target to be used with remote operations. -target addresses given on local CLI are copied into the API request to create a run, which Terraform Cloud will…

Hi - I’m hoping the pros here can help me with understanding how to use dynamic block expression
within the resource aws_autoscaling_group
I’ve went over the TF doc many times and i still can’t seem to wrap my head around understanding how to use this.
Maybe an example would be helpful to shed some light on how to use it.
Here’s what i’m attempting to do with the resource aws_autoscaling_group.
I would like to be able to use mixed_instances_policy
but iterate through multiply instance_type (instance type varies) using the override block.
e.g
resource "aws_autoscaling_group" "example" {
availability_zones = ["us-east-1a"]
desired_capacity = 1
max_size = 1
min_size = 1
mixed_instances_policy {
launch_template {
launch_template_specification {
launch_template_id = "${aws_launch_template.example.id}"
}
override {
instance_type = "c4.large"
weighted_capacity = "3"
}
override {
instance_type = "c3.large"
weighted_capacity = "2"
}
}
}
}

Does this example help? https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/blob/master/main.tf#L125
Terraform module to easily provision CloudFront CDN backed by an S3 origin - cloudposse/terraform-aws-cloudfront-s3-cdn

If I understand you correctly, you’d want to do something like the following:
mixed_instances_policy {
launch_template {
launch_template_specification {
launch_template_id = "${aws_launch_template.example.id}"
}
dynamic "override" {
for_each = var.instance_type_overrides
iterator = i
content {
instance_type = i.instance_type
weighted_capacity = i.weighted_capacity
}
}
}
}

Not sure about using dynamic when nested… but does that look like what you’re looking for @Mr.Devops?

yes thx @Matt Gowie this looks perfect. Maybe i can also var.instance_type_overrides
to use type list(string) to inlcude all diff instance type?

Ah instance_type_overrides
in the above example would be of type:
list(object({
instance_type = string
weighted_capacity = string
}))



i totally forgot to update you (got too excited that it’s working)

Service impact to Terraform runs May 27, 22:24 UTC Investigating - We are currently investigating an issue affecting a subset of Terraform runs.
HashiCorp Services’s Status Page - Service impact to Terraform runs.

Service impact to Terraform runs May 27, 22:51 UTC Identified - The issue has been identified and a fix is being implemented.May 27, 22:51 UTC Update - We are continuing to investigate this issue.May 27, 22:24 UTC Investigating - We are currently investigating an issue affecting a subset of Terraform runs.

Service impact to Terraform runs May 27, 23:45 UTC Resolved - We’ve implemented a confirmed a fix to remedy this issue.May 27, 22:51 UTC Identified - The issue has been identified and a fix is being implemented.May 27, 22:51 UTC Update - We are continuing to investigate this issue.May 27, 22:24 UTC Investigating - We are currently investigating an issue affecting a subset of Terraform runs.

Manage AWS EKS clusters using Terraform and eksctl - mumoshu/terraform-provider-eksctl
2020-05-28

Has anyone done this Terraform certification, how much Terraform Cloud is in there ?

I have done certification, if you are already working with terraform I think it should be pretty straight forward to pass. They don’t ask much about cloud, just overview. Atleast I didn’t see deep questions around cloud

how long did it take you ?

I completed exam in 30 minutes with minimal to no preperation

ah perfect, thanks

You should know the difference between a terraform cloud workspace vs. a standard terraform workspace for certain

I spent less time preparing for the terraform exam than any exam I’ve ever taken. But I’d also been using tf for over a year prior

For all the normal stuff I’m confident, terraform cloud never interested me much so .. Thanks for the heads-up on workspaces So can you have different workspaces in your workspace ?

more that it’s one workspace per repo in tf cloud

so it does not work with multiple tfvars ?

as in, one for dev, one for staging etc.

I’d just setup tf cloud and do a few things in it to get the feel for it. Workspaces is a fundamentally changed term

Interesting to hear others say they didn’t need to study much. I took a practice exam and passed with a 90% but figured that must’ve been easy…. Maybe I’m wrong and should just take that cert.

it’s pretty cheap to take

Yeah — good point

95% correct, and it seems I do not understand terraform modules I hope I get to know what I did wrong..

@maarten Are those your final exam results? That’s awesome if so — quick turnaround.

sooo, tfmask only supports masking variables when they’re changed and not created?

Not by design - it might be a bug. Someone recently (and graciously) updated it to support 0.12 - maybe something got overlooked in the tests. Please have a look at the (very simple) code, if you have a chance. Maybe something stands out?

I am trying to use this git repo for converting yaml to HCL code. But make install
is not working..so I could not proceed further. Any idea how to make it work
https://github.com/jrhouston/tfk8s
A tool for converting Kubernetes YAML manifests to Terraform HCL - jrhouston/tfk8s

I read somewhere with the recent announcement of the new alpha kubernetes provider, that there was a tool for converting the yaml to HCL
A tool for converting Kubernetes YAML manifests to Terraform HCL - jrhouston/tfk8s

is this the tool?


lol

looks like it.

Yes same tool. But not working for me after doing make install. Not sure what am I missing

Below is how I tried it.
auto@auto:~/tfk8s$ ls
CODEOWNERS go.mod go.sum LICENSE Makefile README.md tfk8s.go tfk8s_test.go
auto@auto:~/tfk8s$ make install
go install -ldflags "-X main.toolVersion=0.1.3"
auto@auto:~/tfk8s$ tfk8s
tfk8s: command not found
auto@auto:~/tfk8s$

I need value a should be z if var.x is null and y if var.x has some value , does the following statement works? a = var.x != “” ? y || var.x == “” ? z

why don’t you just try it out, @Rajesh Babu Gangula? Set up a quick experiment with a local block and an output.

yep I am little skeptical about it … so just want to get quick inputs from the group before I fire it up ..

Hey guys I have a question, what is the difference between pinning your providers in the terraform block
terraform {
required_providers {
aws = "2.6.0"
}
}
vs doing it like in the provider:
provider "aws" {
version = "2.6.0"
}

The first approach is the recommended one for modules, the second one is what you would use in your own code that utilizes modules

Thanks for the quick response, seems logical indeed, thanks!

I would like to have AWS EKS setup using Terraform in such way that we have 2 VPCs. One VPC where I should deploy AWS control plane and other VPC I should have my worker nodes running. Do we terraform suite for this in cloudposse or any other repo?

Are you sure it will work? As far as I remember your node group and EKS cluster must be in the same VPC.

It worked for meMy setup is Cluster and Node1 running on VPC1 and Node2 running on VPC2. It worked well. Able to launch pods on Node2 as well. One prob yet to solve is not access or login to pod using kubectl exec command. Tried all security group changes between cluster n Node2. still no luck. That is not blocker for though cos pods in Node1 n Node2 able to communicate. Cluster is able to launch n configure things on Node2. I’m good

I guess this can work using VPC Peering or something like this. What is the point? Can’t you just use multiple subnets and network ACLs?

Yep using VPC peering. Our customer has something like this. So we need verify our pods able to work seamlessly on this type of setup.
2020-05-29

Could someone explain me why outputs interpolation works differently if I reference locals in outputs.tf and for example in main.tf? If do something like
locals {
s3_arn = "aws_iam_policy.s3_${var.environment}[0].arn"
}
output "s3_arn" {
value = local.s3_arn
}

"aws_iam_policy.s3_${var.environment}[0].arn"
is like an evaluation of a variable, you can not interpolate the resource names to reference to. You can do that when you use for_each and then you can refer to it by the name of the key.

If i put locals in main it will output literal string, but if i put locals in outputs.tf it will interpolate correctly Update: Strange enough i cant reproduce it now…

hi, May I know if we have latest/working Terraform module available for enabling Azure Diagnosticas logging for all azure resources ?

# terraorm , can you help me provide module for enabling diagnostics logging for all azure resources ?
2020-05-30
2020-05-31

I’m using terragrunt . Let’s say I want to deploy a VPC and an EKS cluster. Is it possible to put the VPC terragrunt code in one module and reference the vpc_id from the eks module?

Learn about all the blocks and attributes supported in the terragrunt configuration file.


Hello, does anyone know of a terraform trick on how to turn a map like this
tags = {
"abc" = "123"
"bbb" = "aaa"
}
into
[{Key=abc,Value=123},{Key=bbb,Value=aaa}]

@Brij S Something like the following should do the trick:
[ for key, val in tags: { "Key" = key, "Value" = val } ]
