#terraform (2020-05)

no need to create those resources again

you can do Data lookups

base on tags

that sounds like it could be easier.

like that

I’m going to have to lookup a lot of tags..

thanks

you need to define enough to find it

not all

if it has a tag that is unique, then that is it

I will use terraformer

https://github.com/GoogleCloudPlatform/terraformer

we recently used to import some stuff

worked very well

we just changed the resources names a bit

Noice

I will take a look at this.

I will say, creating each resource bit by bit will be better, using modules reduces de time for implementation

but it is possible to do it with this

@jose.amengual thanks for u advise

np

+1 for Terraformer, we’re using for moving parts of the current infrastructure that was manually deployed by the previous lead to Terraform Modules.

what about cdk? seems neat, gives you full benefit of a real programming language

It’s a mountain of a task to climb :)

Data lookup where that comes from is this :

data "aws_security_groups" "atlantis" {
  filter {
    name   = "group-name"
    values = ["hds-${var.environment}-atlantis-service"]
  }
  provider = aws.primary
}

Weird looks like a timestamp to me, what happens if u apply it

Do not know why it is under =[]

it complain that is expecting sg- as a value

never seen such thing before

Taint the resource?

If you you can’t cure it, kill it lol

lol

it was created by another TF

is there, it works

I can add it manually in the UI

I used .id instead of .ids.

returns the terraform-2222

stupid

it should not return anything

is there, it works

you mean you could see the id in the AWS UI??

I mean it should give you some error since the data resource is aws_security_groups and now a not a resource “aws_security_group”

very subtle difference

aha! now i get it

you see how easy is to make a mistake ?

yeah that’s a better option than renaming on one hand. On the other hand it requires more steps. In the setup I have in mind (automated testing), it actual might be better .

nvm, got it working with terraform taint actually

personally, we version all our modules, and use dependabot to update the version refs with a pr…

interesting so dependabot will notice that the module UP has changed and somehow find all modules that depend on UP and submit a PR to them to update the git reference tag in the source which will trigger atlantis’ terraform plan comment ?

if i understand that correctly… how do you configure dependabot to do this ?

if it’s a public project, and tf 0.11, then the free dependabot service works great

if you need tf 0.12, we’ve built a fork and a github-action

could you link this ?

im surprised you folks havent’ blogged about this.

https://github.com/plus3it/terraform-aws-codecommit-pr-reminders/blob/master/.github/workflows/main.yml

(note that’s a read-only github token, please don’t flay me)

haha i was just wondering about that. might be good to comment that inline

here’s the repo for the gh-action, https://github.com/plus3it/dependabot-terraform-action/

and our fork of dependabot-core, which the gh-action uses, https://github.com/plus3it/dependabot-core

hopefully dependabot will eventually get around to merging tf012 support upstream… https://github.com/dependabot/dependabot-core/pull/1388

@Mr.Devops You can still use import to update your TFC state from your local environment AFAIK. Once you update it once from your local environment then it will be in your state and TFC should just know it’s there.

ah ok i will look into this thx @Matt Gowie

there are pros and cons with managed node groups

also, the workers module was created long, long before managed node groups were supported

I would start with managed node groups until you find a use-case that doesn’t work for you

for example, if you need to run docker-in-docker for Jenkins, then the managed node groups won’t work well for that and you should use the workers module.

Keep in mind, that a kubernetes cluster can have any number of node pools

and you can mix and match node pools from different modules concurrently in the same cluster

you can have a fargate node pool along side a managed node pool, along side a ASG workers node pool

thanks much appreciated

@Erik Osterman (Cloud Posse) Would you mind elaborating why dind won’t work well with managed node groups? I use dind and was thinking of moving my nodes to managed.

@Jeremy G (Cloud Posse)

@s_slack EKS disables the Docker bridge network by default, and Managed Node Groups to not allow you to turn it back on. This is a problem if your Dockerfile includes a RUN command that installs a package, for example, as it will not have internet connectivity. Another issue for us is that you cannot configure Managed Node Groups to launch nodes with a Kubernetes taint on startup, which we wanted to keep our spot instances from having non-whitelisted applications running on them.

Perfect. Much appreciated. @Jeremy G (Cloud Posse) @Erik Osterman (Cloud Posse)

@breanngielissen Can you tag via the AWS CLI? If you can then you can use the local-exec provisioner to tag it.

resource "null_resource" "tag_aws_ebs_volume" {
  provisioner "local-exec" {
    command = <<EOF
      $YOUR_AWS_CLI_CODE_TO_TAG
EOF
  }
}

@Matt Gowie That worked perfectly. Thank you!

@breanngielissen Glad to hear it!

their docker container description says to use the standalone one …

https://hub.docker.com/r/opengrok/docker

i was looking into ECS using its docker, but it might be better to use an EC2 instance with userdata to setup opengrok, and keep code on an EFS

or just use an EBS

so then the setup would be…

• EC2

• userdata to retrieve ssh key, configure opengrok, clone all github org’s repos

• EBS / EFS for storage of github org repos

• ALB with a target group

• route53 record

• acm cert curious to hear other peoples’ thoughts

ooo let me take a look at this

it looks like it creates an IAM role https://github.com/cloudposse/terraform-aws-emr-cluster/blob/6444ee4f2a48d3d41c2499914ab8b882e454b11c/main.tf#L280

you can use a data source to grab the IAM role that it creates and then do a aws_iam_role_policy_attachment to attach a new policy to it, outside of its module reference.

Yea, this is our typical pattern - provision the role, expose it, then let the caller attach policies

it would be nice if those modules that are already creating the iam role(s) would also output their respective arns. then you wouldn’t need to use a data source to get the arn

@RB ya I’d consider it a “bug” if they don’t

Using data provider shouldn’t be needed.

I think i find it

Have you seen our EKS cluster module? It has a good example to get you started

yes but it has lot of things in root folder there are some .tf files and also in example I could see ‘complete’ folder has lot of files..Not sure from which folder i need to trigger “terraform apply” and also which variable.tf file I need to feed my details

here better

this is for Security Hub PCI standard, which requires for S3 buckets to have cross-region replica and encryption

cross account or cross region ?

cross region

on one account

so then you do not need the account id in the name in that case ?

we did team-nameofthing-region

we added the regions to everything

we even duplicated similar IAM roles for that

because IAM is a global service

but resources are not

which makes things very difficult to troubleshoot in case of access issues and such

and you need to keep in mind that some resources like ALBs can’t have names longer than 32 characters

so is better to keep it short

the bucket is created by AWS automatically like that

and I haven’t found a place where could I specify the bucket for Beanstalk myself

mmm I do not know much about EB

currently, the module has to support it, by accepting a variable that is used in the count/for_each expression on every resource in the module

for example: https://github.com/terraform-aws-modules/terraform-aws-vpc#conditional-creation

follow the create_vpc value through the module .tf files…

thanks. i’ll take a look.

terraform core is working on module-level count/for_each support. it is in-scope for tf 0.13… https://github.com/hashicorp/terraform/issues/17519#issuecomment-605003408

Is that error from running kubectl? What does your kubeconfig look like? It’s probably in ~/.kube/config. (Don’t paste the whole config in here; we don’t need to see your cluster address or certificate) If the access keys on your local system already have permissions to connect to EKS, make sure your kubeconfig doesn’t have a redundant role assumption in the args:

args:
  ...
  - "--role"
  - "<role-arn>"

Actually yes I have added my role again using command.same role which I am using to connect to EKS. This I have done using command aws eks --region us-east-1 update-kubeconfig --name test-xxx --role-arn arn:awsiam:xxxx:user/xxx

this is because when I checked in kubeconfig file it had new iam role created by Terraform but not mine which i connected in my local machine..only due to this i assumed I couldnt connect so I ve added it. So you mean if we dont add it should be able to connect?

It works when if I dont add the redundant roles. Thanks @David Scott for your help. I really saved lot of time

Glad I could help!

I heard of that solution from a colleague who worked on a project with a customer where they did that. I didn’t go down that route, because the things that were shared were things that didn’t change much, like project ids (this is for GCP). I simply hardcoded the outputs from various runs into the vars as I ran the environment from start to finish.

But I can see the central terraform module as either a God Class or as a glorified parameter store.

if they are still in some modules, yes, need to be removed. We tried to clean it up in all modules converted to TF 0.12

if you see providers in modules, let us know

I think this is the only one that I know of. I hit issues with it with the codepipeline module on a project. And I know @jose.amengual had troubles with it a week or two back.

https://github.com/cloudposse/terraform-github-repository-webhooks/blob/master/main.tf#L1

yea, that one will trigger changes across many modules

yes that is the one, I had that problem

@Andriy Knysh (Cloud Posse) Yeah… I could see that being a PITA. Ya’ll doing a good job with versioning won’t make it too bad I would hope?

yes versioning will help not to break everything (if people don’t pin to master )

I bet there is a few millions doing that lol

Ah yes, hopefully people listen to you folks and don’t pin master. But if they do then they’re outta luck.

¯_(ツ)_/¯

I think the examples in the README should be pinned to the latest version

not to master

yes they should (the latest version always changes though)

Sounds like a good automation task for release:

1. Build README with new tag as input, README example is updated to use new tag
2. Tag repo
3. Push repo

but codefresh will have to inject that, right ?

Question for VSCode folks — Is it that much better than Atom?

I’ve switched text editors a few times over the years from emacs => Sublime => Atom. Now I’m finally considering another switch to VSCode cause people love it so much, but I’m not sure if I get the hype.

its pretty good. I made the switch and love it. I’ve even slowly been forgetting about pycharm when working on Python and just defaulting to vscode

id recomend just installing it and checking it out. I can give you a list of good starting extensions to grab if you want too

and my settings.json file with a bunch of customizatiohn, just so you can see how it all works

Those are some nice offers — makes this more attractive… Yeah, I’ll happily take you up on that.

sure give me about 20 mins, on a call until about 5pm cst

Thanks man!

i find vscode significantly faster than atom. or did when i switched away from atom several years ago…

@Matt Gowie sorry about that, completely forgot about you. I forgot I went through and commented my settings file as well the other day, so that may help you understand some of it too. https://gist.github.com/afmsavage/1bf0241472f74fa21112d4d3698bcb80

Awesome — Thanks @Tony

nice! thanks for sharing

Thank you for all your work! Been using your language server for quite a while. Brilliant work!

Thanks @Matt Gowie and @Andriy Knysh (Cloud Posse) - definitely appreciate the 0.12 upgrade!

I have it all within terraform but not as a single run. Cant figure out how best to reinstate the same number of nodes so i swap out to an ondemand scaling group first, drain to it then swap back in the spot asgs

ugh. not a fan of that.

Yeah :(

I’m a big fan of those modules. None of them have a dependancy on each other though. But theres nothing to stop that

Thanks for the pointer. I hadn’t considered looking at some of the other modules in that repo. The atlantis module is basically the model I want. A “top level” module that uses other modules that are outside of that module, but can also use optional sub modules

Np. Glad you found one that is what you’re looking for!

we have a few very specific modules that deploy lambdas for very specific tasks https://github.com/cloudposse?q=lambda-&type=&language=

maybe you can extract what you need from one of them

Yeah, I get it…that’s what I’m doing…I was just thinking that it would be nice to have one module for a generic Lambda; I know it’s (almost) straightforward, but there are a few corner cases that usually modules takes care about it

your contribution to the open-source community would be very welcome @x80486

@randomy has several awesome modules for managing lambdas

(we have no plans right now to publish a module for generic lambda workflows)

I’ve heard good things about https://github.com/claranet/terraform-aws-lambda

but haven’t used it

Thanks. https://github.com/raymondbutcher/terraform-aws-lambda-builder is better than the Claranet one IMO (I made both)

ya, i use that currently in atlantis

What’re your thoughts? Has it caught any serious gotchas for you?

it’s nice and i believe it has some overlap with checkov and tflint

but it does find certain things the others dont.

we really need a nice comparison betw linters and their rules

we always setup the provider credentials in the root module, using multiple providers with aliases if there are multiple credentials (e.g. different roles). we then pass the provider alias to the module that needs it

https://www.terraform.io/docs/configuration/modules.html#passing-providers-explicitly

We compose our modules as well and run tests against them for that larger piece of infrastructure they build. I’m just wondering how I could continue testing the child module w/o having a way to setup some necessary config. We use kitchen-terraform currently. Maybe it entails working w/ that to test the child module w/o baking in a provider block which causes headaches down the road.

ya, this issue is coming up more an more often. we’ve (@Andriy Knysh (Cloud Posse)) fought a few issues in the past week

yeah, i think you’d have to setup the credential through kitchen-terraform

OK - so I’m not crazy then?

At least we’re in agreement - cause after re-reading the docs - I see the issue w/ nested providers - but now need to figure out how we can still test independently our child modules w/o those provider blocks.

https://sweetops.slack.com/archives/CB6GHNLG0/p1588878246122400

see the thread below that message

we also have multi-provider modules, for cross-account actions (e.g. vpc peering). in that case you must have a provider block in the module. but we only define the alias. the caller then passes the aliased providers they define to each aliased provider in the module

https://github.com/plus3it/terraform-aws-tardigrade-pcx/blob/master/main.tf#L1-L6

and the setup in the test config (though in this case it is vpc peering in the same account)… https://github.com/plus3it/terraform-aws-tardigrade-pcx/blob/master/tests/create_pcx/main.tf#L38-L41

Cool. I’ll take a look at all these. Hopefully there’s a way forward that reduces the issues we see when our dev teams remove child modules w/ their own provider modules but still allows us to test the child modules and the composed repos independently

Thanks @loren @Erik Osterman (Cloud Posse) !!

You don’t specify providers in child modules. This is logically correct since your submodules don’t need to know or deal with how they are being provisioned.

You specify providers in root modules

Or in the test fixtures

Those could have different providers with different roles or credentials

Or for different regions or accounts

Right. It’s mainly now how to remove the provider blocks (which I see from the other thread that Erik posted - you guys have been removing from all your modules) and continue to have testing ability for these child modules as well.

I’m only going to add that I personally avoid mapping providers into modules as much as humanly possible

Every time you have to do this I’d ask yourself if you are doing the right thing or not.

sometimes is simply unavoidable

sorry, I’m late to the party and likely missed the point but all this self-isolation made me feeling like proclaiming some stuff out loud….

we hear you and see you @Zachary Loeber;)

right, in almost all cases you don’t need to specify providers in child moduler, nor map providers from the top-level modules

just define them in root modules or test fixtures, they are inherited automatically

i dont find workspaces to be that useful. i’d much rather a module references with an environment argument and an environment directory with those module references

We’ve (cloudposse) switch over to using workspaces. previously we would provision a lot of s3 state backends, but that process is tedious and difficult to automate using pure terraform

I’m not too familiar with workspaces but how does it work with atlantis ? you guys have a repo per environment still or due to using workspaces you are now have one repo and multiple workspaces ? @Erik Osterman (Cloud Posse)

we now use a flat project structure. we define a root module exactly once. we don’t import it anywhere. we define an a terraform varfile for each environment. we don’t use the hierarchical folder structure (e.g. aws/us-east-2/prod/eks/ ) and instead just have projects/eks) with files like projects/eks/conf/prod.tfvars )

this enforces that we have the identical architecture in every environment and do not need to copy and paste changes across folders. what I don’t like about the multiple folders is that architectures drift between the folders without human oversight.

you guys have a repo per environment still or due to using workspaces you are now have one repo and multiple workspaces ?

@jose.amengual: yes, we used to, but now have abandoned that approach.

cool so we are using the same aproach

we adopted that approach before gitops was a thing. it proved to not be very gitops friendly and changes were left behind leading to drift. we moved to the flat folder architecture this year in all new engagements.

almost, we do not have subfolders for the conf/ we have it all int the root

for us, it has worked well, the main reason we ended up with one repo was due to multiregion deployments, if we decided to split by env we could endup with an exponential number of repos

Terraform state supposed to help you with infrastructure drift but by having so many repos or folders then who solves the problem of configuration drift when using multi-folder or env-repos?

pretty much impossible

I’d like what you talk about @Erik Osterman (Cloud Posse) to be a decent norm but the diminishing returns come fast when you try to do all this rigorously with only one or two envs and requirements popping up all the time where they end up having to differ in the end for various reasons.

For the things which I do repeat in a very identical fashion I could probably use workspaces.

I have (pretty much) a structure such as:

• infra/logical_grouping/myprefix-env-stack1
• infra/logical_grouping/myprefix-env-stack2
• infra/logical_grouping/myprefix-env-stack2-substack

where -extension in many cases either utilize remote data from the “parent” stacks, sometimes several of them. Sometimes, in the case where it makes more sense to not tie things so closely, the information sharing is done via regular data sources instead.

I’ve made the mistake of putting too many things in the same “stack” way too many times. I’ve learned to separate early now.

(While also learning to become really proficient in terraform state mv )

@mfridh just to be clear, what I’m describing still breaks terraform state apart so it’s not a terralith. we also have taken this a step further, so we have a single YAML config that describes the entire environment and all dependencies. so as @joshmyers was pointing out in the #terragrunt channel, we don’t have this problem anymore with hundreds of config fragments all over the place.

the key here is the terraform state is stored in workspaces, the desired state is stored in the YAML configuration, and the declarative configuration of infrastructure is in pure terraform HCL2 + modules.

Here’s a hint as to what that looks like: https://archive.sweetops.com/aws/2020/04/#ebb3d28d-3753-417c-a88a-7775f8df633c

but we’ve progressed beyond this point.

All config fragments are in a single repo, but still a lot of them…multi region definitely doesn’t help that

@Erik Osterman (Cloud Posse) not using the https://github.com/cloudposse/testing.cloudposse.co approach still?

Does the use of workspaces mean your states are all in the same s3 bucket? Last time I tried it, I found this to be problematic. (Can’t remember the exact issue/limitation)

@joshmyers using geodesic (and still using that repo), but the 1-repo-per-account proved not very friendly for pipeline style automation

@randomy yes, all in one bucket, but with IAM policies restricting access based on roles

note, workspaces get their own path within the bucket so it’s easy to restrict.

are they any different to the usual environment variable? Could do all states in same s3 bucket without workspaces, right?

@joshmyers, yes, you’re correct. Could implement “workspace” like functionality without actually using the workspaces feature just by manipulating the backend settings for S3 during initialization.

I don’t know what the advantage would be to doing so

That’s what my Makefile does. No advantage over workspaces I guess but to be fair, workspaces didn’t exist when I started to do this ;).

I like that yaml defined ordering you linked to Erik. Looks legit. Although I’m not sure how you actually implement it… variant reads it and runs various targets in order?

That’s one thing to consider…. I’m also considering Atlantis. But not sure if that would actually be a good idea for practical reasons or just “for show”.

I’m using terraform aws_ec2_client_vpn_endpoint to create CVPN and aws_cloudformation_stack to add routes

https://github.com/frimik/tf_module_plugin_example/blob/master/modules/tf_example_module/variables.tf.json

Generate a variables file. Json works fine.

That’s from days of yonder before Terraform had proper provider plugin support.

Another alternative in your case, if you don’t want to pass through files on disk is to run that bash command as an external data source script: https://www.terraform.io/docs/providers/external/data_source.html

Have you considered using yamldecode(file(...)) with a local?

https://www.terraform.io/docs/configuration/functions/yamldecode.html

We use this pattern and it works well.

I’ll look into yamldecode. I’m also wondering if I could have just used a data source for the initial load of values.

if you can output to json instead, you can write <foo>.auto.tfvars.json to your tf directory, and terraform will read it automatically and assign the tfvars

https://www.terraform.io/docs/configuration/variables.html#variable-definitions-tfvars-files

Terraform also automatically loads a number of variable definitions files if they are present:

* Files named exactly terraform.tfvars or terraform.tfvars.json.

• Any files with names ending in .auto.tfvars or .auto.tfvars.json.

i’ve also used terragrunt to do this kind of pre-processing

or maybe try pretf, https://github.com/raymondbutcher/pretf

here’s an example of using terragrunt and hcl string templating to massage some input into a auto.tfvars file… i think .auto.tfvars.json is better, but you can pick your own poison… https://github.com/gruntwork-io/terragrunt/issues/1121#issuecomment-610529140

Thanks @loren Those are awesome options.

I confirmed I could just use a data source for my use case.

Ok… this is what I’m trying to do

Default settings above… new item below

repos = {
    terraform-aws-foobar = {
      description  = ""
      repo_creator = "willy.wonka"
      settings     = merge(local.default_settings, {})
    }
}

But when I some map key values that I want to override, they don’t seem to get picked up, despite the behavior of merge being the last one should replace for simple (not deep) merges.

repos = {
    terraform-aws-foobar = {
      description  = ""
      repo_creator = "willy.wonka"
        settings = merge(local.default_settings, {
                topics = ["terraform", "productivity", "github-templates"]
            },
            {
                is_template = true
            }
        )
    }
}

Any ideas before I go to devops.stackexchange.com or terraform community?

what is posted should work, i think. do you have an error? or an output displaying exactly what is not working?

It just doesn’t seem to pick up the “override” values from the is_template = true when the default at the top was false. No changes detected. Applying my new topics also doesn’t get picked up. This doesn’t align with my understanding of merge in hcl2

i do exactly this quite a bit, and it does indeed work. hence, we need to see what you are seeing

try creating a reproduction case with your locals and just an output so we can see what the merge produces

Well a good example would be running terraform console on this

I just created a new repo. I added topics to it, like you see above. The new topics I’m overriding don’t even show

and when i run doesn’t show topics at all in the properties list

Yeah so I just ran console against on specific on that has the topics override it shows empty on topics, not taking it at all

this item

 devops-stream-analytics = {
      description  = "foobar"
      repo_creator = "sheldon.hull"
      settings = merge(local.default_settings, {
        topics = [
          "analytics",
          "telemetry",
          "azure"
        ]
      })
    }

Here

> github_repository.repo["devops-stream-analytics"]
{
  "allow_merge_commit" = false
  "allow_rebase_merge" = false
  "allow_squash_merge" = true
  "archived" = false
  "auto_init" = true
  "default_branch" = "master"
  "delete_branch_on_merge" = true
  "description" = "foobar"  
  "etag" = "foobar"
  "full_name" = "foobar"
  "git_clone_url" = "foobar"
  "gitignore_template" = "Terraform"
  "has_downloads" = false
  "has_issues" = false
  "has_projects" = false
  "has_wiki" = false
  "homepage_url" = ""
    "html_url" = "<https://github.com/foobar>"
    "http_clone_url" = "<https://github.com/foobar>"
  "id" = "devops-stream-analytics"
  "is_template" = false
  "name" = "devops-stream-analytics"
  "node_id" = "MDEwOlJlcG9zaXRvcnkyNTQ3NTEwMzc="
  "private" = true
    "ssh_clone_url" = "[email protected]:foobar.git"
    "svn_url" = "<https://github.com/foobar>"
  "template" = []
  "topics" = [] <------ this should be overriden by my logic?
}

see the last item. It’s empty. Not sure why my merge syntax would fail as you see above.

looks fine to me?

$ cat main.tf
locals {
  default_settings = {
    topics                          = ["infrastructure-as-code", "terraform"]
    gitignore_template              = "Terraform"
    has_issues                      = false
    has_projects                    = false
    auto_init                       = true
    default_branch                  = "master"
    allow_merge_commit              = false
    allow_squash_merge              = true
    allow_rebase_merge              = false
    archived                        = false
    template                        = false
    enforce_admins                  = false
    dismiss_stale_reviews           = true
    require_code_owner_reviews      = true
    required_approving_review_count = 1
    is_template                     = false
    delete_branch_on_merge          = true
  }
}

locals {
  repos = {
    devops-stream-analytics = {
      description  = "foobar"
      repo_creator = "sheldon.hull"
      settings = merge(local.default_settings, {
        topics = [
          "analytics",
          "telemetry",
          "azure"
        ]
      })
    }
  }
}

output "repos" {
  value = local.repos
}

hmm. What terraform version are you running

Personally I prefer to have unique IAM resources for each module as it’s easily repeatable and self-contained solution. IMO the label module helps to avoid the naming collision

encourage them to use name_prefix instead of name ?

ask them what they do when they deploy the same tf module more than once?

This is not an issue with modules having hardcodes names. The “submodules” are very configurable but the root module in this case defines a very specific configuration and should only be deployed once per account

My worry is that if IAM resources are created in dozens of states it could become very hard to reason about the state of IAM in an account

I hope that makes sense

@vFondevilla That is an interesting solution. Although for example if that module defines a simple barebones ecs_task_execution_role and you deploy 20 copies of that module

well alright, but that was the scenario presented

You end up with 20 ecs_task_execution_role_some_id with identical policies attached? That always felt a bit wrong to me. If they have different policies great but just having many copies of identical IAM resources seems redundant

iam policies have no costs. multiple instances of a policy with the same permissions is worth it to us, for the flexibility of a self-contained module. i suppose we tend to reason about the service we are creating. not the state of an aws service as a whole

That is interesting. Its great to talk about this stuff. Historically I have had a very “service centric” split of state

Yes, that’s the situation in my accounts. Each ECS service has its own IAM execution role managed by the terraform module. This enables us to repeatedly deploy the module in one of the multiple aws accounts we’re using

While it can be easier to reason about / delegate responsibility it does also lead to a lot of remote state hell and dependancy hell

e.g I need to update the vpc state to add a security group and then I need to use remote state or data sources in my ecs state to now refer to that security group

Maybe I should relax my service per state thinking

we do separate IAM for humans into its own module and control it centrally. but IAM for services goes into the service module

and move towards an “application” or “logical grouping of resources” per state

Yes for IAM for humans I tend to just create a module that uses lots of the modules in https://github.com/terraform-aws-modules/terraform-aws-iam

And that is deployed in a single account

Thanks for rubber ducking this. Do you ever end up with the opposite problem to me? i.e because IAM roles are defined alongside the service, can it be difficult to for example allow access to another principal? Because its created in another module and you can have chicken egg problems?

Or would a scenario like that be moved out to another state

I’m probably overthinking this a bit. Its a greenfields project so trying not to make any decisions that will shoot me in the foot in 12 months

I’m trying to avoid state dependencies just to simplify my life and reduce the steep learning curve for the rest of the team, so sometimes instead of referencing the remote state managed via another module, I’m using data resources as variables for the modules. In our case is working awesome with a team which before me doesn’t knew anything about terraform.

the thing that shoots us in the foot is when we do use iam resources (or most anything else, really) managed in a different state. when we keep all the resources contained in the services modules, it stays clean

Do you mean you use data sources for most module inputs @vFondevilla something like this?

data "aws_vpc" "default" {
  default = true
}

data "aws_subnet_ids" "all" {
  vpc_id = data.aws_vpc.default.id
}

module "db" {
  vpc_id = data.aws_vpc.default.id
  subnets = data.aws_subnet_ids.all.ids
  #.......
}

yep

Sounds good @loren I will definitely try out this approach. I think it will save me some pain going forward

Much easier than explain someone with barely 0 experience working with Terraform modules how to use remote state and the possible ramifications. It’s pretty manual for deploying new modules (as you have to change the data and/or variables of the module), but in our case is working well.

Yeah I find data sources a cleaner solution than remote state where possible. Nothing worse than the “output plumbing” required when you need a new output from a nested module for example

this are same with that Error: Invalid for_each argument

on resource.tf line 63, in resource “aws_efs_mount_target” “efs”: 63: for_each = toset(compact(module.vpc-1.private_subnets))

The “for_each” value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created. To work around this, use the -target argument to first apply only the resources that the for_each depends on.

Do you already created routes in your vpc? Count depend on output of:

• data.aws_route_tables.requestor.0.ids
• data.aws_vpc.acceptor.0.cidr_block_associations

I am using aws vpc module of cloudpoose.

try to use https://github.com/cloudposse/terraform-aws-vpc-peering/blob/master/examples/complete/main.tf

they create

• vpc

• subnets

• peering

probably you are missing subnets creation

RTFM

forecast metric and a monitor on that metric ?

usually that is how it works

yeah, I think I got it

an example would be nice though. . . for that provider

Push plan to Jira?

Hahah that strikes a cord — I do PM / Architecture consulting for one client and I use Jira too much as it is. I think that’d be my nightmare.

I’m going to guess after my quick google search that there is no such tool. Which is interesting to me… Too many ideas, not enough time.

You get that with terraform cloud

Use your ci platform to report the plan as a test result on the pr in Github?

the ability to comment on the plan

Huh I thought and looked at my TF cloud project for one client. It doesn’t go into the discussion level that I wanted, but maybe that solves enough of the problem. I just haven’t used it enough.

I was hoping to see something that has the ability to comment on a line by line change in a plan so I can explain / discuss the specific results of resource change to non-ops team members before we move it forward.

runatlantis.io ?

That’s an interesting idea. Atlantis had to work around Terraform plans hitting the max character length in GitHub comments by splitting up the plan into multiple comments. Sticking the plan output into a file in the pull request would get around the comment length limitation, and allow for inline commenting.

I have a to-do item to implement terraform in GitHub Actions in my org. When I get around to that task I’ll see if I can add the plan to the pull request as a file when the PR is opened.

Allowing the CI/CD process to change the contents of a commit or add a commit poses its own challenges, but we already have to deal with it using @lerna/version

I guess you could have a dedicated repo for plan artifacts and commit to that via automation and open pull request and all. That sounds like what you need, but dang the house of cards starts getting higher. How about just the review in page and then fail fast and fix if an issue :-)

you could fork atlantis and add what you need

time to learn golang

slick

What’s the problem? Terraform’s AWS R53 resources work as advertised.

resource "aws_route53_zone" "main" { count = length(var.domains) > 0 ? length(var.domains) : 0 name = element(var.domains, count.index) }

for CNAME record, how can i create for specific domain?

resource “aws_route53_record” “www” { count = length(aws_route53_zone.main) zone_id = element(aws_route53_zone.main.*.zone_id, count.index) name = “www” type = “CNAME” ttl = “300” records = [“test.example.com”] }

don’t know if I understand specifically but: count and index are blunt instruments and often will fail you when you try to implement more complex use cases

I’ve had success with maps and for_each statements

my use cases is, i’m working on creating a poc with route53 using terraform.

I had to create multiple domains in route53 and take care of operation tasks like adding new/updating/deleting records

i haven’t tried with map and for_each. i will give a try

yeah, for_each is better. count would be a disaster there, it would likely want to delete and recreate your zones when you change var.domains

Thank you, i will give a try and come back

go into dynamodb and clear the lock table…

Okay, thank you. I haven’t worked with dynamo yet and was worried I may break something else by clearing it. I’m especially cautious because I’ve already dug myself in to a deep enough hole.

nah, much more likely to break something on the s3 side. the dynamodb table is very lightweight

if you haven’t already, highly recommend turning on versioning for any s3 bucket you use for tfstate

yeah, as soon as I realized what was going on, I wished that I had.

you might have a copy of the tfstate locally, from your last apply

check the .terraform directory wherever you ran terraform from

I wish. It was a new terraform script that I had only applied in Jenkins.

ah, well, jenkins. there’s the problem

I think I actually have my script working fine at this point, but I ran taint locally (which was on .25) and it bumped the version of the state file. I can only automatically install up to .24 on Jenkins at the moment. :x

Oof. Yeah, might also want to add a terraform version constraint…

we do this in all our root modules:

terraform {
  required_version = "0.12.24"
}

pin the version, then update it intentionally as a routine task

I don’t know if you could do this but maybe you could fork runatlantis.io and instead of receiving a web hook payload change it to receive a regular Rest request

That’s an idea. I remember there is a company that created a service catalog for Terraform templates. I couldn’t find the company, but I did run into this when I did a Google search for it. https://www.hashicorp.com/products/terraform/self-service-infrastructure/

It doesn’t go into details on how the self-service part works.

Self-service = pay for nomad and terraform enterprise

If Terraform Enterprise can provide self-service, that would be a plus if you can afford it. For larger companies, it would pay for itself in not having engineers reinvent the wheel.

my old team did some work on addressing scaling issues on one of the largest terraform enterprise installations deployed - i’d say if you’re getting to that size you want to divide the scope of what an individual deployment of TFE is addressing - maybe broken down by business units or some other logical boundary

there is an upper limit and it gets ugly when things go wrong at that limit

so i’m a bit confused here… where is the concern about scaling?
when working with thousands of internal customer or
when you’re dealing with 100k servers, so there’s some upper limit. And I think they are different problems to optimize for.

If you have thousands of internal customers (e.g. developers), I still would imagine it scales well, if:

• massively bifurcated terraform state across projects and AWS accounts

• teams have their own projects, aws accounts

• teams are < 20 developers on average

then i really don’t see how terraform would have technical scaling challenges e.g. by hitting AWS rate limits.

terraform itself yes - terraform enterprise (as in the server) can hit scaling issues at high usage

aha, gotcha - i could see that with something like terraform enterprise.

and (at least when we were working on it a while back - older versions) did not scale out well

i imagine that they’ve implemented a lot of lessons learned in deploying the new terraform cloud platform

but with standard CI/CD systems that can fan out to workers, I see it less of a problem.

yeh - as long as you can scale your worker fleet then it should be easier

the practical problem I see is enforcing policies at that scale.

i’m in a bit of a quandary there between opa and sentinel - i really really like opa

Ya, same. OPA is where I’d want to put my investment.

but then going against the TFE ecosystem

i mean if you’re not going to use TFE as your runner it makes sense

the other issue i see is getting support from hashicorp

I was also thinking of scaling in terms of management and implementation. For example if a core team has say 10 developers, writing code for thousands of internal teams, each with their own accounts/projects doesn’t scale well. So the urge is to create some system where the end user has a gui they can go to request their infrastructure while using Terraform. What I don’t get is how to use Terraform when you scale beyond what manually writing code in a GitOps fashion does.

generally i’d say you want to scale out to teams via code rather than a gui

but i realise that that’s not always a possible reality for some enterprises

somewhat embarrassed to admit i built a demo PoC for basically this before it existed: https://www.terraform.io/docs/cloud/integrations/service-now/index.html

I was reading about that. It looks pretty cool. I think what the company I’m working with now wants to do is integrate Terraform with their own infrastructure request toolchain.

that’s something that i’ve seen asked for a lot - i’ve worked on projects to build it a couple of times. i understand why it’s a reality - i just don’t think i’d ever want to have to request infrastructure in that way

as an engineer on the other side of the gui i’d probably hate it

but that’s possibly because i’ve always had the privilege of not being blocked by systems of approval like that

enterprise realities kind of suck

which is why i don’t work in enterprise

i know realistically that a system to request from a service catalogue like that is 1000x better than what a lot of enterprises are dealing with today

@Erik Osterman (Cloud Posse) can you expand on this : https://sweetops.slack.com/archives/CB6GHNLG0/p1589502664287900?thread_ts=1589496149.282000&cid=CB6GHNLG0

I have read about OPA, and every time I go the the website it looks to good to be true

I keep thinking in how I’m to somehow get charged, of screwed for lack of support or something

I ended up outputing my kubeconfig(s) to local_file and storing them back in source control. I reference them in null_resource with --kubeconfig=${local_file.kubeconfig.filename}. It’s not the most elegant solution, but neither is using null_resource for kubectl apply.

Thanks for the reply. Can this be achieved in single terraform run like generating kubeconfig n copying in config path? Also can we use --kubeconfig=${local_file.kubeconfig.filename} like this inside null resources?

Please ignore my previous msg.. I understood your answer thanks for the help. Will try it n get back if any issues

Keeping my eyes on this. You can see some stuff on this in some repos but I haven’t yet had capacity to dive into tests yet.

Really appreciate if any help on this?

I’m not sure you would use the pod IP for that.

What do you mean by:
I need to use that POD IP address to bring another POD

In Kubernetes pods get assigned ephemeral IP addresses that change when the pod is moved, restarted, scaled. Using the IP of the pod to communicate with would not be a good experience. instead, Kubernetes depends on DNS names and Kubernetes Service resources.

A good place to start to understand this is: http://www.youtube.com/watch?v=0Omvgd7Hg1I&t=10m0s

The whole talk is great but the relevant part is at 10 minutes in.

Actually I need that POD to get some information from that POD..anyways let me check on DNS config..that would be appropriate as you say..

I will go over this video..

one basic question..i never tried with dns config..is it default dns config we can give are we need to have dns server configured in K8s cluster then set a dns for a particular pod..

BTW Thank you for the response..really appreciate

A good place to start to understand DNS and services in Kubernetes is: https://kubernetes.io/docs/concepts/services-networking/dns-pod-service

If your deployments (and the pods they create) are in the same namespace, once you’ve created the service, you’ll be able to access the pods with the host part of the DNS name.

hmm I get it..it really makes sense to go with service

Say I have a Ghost blog in namespace my-blog and a Percona DB server in my-blog - I’d create a Service of type ClusterIP for the database server named db and my Ghost blog could access it with the host name db

i was trying out raw thing out here..this looks complete solution..

If I were to deploy the DB server into another namespace I could reference it with it’s FQDN: db.other-namespace.svc.cluster.local

Another fantastic talk to watch to understand how all these Kubernetes resources interact is this video: https://www.youtube.com/watch?v=90kZRyPcRZw

Ok that sounds clear.. for now I can go with same namespace that should be fine for my case..so that it would be simple for me to refer that..

Thanks Tim..guess these details would really help me kick start on these areas..which is new to me as a k8s beginner..

mm we do have lambdas that interact with the db but not to create them

usually we create from a snapshot

can you just create a snapshot with the schema and whatever data you need and put it on a s3 bucket ?

or share the snapshot

Ohh that is an interesting idea. The s3 import option.

@Jeremy G (Cloud Posse)

Only downside is that it needs to be in the percona xtradb format, but haven’t looked at it yet.

http://docs.aws.amazon.com/AmazonRDS/latest/APIReference/API_RestoreDBInstanceFromS3.html

is just the percona tools, you can use the tool to dump in that format and you are set

it is fairly simple

We need to provision users, too.

Limitations and Recommendations for Importing Backup Files from Amazon S3 to Amazon RDS
The following are some limitations and recommendations for importing backup files from Amazon S3:
You can only import your data to a new DB instance, not an existing DB instance.  (emphasis added) That is a deal-killer

can’t you add the users before you do the dump?

we do that too, we never had to create the users

plus you can use IAM auth and have “shared” users OR use SM autorotation secrets for apps ( requires code changes in app)

IAM auth is not recommended for apps with more than 200 connections per second

you can use the lambda examples for autorotate secrets to add users too since you basically need to setup the lambda access to the DB

https://github.com/aws-samples/aws-secrets-manager-rotation-lambdas/blob/master/SecretsManagerRDSMySQLRotationSingleUser/lambda_function.py

https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-rds.html

I’m working on making all this to work on TF

based from this module : https://github.com/giuseppeborgese/terraform-aws-secret-manager-with-rotation/

Once the private database is provisioned how do you apply schema changes, serverless?

https://blog.codecentric.de/en/2019/10/rds-database-migration-with-lambda/ Doesn’t sound too bad…

for the original use case I think once is provisioned is done, some one else takes over

we are going to start using liquibase

You can only import your data to a new DB instance, not an existing DB instance. @Jeremy G (Cloud Posse) this is a cold-start problem for us and it’s a new instance. As a compromise we can provision all 3 databases this way at start. Agree, it would be nicer to decouple each database creation from the RDS instance creation, but it’s less a deal-breaker than it is an inconvenience.

A lambda like that would be very cool, especially when combined with a new postgres / mysql provider

@Erik Osterman (Cloud Posse)
User accounts are not imported automatically. Save your user accounts from your source database and add them to your new DB instance later. This is also a deal killer.

Remember, all we need to do is run 3 MySQL commands per database: CREATE DATABASE, CREATE USER, and GRANT. This does not need to be overly complicated.

What we need are

• a secure way to get master credentials to the script runner

• a way to get other dynamic parameters to the script runner

• a way for the script runner to have network connectivity to the database cluster

• a way for the script runner to communicate status

• a way to integrate and synchronize running the script with the rest of our Terraform operations

Yea, given these requirements, the lambda will be the most generalizable solution that I can think about. Or a lambda that listens to an SNS topic to run permitted psql commands.

I do not agree with that paragraph about user accounts not imported , we use the snapshots and we never had to recreate the users

but that is from a dump that already had the users in it

Have you thought of launching one off task containers for these types DB administration use cases? Lambda always feels like overkill to me when what you really need is an ECS container with a bash script and a bunch of SQL.

ECS is far more convoluted to setup than lambda

in my opinion

you could run and EC2 instance with session manager and run stuff manually trough the web console which does not require VPC conectivity

actually you can use a SSM document to run a script that will do the mysql stuff for you in an instance controlled by SSM

@Erik Osterman (Cloud Posse) I recently developed such a module. It is a lambda function that can provision a database and optionally a user in mysql and postgres in RDS instances without public access in VPC. Not very polished, but does it job perfectly. I can open source it on github within the next 12 hours or so if that helps.

https://github.com/aleks-fofanov/terraform-aws-rds-lambda-db-provisioner

https://github.com/hashicorp/terraform/issues/24010

done everything in that but no solution

when you run direclty /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4 can you provide output?

if the same permission denied, then chmod +x /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4 and try again

same output

given 777 still same issue

use strace

can you help me regarding strace idk how to use it and what m looking for

use google

i mount another drive to this folder , it may be the case ?

strace terraform init

directly on /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4

strace /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/terraform-provider-local_v1.4.0_x4

i mount another drive to this folder , it may be the case possibly, try some other writable location

strace permission denied

on that folder

:joy: and maybe ls /home/kratos/Documents/Projects/terra/.terraform/plugins/linux_amd64/

fix your permissions on documents folder

i mount on Document folder

no bad words

use mount properly

THNX MAN

@Maxim Mironenko (Cloud Posse) @Andriy Knysh (Cloud Posse)

I was looking at the Terraform modules tutorial and I saw this :

source  = "terraform-aws-modules/ec2-instance/aws"
  version = "2.12.0"

and I was wondering about the benefit of using the registry over git

Nope, no such concept. The registery is just a proxy for GitHub

ahhhhhh I did not know what

It doesn’t work like other registries like Docker hub or package registries that you upload to

so how do they know what to pull ?

The registry requires a strict repository naming convention

Modules still need to be manually added

But from that point on all “updates” are in real-time

but there is a registration step somewhere I guess ?

I will read the registry docs

Using the terraform registry is not the same as git. It is a proxy, but it only gets releases with the right name format. It cannot get branches, tags, or other git references. Because of this, it is faster than using git. Git does a clone (which is larger) and if used mutiple time will do multipl clones. Where using registry will only download once per versions

Interesting thanks

Yes, good distinctions! Wasn’t thinking about that. @Steven

look up for_each

for_each is if you’re iterating over a map. If you just want to create n number of resources, you would still use count.

for_each also works with a unique list of things (cast to a set with toset()), as well as maps… I’d only recommend using count if it is literally just creating multiples of the same thing, i.e. it’s original, literal purpose. Otherwise when the the list/map changes, count will want to destroy/create way more than expected since the index changes for all items in the list

I have no idea what do you mean with this ?

what does it mean to be dependant in gitops ?

Yes, ci/cd can pass parameters to Terraform. This is not uncommon. Terraform state needs to be remote or you risk losing it. Terraform manages state. What you pass to it should be things that change the state. If that is not what you are trying to do, then job either needs to change to different state files or use workspaces.

Terraform doesn’t care about GitOps. You can use with or without

That’s useful information. I’m so used to thinking of GitOps as the goal that I’m trying to wrap my mind around other ways of interacting with Terraform besides through git commits.

gitops is a methodology not a tool

Terraform Cloud is fundamentally “GitOps” driven.

The config map has probably been created before (e.g. by using older version of the module). You need to manually delete it using kubectl. The module uses kubernetes provider to create the config map, and the provider just can’t update it, only create a new one

@Andriy Knysh (Cloud Posse) I’ve done a terraform run from scratch into a completely clean environment, still I consistently get this error. I’ve cleaned up the state files, .terraform folder .. pretty much everything. And still keep getting this error.

the config map is already provisioned (can’t say how). To fix this, there are only two ways: 1) destroy the EKS cluster and provision again; 2) from the command line, find the config map and delete it using kubectl

Andriy, I’m spinning an EKS cluster from scratch. It wasn’t pre-existing. Still I continue to hit this. There is no other piece of code that is creating aws_auth configmap.

If I set the following flag to false will it hurt? apply_config_map_aws_auth

looking at the code it seems that will stop the code from creating the aws_auth map altogether.

@Andriy Knysh (Cloud Posse), What do you think?

@Harshal Vaidya did you look at this example https://github.com/cloudposse/terraform-aws-eks-cluster/tree/master/examples/complete

when you create the cluster first time, there should be no config map

did you delete all the resources and start again?

That is correct .. I deleted the entire cluster and then started again ..

I’ll look at the example tomorrow

A good question for todays office hours but I’ll say that anything you can shift to pure terraform provider code will make your life much easier in the long run

you have basically 2 options.

One is to add any dependencies to the git repo itself. PIA if you need to copy that 100x over.

the other is to use the exec-provider to install what you need. we’ve done this in the past. @Andriy Knysh (Cloud Posse) probably has a snippet somewhere.

either way, it’s a slippery slope.

What terraform cloud needs (and the competitors like Spacelift and Scalr support) is a model where you bring your own docker image

@Jake Lundberg (HashiCorp) @sarkis

Terraform Enterprise does have this ability. I’ll check if we’ll enable this for TFC.

We may only allow that for paid accounts (in the future).

Right - that’s fair. But supporting it in some capacity I think is needed. Though support for thirdparty providers is improving (i think?) which mitigates it to some degree.

We had a recent use case of using a third-party provider terraform-helmfile-provider, which depends on helmfile binary, which depends on the helm binary and helm-diff , helm-secrets, etc plugins. Getting all that to run on terraform cloud was a major PIA.

:smiley: oh and kubectl

@Erik Osterman (Cloud Posse) can you go through the process to get this to work? I’d like to make sure we’re capturing this from the field.

@Jake Lundberg (HashiCorp) Here’s what we had… (not proud of this!)

provider "shell" {}

data "shell_script" "install_external_packages" {
  lifecycle_commands {
    read = <<EOF
        set -e
        export PATH=$PATH:${local.external_packages_install_path}:${local.external_packages_install_path}/bin
        mkdir -p ${local.external_packages_install_path}
        cd ${local.external_packages_install_path}
        echo Installing AWS CLI ...
        curl -LO <https://s3.amazonaws.com/aws-cli/awscli-bundle.zip>
        unzip ./awscli-bundle.zip
        ./awscli-bundle/install -i ${local.external_packages_install_path}
        aws --version
        echo Installed AWS CLI
        echo Installing kubectl ...
        curl -LO <https://storage.googleapis.com/kubernetes-release/release/${local.kubectl_version}/bin/linux/amd64/kubectl>
        chmod +x ./kubectl
        echo Installed kubectl
        echo Installing helm ...
        curl -LO <https://get.helm.sh/helm-v${var.helm_version}-linux-amd64.tar.gz>
        tar -zxvf ./helm-v${var.helm_version}-linux-amd64.tar.gz
        mv ./linux-amd64/helm ./helm
        chmod +x ./helm
        helm version --client
        echo Installed helm
        echo Installing helmfile ...
        curl -LO <https://github.com/roboll/helmfile/releases/download/v${var.helmfile_version}/helmfile_linux_amd64>
        mv ./helmfile_linux_amd64 ./helmfile
        chmod +x ./helmfile
        which helmfile
        helmfile --version
        echo Installed helmfile
        aws_cli_assume_role_arn="${var.aws_assume_role_arn}"
        aws_cli_assume_role_session_name="${module.label.id}"
        echo Assuming role "$aws_cli_assume_role_arn" ...
        curl -L <https://github.com/stedolan/jq/releases/download/jq-${var.jq_version}/jq-linux64> -o jq
        chmod +x ./jq
        # source <(aws --output json sts assume-role --role-arn "$aws_cli_assume_role_arn" --role-session-name "$aws_cli_assume_role_session_name"  | jq -r  '.Credentials | @sh "export AWS_SESSION_TOKEN=\(.SessionToken)\nexport AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) "')
        echo Assumed role "$aws_cli_assume_role_arn"
        echo Getting kubeconfig from the cluster...
        aws eks update-kubeconfig --name=${var.cluster_name} --region=${var.region} --kubeconfig=${var.kubeconfig_path} ${var.aws_eks_update_kubeconfig_additional_arguments}
        export KUBECONFIG=${var.kubeconfig_path}
        kubectl version --kubeconfig ${var.kubeconfig_path}
        kubectl get nodes --kubeconfig ${var.kubeconfig_path}
        kubectl get pods --all-namespaces --kubeconfig ${var.kubeconfig_path}
        echo Got kubeconfig from the cluster
		echo '{"helm_version": "$(helm version --client)", "helmfile_version": "$(helmfile --version)"}' >&3
      EOF
  }
}

output "helm_version" {
  value = data.shell_script.install_external_packages.output["helm_version"]
}

output "helmfile_version" {
  value = data.shell_script.install_external_packages.output["helmfile_version"]
}

so, if we just had a docker image with our toolchain inside of it, it would have been a lot easier.

Thanks for this Erik. What’s strange is we have custom workers in TFE. I think there’s some major concerns about security, but that you can install anything you want anyway, just slows things down.

I’ll pass this on to the developers so they see what kind of gymnastics folks are going through.

Terraform cloud is more than that. That’s just the state file stuff. I use for running plan and apply in their hosted containers

https://www.terraform.io/docs/providers/aws/r/launch_configuration.html#block-devices

you can have multiple ebs_block_device blocks

does this work

I mean we need to have the functionality that can have any number of devices added based on the requirement

no count wont work

you’ll need to use dynamic blocks for that and terraform 0.12

https://www.terraform.io/docs/configuration/expressions.html#dynamic-blocks

Yep, let’s move to #pr-reviews

sure

Yep, check out https://github.com/cloudposse/terraform-aws-ecs-web-app

Awesome example. Out of interest in this case what updates / registers the new task definition after the latest image is built in CI? And how do you stop that from conflicting with the initial task definition created by terraform?