#terragrunt (2021-02)

terragrunt

Terragrunt discussions Archive: https://archive.sweetops.com/terragrunt/

2021-02-28

Christian avatar
Christian

How do you guys structure your folders when dealing with iam users in terragrunt? One terragrunt.hcl file per iam user?

David avatar
David

I have one module that makes all users, and then I have separate IAM role modules that assign users to roles

2021-02-26

Tim Gourley avatar
Tim Gourley

Hi All, I’ve been looking at Atlantis for automation with our Terragrunt configurations. The recommended approach seems to be to plan individual projects. Atlantis can do some parallel operations but it is real fuzzy to me if that actually works (workspaces get pulled into the conversation etc) I have a test configuration with about 24 modules and the Atlantis plan takes ~30 minutes since it processes each model in sequence. I did get some parallel operations but then there were lock contentions, and the workspace topic freaks me out as I understood that is not something you do when using Terragrunt.

So, what am I missing, am I using a hammer to drive a screw? Why is plan-all not discussed? I do understand the shortcomings of unapplied modules affecting dependent modules in the plan but I think this is one of the choices we make when we move to Terragrunt. Any suggestions on a more appropriate automation tooling for Terragrunt managed envs?

TIA, Tim

ohad avatar

Check out env0 support of Terragrunt - https://www.env0.com/blog/terragrunt-release (disclaimer - I am co-founder and CEO of env0)

Time to DRY those IaC configurations with env0 and Terragrunt! | env0 blog attachment image

Hello, env0 and Terragrunt fans alike! It’s new-feature-day, yet again! But this time, we have something really special for you. We’re giving you the ability to completely change the game on the Infrastructure as Code files that you use to deploy and manage environments with our platform! Introducing remote-run support for Terragrunt workflows, now available in env0!

David avatar
David

You can also still use Atlantis and just use https://github.com/transcend-io/terragrunt-atlantis-config to automatically make atlantis work for terragrunt with parallelization and automatic atlantis.yaml generation so that all affected modules are planned

transcend-io/terragrunt-atlantis-config

Generate Atlantis config for Terragrunt projects. Contribute to transcend-io/terragrunt-atlantis-config development by creating an account on GitHub.

Tim Gourley avatar
Tim Gourley

I tried the terragrunt-atlantis-config and it did not seem help much. Isn’t atlantis still going to process modules individually and sequentially?

David avatar
David

If you use the --parallel and ``–create-workspace` flags it will process in parallel. I have about 600 terragrunt modules that can all be planned in about 15 minutes

David avatar
David
transcend-io/terragrunt-atlantis-config

Generate Atlantis config for Terragrunt projects. Contribute to transcend-io/terragrunt-atlantis-config development by creating an account on GitHub.

David avatar
David

There is a concept of Atlantis workspaces, and there is a concept of terraform workspaces, but they are entirely unrelated, so --create-workspace is safe to use. It only refers to creating Atlantis workspaces

Tim Gourley avatar
Tim Gourley

Thanks for the responses! I’ll give this another go…

2021-02-17

2021-02-11

Asis avatar

Hi Everyone Our team runs terragrunt modules locally , what are the best solutions/ practice to run modules in more unified pattern Note we have s3bucket + dynamodb for locking state Thanks in Advance

omry avatar

hi @ - you can give env0 a try, we support deployments and running terragrunt and terraform modules in a unified solution. You can read mode here - https://www.env0.com/

  • Disclaimer - I am the co-founder and CTO of env0.
Automated remote-run workflows management | env0 attachment image

Automate the provisioning of Terraform workflows and Infrastructure as Code with env0. Get visibility, predictability and governance on your cloud deployments.

2021-02-10

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Am new to Terragrunt. Trying to run Cloudrail as part of my Terragrunt flow to test plans before they are applied. I have three modules - item1, item2, item3. Cloudrail needs to take in the plan.out out as its input, and spits out some stdout content as well as an exit code. Based on this exit code, I want to move forward to apply.

So am running this:

terragrunt run-all plan -out plan.out
terragrunt run-all apply -auto-approve plan.out

And I’ve added this after_hook:

terraform {
  after_hook "cloudrail_after_hook" {
    commands     = ["plan"]

    execute      = [
      "docker", 
      "run", 
      "--rm", 
      "-v", "${get_env("PWD", "")}:/data", 
      "indeni/cloudrail-cli", 
      "run", 
      "-d", "${path_relative_to_include()}",
      "--tf-plan", "${path_relative_to_include()}/plan.out",
      "--origin", "ci",
      "--build-link", "<https://somelink>",
      "--execution-source-identifier", "somebuildnumber - tg module ${path_relative_to_include()}",
      "--api-key", "${get_env("CLOUDRAIL_API_KEY", "")}",
      "--auto-approve"
      ]
  }
}

But:

  1. The execution of the Cloudrail container is all over the place. It’s hard to know which module the Cloudrail output belongs to without using --terragrunt-log-level info.
  2. The whole concept of passing to apply a plan I created in a previous step doesn’t seem to be working. Am I missing something?
Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

@ help!!!

loren avatar
loren

the -all commands are a hot mess. it’s a nice idea, but seems to only work in very restrictive use cases

1
Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

So if you have multiple modules, you run each plan separately?

loren avatar
loren

personally, yes, i manage that execution/orchestration outside terragrunt.

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

In that case, a simple plan -out plan.out (with after_hook) and apply plan.out, just done multiple times.

loren avatar
loren

yes, exactly

loren avatar
loren

i also agree on the terragrunt output being interleaved and difficult to understand when using -all… i have an open issue for addressing it… https://github.com/gruntwork-io/terragrunt/issues/1194

Option to prefix *-all terraform output with working-dir label · Issue #1194 · gruntwork-io/terragrunt

Right now, all terraform output is unmodified, which makes a lot of sense for regular terraform commands because they are only working on one config at a time and the output can be captured/used fo…

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Ah! I was looking for the same thing. This interleaving was messing with my mind.

loren avatar
loren

i got so sick of it, i had to write a custom command runner to inject the paths for our own usage i’d try to fix it in terragrunt, but golang makes my head hurt

loren avatar
loren

python asyncio ftw!

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Amen to that

loren avatar
loren

semi-related, absolute paths are too noisy… https://github.com/gruntwork-io/terragrunt/issues/1515

[feature request] Print relative paths instead of absolute paths in console output · Issue #1515 · gruntwork-io/terragrunt

Instead of showing absolute paths, is it possible to make Terragrunt&#39;s console output show paths relative to where the command was run? For example, commands like validate, validate-all, plan e…

loren avatar
loren

Also, the change adding –terragrunt-log-level is very new, and there is a new pr making “info” the default, basically reverting the output back to before they had log levels… Basically, output of -all is a work in progress

loren avatar
loren

One thing you can try, is to redirect the cloudrail output to a known location, with a filename based on the terragrunt-working-dir, so you can review it afterwards

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Oh yeah that could work too. There’s the --output-file and --output-format options for Cloudrail.

1
joshmyers avatar
joshmyers

Don’t use plan-all / apply-all

joshmyers avatar
joshmyers

not production safe

Mohammed Yahya avatar
Mohammed Yahya

@ first make it more simple by creating a function alias in .bashrc or .zshrc as this example:

slscan() {
    docker run --rm -e "WORKSPACE=$(pwd)" -e GITHUB_TOKEN -v "$(pwd):/app" shiftleft/scan scan $*
}

add there what ever logic you need, then cal slscan in your terragrunt

Mohammed Yahya avatar
Mohammed Yahya

IMHO I’m using a Taskfile (modern Makefile alternative) for all the logic I need before calling Terraform or Terragrunt, example:

task init
task plan # this will write plan.out
task cloudrail # this will read/scan plan.out
task apply # this will apply plan.out
task inframap # this will draw plan.out
joshmyers avatar
joshmyers

Why wrap Terrragrunt (a wrapper) in another wrapper?

Mohammed Yahya avatar
Mohammed Yahya

Good one, but when you work so deep with IaC you need your own wrapper

joshmyers avatar
joshmyers

Not true

joshmyers avatar
joshmyers

Mohammed Yahya avatar
Mohammed Yahya

I don’t use Terragrunt anymore, I’m using my own bananadance

Mohammed Yahya avatar
Mohammed Yahya
same for CloudPosse’s [atmos> and <https://github.com/cloudposse/turf turf](https://github.com/cloudposse/atmos)
cloudposse/atmos

Universal Tool for DevOps and Cloud Automation (works with terraform, helm, helmfile, istioctl, etc) - cloudposse/atmos

cloudposse/turf

CLI Tool to help with various automation tasks (mostly all that stuff we cannot accomplish with native terraform) - cloudposse/turf

Mohammed Yahya avatar
Mohammed Yahya

you reach a point where you have complex logic and need your own tool

Mohammed Yahya avatar
Mohammed Yahya

I agree with you but the point is all of these tools are just wrapper around Terraform, so used what ever you prefer, and match your use case

joshmyers avatar
joshmyers

Need a tool, don’t need own tool….

joshmyers avatar
joshmyers

Using Terragrunt/Atlantis without issues on a large codebase, multi region, multi account …

1
joshmyers avatar
joshmyers

I’d rather take something sane off the shelf, than build your own and have to write fancy bespoke adapters for everything

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Thank you for the feedback everyone. I feel much better now (I thought it was me!).

loren avatar
loren

atlantis is nice, but you still need to run it and maintain it. it’s not actually free. a cli wrapper can run in any existing CI or local system, and doesn’t consume resources when not in use. most of our implementations are in codecommit, with event hooks to trigger codebuild

loren avatar
loren

so pros and cons both ways

loren avatar
loren

@ might be worth revisiting the -all workflow… terragrunt has published a few versions already that are trying to address some of your pain….

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Yeah I noticed

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

Did you give it a shot?

loren avatar
loren

i have not… my workflows tend to avoid -all

Mohammed Yahya avatar
Mohammed Yahya
11:26:23 PM

@ has joined the channel

2021-02-09

Amit Karpe avatar
Amit Karpe

Do we have AWS Profile option using Terragrunt?

I am ware that using Terraform, we can use provider block:

provider "aws" {
  profile = "default"
  region  = "us-west-2"
}

Where we can specify profile name. How I can change that in the terragrunt.hcl? Is there any way we can override it during terragrunt apply or any run time operation?

aaratn avatar
aaratn
remote_state {
  backend = "s3"
  config = {
    bucket  = "foo-bar-tfstate"
    key     = "<tfstatename>.tfstate"
    region  = "us-east-1"
    encrypt = true
    profile = "<aws-profile-name>"
  }
}
aaratn avatar
aaratn

Something like this ?

    keyboard_arrow_up