#terragrunt (2020-05)

terragrunt

Terragrunt discussions Archive: https://archive.sweetops.com/terragrunt/

2020-05-29

Brij S avatar
Brij S

Hey all, I’m using the generate block to create a backend for me, however I notice the tags dont get passed to it.

remote_state {
  backend = "s3"
  generate = {
    path      = "[backend.tf](http://backend\.tf)"
    if_exists = "overwrite"
  }

  config = {
    bucket                    = format("%s-state-%s", local.role_name, local.aws_region)
    key                       = "terraform.tfstate"
    region                    = local.aws_region
    encrypt                   = true
    dynamodb_table            = format("%s-locks", local.role_name)
    s3_bucket_tags            = local.tags
    dynamodb_table_tags       = local.tags
    skip_bucket_accesslogging = true
  }
}

any way I can pass the tags?

2020-05-28

muhaha avatar
muhaha

Is possible to define “projects” in terragrunt.hcl ? Like project: foo with path: a/b/c and then if is possible to run terragrunt apply –project foo

Tim Birkett avatar
Tim Birkett

Not that I’m aware of @muhaha but that does sound interesting… I’ve wrapped calling terragrunt in a rake task so that I can do something like rake tg:apply[<account>/<module>] or rake tg:apply[<account>] for an apply-all. Our directory structure is rather deep to allow for expansion into multiple regions and we have multiple accounts per environment which presents interesting challenges. I think the CloudPosse team would be open to discussing the feature and possibly accept a PR?

muhaha avatar
muhaha

here is more info https://sweetops.slack.com/archives/CQCDCLA1M/p1590742750321200 , but … I just found https://mbtproject.github.io/mbt/ Now I at least know what I am looking for … “Monorepo build tool language agnostic”

muhaha avatar
muhaha

btw, terragrunt is by cloudposse?

Tim Birkett avatar
Tim Birkett

Or is it Gruntworks?? I forget

muhaha avatar
muhaha

did not find any community forum/slack channel thoi

muhaha avatar
muhaha
(feat): Monorepo build options idea · Issue #1206 · gruntwork-io/terragrunt
Hello, Would be great to have something for handling monorepo use cases ( especially in CI ): Folder structure: azure(dir) westeurope project1 [main.tf> terragrunt.hcl project2 <http://main.tf main.tf](http://main.tf) terragrunt.hcl…
loren avatar
loren

not too sure i understand your definition of project, but have you seen --terragrunt-working-dir? https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-working-dir

CLI options

Terragrunt forwards all arguments and options to Terraform. Learn more about CLI options in Terragrunt.

loren avatar
loren
Execute Terraform commands on multiple modules at once

Learn how to avoid tedious tasks of running commands on each module separately.

muhaha avatar
muhaha

@loren you should be able to define projects and paths in yaml, toml, json

loren avatar
loren

i don’t understand the “should”?

muhaha avatar
muhaha

then triggering terraform apply -p PROJECT1

muhaha avatar
muhaha

in my proposal

loren avatar
loren

i’m just pointing you towards the features that terragrunt supports today that seem to get you closer to your goal, you can use them however you like

muhaha avatar
muhaha

terragrunt does handle git diff ?

muhaha avatar
muhaha

how terragrunt will get info about what project has been changed?

muhaha avatar
muhaha

i dont event need --terragrunt-working-dir flag, i can use cd $working_dir in shell

Tim Birkett avatar
Tim Birkett

That would be your CI tools job really. To work out from changed files whether it should run something.

loren avatar
loren

terragrunt has no integration with git or any source control system

muhaha avatar
muhaha

of course it can be inotify

muhaha avatar
muhaha

point is that terragrunt is not ready for larger monorepos and CI

loren avatar
loren
12:57:04 PM

i have a setup not unlike your folder structure, and works fine for me ¯_(ツ)_/¯

loren avatar
loren

i just use a small custom utility to control what modules i want it to apply. terragrunt doesn’t care about all that

muhaha avatar
muhaha

how are you handling situation if someone will create PR

muhaha avatar
muhaha

then CI mechanism must determine what project is changed

loren avatar
loren

yes, my utility is integrated with git

muhaha avatar
muhaha

to trigger terragrunt on those project/module

muhaha avatar
muhaha

is it avaialbe as opensource? i am interested how are you doingit

Tim Birkett avatar
Tim Birkett

That is not difficult, most CI utilities have that functionality.

loren avatar
loren

nope

muhaha avatar
muhaha

oh sure

muhaha avatar
muhaha

muhaha avatar
muhaha

@ what utilities? there is no one

loren avatar
loren

but the base of the logic is this function:

def _get_changed_dirs(include, commit_range):
    changed_files = utils.check_output(
        ['git', 'diff', '--name-only', commit_range],
        stdout_callback=None
    ).decode('utf-8').split()

    changed_dirs = {os.path.dirname(f) for f in changed_files}

    return set(itertools.chain.from_iterable([
        wcmatch.glob.globfilter(
            changed_dirs, item, flags=wcmatch.glob.EXTGLOB)
        for item in include
    ]))
muhaha avatar
muhaha

you have to depend on raw git command

loren avatar
loren

yes, git libraries are pretty horrible in general

muhaha avatar
muhaha

i am using same mechanism, i have defined projects in json with name and path, and if diff is from that paths => run terragrunt with that path

Tim Birkett avatar
Tim Birkett

What is wrong with that solution?

muhaha avatar
muhaha

atlantis handle it very well, so why not to use terragrunt for it

muhaha avatar
muhaha

if its not git based, it can still calculate sha

Tim Birkett avatar
Tim Birkett

I thought Atlantis did it based on the info in the webhook it receives. You can run Terragrunt in Atlantis and configure Atlantis to trigger terragrunt in the correct “project” can’t you?

2020-05-22

2020-05-21

Brij S avatar
Brij S

Hello! Does anyone here have any clever ways of actually managing the remote state infra thats created by terragrunt? (s3/dynamo). How does one remove/delete it in a sane manner

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Don’t know how much this will go towards alleviating your problem/request, but we’re managing terraform state buckets with terraform.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-tfstate-backend

Terraform module that provision an S3 bucket to store the terraform.tfstate file and a DynamoDB table to lock the state file to prevent concurrent modifications and state corruption. - cloudposse…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

See PR#46 which is a clever way of doing this. Haven’t yet fully reviewed it…

Tim Birkett avatar
Tim Birkett

Interesting topic… I think it sort of lives forever and requires manual cleanup. I do believe there is a generate command that can help you generate the code to create the state resources: https://terragrunt.gruntwork.io/docs/features/keep-your-remote-state-configuration-dry/#using-the-generate-property-to-generate-terraform-code-for-managing-remote-state - not sure how useful that would be. You’d then have local state for those resources.

Tim Birkett avatar
Tim Birkett

Hmm… that doesn’t create the actual bucket or table though. Maybe there should be a PR for a destroy command that cleans up those resources.

Brij S avatar
Brij S
Clean command : Remove S3 bucket and DynamoDB table · Issue #250 · gruntwork-io/terragrunt

Terragrunt is a really nice wrapper to automate the creation of S3 bucket and DynamoDB table. I&#39;m actually using terragrunt to spawn demo enviroment for different development teams. So each tea…

Tim Birkett avatar
Tim Birkett

What is it you want to achieve? I’m leaning towards it being uncommon that you’d actually want to remove the state bucket / dynamodb lock table.

Brij S avatar
Brij S

we have preview environments that use terragrunt to spin up some infra. Once they are torn down it would be nice to have some way to clean up the state resources as well

Tim Birkett avatar
Tim Birkett

That’s what I thought might be the answer - I think you could simply use the aws cli and the parameters from the terragrunt.hcl files or something?

Brij S avatar
Brij S

and thats what I thought might be the easiest answer as well

Brij S avatar
Brij S

just thought there might be something more nicer out there, guess not

Tim Birkett avatar
Tim Birkett

If you could create your environments in their own account, you could nuke everything using something like: https://github.com/gruntwork-io/cloud-nuke

Tim Birkett avatar
Tim Birkett

There are hooks in Terragrunt, so you could programmatically clean up the bucket and table in an after hook on destroy-all or something? https://terragrunt.gruntwork.io/docs/features/before-and-after-hooks/

Tim Birkett avatar
Tim Birkett

I think the problem is that you could destroy the bucket on an infrastructure that contains many states and end up with a lot of left over infrastructure to clean up.

Tim Birkett avatar
Tim Birkett

If you were running this alongside other important environments you’d have a fun task on your hands.

Tim Birkett avatar
Tim Birkett

Personally, I wrap my Terragrunt commands in rake / ruby so I can be a bit more flexible and add extra validations to running things.

2020-05-20

aaratn avatar
aaratn

Checkout terraform version manager written by me, supports pip, docker and homebrew !!

Its already becoming popular with 40  .

https://github.com/aaratn/terraenv

aaratn/terraenv

Terraform & Terragrunt Version Manager. Contribute to aaratn/terraenv development by creating an account on GitHub.

:--1:3
joey avatar

related, i had been using tfenv, tgenv, helmenv, pyenv, and rvm until i recently learned of https://github.com/asdf-vm/asdf which also [has support for many more things>. </i](https://asdf-vm.com/#/plugins-all)

asdf-vm/asdf

Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more - asdf-vm/asdf

2
:--1:2
aaratn avatar
aaratn

@ So this supports brew and raw git installation, its hard to ensure consistency

which is the case with tfenv and tgenv too and thats the reason I chose to write this in python and produce binaries which are available in form of brew, pip and docker images

asdf-vm/asdf

Extendable version manager with support for Ruby, Node.js, Elixir, Erlang & more - asdf-vm/asdf

joshmyers avatar
joshmyers

lol

2020-05-18

2020-05-17

Tim Birkett avatar
Tim Birkett

Hey :wave:  - Terragrunt v0.23.18 brought a nice new function: sops_decrypt_file  it allows you to decrypt either a JSON or YAML file that has been created with Mozilla’s sops and use it as inputs to Terraform. Looks like a reasonable solution for those initial an innoying static keys and passwords that you’d otherwise have to manually create somewhere. https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#sops_decrypt_file

Built-in functions

Terragrunt allows you to use built-in functions anywhere in terragrunt.hcl, just like Terraform.

6

2020-05-11

joshmyers avatar
joshmyers

read_terragrunt_config doesn’t let you share locals between files, you can do stuff with inputs….but not locals…

joshmyers avatar
joshmyers

You can use locals in inputs, but they aren’t quite the same

2020-05-09

joshmyers avatar
joshmyers

Yeah, not keen. I want a DSL, not a full fledged language and it still uses the TF providers under the covers

1

2020-05-08

joshmyers avatar
joshmyers

@David Nice! Can I ask about what your Terragrunt on disk structure looks like?

David avatar
David

Sure thing! “on disk” as in what our file structure looks like?

joshmyers avatar
joshmyers

Yup

joshmyers avatar
joshmyers
terraform
├── coreeng
│   ├── account.hcl
│   └── global
│       ├── env.hcl
│       └── us-east-1
│           ├── atlantis
│           │   └── terragrunt.hcl
│           ├── ecr
│           │   └── terragrunt.hcl
│           └── region.hcl
├── globals.hcl
├── prod
│   ├── account.hcl
│   └── prod
│       ├── env.hcl
│       ├── eu-central-1
│       │   ├── account-service
│       │   │   └── terragrunt.hcl
│       │   ├── device-service
│       │   │   └── terragrunt.hcl
│       │   ├── graph-service
│       │   │   └── terragrunt.hcl
│       │   ├── idp-service
│       │   │   └── terragrunt.hcl
│       │   ├── profile-service
│       │   │   └── terragrunt.hcl
│       │   ├── region.hcl
│       │   └── resource-service
│       │       └── terragrunt.hcl
│       ├── eu-west-1
│       │   ├── account-service
│       │   │   └── terragrunt.hcl
│       │   ├── buckets
│       │   │   ├── [main.tf](http://main\.tf)
│       │   │   └── terragrunt.hcl
│       │   ├── compliance-service
│       │   │   └── terragrunt.hcl
│       │   ├── device-service
│       │   │   └── terragrunt.hcl
│       │   ├── graph-service
│       │   │   └── terragrunt.hcl
│       │   ├── idp-service
│       │   │   └── terragrunt.hcl
│       │   ├── profile-service
│       │   │   └── terragrunt.hcl
│       │   ├── region.hcl
│       │   └── resource-service
│       │       └── terragrunt.hcl
│       ├── us-east-1
│       │   ├── account-service
│       │   │   └── terragrunt.hcl
│       │   ├── buckets
│       │   │   ├── [main.tf](http://main\.tf)
│       │   │   ├── terragrunt.hcl
│       │   │   └── tfplan
│       │   ├── compliance-service
│       │   │   └── terragrunt.hcl
│       │   ├── device-service
│       │   │   └── terragrunt.hcl
│       │   ├── graph-service
│       │   │   └── terragrunt.hcl
│       │   ├── idp-service
│       │   │   └── terragrunt.hcl
│       │   ├── profile-service
│       │   │   └── terragrunt.hcl
│       │   ├── region.hcl
│       │   └── resource-service
│       │       └── terragrunt.hcl
│       └── us-west-2
│           ├── account-service
│           │   └── terragrunt.hcl
│           ├── device-service
│           │   └── terragrunt.hcl
│           ├── graph-service
│           │   └── terragrunt.hcl
│           ├── idp-service
│           │   └── terragrunt.hcl
│           ├── profile-service
│           │   └── terragrunt.hcl
│           ├── region.hcl
│           └── resource-service
│               └── terragrunt.hcl
├── terragrunt.hcl
└── test
    ├── account.hcl
    ├── dev
    │   ├── account-service.hcl
    │   ├── compliance-service.hcl
    │   ├── device-service.hcl
    │   ├── env.hcl
    │   ├── eu-central-1
    │   │   ├── account-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── device-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── graph-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── idp-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── profile-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── region.hcl
    │   │   └── resource-service
    │   │       └── terragrunt.hcl
    │   ├── eu-west-1
    │   │   ├── account-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── compliance-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── device-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── graph-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── idp-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── profile-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── region.hcl
    │   │   └── resource-service
    │   │       └── terragrunt.hcl
    │   ├── graph-service.hcl
    │   ├── idp-service.hcl
    │   ├── profile-service.hcl
    │   ├── resource-service.hcl
    │   ├── us-east-1
    │   │   ├── account-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── compliance-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── device-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── graph-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── idp-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── profile-service
    │   │   │   └── terragrunt.hcl
    │   │   ├── region.hcl
    │   │   └── resource-service
    │   │       └── terragrunt.hcl
    │   └── us-west-2
    │       ├── account-service
    │       │   └── terragrunt.hcl
    │       ├── device-service
    │       │   └── terragrunt.hcl
    │       ├── graph-service
    │       │   └── terragrunt.hcl
    │       ├── idp-service
    │       │   └── terragrunt.hcl
    │       ├── profile-service
    │       │   └── terragrunt.hcl
    │       ├── region.hcl
    │       └── resource-service
    │           └── terragrunt.hcl
    └── qa
        ├── env.hcl
        ├── eu-central-1
        │   ├── account-service
        │   │   └── terragrunt.hcl
        │   ├── device-service
        │   │   └── terragrunt.hcl
        │   ├── graph-service
        │   │   └── terragrunt.hcl
        │   ├── idp-service
        │   │   └── terragrunt.hcl
        │   ├── profile-service
        │   │   └── terragrunt.hcl
        │   ├── region.hcl
        │   └── resource-service
        │       └── terragrunt.hcl
        ├── eu-west-1
        │   ├── account-service
        │   │   └── terragrunt.hcl
        │   ├── compliance-service
        │   │   └── terragrunt.hcl
        │   ├── device-service
        │   │   └── terragrunt.hcl
        │   ├── graph-service
        │   │   └── terragrunt.hcl
        │   ├── idp-service
        │   │   └── terragrunt.hcl
        │   ├── profile-service
        │   │   └── terragrunt.hcl
        │   ├── region.hcl
        │   └── resource-service
        │       └── terragrunt.hcl
        ├── us-east-1
        │   ├── account-service
        │   │   └── terragrunt.hcl
        │   ├── compliance-service
        │   │   └── terragrunt.hcl
        │   ├── device-service
        │   │   └── terragrunt.hcl
        │   ├── graph-service
        │   │   └── terragrunt.hcl
        │   ├── idp-service
        │   │   └── terragrunt.hcl
        │   ├── profile-service
        │   │   └── terragrunt.hcl
        │   ├── region.hcl
        │   └── resource-service
        │       └── terragrunt.hcl
        └── us-west-2
            ├── account-service
            │   └── terragrunt.hcl
            ├── device-service
            │   └── terragrunt.hcl
            ├── graph-service
            │   └── terragrunt.hcl
            ├── idp-service
            │   └── terragrunt.hcl
            ├── profile-service
            │   └── terragrunt.hcl
            ├── region.hcl
            └── resource-service
                └── terragrunt.hcl

102 directories, 114 files
joshmyers avatar
joshmyers

Sharing locals between terragrunt.hcl’s still isn’t a thing in TG AFAIK. There are a few interesting PRs/ideas how to DRY up this somewhat

joshmyers avatar
joshmyers

account>env>region>thing

joshmyers avatar
joshmyers

I suppose with Atlantis involved, DRY code isn’t necessarily what you want given triggers off an individual file

joshmyers avatar
joshmyers

Have you come up with any nice ways to layout your TG code while mainting Atlantis triggering on scoped events? convention over configuration?

David avatar
David

I believe you can share locals through read_terragrunt_config, unless I’m misunderstanding what you’re trying to share. And yep, we like lots and lots of small files because our Atlantis config generation is automated.

Here’s a directory tree (with some DNS and other duplicated things removed) from one of our projects:

David avatar
David
infra
├── README.md
├── admin
│   ├── common.block
│   ├── datadog
│   │   ├── env-dev
│   │   │   ├── secrets
│   │   │   │   └── terragrunt.hcl
│   │   │   └── terragrunt.hcl
│   │   └── provider.hcl
│   ├── organizations
│   │   ├── atlantis
│   │   │   └── terragrunt.hcl
│   │   ├── commons
│   │   │   └── terragrunt.hcl
│   │   ├── identity
│   │   │   └── terragrunt.hcl
│   │   ├── provider.block
│   │   ├── sandbox
│   │   │   └── terragrunt.hcl
│   │   ├── staging
│   │   │   └── terragrunt.hcl
│   │   ├── testing
│   │   │   └── terragrunt.hcl
│   │   └── transcend
│   │       ├── provider.block
│   │       └── terragrunt.hcl
│   ├── people.json
│   ├── policies
│   │   ├── [admin.tf](http://admin\.tf)
│   │   ├── admin_no_user_[iam.tf](http://iam\.tf)
│   │   ├── ci_[policy.tf](http://policy\.tf)
│   │   └── terragrunt.hcl
│   ├── roles
│   │   ├── atlantis_org
│   │   │   ├── admin
│   │   │   │   └── terragrunt.hcl
│   │   │   └── atlantis
│   │   │       └── terragrunt.hcl
│   │   ├── commons_org
│   │   │   ├── atlantis
│   │   │   │   └── terragrunt.hcl
│   │   │   └── ecr_role
│   │   │       └── terragrunt.hcl
│   │   ├── identity_org
│   │   │   ├── access_admin_role
│   │   │   │   └── terragrunt.hcl
│   │   │   └── atlantis
│   │   │       └── terragrunt.hcl
│   │   ├── role_provider.block
│   │   ├── sandbox_org
│   │   │   ├── admin
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── atlantis
│   │   │   │   └── terragrunt.hcl
│   │   │   └── ci
│   │   │       └── terragrunt.hcl
│   │   ├── staging_org
│   │   │   ├── admin
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── atlantis
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── ci
│   │   │   │   └── terragrunt.hcl
│   │   │   └── integrations_admin
│   │   │       └── terragrunt.hcl
│   │   ├── testing_org
│   │   │   ├── admin
│   │   │   │   └── terragrunt.hcl
│   │   │   └── atlantis
│   │   │       └── terragrunt.hcl
│   │   └── transcend_org
│   │       ├── admin
│   │       │   └── terragrunt.hcl
│   │       ├── admin_no_iam
│   │       │   └── terragrunt.hcl
│   │       ├── atlantis
│   │       │   └── terragrunt.hcl
│   │       ├── ci
│   │       │   └── terragrunt.hcl
│   │       ├── integrations_admin
│   │       │   └── terragrunt.hcl
│   │       └── role_provider.block
│   ├── terragrunt-config-admin.hcl
│   └── users
│       ├── datadog
│       └── identity_org
│           ├── access_key_management
│           │   └── terragrunt.hcl
│           ├── not_robots
│           │   └── terragrunt.hcl
│           └── robots
│               └── terragrunt.hcl
├── admin-dashboard
│   └── env-dev
│       └── terragrunt.hcl
├── atlantis
│   ├── ami
│   │   ├── README.md
│   │   ├── main.pkr.hcl
│   │   ├── terragroan.js
│   │   └── user_whitelist.js
│   ├── instance
│   │   └── terragrunt.hcl
│   └── secrets
│       └── terragrunt.hcl
├── backend
│   ├── README.md
│   ├── ecr
│   │   └── terragrunt.hcl
│   ├── env-dev
│   │   ├── alb
│   │   │   └── terragrunt.hcl
│   │   ├── buckets
│   │   │   └── terragrunt.hcl
│   │   ├── database
│   │   │   ├── primary
│   │   │   │   └── terragrunt.hcl
│   │   │   └── recovery
│   │   │       ├── kms_lookup
│   │   │       │   └── terragrunt.hcl
│   │   │       └── terragrunt.hcl
│   │   ├── ecs
│   │   │   ├── cluster
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── common.hcl
│   │   │   ├── container
│   │   │   │   ├── secrets
│   │   │   │   │   └── terragrunt.hcl
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── daemon_container
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── daemon_service
│   │   │   │   └── terragrunt.hcl
│   │   │   ├── service
│   │   │   │   └── terragrunt.hcl
│   │   │   └── task_permissions
│   │   │       └── terragrunt.hcl
│   │   ├── security_groups
│   │   │   ├── alb
│   │   │   │   └── terragrunt.hcl
│   │   │   └── rds
│   │   │       ├── primary
│   │   │       │   └── terragrunt.hcl
│   │   │       └── recovery
│   │   │           └── terragrunt.hcl
│   │   ├── ses_forwarding
│   │   │   └── terragrunt.hcl
│   │   ├── ssm_bastion
│   │   │   └── terragrunt.hcl
│   │   └── vpc
│   │       ├── main
│   │       │   └── terragrunt.hcl
│   │       └── recovery
│   │           └── terragrunt.hcl
│   └── env-local
│       ├── comment_attachments_bucket
│       │   └── terragrunt.hcl
│       ├── communication_emails_bucket
│       │   └── terragrunt.hcl
│       ├── company_uploads_bucket
│       │   └── terragrunt.hcl
│       ├── dsr_assembly_bucket
│       │   └── terragrunt.hcl
│       └── ses_forwarding
│           └── terragrunt.hcl
├── cdn_transcend_io
│   └── env-dev
│       └── terragrunt.hcl
├── codelabs
│   ├── bower_deps_bucket
│   │   └── terragrunt.hcl
│   ├── env-dev
│   │   └── terragrunt.hcl
│   └── homepage_images
│       └── terragrunt.hcl
├── cognito
│   ├── README.md
│   └── user_group
│       └── env-dev
│           ├── secrets
│           │   └── terragrunt.hcl
│           └── terragrunt.hcl
├── [common.tf](http://common\.tf)
├── commons
│   └── s3_state
│       ├── [main.tf](http://main\.tf)
│       ├── terragrunt.hcl
│       └── [vars.tf](http://vars\.tf)
├── constants
│   ├── README.md
│   ├── atlantis_org_[id.tf](http://id\.tf)
│   ├── commons_org_[id.tf](http://id\.tf)
│   ├── identity_org_[id.tf](http://id\.tf)
│   ├── prod_admin_no_iam_role_[arn.tf](http://arn\.tf)
│   ├── prod_admin_role_[arn.tf](http://arn\.tf)
│   ├── sandbox_admin_role_[arn.tf](http://arn\.tf)
│   ├── sandbox_org_[id.tf](http://id\.tf)
│   ├── staging_admin_role_[arn.tf](http://arn\.tf)
│   ├── staging_org_[id.tf](http://id\.tf)
│   ├── terragrunt.hcl
│   ├── testing_org_[id.tf](http://id\.tf)
│   └── transcend_org_[id.tf](http://id\.tf)
├── datadog
│   └── agent_container
│       └── env-dev
│           ├── secrets
│           │   └── terragrunt.hcl
│           └── terragrunt.hcl
├── externally_available_ecr
│   ├── ci-base
│   │   ├── Dockerfile
│   │   └── terragrunt.hcl
│   ├── ci-base-docker
│   │   ├── Dockerfile
│   │   └── terragrunt.hcl
│   ├── ci-base-go
│   │   ├── Dockerfile
│   │   └── terragrunt.hcl
│   ├── ci-base-pre-commit
│   │   ├── Dockerfile
│   │   └── terragrunt.hcl
│   ├── ci-base-puppeteer
│   │   ├── Dockerfile
│   │   └── terragrunt.hcl
│   └── sombra
│       └── terragrunt.hcl
├── firelens
│   └── container_definition
│       └── terragrunt.hcl
├── get_dot_env.sh
├── guardduty
│   ├── sandbox_org
│   │   ├── california
│   │   │   └── terragrunt.hcl
│   │   ├── frankfurt
│   │   │   └── terragrunt.hcl
│   │   ├── ireland
│   │   │   └── terragrunt.hcl
│   │   ├── ohio
│   │   │   └── terragrunt.hcl
│   │   └── virginia
│   │       └── terragrunt.hcl
│   ├── staging_org
│   │   ├── california
│   │   │   └── terragrunt.hcl
│   │   ├── frankfurt
│   │   │   └── terragrunt.hcl
│   │   ├── ireland
│   │   │   └── terragrunt.hcl
│   │   ├── ohio
│   │   │   └── terragrunt.hcl
│   │   └── virginia
│   │       └── terragrunt.hcl
│   └── transcend_org
│       ├── california
│       │   └── terragrunt.hcl
│       ├── frankfurt
│       │   └── terragrunt.hcl
│       ├── ireland
│       │   └── terragrunt.hcl
│       ├── ohio
│       │   └── terragrunt.hcl
│       └── virginia
│           └── terragrunt.hcl
├── homebrew
│   └── README.md
├── homepage
│   └── www_redirect
│       └── terragrunt.hcl
├── lambda_at_edge
│   └── env-dev
│       ├── 404_redirect
│       │   └── terragrunt.hcl
│       ├── cognito_auth
│       │   └── terragrunt.hcl
│       └── version
│           └── terragrunt.hcl
├── log_buckets
│   ├── commons
│   │   └── ireland
│   │       └── terragrunt.hcl
│   └── env-dev
│       ├── ireland
│       │   └── terragrunt.hcl
│       └── virginia
│           └── terragrunt.hcl
├── misc-assets-transcend
│   └── terragrunt.hcl
├── package.json
├── privacy-center
│   └── env-dev
│       └── terragrunt.hcl
├── spa-static-server
│   └── terragrunt.hcl
├── terragrunt-config-atlantis.hcl
├── terragrunt-config-commons.hcl
├── terragrunt-config-dev.hcl
├── terragrunt-config-local.hcl
├── terragrunt-config-prod.hcl
├── terragrunt-config-staging.hcl
├── terragrunt-config-testing.hcl
└── vault
    ├── README.md
    ├── ami
    │   ├── README.md
    │   └── main.pkr.hcl
    ├── env-dev
    │   ├── cluster
    │   │   ├── secrets
    │   │   │   └── terragrunt.hcl
    │   │   └── terragrunt.hcl
    │   └── settings
    │       ├── audit
    │       │   └── terragrunt.hcl
    │       ├── auth_backends
    │       │   └── aws
    │       │       └── terragrunt.hcl
    │       ├── mounts
    │       │   └── kv
    │       │       └── terragrunt.hcl
    │       └── roles
    │           ├── atlantis_role
    │           │   └── terragrunt.hcl
    │           └── developer_role
    │               └── terragrunt.hcl
    ├── provider.block
    ├── signRequest.js
    ├── vault-config-dev.block
    ├── vault-config-prod.block
    └── vault-config-staging.block
David avatar
David

Those are just directories with terragrunt files, so it doesn’t include any of our terraform (though we mostly use open source modules).

I also removed all the directories for staging/prod, as they are almost always the exact same as our env-dev directories. I’m not sure this is the best possible directory setup, but we haven’t had any troubles with it really.

At the very top level of this structure, we have files named “terragrunt-config-dev”, “terragrunt-config-staging”, etc. that set up remote state, common inputs, tags, AWS infro, etc.

We use the *.block files as common code (like providers) to generate and insert into our other terragrunt modules

David avatar
David

We don’t use as many regions as it looks like you do yet, but we often have duplication across accounts/envs, and for those we do essentially duplicate the terragrunt config and let Atlantis take care of it

joshmyers avatar
joshmyers

e.g. to keep all the services using the same source remote release in lockstep across regions (so you only have to bump versions in one place) in the env.yaml for a given environment there is a place holder for each service version, but then bumping anything in there triggers all Atlantis projects to plan

joshmyers avatar
joshmyers

Not sure if that makes sense

joshmyers avatar
joshmyers

OK, so you have a shit ton of dirs and individual files to toggle too

joshmyers avatar
joshmyers

(not just me then)

joshmyers avatar
joshmyers

So you have thing>environment>region, interesting

David avatar
David

Yep, I love my small files. All the modules have really small blast radius in case anything goes wrong, or if I ever need to refactor/migrate resources to different terraform state

joshmyers avatar
joshmyers

Aye, nice

David avatar
David

For that particular project, here is the cloc results:

cloc –vcs git .

cloc --vcs git .
     533 text files.
     510 unique files.                                          
Complex regular subexpression recursion limit (32766) exceeded at /usr/local/Cellar/cloc/1.84/libexec/bin/cloc line 9879.
Complex regular subexpression recursion limit (32766) exceeded at /usr/local/Cellar/cloc/1.84/libexec/bin/cloc line 9879.
      38 files ignored.

[github.com/AlDanial/cloc](http://github\.com/AlDanial/cloc) v 1.84  T=0.44 s (1133.1 files/s, 48260.4 lines/s)
--------------------------------------------------------------------------------
Language                      files          blank        comment           code
--------------------------------------------------------------------------------
HCL                             451           3267           2393          13403
JavaScript                       18            130            365            627
Markdown                          9            134              0            222
JSON                              7              0              0            191
Dockerfile                        5             33             36            128
Bourne Again Shell                2             28             16             90
Bourne Shell                      3              5              6              8
--------------------------------------------------------------------------------
SUM:                            495           3597           2816          14669
--------------------------------------------------------------------------------

So while we have tons of HCL, the files on average are <30 lines of actual code

joshmyers avatar
joshmyers

I’m waiting for some PRs to drop but I think about a DRY refactor

joshmyers avatar
joshmyers

Yeah same, but still a few :p

joshmyers avatar
joshmyers

I want something like Hiera for Terraform

Andy avatar

Hey :wave:  - Terragrunt v0.23.18 brought a nice new function: sops_decrypt_file  it allows you to decrypt either a JSON or YAML file that has been created with Mozilla’s sops and use it as inputs to Terraform. Looks like a reasonable solution for those initial an innoying static keys and passwords that you’d otherwise have to manually create somewhere. https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#sops_decrypt_file

joshmyers avatar
joshmyers

Terragrunt 2 layer approach isn’t ideal

David avatar
David

Have you played around with pulumi at all? You could probably reduce a lot of boilerplate with the higher level language constructs

2020-05-07

maarten avatar
maarten

Is anyone using any kind of IDE with Terragrunt autocompletion on dependencies and inputs ? I’m curious.

maarten avatar
maarten

@antonbabenko are you using an IDE ?

David avatar
David

I wrote a blog about why I find prefer Terragrunt to just Terraform: https://transcend.io/blog/why-we-use-terragrunt

Enhancing the Terraform Experience: Why we use Terragrunt attachment image

How Transcend deploys high-stakes infrastructure with confidence.

:--1:4
:100:1

2020-05-04

    keyboard_arrow_up