#terragrunt (2020-06)

terragrunt

Terragrunt discussions Archive: https://archive.sweetops.com/terragrunt/

2020-06-25

Tim Birkett avatar
Tim Birkett

I’m using terragrunt heavily at the moment but one thing is eluding me… During development of modules I pin to a branch and sometimes branches of Terraform modules under that. Currently, I’m deleting the .terragrunt-cache before every apply which is a bit frustrating, is there anyway to force the changes to be pulled in rather than deleting the cached modules every time I want to apply changes? Maybe there’s soe magic option I’ve missed, re-running an “init” doesn’t pull the newest changes to a branch in either

loren avatar
loren

pass the flag --terragrunt-source-update and it will automatically delete the cache for that config

Sam Holton avatar
Sam Holton

I normally use https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-source if I’ve actively developing a module, that way you don’t even have to commit/push to test it

CLI options

Terragrunt forwards all arguments and options to Terraform. Learn more about CLI options in Terragrunt.

:100:1
Tim Birkett avatar
Tim Birkett

Thanks @loren @Sam Holton - TERRAGRUNT_SOURCE=... works a charm

:--1:1

2020-06-24

muhaha avatar
muhaha

Guys, I need help: /project1/terragrunt.hcl file contains ( its main file ):

include {
  path = find_in_parent_folders("templates/aws.hcl")
}

/templates/aws.hcl file contains:

terraform {}

remote_state {
    disable_init = tobool(get_env("TERRAGRUNT_DISABLE_INIT", "false"))
    backend = "s3"
      config = {
      bucket = "${local.bucket}"
      region = "${local.backend_region}"
      key    = "${get_env("PPATH")}/terraform.tfstate"
      encrypt = true
      dynamodb_table = "${local.dynamodb_table}"
    }
}

locals {
  bucket = "${get_env("bucket")}"
  backend_region = "${get_env("backend_region")}"
  dynamodb_table = "${get_env("dynamodb_table")}"
}

generate "provider" {
  path      = "[provider.tf](http://provider\.tf)"
  if_exists = "overwrite"
  contents = <<EOF
provider "aws" {
  version = "2.67.0"
  region = ???
}
EOF
}

generate "backend" {
  path      = "[backend.tf](http://backend\.tf)"
  if_exists = "overwrite"
  contents = <<EOF
terraform {
  backend "s3" {}
}
EOF

How can I set region in /project1/terragrunt.hc with minimum effort?

• its not possible to set it via locals

• maybe with env_vars ? Thanks

David avatar
David

Yeah, the locals block their is scoped to that terragrunt file, while the provider block inside the generate block is injected as a terraform file, so you can’t directly use any locals there.

I would suggest using templatefile, and injecting in get_env("backend_region") in the region = ??? spot

muhaha avatar
muhaha

templatefile ? which one? I dont follow

David avatar
David

No worries! I was suggesting you use this function here: https://www.terraform.io/docs/configuration/functions/templatefile.html

So I’d add a /templates/provider.tf file with contents:

provider "aws" {
  version = "2.67.0"
  region = ${REGION}
}

and then in your /templates/aws.hcl file I would change the generate block to be:

generate "provider" {
  path      = "[provider.tf](http://provider\.tf)"
  if_exists = "overwrite"
  contents  = templatefile("./provider.tf", {
    REGION = get_env("backend_region")
  })
}

The other option is to just not specify region at all in your provider, and terraform will default to the AWS_REGION env var

templatefile - Functions - Configuration Language - Terraform by HashiCorp

The templatefile function reads the file at the given path and renders its content as a template.

muhaha avatar
muhaha

No, thats not a problem.. I dont want to preset region with ENV var, I just want pure gitops, but it should not be hardcoded in template, which is shared … So something like this:

/project1/terragrunt.hcl file contains ( its main file ):

include {
  path = find_in_parent_folders("templates/aws.hcl")
}

locals {
  region = "eu-central-0"
}

/templates/aws.hcl file contains:

terraform {}

remote_state {
    disable_init = tobool(get_env("TERRAGRUNT_DISABLE_INIT", "false"))
    backend = "s3"
      config = {
      bucket = "${local.bucket}"
      region = "${local.backend_region}"
      key    = "${get_env("PPATH")}/terraform.tfstate"
      encrypt = true
      dynamodb_table = "${local.dynamodb_table}"
    }
}

locals {
  bucket = "hardcoded"
  backend_region = "hardcoded"
  dynamodb_table = "hardcoded"
}

generate "provider" {
  path      = "[provider.tf](http://provider\.tf)"
  if_exists = "overwrite"
  contents = <<EOF
provider "aws" {
  version = "2.67.0"
  region = ${local.region}"
}
EOF
}

generate "backend" {
  path      = "[backend.tf](http://backend\.tf)"
  if_exists = "overwrite"
  contents = <<EOF
terraform {
  backend "s3" {}
}
EOF
David avatar
David

Ah, that is not possible. Terragrunt scopes locals to just the file you are working in, unfortunately. I’m not sure if there’s a bug or not for it, but you can’t to parent -> child or child -> parent locals through include blocks if I remember right

muhaha avatar
muhaha

loren avatar
loren

it feels backwards from the terragrunt paradigm to try to pass values from the child config to the parent config

loren avatar
loren

you can do things like define common values in a .yml file and read them with yamldecode(), or keep them in a “common” terragrunt config and use read_terragrunt_config()https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config

Built-in functions

Terragrunt allows you to use built-in functions anywhere in terragrunt.hcl, just like Terraform.

muhaha avatar
muhaha

@loren like this https://pastebin.com/raw/Q7UzBybA ? Will it work? :X

loren avatar
loren

i still think you have the paradigm backwards, and need to flip things around. what you are calling a “template” is what terragrunt refers to as a “parent”, and the relationship works the other way around

loren avatar
loren

at least, for the “include” functionality

loren avatar
loren

i’m not really sure how to do what you want, exactly how you want it. i’m just pointing you towards the tools that might get you to an equivalent setup. but it probably requires rethinking your approach

muhaha avatar
muhaha

tools?

loren avatar
loren

functions as tools, in this case

2020-06-22

2020-06-19

2020-06-18

Brij S avatar
Brij S

Hey all, I was wondering if theres any way to combine tags from the child into the parent hcl? For example, I’ve got the following in the my parent hcl file

locals {
  env           = element(split("/", local.relative_path), 1)
  base_tags = {
    "accountid"      = get_aws_account_id(),
    "environment"    = local.env,
  }
}

and in my child hcl file I have

locals {
  # This loads base tags from parent hcl and adds additional ones

  # Ideally I want to do the opposite - define some tags here, and combine
  # them in the parent hcl so that we have access to the full set of tags there
  # to use for s3_bucket_tags and dynamodb_table_tags
  # I'm not sure if/how that's possible
  # See <https://terragrunt.gruntwork.io/docs/features/locals/>
  # Their solution with yaml doesn't help us because 2 of the 3 base tags are
  # dynamically computed

  base_config = read_terragrunt_config(find_in_parent_folders())

  tags = merge(local.base_config.locals.base_tags, {
    "abc" = "0123"
    "dept"  = "engineering"
  })
}

any ideas

Adrian avatar
Adrian

So you want in your child merged tags from parent and child?

Brij S avatar
Brij S

other way around

Adrian avatar
Adrian

you can merge multiple values from mutliple level

locals {
  base_tags = {
    "accountid"      = "123123123",
    "environment"    = "test",
  }
  other_tags = {
    "test" = "asd"
  }
}

output "tags" {
  value = merge(local.base_tags, local.other_tags)
}
Brij S avatar
Brij S

I understand that, what I want is to take the tags declared in a child.hcl and use them in the parent.hcl

Brij S avatar
Brij S

two different files on different levels

2020-06-16

Adrian avatar
Adrian

What’s do you think about that type of approach to put this data in input section of HCL file?

  custom_task_policy_statement = [
    {
      actions = [
        "s3:GetObject",
        "s3:ListBucketVersions",
        "s3:ListBucket",
        "s3:GetObjectTagging",
        "s3:GetBucketVersioning",
        "s3:GetBucketAcl",
        "s3:GetBucketCORS",
        "s3:GetBucketLocation",
        "s3:GetObjectVersion",
        "s3:PutObject",
        "s3:GetAccountPublicAccessBlock",
        "s3:ListAllMyBuckets",
        "s3:HeadBucket"
      ]
      effect = "Allow"
      resources = [
        dependency.s3.outputs.this_s3_bucket_arn,
        "${dependency.s3.outputs.this_s3_bucket_arn}/*"
      ]
    },
    {
      actions = [
        "kms:DescribeKey",
        "kms:GenerateDataKey*",
        "kms:Encrypt",
        "kms:ReEncrypt*",
        "kms:Decrypt"
      ]
      effect    = "Allow"
      resources = [dependency.kms.outputs.key_arn]
    }
  ]
David avatar
David

I usually either:

• ask for a string policy, which can be made in inputs with jsonencode

• ask for an array of existing IAM Policy arns Your approach here could definitely work, but I’d make sure it works with things like condition clauses. Terraform’s typing isn’t great for working with maps of semi-structured data, but it can work

joshmyers avatar
joshmyers

I have kinda similar to this

joshmyers avatar
joshmyers

Without the dependency usage

2020-06-08

muhaha avatar
muhaha

Is possible to get path ( absolute ) if I am using terragrunt --terragrunt-working-dir $mydir ? For example: Terragrunt is triggered from /tmp/ and is using --terragrunt-working-dir foo/bar/project I need relative path like foo/bar/project

In fact foo/bar/project is located in /tmp, but on other enviroments, this path may be different

loren avatar
loren

i’m confused a bit, it sounds like you are looking for a relative path, but you are asking about an absolute path

muhaha avatar
muhaha

*relative, sorry

loren avatar
loren

get_terragrunt_dir() will give you the absolute path

loren avatar
loren

you might be able to do it with some creative use of path_relative_to_include()

loren avatar
loren
Built-in functions

Terragrunt allows you to use built-in functions anywhere in terragrunt.hcl, just like Terraform.

muhaha avatar
muhaha

I am not using include

muhaha avatar
muhaha

path_relative_to_include() will always return .

muhaha avatar
muhaha

maybe run_cmd() ?

loren avatar
loren

lolol, all part of what i meant by “creative use”

loren avatar
loren

sometimes you have to adjust how you have things setup, in order to use the features a tool offers…

muhaha avatar
muhaha

well, its should be possible to do: run_cmd("echo","${get_terragrunt_dir()#pwd}") , does not work yet .. :X

Adrian avatar
Adrian

Where do you put your provider settings?

provider "aws" {
  region  = var.aws_region

  # Make it faster by skipping something
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_credentials_validation = true
}

also this setting?

terraform {
  backend "s3" {}
}

What the best way and how do you handle this? I’ve created terragrunt.tf> in each module with both of this settings. Is this correct way? Previously I was using hook to copy main_<http://providers.tf|providers.tf

provider "aws" {
  region  = var.aws_region

  # Make it faster by skipping something
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_credentials_validation = true
}

provider "aws" {
  alias   = "virginia"
  region  = "us-east-1"
}

terraform {
  backend "s3" {}
  required_version = ">= 0.12.0"
}

variable "aws_region" {
  type        = string
  default     = "eu-central-1"
  description = "AWS region to use for all resources"
}
Adrian avatar
Adrian
gruntwork-io/terragrunt-infrastructure-live-example

A repo used to show examples file/folder structures you can use with Terragrunt and Terraform - gruntwork-io/terragrunt-infrastructure-live-example

loren avatar
loren

These days we use the new generate block…

:100:1
loren avatar
loren

Yep, that file is a good example

2020-06-04

Adrian avatar
Adrian
Calling apply does not remove `-var` arguments if the actual variables are a separate argument · Issue #1035 · gruntwork-io/terragrunt

I am using Terragrunt v0.21.11 If I say: terraform { extra_arguments &quot;common_vars&quot; { commands = get_terraform_commands_that_need_vars() arguments = [ &quot;-var&quot;, foo=bar&quot;, ] } …

David avatar
David

This seems related to https://github.com/gruntwork-io/terragrunt/issues/983, which I issued a long ways back.

Terragrunt doesn’t discriminate between apply calls with vs without a planfile, and doing so seems non-trivial

At my company the problem disappeared when we switched from using local secrets files to fetching secrets from Vault

Add support to include tfvars only for `apply`s without plan output files · Issue #983 · gruntwork-io/terragrunt

Right now, writing code such as terraform { extra_arguments &quot;common_var&quot; { commands = get_terraform_commands_that_need_vars() arguments = [&quot;-var-file=${get_aws_account_id()}.tfvars&q…

Adrian avatar
Adrian

Ye, I need to figure out how to get rid of var files inlcuded in every apply. I have region.tfvars and env.tfvars included in tree structure

Adrian avatar
Adrian
From what I can tell on Terragrunt 0.21.11, this is not an issue. When you call apply with a plan file, it removes all arguments starting with "-var" or "-var-file". This is a problem for me because I pass ["-var", "foo=bar"] and it ends up calling terraform with just foo=bar, so I have to use the form -var=foo=bar to make it correctly remove that when the apply is called with a plan.

?

David avatar
David

what does your terraform block look like now?

Adrian avatar
Adrian
terraform {
  extra_arguments "common_vars" {
    commands = get_terraform_commands_that_need_vars()

    optional_var_files = [
      "${get_parent_terragrunt_dir()}/terraform.tfvars",
      "${get_parent_terragrunt_dir()}/common.tfvars",
      "${find_in_parent_folders("region.tfvars", "skip-env-if-does-not-exist")}",
      "${find_in_parent_folders("env.tfvars", "skip-env-if-does-not-exist")}",

    ]
  }

  extra_arguments "init-no-color" {
    commands = ["init"]

    arguments = ["-no-color"]
  }

  extra_arguments "disable_input" {
    commands  = get_terraform_commands_that_need_input()
    arguments = ["-input=false"]
  }

  before_hook "copy_common_main_providers" {
    commands = ["init-from-module"]
    execute  = ["cp", "${get_parent_terragrunt_dir()}/common/main_providers.tf", "."]
  }

  after_hook "remove_useless_copy_of_main_providers" {
    commands = ["init"]
    execute  = ["rm", "-f", "${get_parent_terragrunt_dir()}/${path_relative_to_include()}/main_providers.tf"]
  }
}
Adrian avatar
Adrian

terragrunt should remove var or var-file when apply with plan

2020-06-01

Brij S avatar
Brij S

does anyone know how one is able to run cli commands using after_hook? Im trying the following

  after_hook "after_hook" {
    commands     = ["apply"]
    execute      = ["aws", "s3 ls"]
  }

but I get the following error

[terragrunt] 2020/06/01 10:03:21 Detected 1 Hooks
[terragrunt] 2020/06/01 10:03:21 Executing hook: after_hook
[terragrunt] 2020/06/01 10:03:21 Running command: aws s3 ls
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument command: Invalid choice, valid choices are:

seems its not recognizing the command

loren avatar
loren

because you ran aws "s3 ls" and not aws s3 ls

  after_hook "after_hook" {
    commands     = ["apply"]
    execute      = ["aws", "s3", "ls"]
  }
Brij S avatar
Brij S

:sweat_smile: ah, yeah thats it - one more question for you if you dont mind! I’m trying to run the following cli command to tag the state bucket thats created

    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", "TagSet=[ for key, val in local.tags: { \"Key\" = key, \"Value\" = val } ]"]

but it seems I cant do the interpolation there :thinking_face: am I missing anything here or can I not use local.tags here

loren avatar
loren

create the tag-string in another local

loren avatar
loren

then use

    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", "TagSet=${local.tag_set}"]
Brij S avatar
Brij S

I added

tag_set = [ for key, val in local.tags: { "Key" = key, "Value" = val } ]

as a local and ran again with your suggestion but got

Invalid template interpolation value; Cannot include the given value in a string template: string required.
loren avatar
loren

because you didn’t create a string, you created a list of maps

Brij S avatar
Brij S

oh, right - the awscli docs requires a list of maps https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-tagging.html

loren avatar
loren

well and it requires json not hcl

loren avatar
loren

try to jsonencode it…

    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", "TagSet=${jsonencode(local.tag_set)}"]
Brij S avatar
Brij S

ah was just about to try that

Brij S avatar
Brij S

ahh so close! looks like jsonencode turns = into : !

Error parsing parameter '--tagging': Expected: '=', received: '"' for input:
TagSet=[{"Key":"abc:account:accountid","Value":"123456789012"},{"Key":"abc:business:cost","Value":"0000"}]
Brij S avatar
Brij S

any ideas on how to preserve the =

loren avatar
loren

Don’t. Just use the json syntax. The TagSet key needs to be part of the json dictionary… https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-tagging.html

Brij S avatar
Brij S

what do you mean?

Brij S avatar
Brij S

my string looks exactly like the example

TagSet=[{Key=string,Value=string},{Key=string,Value=string}]
loren avatar
loren

Look at the json syntax

loren avatar
loren
locals {
  tag_set = {
    TagSet = [ for key, val in local.tags: { "Key" = key, "Value" = val } ]
  }
}
loren avatar
loren
    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", jsonencode(local.tag_set)]
Brij S avatar
Brij S

oh man! that did it!

Brij S avatar
Brij S

thank you

    keyboard_arrow_up