#terragrunt (2020-06)

terragrunt

Terragrunt discussions

Archive: https://archive.sweetops.com/terragrunt/

2020-06-01

Brij S avatar

does anyone know how one is able to run cli commands using after_hook? Im trying the following

  after_hook "after_hook" {
    commands     = ["apply"]
    execute      = ["aws", "s3 ls"]
  }

but I get the following error

[terragrunt] 2020/06/01 10:03:21 Detected 1 Hooks
[terragrunt] 2020/06/01 10:03:21 Executing hook: after_hook
[terragrunt] 2020/06/01 10:03:21 Running command: aws s3 ls
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: argument command: Invalid choice, valid choices are:

seems its not recognizing the command

loren avatar

because you ran aws "s3 ls" and not aws s3 ls

  after_hook "after_hook" {
    commands     = ["apply"]
    execute      = ["aws", "s3", "ls"]
  }
Brij S avatar

:sweat_smile: ah, yeah thats it - one more question for you if you dont mind! I’m trying to run the following cli command to tag the state bucket thats created

    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", "TagSet=[ for key, val in local.tags: { \"Key\" = key, \"Value\" = val } ]"]

but it seems I cant do the interpolation there :thinking_face: am I missing anything here or can I not use local.tags here

loren avatar

create the tag-string in another local

loren avatar

then use

    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", "TagSet=${local.tag_set}"]
Brij S avatar

I added

tag_set = [ for key, val in local.tags: { "Key" = key, "Value" = val } ]

as a local and ran again with your suggestion but got

Invalid template interpolation value; Cannot include the given value in a string template: string required.
loren avatar

because you didn’t create a string, you created a list of maps

loren avatar

well and it requires json not hcl

loren avatar

try to jsonencode it…

    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", "TagSet=${jsonencode(local.tag_set)}"]
Brij S avatar

ah was just about to try that

Brij S avatar

ahh so close! looks like jsonencode turns = into : !

Error parsing parameter '--tagging': Expected: '=', received: '"' for input:
TagSet=[{"Key":"abc:account:accountid","Value":"123456789012"},{"Key":"abc:business:cost","Value":"0000"}]
Brij S avatar

any ideas on how to preserve the =

loren avatar

Don’t. Just use the json syntax. The TagSet key needs to be part of the json dictionary… https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-tagging.html

Brij S avatar

what do you mean?

Brij S avatar

my string looks exactly like the example

TagSet=[{Key=string,Value=string},{Key=string,Value=string}]
loren avatar

Look at the json syntax

loren avatar
locals {
  tag_set = {
    TagSet = [ for key, val in local.tags: { "Key" = key, "Value" = val } ]
  }
}
loren avatar
    execute      = ["aws", "s3api", "put-bucket-tagging", "--bucket", "my-bucket", "--tagging", jsonencode(local.tag_set)]
Brij S avatar

oh man! that did it!

Brij S avatar

thank you

2020-06-04

Adrian avatar
Calling apply does not remove `-var` arguments if the actual variables are a separate argument · Issue #1035 · gruntwork-io/terragrunt

I am using Terragrunt v0.21.11 If I say: terraform { extra_arguments &quot;common_vars&quot; { commands = get_terraform_commands_that_need_vars() arguments = [ &quot;-var&quot;, foo=bar&quot;, ] } …

David avatar

This seems related to https://github.com/gruntwork-io/terragrunt/issues/983, which I issued a long ways back.

Terragrunt doesn’t discriminate between apply calls with vs without a planfile, and doing so seems non-trivial

At my company the problem disappeared when we switched from using local secrets files to fetching secrets from Vault

Add support to include tfvars only for `apply`s without plan output files · Issue #983 · gruntwork-io/terragrunt

Right now, writing code such as terraform { extra_arguments &quot;common_var&quot; { commands = get_terraform_commands_that_need_vars() arguments = [&quot;-var-file=${get_aws_account_id()}.tfvars&q…

Adrian avatar

Ye, I need to figure out how to get rid of var files inlcuded in every apply. I have region.tfvars and env.tfvars included in tree structure

Adrian avatar
From what I can tell on Terragrunt 0.21.11, this is not an issue. When you call apply with a plan file, it removes all arguments starting with "-var" or "-var-file". This is a problem for me because I pass ["-var", "foo=bar"] and it ends up calling terraform with just foo=bar, so I have to use the form -var=foo=bar to make it correctly remove that when the apply is called with a plan.

?

David avatar

what does your terraform block look like now?

Adrian avatar
terraform {
  extra_arguments "common_vars" {
    commands = get_terraform_commands_that_need_vars()

    optional_var_files = [
      "${get_parent_terragrunt_dir()}/terraform.tfvars",
      "${get_parent_terragrunt_dir()}/common.tfvars",
      "${find_in_parent_folders("region.tfvars", "skip-env-if-does-not-exist")}",
      "${find_in_parent_folders("env.tfvars", "skip-env-if-does-not-exist")}",

    ]
  }

  extra_arguments "init-no-color" {
    commands = ["init"]

    arguments = ["-no-color"]
  }

  extra_arguments "disable_input" {
    commands  = get_terraform_commands_that_need_input()
    arguments = ["-input=false"]
  }

  before_hook "copy_common_main_providers" {
    commands = ["init-from-module"]
    execute  = ["cp", "${get_parent_terragrunt_dir()}/common/main_providers.tf", "."]
  }

  after_hook "remove_useless_copy_of_main_providers" {
    commands = ["init"]
    execute  = ["rm", "-f", "${get_parent_terragrunt_dir()}/${path_relative_to_include()}/main_providers.tf"]
  }
}
Adrian avatar

terragrunt should remove var or var-file when apply with plan

2020-06-08

muhaha avatar

Is possible to get path ( absolute ) if I am using terragrunt --terragrunt-working-dir $mydir ? For example: Terragrunt is triggered from /tmp/ and is using --terragrunt-working-dir foo/bar/project I need relative path like foo/bar/project

In fact foo/bar/project is located in /tmp, but on other enviroments, this path may be different

loren avatar

i’m confused a bit, it sounds like you are looking for a relative path, but you are asking about an absolute path

muhaha avatar

*relative, sorry

loren avatar

get_terragrunt_dir() will give you the absolute path

loren avatar

you might be able to do it with some creative use of path_relative_to_include()

loren avatar
Built-in functions

Terragrunt allows you to use built-in functions anywhere in terragrunt.hcl, just like Terraform.

muhaha avatar

I am not using include

muhaha avatar

path_relative_to_include() will always return .

muhaha avatar

maybe run_cmd() ?

loren avatar

lolol, all part of what i meant by “creative use”

loren avatar

sometimes you have to adjust how you have things setup, in order to use the features a tool offers…

muhaha avatar

well, its should be possible to do: run_cmd("echo","${get_terragrunt_dir()#pwd}") , does not work yet .. :X

Adrian avatar

Where do you put your provider settings?

provider "aws" {
  region  = var.aws_region

  # Make it faster by skipping something
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_credentials_validation = true
}

also this setting?

terraform {
  backend "s3" {}
}

What the best way and how do you handle this? I’ve created terragrunt.tf in each module with both of this settings. Is this correct way? Previously I was using hook to copy main_providers.tf

provider "aws" {
  region  = var.aws_region

  # Make it faster by skipping something
  skip_get_ec2_platforms      = true
  skip_metadata_api_check     = true
  skip_region_validation      = true
  skip_credentials_validation = true
}

provider "aws" {
  alias   = "virginia"
  region  = "us-east-1"
}

terraform {
  backend "s3" {}
  required_version = ">= 0.12.0"
}

variable "aws_region" {
  type        = string
  default     = "eu-central-1"
  description = "AWS region to use for all resources"
}
Adrian avatar
gruntwork-io/terragrunt-infrastructure-live-example

A repo used to show examples file/folder structures you can use with Terragrunt and Terraform - gruntwork-io/terragrunt-infrastructure-live-example

loren avatar

These days we use the new generate block…

1
loren avatar

Yep, that file is a good example

2020-06-16

Adrian avatar

What’s do you think about that type of approach to put this data in input section of HCL file?

  custom_task_policy_statement = [
    {
      actions = [
        "s3:GetObject",
        "s3:ListBucketVersions",
        "s3:ListBucket",
        "s3:GetObjectTagging",
        "s3:GetBucketVersioning",
        "s3:GetBucketAcl",
        "s3:GetBucketCORS",
        "s3:GetBucketLocation",
        "s3:GetObjectVersion",
        "s3:PutObject",
        "s3:GetAccountPublicAccessBlock",
        "s3:ListAllMyBuckets",
        "s3:HeadBucket"
      ]
      effect = "Allow"
      resources = [
        dependency.s3.outputs.this_s3_bucket_arn,
        "${dependency.s3.outputs.this_s3_bucket_arn}/*"
      ]
    },
    {
      actions = [
        "kms:DescribeKey",
        "kms:GenerateDataKey*",
        "kms:Encrypt",
        "kms:ReEncrypt*",
        "kms:Decrypt"
      ]
      effect    = "Allow"
      resources = [dependency.kms.outputs.key_arn]
    }
  ]
David avatar

I usually either: • ask for a string policy, which can be made in inputs with jsonencode

• ask for an array of existing IAM Policy arns Your approach here could definitely work, but I’d make sure it works with things like condition clauses. Terraform’s typing isn’t great for working with maps of semi-structured data, but it can work

joshmyers avatar
joshmyers

I have kinda similar to this

joshmyers avatar
joshmyers

Without the dependency usage

2020-06-18

Brij S avatar

Hey all, I was wondering if theres any way to combine tags from the child into the parent hcl? For example, I’ve got the following in the my parent hcl file

locals {
  env           = element(split("/", local.relative_path), 1)
  base_tags = {
    "accountid"      = get_aws_account_id(),
    "environment"    = local.env,
  }
}

and in my child hcl file I have

locals {
  # This loads base tags from parent hcl and adds additional ones

  # Ideally I want to do the opposite - define some tags here, and combine
  # them in the parent hcl so that we have access to the full set of tags there
  # to use for s3_bucket_tags and dynamodb_table_tags
  # I'm not sure if/how that's possible
  # See <https://terragrunt.gruntwork.io/docs/features/locals/>
  # Their solution with yaml doesn't help us because 2 of the 3 base tags are
  # dynamically computed

  base_config = read_terragrunt_config(find_in_parent_folders())

  tags = merge(local.base_config.locals.base_tags, {
    "abc" = "0123"
    "dept"  = "engineering"
  })
}

any ideas

Adrian avatar

So you want in your child merged tags from parent and child?

Brij S avatar

other way around

Adrian avatar

you can merge multiple values from mutliple level

locals {
  base_tags = {
    "accountid"      = "123123123",
    "environment"    = "test",
  }
  other_tags = {
    "test" = "asd"
  }
}

output "tags" {
  value = merge(local.base_tags, local.other_tags)
}
Brij S avatar

I understand that, what I want is to take the tags declared in a child.hcl and use them in the parent.hcl

Brij S avatar

two different files on different levels

2020-06-19

2020-06-22

2020-06-24

muhaha avatar

Guys, I need help: /project1/terragrunt.hcl file contains ( its main file ):

include {
  path = find_in_parent_folders("templates/aws.hcl")
}

/templates/aws.hcl file contains:

terraform {}

remote_state {
    disable_init = tobool(get_env("TERRAGRUNT_DISABLE_INIT", "false"))
    backend = "s3"
      config = {
      bucket = "${local.bucket}"
      region = "${local.backend_region}"
      key    = "${get_env("PPATH")}/terraform.tfstate"
      encrypt = true
      dynamodb_table = "${local.dynamodb_table}"
    }
}

locals {
  bucket = "${get_env("bucket")}"
  backend_region = "${get_env("backend_region")}"
  dynamodb_table = "${get_env("dynamodb_table")}"
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents = <<EOF
provider "aws" {
  version = "2.67.0"
  region = ???
}
EOF
}

generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite"
  contents = <<EOF
terraform {
  backend "s3" {}
}
EOF

How can I set region in /project1/terragrunt.hc with minimum effort?

• its not possible to set it via locals

• maybe with env_vars ? Thanks

David avatar

Yeah, the locals block their is scoped to that terragrunt file, while the provider block inside the generate block is injected as a terraform file, so you can’t directly use any locals there.

I would suggest using templatefile, and injecting in get_env("backend_region") in the region = ??? spot

muhaha avatar

templatefile ? which one? I dont follow

David avatar

No worries! I was suggesting you use this function here: https://www.terraform.io/docs/configuration/functions/templatefile.html

So I’d add a /templates/provider.tf file with contents:

provider "aws" {
  version = "2.67.0"
  region = ${REGION}
}

and then in your /templates/aws.hcl file I would change the generate block to be:

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = templatefile("./provider.tf", {
    REGION = get_env("backend_region")
  })
}

The other option is to just not specify region at all in your provider, and terraform will default to the AWS_REGION env var

templatefile - Functions - Configuration Language - Terraform by HashiCorp

The templatefile function reads the file at the given path and renders its content as a template.

muhaha avatar

No, thats not a problem.. I dont want to preset region with ENV var, I just want pure gitops, but it should not be hardcoded in template, which is shared … So something like this:

/project1/terragrunt.hcl file contains ( its main file ):

include {
  path = find_in_parent_folders("templates/aws.hcl")
}

locals {
  region = "eu-central-0"
}

/templates/aws.hcl file contains:

terraform {}

remote_state {
    disable_init = tobool(get_env("TERRAGRUNT_DISABLE_INIT", "false"))
    backend = "s3"
      config = {
      bucket = "${local.bucket}"
      region = "${local.backend_region}"
      key    = "${get_env("PPATH")}/terraform.tfstate"
      encrypt = true
      dynamodb_table = "${local.dynamodb_table}"
    }
}

locals {
  bucket = "hardcoded"
  backend_region = "hardcoded"
  dynamodb_table = "hardcoded"
}

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents = <<EOF
provider "aws" {
  version = "2.67.0"
  region = ${local.region}"
}
EOF
}

generate "backend" {
  path      = "backend.tf"
  if_exists = "overwrite"
  contents = <<EOF
terraform {
  backend "s3" {}
}
EOF
David avatar

Ah, that is not possible. Terragrunt scopes locals to just the file you are working in, unfortunately. I’m not sure if there’s a bug or not for it, but you can’t to parent -> child or child -> parent locals through include blocks if I remember right

loren avatar

it feels backwards from the terragrunt paradigm to try to pass values from the child config to the parent config

loren avatar

you can do things like define common values in a .yml file and read them with yamldecode(), or keep them in a “common” terragrunt config and use read_terragrunt_config()https://terragrunt.gruntwork.io/docs/reference/built-in-functions/#read_terragrunt_config

Built-in functions

Terragrunt allows you to use built-in functions anywhere in terragrunt.hcl, just like Terraform.

muhaha avatar

@loren like this https://pastebin.com/raw/Q7UzBybA ? Will it work? :X

loren avatar

i still think you have the paradigm backwards, and need to flip things around. what you are calling a “template” is what terragrunt refers to as a “parent”, and the relationship works the other way around

loren avatar

at least, for the “include” functionality

loren avatar

i’m not really sure how to do what you want, exactly how you want it. i’m just pointing you towards the tools that might get you to an equivalent setup. but it probably requires rethinking your approach

muhaha avatar

tools?

loren avatar

functions as tools, in this case

2020-06-25

Tim Birkett avatar
Tim Birkett

I’m using terragrunt heavily at the moment but one thing is eluding me… During development of modules I pin to a branch and sometimes branches of Terraform modules under that. Currently, I’m deleting the .terragrunt-cache before every apply which is a bit frustrating, is there anyway to force the changes to be pulled in rather than deleting the cached modules every time I want to apply changes? Maybe there’s soe magic option I’ve missed, re-running an “init” doesn’t pull the newest changes to a branch in either

loren avatar

pass the flag --terragrunt-source-update and it will automatically delete the cache for that config

Sam Holton avatar
Sam Holton

I normally use https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-source if I’ve actively developing a module, that way you don’t even have to commit/push to test it

CLI options

Terragrunt forwards all arguments and options to Terraform. Learn more about CLI options in Terragrunt.

1
Tim Birkett avatar
Tim Birkett

Thanks @loren @Sam Holton - TERRAGRUNT_SOURCE=... works a charm

1
    keyboard_arrow_up