#terraform (2021-05)

terraform Discussions related to Terraform or Terraform Modules

Archive: https://archive.sweetops.com/terraform/

2021-05-11

Jason avatar
Jason

Hi all I’m using terraform-aws-transit-gateway (https://github.com/cloudposse/terraform-aws-transit-gateway) to create TGW and share it with external principals.

I faced an issue when sharing TGW with external principal as below

rror: error reading EC2 Transit Gateway: InvalidTransitGatewayID.NotFound: Transit Gateway tgw-090ff1710310403a7 was deleted or does not exist.
        status code: 400, request id: 836b5c87-7b76-44f9-b318-f1fbf47fa785

  on ../../../modules/tgw/main.tf line 49, in data "aws_ec2_transit_gateway" "this":
  49: data "aws_ec2_transit_gateway" "this" {

The reason is this module has a check

data "aws_ec2_transit_gateway" "this" {
  id = local.transit_gateway_id
}

As you may know, for external principal, it needs to be accepted from second aws account then it can be seen and define as data_source

My question is whether we have timeout/delay in data_source or dependency to wait for the accepter to accept the sharing, then it can process for next steps. ?

Thanks everyone

cloudposse/terraform-aws-transit-gateway attachment image

Terraform module to provision AWS Transit Gateway, AWS Resource Access Manager (AWS RAM) Resource, and share the Transit Gateway with the Organization or another AWS Account. - cloudposse/terraform…

2021-05-10

François Davier avatar
François Davier

Hi, want to use https://registry.terraform.io/modules/cloudposse/cloudwatch-events/aws/latest. Target is to monitor if aws backup job is ok/ko when copying restauration point from source region to vault target region. Have you some example please ? Thank you

Rhys Davies avatar
Rhys Davies

Hey can anyone recommend an article or series or blog post about how to correctly structure the layers in a terraform project

2
David Morgan avatar
David Morgan

hi i am trying to use terraform-aws-modules/dynamodb-table/aws 0..13.0 when i specify ttl_attribute = “ttl” i get the following message

An argument named "ttl_attribute" is not expected here

this is what i’m an trying to run

module "cache_dynamo_table_forum_post_count" {
  source  = "terraform-aws-modules/dynamodb-table/aws"
  version = "0.13.0"

  name      = "mytable_name"
  hash_key  = "my_id"
  billing_mode   = "PAY_PER_REQUEST"
  ttl_attribute = "ttl"
}

thanks

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

ttl_atribute -> ttl_attribute_name

Oliver avatar
Oliver

Hi David, I thinnk the variable name is ttl_attribute_name

Oliver avatar
Oliver

oh you beat me to it

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

The fastest in the West

1
David Morgan avatar
David Morgan

verified - thanks for the quick response!

2021-05-09

mcseoliver avatar
mcseoliver

Hi everyone

Building a VPC with 3 subnets using terraform, I have some more to add to it but it’s want to know if anyone has a ready receipt so I can check to see any mistakes I may made

2021-05-08

Alec Fong avatar
Alec Fong
Hello! What’s the differences between [terraform-aws-multi-az-subnets> and <https://github.com/cloudposse/terraform-aws-dynamic-subnets terraform-aws-dynamic-subnets](https://github.com/cloudposse/terraform-aws-multi-az-subnets)? Why/when would I use one over the other?
cloudposse/terraform-aws-multi-az-subnets attachment image

Terraform module for multi-AZ public and private subnets provisioning - cloudposse/terraform-aws-multi-az-subnets

cloudposse/terraform-aws-dynamic-subnets attachment image

Terraform module for public and private subnets provisioning in existing VPC - cloudposse/terraform-aws-dynamic-subnets

Alec Fong avatar
Alec Fong

Ah I see multi-az is more explicit in defining each public and private subnets where dynamic creates both subnets for you.

cloudposse/terraform-aws-multi-az-subnets attachment image

Terraform module for multi-AZ public and private subnets provisioning - cloudposse/terraform-aws-multi-az-subnets

cloudposse/terraform-aws-dynamic-subnets attachment image

Terraform module for public and private subnets provisioning in existing VPC - cloudposse/terraform-aws-dynamic-subnets

Alec Fong avatar
Alec Fong

If starting fresh would there be any benefits to using multi-az?

jose.amengual avatar
jose.amengual

we usually use dynamic subnets

1
Brij S avatar
Brij S

Hey all, I’m trying to setup EKS with managed node groups with the following config

module "vpc" {
  source = "terraform-aws-modules/vpc/aws"

  name = "eks-vpc"
  cidr = "172.21.0.0/16"

  azs            = [data.aws_availability_zones.available.names[0], data.aws_availability_zones.available.names[1]]
  public_subnets = ["172.21.0.0/20", "172.21.16.0/20"]

  enable_nat_gateway                = false
  enable_vpn_gateway                = true
  propagate_public_route_tables_vgw = true

  tags = merge(var.tags, {
    "kubernetes.io/cluster/eks" = "shared"
  })

  public_subnet_tags = merge(var.tags, {
    "kubernetes.io/cluster/eks" = "shared"
  })
}

module "eks" {
  source                          = "terraform-aws-modules/eks/aws"
  version                         = "15.2.0"
  cluster_name                    = "eks"
  cluster_version                 = "1.19"
  subnets                         = module.vpc.private_subnets
  vpc_id                          = module.vpc.vpc_id
  cluster_enabled_log_types       = ["scheduler"]
  tags                            = var.tags
  cluster_endpoint_private_access = true

  node_groups_defaults = {
    ami_type  = "AL2_x86_64"
    disk_size = 20
    # subnets   = module.vpc.private_subnets
  }
  node_groups = {
    gitlab-eks = {
      name             = "gitlab-eks"
      desired_capacity = 3
      max_capacity     = 5
      min_capacity     = 3
      instance_types   = ["t3.2xlarge"]
      capacity_type    = "ON_DEMAND"
    }
  }
}

However, I keep running into the following error;

Error: List shorter than MinItems
  on .terraform/modules/eks/modules/node_groups/node_groups.tf line 8, in resource "aws_eks_node_group" "workers":
   8:   subnet_ids    = each.value["subnets"]
Attribute supports 1 item minimum, config has 0 declared

Has anyone else run into this? I’ve looked in the eks module issues and havent found anything and I also tried adding/removing the subnet in the nodegroup defaults, with no success

loren avatar
loren

It looks like you’re creating public subnets in the vpc module, but using private subnets in the eks module, so subnets is indeed an empty list

Brij S avatar
Brij S

oh man, good eye! thanks

1

2021-05-07

Paul Robinson avatar
Paul Robinson

Hi @Matt Gowie I’ve just joined following a couple of PRs to the terraform-aws-multi-az-subnets module. I have a question about this if you can explain please? https://github.com/cloudposse/terraform-aws-multi-az-subnets/pull/48#pullrequestreview-649820152
We are not going to support the use case of private subnets without NAT gateways, at least not in this module.
I saw you reviewed the follow up PR #50. Is there any contextual design discussion that I can read up on please?

Fix nat_gateway_enabled=false Invalid index error by paulrob-100 · Pull Request #48 · cloudposse/terraform-aws-multi-az-subnets attachment image

what Fix #44 Same test from #45 applied and still fails Readded test after merge of #47 and using us-east-2 as advised Tried to retain the output_map AZ => null design choice when nat_gateway_e…

Matt Gowie avatar
Matt Gowie

Hey @ — I don’t believe that was discussed. More just a unilateral decision. But your point about gateway load balancers in your final comment seems like a valid one one to me.

@Jeremy (Cloud Posse) can you please review Paul’s comment and discuss? If the functionality is disabled by default and it’s supporting a valid use-case with newer AWS patterns then I don’t see why we would turn away an eager contributor who is willing to implement.

Fix nat_gateway_enabled=false Invalid index error by paulrob-100 · Pull Request #48 · cloudposse/terraform-aws-multi-az-subnets attachment image

what Fix #44 Same test from #45 applied and still fails Readded test after merge of #47 and using us-east-2 as advised Tried to retain the output_map AZ => null design choice when nat_gateway_e…

Paul Robinson avatar
Paul Robinson

Thanks both. Yeah it was one of the reasons for choosing this module.

Private subnets without routes to nat gateways are standard with the advent of transit gateway and the gateway load balancer.

Linking again to an aws blog with reference VPCs. https://aws.amazon.com/blogs/networking-and-content-delivery/introducing-aws-gateway-load-balancer-supported-architecture-patterns/

It doesn’t seem like the existing module is incompatible here.

Introducing AWS Gateway Load Balancer: Supported architecture patterns | Amazon Web Services attachment image

Customers often ask me how they can maintain consistent policies and practices as they move to the cloud, especially as it relates to using the network appliances. They trust third-party hardware and software appliances to protect and monitor their on-premises traffic, but traditional appliance deployment models are not always well suited to the cloud. Last […]

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@ Thank you for bringing this up.

First, this pattern of using a Gateway Load Balancer appliance is new to me, and AFAIK, new to Cloud Posse. In general, as the saying goes, we like to “eat our own dog food”, which means we only publish modules and features that we use so we can design the features appropriately and ensure that the module works in practice. This is not a hard-and-fast rule, but rather a reflection of our values. We do often accept contributions to modules that add features we have not used and do not plan on using, and we are grateful for the community support in enhancing the modules to be more useful to more people.

We recently had an internal discussion on the topic of private subnets without gateways due to a different PR and decided not to support them, because it adds a surprising amount of complexity to a module to make them completely optional. This was also in part due to the fact that we had never seen a use case for them and did not contemplate one, so this was not dog food we were ever going to eat. Thank you for educating me on this emerging use case; I will keep it in mind in future PR reviews.

Also, it was not at all clear to me that part of the enhancement you were seeking with your PR was enabling private subnets without NAT. I did not think it was important to you, I thought you were just trying to generalize.

We have 3 modules for creating subnets: • terraform-aws-named-subnetsterraform-aws-dynamic-subnetsterraform-aws-multi-az-subnets As far as I can recall, for the past 2 years we have only used terraform-aws-dynamic-subnets in client engagements, which makes me personally (and this is definitely me and not Erik or Cloud Posse in general) less interested in maintaining or enhancing the other 2 modules, because we have limited resources and are having trouble keeping up with all the PRs across all our modules, so I would rather we not try to keep 3 modules whose differences are difficult to articulate. (See, for comparison, our decision to deprecate and eventually freeze terraform-terraform-label> in favor of <https://github.com/cloudposse/terraform-null-label|terraform-null-label ).

Furthermore, when it comes to creating subnets, if you are only creating private subnets without gateways, I personally (and again, not speaking for Cloud Posse) do not see much point in using a Terraform module. It is easy enough to do just using the AWS provider directly and the Terraform built-in function cidrsubnet.

So that is the long story behind the terse
We are not going to support the use case of private subnets without NAT gateways, at least not in this module.
I suggest you look at terraform-aws-named-subnets if you want to create private subnets without gateways. If that doesn’t work for you, then we can discuss what you need and how to get it done in the most appropriate way.

cloudposse/terraform-aws-named-subnets attachment image

Terraform module for named subnets provisioning. Contribute to cloudposse/terraform-aws-named-subnets development by creating an account on GitHub.

cloudposse/terraform-aws-dynamic-subnets attachment image

Terraform module for public and private subnets provisioning in existing VPC - cloudposse/terraform-aws-dynamic-subnets

cloudposse/terraform-aws-multi-az-subnets attachment image

Terraform module for multi-AZ public and private subnets provisioning - cloudposse/terraform-aws-multi-az-subnets

cloudposse/terraform-terraform-label attachment image

Terraform Module to define a consistent naming convention by (namespace, stage, name, [attributes]) - cloudposse/terraform-terraform-label

cloudposse/terraform-null-label attachment image

Terraform Module to define a consistent naming convention by (namespace, stage, name, [attributes]) - cloudposse/terraform-null-label

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)
Do not require NAT gateway IDs for private subnets by Nuru · Pull Request #51 · cloudposse/terraform-aws-multi-az-subnets attachment image

what Do not require NAT gateway IDs for private subnets why Users should be able to create subnets without NAT gateways Implemented in #48, which was closed in favor of #50, but #50 left this fe…

Paul Robinson avatar
Paul Robinson

Thanks @Jeremy (Cloud Posse) for your detailed and insightful response and also your PR.

I can appreciate the pain of maintaining so many modules and the need to deprecate the lesser used/older ones. I also appreciate the time you spend building community and encouraging contributions :raised_hands:

I found a few people asking about the differences between the various subnets modules in this channel. Indeed, there is an open issue on the terraform-aws-multi-az-subnets module. Perhaps I can add my thinking which informed my choice of module.

I did also consider the terraform-aws-dynamic-subnets and terraform-aws-named-subnets modules, but found the former to be simplistic for my needs and the latter to be limited to a single AZ. It does seem at least 2 modules are needed here.

The low barrier to entry terraform-aws-dynamic-subnets module is perfect as a starter/utility/shared services VPC . It splits the VPC CIDR range equally into public and private subnets. This means you are likely to choose a larger CIDR range due to generally needing more ips in the private subnet ranges. Also if you use VPC endpoints, they would probably be located in the public subnets since there are likely spare ips in those subnets.

However I like to use smaller VPC CIDR ranges and make best use of the available CIDR range by having smaller public subnets and relatively larger private subnets. I usually split the private subnets into app/data and a separate one for vpc endpoints. This is a safety design such that, for example, scaling lambda VPC ips cannot exhaust the data subnet range. Also a security design since AWS service traffic and gateway traffic is routed internally to the AWS backbone (which can also lead to a design with no NAT gateway as with this thread). Also a separate VPC endpoints subnet allows NACL rules and/or security groups on the endpoints. For that VPC design, the terraform-aws-multi-az-subnets is perfect.

As to the other benefits of the cloudposse modules, there’s the tagging support and features of the terraform-null-label, the clearly thoughtful implementation and versioning, the CI github actions/PR process, community and terraform registry. Many reasons to contribute rather than roll my own .

I also note the similarities between multi-az-subnets and dynamic-subnets. Your recent switch to using for_each from count in multi-az-subnets is a good example of an important contribution to robustness btw . I can see how some of these PRs would ideally be done on each subnet module (eg/ #49).

I did also consider the named-subnets module but would have had to use for_each at the module level (one for each AZ), and I saw the multi-az-subnets properly calculates the subnet ranges for n AZs for the user, so multi-az-subnets was the winner for our needs.

All considered, it’s difficult to see how to reduce the number of modules to reduce maintenance burden. Perhaps if dynamic-subnets was extended to support multiple categories of private subnets and varying subnet cidr ranges per category, possibly using a sub-module? And make the public subnet optional? And incorporate the named subnets differences if any. This might complicate the interface tbh. I like the simplicity of dynamic-subnets, which I’m sure contributes to its popularity.

Thanks again!

Document the difference between this module and terraform-aws-dynamic-subnets · Issue #23 · cloudposse/terraform-aws-multi-az-subnets attachment image

This org offers two repo&#39;s with Terraform modules that almost seem to do similar things: https://github.com/cloudposse/terraform-aws-dynamic-subnets and this repo. Is it an idea to document wha…

Added Assign Public IP on Launch by nadenf · Pull Request #49 · cloudposse/terraform-aws-multi-az-subnets attachment image

what Provides option to auto-assign public IP addresses for public subnets. why EKS needs auto-assign public IP enabled for public subnets.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Thank you @ for your review of the 3 modules and the differences between them. I would appreciate it if you would edit it slightly and post it as a comment to https://github.com/cloudposse/terraform-aws-multi-az-subnets/issues/23 to help us get started on the requested documentation.

Thank you also for educating me about uses of subnets without gateways.

Document the difference between this module and terraform-aws-dynamic-subnets · Issue #23 · cloudposse/terraform-aws-multi-az-subnets attachment image

This org offers two repo&#39;s with Terraform modules that almost seem to do similar things: https://github.com/cloudposse/terraform-aws-dynamic-subnets and this repo. Is it an idea to document wha…

Paul Robinson avatar
Paul Robinson

great yes I think that I can help there

managedkaos avatar
managedkaos

anyone used this app to go from infra back to code? I’ve used terraformer but this looks slicker since it generates TF, CFN, CDK, and even Pulumi

https://former2.com/

Former2 attachment image

Convert your existing cloud resources into CloudFormation / Terraform / Troposphere

jose.amengual avatar
jose.amengual

only Terraformer from google

Former2 attachment image

Convert your existing cloud resources into CloudFormation / Terraform / Troposphere

1
Sachin c avatar
Sachin c

Hi Team, I was trying to use latest version of cloudposse/ec2-autoscale-group/aws module and found cloudwatch alarm name is duplicating in default alarms.

Expected behavior:

  # module.autoscale_group.aws_cloudwatch_metric_alarm.all_alarms["cpu_high"] will be created
  + resource "aws_cloudwatch_metric_alarm" "all_alarms" {
      + actions_enabled                       = true
      + alarm_actions                         = (known after apply)
      + alarm_description                     = "Scale up if CPU utilization is above 70 for 120 seconds"
      + alarm_name                            = "appname-prod-backend-cpu-utilization-high"

Actual Result:

  # module.autoscale_group.aws_cloudwatch_metric_alarm.all_alarms["cpu_high"] will be created
  + resource "aws_cloudwatch_metric_alarm" "all_alarms" {
      + actions_enabled                       = true
      + alarm_actions                         = (known after apply)
      + alarm_description                     = "Scale up if CPU utilization is above 70 for 120 seconds"
      + alarm_name                            = "appname-prod-backend-appname-prod-backend-cpu-utilization-high"

2021-05-06

SecOH avatar
SecOH

Hello, guys. I have a question. Is it mandatory that amazonmq(rabbitmq) security group’s egress rule should allow all outbound traffic? (egress 0.0.0.0/0) https://github.com/cloudposse/terraform-aws-mq-broker/blob/3951c8e1cf4faf94c3c92b2b01d26b078bc60d88/sg.tf#L8 If I create security group for mq, the egress rule must be added to my SG.

cloudposse/terraform-aws-mq-broker attachment image

Terraform module for provisioning an AmazonMQ broker - cloudposse/terraform-aws-mq-broker

Matt Gowie avatar
Matt Gowie

@ not totally sure. I’m not sure if the AMQ service requires an outside connection to do updates or similar… but I would guess so. If you want to put up a PR to add an additional _enabled bool flag which disabling adding the egress rule then we’d be happy to review and get merged I’m sure.

cloudposse/terraform-aws-mq-broker attachment image

Terraform module for provisioning an AmazonMQ broker - cloudposse/terraform-aws-mq-broker

SecOH avatar
SecOH

@Matt Gowie Happy to get your reply, Thank you. I will make a PR if I get some free time.!

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

If any of y’all use GitHub and you want to see Dependabot support for Terraform, there is a way to help: https://github.com/dependabot/dependabot-core/issues/1176#issuecomment-833383992

Tl;dr: dependabot is working on HCL2/tf0.14/tf0.15 support and is asking for any people with public repos interested in testing

1
1
greg n avatar
greg n

Hello guy, I just ran into https://github.com/cloudposse/terraform-aws-multi-az-subnets not supporting var vpc_default_route_table_id like https://github.com/cloudposse/terraform-aws-dynamic-subnets#input_vpc_default_route_table_id. Could that be useful so worth raising a GH issue for ?

cloudposse/terraform-aws-multi-az-subnets attachment image

Terraform module for multi-AZ public and private subnets provisioning - cloudposse/terraform-aws-multi-az-subnets

cloudposse/terraform-aws-dynamic-subnets attachment image

Terraform module for public and private subnets provisioning in existing VPC - cloudposse/terraform-aws-dynamic-subnets

Matt Gowie avatar
Matt Gowie

@ GH issue or a small PR to help support would be much appreciated!

cloudposse/terraform-aws-multi-az-subnets attachment image

Terraform module for multi-AZ public and private subnets provisioning - cloudposse/terraform-aws-multi-az-subnets

cloudposse/terraform-aws-dynamic-subnets attachment image

Terraform module for public and private subnets provisioning in existing VPC - cloudposse/terraform-aws-dynamic-subnets

Michael Dizon avatar
Michael Dizon

anyone know of a way to conditionally create a resource (in my case a aws_lambda_function) only if a resource exists (an aws_ecr_image that gets uploaded as a separate process outside of TF) ?

pjaudiomv avatar
pjaudiomv

For stuff like that I usually set a var from a bash script in my pipeline before terraform runs and have it conditionally create based off that

Michael Dizon avatar
Michael Dizon

interesting, do you have an example?

pjaudiomv avatar
pjaudiomv

you would have a variable say a bool create_lambda and only create if that var is true before terraform runs for ecs you could do something like

aws ecr list-images --repository-name anchore | jq -r '.imageIds[].imageTag' | grep -w -q 1.0.0 && export TF_VAR_create_lambda=true || export TF_VAR_create_lambda=false

so if the tag 1.0.0 exists it sets the var to true otherwise false

1
Michael Dizon avatar
Michael Dizon

i’m using TF Cloud, does that make a difference?

pjaudiomv avatar
pjaudiomv

idk ive never used TF cloud

Michael Dizon avatar
Michael Dizon

I’ll figure it out. Thanks for the tip!

1
greg n avatar
greg n
#!/usr/bin/env bash
set -eu
set -o pipefail
# set -x
###
# A shell script to be used as a Terraform external datasource to fetch AMI as created by ImageBuilder pipeline.
# Can't use a normal aws_ami datasource as that errors out if there's no result, giving a chicken&egg situation.
#
# Usage:
# data "external" "imagebuilder_ami" {
#   program = ["bash", "${path.module}/files/jq-ext-latest-imagebuilder-arn.sh"]
#   query = {
#     JQ_AWS_REGION = "eu-west-2"
#     JQ_IMAGE_NAME = "xxxx-ami-builder"
#   }
# }
#
# Unit Testing:
#   echo '{"JQ_AWS_REGION": "eu-west-2", "JQ_IMAGE_NAME": "xxxxx-ami-builder"}' | \
#     ./files/jq-ext-latest-imagebuilder-arn.sh
###

# Use JQ's @sh to escape the datasource arguments & eval to set env vars
eval "$(jq -r '@sh "JQ_AWS_REGION=\(.JQ_AWS_REGION) JQ_IMAGE_NAME=\(.JQ_IMAGE_NAME)"')"

aws imagebuilder list-images  --output json --owner 'Self'                               | \
    jq -r --arg JQ_IMAGE_NAME "${JQ_IMAGE_NAME}" '.imageVersionList[] |
        select(.name == $JQ_IMAGE_NAME) | [.] |
        max_by(.dateCreated).arn'                                                        | \
    xargs -n1 -I% aws imagebuilder --output json get-image --image-build-version-arn %   | \
    jq --arg JQ_AWS_REGION "${JQ_AWS_REGION}"                                              \
      '.image.outputResources.amis[] | select( .region == $JQ_AWS_REGION)'              || \
    true
greg n avatar
greg n

That will give you an external data source that won’t fail if an AMI isn’t found.

data "external" "imagebuilder_ami" {
  program = ["bash", "${path.module}/files/jq-ext-latest-imagebuilder-arn.sh"]
  query = {
    JQ_AWS_REGION = var.AWS_REGION
    JQ_IMAGE_NAME = "${local.full_name}-ami-builder"
  }
}
1
Michael Dizon avatar
Michael Dizon

amazing, i’m going to look at this over the weekend

greg n avatar
greg n

and I use it l like this:

image_id  = length(keys(data.external.imagebuilder_ami.result)) > 0 ? data.external.imagebuilder_ami.result.image : data.aws_ami.ubuntu.id
Steve Wade avatar
Steve Wade

could this be an asymmetric routing issue?

Release notes from terraform avatar
Release notes from terraform
07:03:45 PM

v0.15.3 0.15.3 (May 06, 2021) ENHANCEMENTS: terraform show: Add data to the JSON plan output describing which changes caused a resource to be replaced (#28608) BUG FIXES: terraform show: Fix crash for JSON plan output of new resources with sensitive attributes in nested blocks (<a href=”https://github.com/hashicorp/terraform/issues/28624“…

2021-05-05

Adrian avatar
Adrian

hey, I used terraform-aws-elastic-beanstalk-environment to create an Elastic Benstalk env. I want to upload an new docker image. Is there an bucket for this? didnt see any reference for this.

cloudposse/terraform-aws-elastic-beanstalk-environment attachment image

Terraform module to provision an AWS Elastic Beanstalk Environment - cloudposse/terraform-aws-elastic-beanstalk-environment

Saichovsky avatar
Saichovsky

I have set up security hub using terraform and part of the resources include a lambda which gets triggered by an EventBridge rule. So whenever I run terraform apply, a new aws_cloudwatch_event_target resource is created as a trigger attached to the existing lambda. So we have a duplication of triggers to the lambda, with the latest one being the active one and the former being disabled. Both triggers have the same ARN, but they have separate IDs

resource "aws_cloudwatch_event_target" "event_target" {
    arn            = "arn:aws:lambda:eu-west-1:123456789012:function:service-security_hub_to_jira"
    event_bus_name = "default"
    id             = "eng-security_hub_to_jira_rule-terraform-20210505102803432000000001"
    rule           = "eng-security_hub_to_jira_rule"
    target_id      = "terraform-20210505102803432000000001"
}

This is the output from terraform state show. It only lists one resource when i provide the resource address, but in the lambda console, under triggers, I have two EventBridge resources with the same ARN, but one is enabled and the other disabled.

  1. Is this a bug in terraform?
  2. Is there a way to have terraform apply ID the event rule by ARN and not by id which is not even viewable on the AWS console?
Sergey Kvetko avatar
Sergey Kvetko

Hi! Could somebody makes release https://github.com/cloudposse/terraform-provider-utils with darwin_arm64 support?

cloudposse/terraform-provider-utils attachment image

The Cloud Posse Terraform Provider for various utilities (e.g. deep merging, stack configuration management) - cloudposse/terraform-provider-utils

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@matt

cloudposse/terraform-provider-utils attachment image

The Cloud Posse Terraform Provider for various utilities (e.g. deep merging, stack configuration management) - cloudposse/terraform-provider-utils

matt avatar

Sure, I’ll look into that

matt avatar

Although I think it won’t be possible right now as discussed in this issue

darwin/arm64 build · Issue #27257 · hashicorp/terraform attachment image

I didn&#39;t see an existing issue, so I thought I&#39;d open an issue to track building an arm64 (Apple Silicon) binary for macOS. After migrating to a new Mac, I have seen at least one issue usin…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh interesting. @dansimau used to be at uber. He wrote https://github.com/uber/astro. I guess that’s why development stalled on it.

uber/astro attachment image

Astro is a tool for managing multiple Terraform executions as a single command - uber/astro

Steve Wade avatar
Steve Wade

does anyone know the minimal IAM policy required to read SQS messages ?

wannafly37 avatar
wannafly37

I’m using a queue policy with:

      "Action": [
        "sqs:SendMessage",
        "sqs:ReceiveMessage",
        "sqs:DeleteMessage"
      ],
Steve Wade avatar
Steve Wade

does it need Send to read SQS messages?

wannafly37 avatar
wannafly37

Probably not, but my app uses the same perms for producer and consumers

Alex Jurkiewicz avatar
Alex Jurkiewicz

You may need KMS key access too if you are using a cmk

paultath81 avatar
paultath81
04:42:30 PM

Reposting just in case anyone missed and willing to help

hoping someone can help. I’m using the azurerm_role_assignment resource. What i like to be able to do is have a list of resource i can scope to. I did something using map types, but using it in this way

variable "role" {
  type        = map(any)
  description = "The permission block for roles assignment"
  default = {
    "default_assignment" = {
      scope                = ""
      role_definition_name = "Reader"
      principal_id         = ""
    }
  }
}

would result me to setting my inputs as

role = {
    "scope-001" = {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Contributor"
      principal_id         = dependency.identity.outputs
    },
    "scope-002" = {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Reader"
      principal_id         = dependency.identity
    }

where instead i would like to use it something like this

role = {
    "scope-001" = {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Contributor"
      principal_id         = dependency.identity.outputs
    },
    {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Reader"
      principal_id         = dependency.identity
    }
Release notes from terraform avatar
Release notes from terraform
10:03:41 PM

v0.15.2 0.15.2 (May 05, 2021) ENHANCEMENTS: terraform plan and terraform apply: Both now support a new planning option -replace=… which takes the address of a resource instance already tracked in the state and forces Terraform to upgrade either an update or no-op plan for that instance into a “replace” (either destroy-then-create or create-then-destroy depending on configuration), to allow replacing a degraded object with a new object of the same configuration in a single action and preview the…

Matt Gowie avatar
Matt Gowie

Huh -replace is interesting. Figure it’s the same as doing a terraform state rm $RESOURCE && terraform apply -target=$RESOURCE. I’m sure I’ve done that… but not too many times.

Zach avatar

more like ‘terraform taint <resource>; terraform apply’

Zach avatar

doing state rm doesn’t destroy the resource, you’d have a conflict with anything that had the same names/ids on the apply

loren avatar
loren

From the hangops slack…

Matt Gowie avatar
Matt Gowie

Ah yeah, that is my goof in regards to rm > taint

Matt Gowie avatar
Matt Gowie

Is there a big presence of terraform folks in the hangops slack? I haven’t poked my head in there in a couple years.

loren avatar
loren

Mostly just apparentlymart, with any regulatory, but that’s just enough to be useful

Matt Gowie avatar
Matt Gowie

Yeah I could see just him being around being a big benefit.

loren avatar
loren

I muted every other channel there, that slack has a way low signal:noise otherwise

Zach avatar

terraform and aws channels are incredibly helpful on that slack though

Dhaval Dedhia avatar
Dhaval Dedhia

Hi, I am trying to make create multiple DataDog monitors via Terraform. And I am faced with a weird issue. Can anyone please help me out here?? My resource block looks like this (which is inspired from Cloudposse’s module

resource "datadog_monitor" "monitor" {
  for_each = var.datadog_monitors

  name                = each.value.name
  type                = each.value.type
  query               = each.value.query
  message             = format("%s%s", each.value.message, var.alert_tags)
.
.
.
.
.
.
.
}

And in my tfvars file, I have a map of monitors configs which I pass via tfvars.

datadog_monitors {
high-error-logs = {
    name                = "[P2] [uds] [prod] [naea1] monitor name here"
    type                = "log alert"
    query               = "logs("service:service-name platform:platform-name environment:prod status:error env:env-name region:us-east-1").index("*").rollup("count").last("10m") > 50"
    tags                = ["managed_by:Terraform", "platform:platform-name", "environment:prod", "env:env-name", "service:service-name", "region:us-east-1"]
  }
}

I am not able to pass in the query exactly like this because of the quotes in the query value (“) I tried to replace “ with ‘, but that won’t work because the query then become invalid. I even tried to prefix the quotes in the middle with an " but that gives me errors well. I am stuck. Has anybody else faced a similar issue before and can help me out please?

Matt Gowie avatar
Matt Gowie

You can try doing the following:

 query               = <<-EOT
logs("service:service-name platform:platform-name environment:prod status:error env:env-name region:us-east-1").index("*").rollup("count").last("10m") > 50
EOT
1
Matt Gowie avatar
Matt Gowie

I believe that will do it.

Matt Gowie avatar
Matt Gowie

Or use the https://github.com/cloudposse/terraform-datadog-monitor module and define your monitors via YAML, which is great.

cloudposse/terraform-datadog-monitor attachment image

Terraform module to configure and provision Datadog monitors from a YAML configuration, complete with automated tests. - cloudposse/terraform-datadog-monitor

Dhaval Dedhia avatar
Dhaval Dedhia

Yup that worked. I had tried to use the heredoc style, but i got an error because i put everything in the same single line. This works however. Thank you so much!

Dhaval Dedhia avatar
Dhaval Dedhia

And the reason why i cannot use cloudposse’s module is because of the yaml configuration. I will have to reference that for my monitor values. And almost all the remaining monitors/infra that we have uses tfvars in its native form.

managedkaos avatar
managedkaos

Hey, Team!  Question:  When you encounter a catastrophic error in TF (crash or resource conflict), what’s the best way to find and/or recover any resources that  have been created but not written to state?

Example, the TF config says create resource named X but resource X already exists (manually created, from another TF project, etc).  So TF encounters an error and stops processing (at best) or crashes (at worst).  The resources created up to that point may have not been written to state prior to the stop/crash.

On small projects, I’ve gone through the console or CLI and manually removed things.  But I’m wondering if there’s a better way in the event a project contains hundreds (or more!) resources all over the place.  TIA!

ms16 avatar

I usually start by TG_LOG=DEBUG and running plan to observe the logs

2021-05-04

Pierre-Yves avatar
Pierre-Yves

Hello, how do you initialize a new disk on an Azure VM with Terraform ? I am looking to automatize the next step which have to be done after the two steps:

• azurerm_managed_disk

• azurerm_virtual_machine_data_disk_attachment The point I want to do with terraform is mounting and formating the disk on windows which is describe here: Initialize a new data disk

mudiki08 avatar
mudiki08

Hi folks, does anyone can provide me details on how to spin up a full blown AWS EKS cluster using terraform with self managed nodes and fargate profile. There was no clear documentation on how to get started with it! I would love start using the modules that has been already built. Thanks for your help in advance!

paultath81 avatar
paultath81

hoping someone can help. I’m using the azurerm_role_assignment resource. What i like to be able to do is have a list of resource i can scope to. I did something using map types, but using it in this way

variable "role" {
  type        = map(any)
  description = "The permission block for roles assignment"
  default = {
    "default_assignment" = {
      scope                = ""
      role_definition_name = "Reader"
      principal_id         = ""
    }
  }
}

would result me to setting my inputs as

role = {
    "scope-001" = {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Contributor"
      principal_id         = dependency.identity.outputs
    },
    "scope-002" = {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Reader"
      principal_id         = dependency.identity
    }

where instead i would like to use it something like this

role = {
    "scope-001" = {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Contributor"
      principal_id         = dependency.identity.outputs
    },
    {
      scope = "/subscriptions/${local.sub_id}"
      role_definition_name = "Reader"
      principal_id         = dependency.identity
    }
Pierre-Yves avatar
Pierre-Yves

hello, you say you want to use a list of map and you are using a map of map.

it will be simplier with this

variable "role" {
  type        = list(any)
  description = "The permission block for roles assignment"
  default = [
      {
      scope                = ""
      role_definition_name = "Reader"
      principal_id         = ""
    }
    ]
 }

then you can use a for_each for loop to go through it

2021-05-03

ms16 avatar

Starting a greenfield Terraform env - Customer has Bitbucket and Bamboo , Would you recommend Atlantis over Bamboo Pipeline (YAML Specs) for Terraform Automation ?

Matt Gowie avatar
Matt Gowie

Yeah save yourself re-implementing the wheel.

1
Matt Gowie avatar
Matt Gowie

There is an #atlantis channel if you have any questions.

1
Jeff Behl avatar
Jeff Behl

alright, apologies for this being so general, but.. how/where are folks registering the outputs of terraform for use in app configurations? eg: my app needs to use the SQS queue created by terraform, and it uses a env file with vars and values. env file could/should be a template of some sorts, but how to get the values? CI/CD on terraform output that looks for specific outputs and pushes to consul (or any persistent place)? some place where ansible could get facts for template generation? but for both, storing the results somewhere accessible seems to be the question for us. parsing the state file seems like a horrible idea, so I’ll discount that.. thx

Yoni Leitersdorf (Indeni Cloudrail) avatar
Yoni Leitersdorf (Indeni Cloudrail)

We use terraform output. Note that it parses the state file… you shouldn’t do it yourself

loren avatar
loren

people also push to key/value stores directly, instead of outputs. for example, there is a consul provider… https://registry.terraform.io/providers/hashicorp/consul/latest/docs/resources/keys

aws parameter store is also popular. or s3, or dynamodb…

Joe Niland avatar
Joe Niland

+1 for SSM param store

Jeff Behl avatar
Jeff Behl

thanks gents.

Jeff Behl avatar
Jeff Behl

@loren we’d considering adding to dynamodb (don’t ask) just this way. just seems…laborious? but don’t think there’s an easy way out of it

loren avatar
loren

Haha the dynamodb item syntax is annoying, could use some sugar

loren avatar
loren

We have a keystore module that tries to be a wrapper for this kind of thing… https://github.com/plus3it/terraform-aws-tardigrade-keystore

plus3it/terraform-aws-tardigrade-keystore attachment image

Terraform module to create a keystore within S3/SSM - plus3it/terraform-aws-tardigrade-keystore

Alex Jurkiewicz avatar
Alex Jurkiewicz

+1 to using another service to store your Terraform outputs. It lets you decouple the consumer and use another technology.

We use Terraform outputs purely as diagnostic help in our CD logs. Nothing automated reads them, even other Terraform configurations

Jeff Behl avatar
Jeff Behl

@ meaning using a terraform resource to store the results, correct? This has the definite advantage as well of not having to pipe module output up the stack, effectively making one declare it multiple times. thx

1
Alex Jurkiewicz avatar
Alex Jurkiewicz

sorry, I think terraform outputs are useful to send data from a module to the calling Terraform configuration. I meant “outputs” as in top-level outputs only

1
Jeff Behl avatar
Jeff Behl

i see - thx. just trying to figure out the easiest way to gather and store these outputs. seems it’s either parse outputs and push them to an external service, or use a terraform resource to store them in one..

Petro Gorobchenko avatar
Petro Gorobchenko

Hello everyone, looking for support on this issue. looking to utilize terraform-aws-ecs-web-app running into this issue - cache location is required when cache type is “S3” , seems like it may be coming from

on .terraform/modules/ecs_web_app.ecs_codepipeline.codebuild/main.tf line 292, in resource "aws_codebuild_project" "default":
 292: resource "aws_codebuild_project" "default" {

I can’t see what configuration may be causing this. Any help on this is greatly appreciated.

Matt Gowie avatar
Matt Gowie

Hey Petro, I’d open an issue if nobody response and try digging in yourself. If the type is S3 then you should look at the corresponding resource that is failing and look where you need to add a new variable / value to provide the bucket name / ARN.

Petro Gorobchenko avatar
Petro Gorobchenko

hey @Matt Gowie, thanks for the input. noob question, but are you referring to modifying the modules that are imported to resolve the issue? or are you mentioning that within terraform-aws-ecs-web-app

Joe Niland avatar
Joe Niland

the ecs-codepipeline module has a cache_type variable which defaults to ‘S3’. ecs-web-app doesn’t set it explicitly or expose it as a variable. @ you could open a PR to make this change, if you are able.

Petro Gorobchenko avatar
Petro Gorobchenko

sounds good. Ill play around with it and see if I can create a PR for it.

Petro Gorobchenko avatar
Petro Gorobchenko

added a PR, unsure if I’m missing any steps for the process. https://github.com/cloudposse/terraform-aws-ecs-web-app/pull/147

Making s3_cache_type pass through for module ecs_codepiple by pgbce · Pull Request #147 · cloudposse/terraform-aws-ecs-web-app attachment image

what S3 CacheType on is explicitly set and not exposed. why Attempting to run the module is causing on .terraform/modules/ecs_web_app.ecs_codepipeline.codebuild/main.tf line 292, in resource &qu…

1

2021-05-02

marc slayton avatar
marc slayton

Multi-AZ subnets occur in more than one reliability zone.

2021-05-01

Matt Gowie avatar
Matt Gowie

If anybody is interested in contributing to a good open source module — We have a few good first issues in our terraform-aws-multi-az-subnets repository. Ranges from super easy change a variable to a different type + rename to figure out the difference between two modules and write some quick docs. Check em out if you’re interested!

cloudposse/terraform-aws-multi-az-subnets attachment image

Terraform module for multi-AZ public and private subnets provisioning - cloudposse/terraform-aws-multi-az-subnets

1
1
Alex Jurkiewicz avatar
Alex Jurkiewicz

How does that differ from dynamic subnets?

cloudposse/terraform-aws-multi-az-subnets attachment image

Terraform module for multi-AZ public and private subnets provisioning - cloudposse/terraform-aws-multi-az-subnets

Zach avatar


How does that differ from dynamic subnets?
That’s actually one of the documentation issues https://github.com/cloudposse/terraform-aws-multi-az-subnets/issues/23

Alex Jurkiewicz avatar
Alex Jurkiewicz

nice, didn’t even know about this aws feature

    keyboard_arrow_up