#terraform-aws-modules (2019-11)

terraform Terraform Modules

Discussions related to https://github.com/terraform-aws-modules

Archive: https://archive.sweetops.com/terraform-aws-modules/

2019-11-01

Sharanya avatar
Sharanya

Hey Guys - Quick Question — if any one has every come across this “Is there any way we can make our S3 bucket Private and then Cloudfront provides - URL , Which can be viewed from this Private S3 Bucket” ?

2019-11-02

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yes see our terraform module for hosting an s3 website

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh with cloud front.. we have something for that too

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/terraform-aws-cloudfront-s3-cdn

Terraform module to easily provision CloudFront CDN backed by an S3 origin - cloudposse/terraform-aws-cloudfront-s3-cdn

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think that does what you want

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(Not a github.com/terraform-aws-module though which is what this channel is for)

2019-11-03

navdeep avatar
navdeep

Hey guys, I have been using cloudposse for getting our emr cluster up, though I am able to get my cluster up in our vpc , but our autoscaling policy is unable get triggered with the error Failed to provision the AutoScaling policy: Unable to assume IAM role: arn:aws:iam::216727*****:role/emr-stage-dataorc-emr-ec2-autoscaling , though in the module <https://github.com/cloudposse/terraform-aws-emr-cluster> i can see most of the assume policy requirement given, any direction I can look up or something I am missing

navdeep avatar
navdeep

i gues i found the issue , below policy requires one more trustee application-autoscaling.amazonaws.com

data "aws_iam_policy_document" "assume_role_ec2" {
  count = var.enabled ? 1 : 0

  statement {
    effect = "Allow"

    principals {
      type        = "Service"
      identifiers = ["elasticmapreduce.amazonaws.com","application-autoscaling.amazonaws.com"]
    }

    actions = ["sts:AssumeRole"]
  }
}
navdeep avatar
navdeep

good for a PR ?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

thanks @navdeep, we’ll review it

2019-11-04

Yuriy avatar

Hi! I use this module https://github.com/cloudposse/terraform-aws-dynamic-subnets and can’t figure out how to allow connections from public subnets to private ones (I am setting up bastion host) - what am I missing?

cloudposse/terraform-aws-dynamic-subnets

Terraform module for public and private subnets provisioning in existing VPC - cloudposse/terraform-aws-dynamic-subnets

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This is not a github.com/terraform-aws-module (which this channel is for)

cloudposse/terraform-aws-dynamic-subnets

Terraform module for public and private subnets provisioning in existing VPC - cloudposse/terraform-aws-dynamic-subnets

Yuriy avatar

Sorry, I misread

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Better to just use #terraform which is a catch all for everything else

Yuriy avatar

sure

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Ya, it’s confusing. :-)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Make sure they are all in the same route table

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Make sure your bastion’s security group permits the relative networks

Yuriy avatar

I can ssh to bastion already

Yuriy avatar

RDS security group allows incoming 5432 port (postgres)

Yuriy avatar

but psql connection from bastion on that port times out

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)


RDS security group allows incoming 5432 port (postgres)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

via network CIDR ACLs or security group id of the bastion?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and the route table for the vpc has routes for both public and private subnets?

Yuriy avatar

No, I didn’t explicitly create a routing table for vpc

Yuriy avatar

Did I understand you correctly, that by adding routes to the VPC routing table for public and private subnets I’ll make it possible to connect from EC2 instance in public subnet to the RDS in private?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yep, you got it

Yuriy avatar

RDS security group allows incoming 5432 port (postgres) via RDS security group (I didn’t explicitly add ACLs and as far as I can see - they allow everything)

Yuriy avatar

Thank you Erik, I’ll look into the routing table for the VPC

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

hrmm…

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

actually, I might have misspoke. it’s been a while since I looked at this module.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

looks like it creates both public/private subnets at the same time.

Yuriy avatar

yes

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ok, so looksl ike we create a public route table, and a private route table

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and we don’t permit routing out of the box between them

Yuriy avatar

Aha

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I guess there’s not much profit by keeping them seperate if we at routes to both

Yuriy avatar

yeah, what I actually want is for private nets not to be accessed from the internet but only from pubnets

Yuriy avatar

maybe even from specific EC2 ip

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think what we did was use an ELB/TCP in public

Yuriy avatar

(bastion)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

and then deploy bastion in private

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ELB was pointed at bastion

Yuriy avatar

I c

Yuriy avatar

Thanks, than my plan is to move bastion from pub to priv net, add elb and point it to the bastion

Yuriy avatar

I’ve just found something else:

Yuriy avatar

Session Manager is a managed service that provides you with one-click secure access to your instances without the need to open inbound ports and manage bastion hosts. You have centralized access control over who can access your instances and full auditing capabilities to ensure compliance with corporate policies.

Gabe avatar

have you guys run into any issues with this release of terraform-aws-vpc? https://github.com/cloudposse/terraform-aws-vpc/releases/tag/0.8.0

cloudposse/terraform-aws-vpc

Terraform Module that defines a VPC with public/private subnets across multiple AZs with Internet Gateways - cloudposse/terraform-aws-vpc

Gabe avatar

when we upgraded to TF 12 and applied this module it removed all of our default security group rules, which included rules to allow egress traffic

Gabe avatar

according to the docs

When Terraform first adopts the Default Security Group, it immediately removes all ingress and egress rules in the Security Group. It then proceeds to create any rules specified in the configuration. This step is required so that only the rules specified in the configuration are created.
Gabe avatar

I understand best practice is to not use the default security group, but in our case we have to because of legacy infrastructure

Gabe avatar

seems like this should have been a 1.0.0 release instead of a minor release with some documentation in the readme describing how the new version will remove any SG rules on your existing default SG

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We don’t have a single 1.0 module :-)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In semver pre-1.0 has a special meaning

Gabe avatar
Semantic Versioning 2.0.0

Semantic Versioning spec and website

Gabe avatar

do you guys plan on releasing any 1.0.0 modules? or is that too much effort to maintain for you guys?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

With even terraform itself being pre-1.0 there is some debate as to if it is even possible ;)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But it’s largely because they way all this fits together it’s really hard to solve the calculus of when to release 1.0. It’s a judgement call at best. I think it should be somewhat automatic. If we don’t need to change the interface for a module in 12 months, then cut 1.0 or something

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Open to discussion

Gabe avatar

ahhhh okay that makes sense

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The change came about because we are working to achieve compliance with CIS Benchmarks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The default group was flagged

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

that needs to be fixed, maybe by adding default_security_group_enabled flag

2
Gabe avatar

i think a variable will also be needed to allow the user to pass in any ingress and egress rules they need

This resource treats its inline rules as absolute; only the rules defined inline are created, and any additions/removals external to this resource will result in diff shown. For these reasons, this resource is incompatible with the aws_security_group_rule resource.
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

you mean to the default SG?

Gabe avatar

correct

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

maybe better to expose the default SG ID as output

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so the user could do whatever they want with it

Gabe avatar

i don’t think that’ll work based on the docs… where it’s saying the inline rule is treated as the source of truth and aws_default_security_group is incompatible with aws_security_group_rule

Gabe avatar

at least that’s how i interpret what they’re trying to say

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so yes, I agree. What was done in https://github.com/cloudposse/terraform-aws-vpc/blob/master/main.tf#L22 is not usable w/o inline rules since you can’t use external rules with default SG

cloudposse/terraform-aws-vpc

Terraform Module that defines a VPC with public/private subnets across multiple AZs with Internet Gateways - cloudposse/terraform-aws-vpc

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

and default SG is created in a VPC

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

so for that to be usable, we need to provide variables to specify ingress and egress inline rules

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

we did not catch that (visually and in CI/CD tests) b/c we actually don’t use the default SG in VPCs (always create SGs specific to resources)

Gabe avatar

that makes sense… ideally we wouldn’t be either but unfortunately we are

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

On the other hand, that was created specifically for CIS probably to completely restrict the default SG

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

So adding the inline rules back prob defeats the purpose

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

enabled flag might be better

Gabe avatar

oh yeah i agree with that

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

So TF will not touch the default SG at all if not needed for CIS

Gabe avatar

with the enabled flag the user can manage the default SG outside of the module and do as they wish

Gabe avatar

i also don’t think the way terraform implemented this is very intuitive either … this is the only case i’ve seen where they automatically import the resource and remove things from it

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it’s better to specify default egress in any SG

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
Update: Issue affecting HashiCorp Terraform resource deletions after the VPC Improvements to AWS Lambda | Amazon Web Servicesattachment image

On September 3, 2019, we announced an exciting update that improves the performance, scale, and efficiency of AWS Lambda functions when working with Amazon VPC networks. You can learn more about the improvements in the original blog post. These improvements represent a significant change in how elastic network interfaces (ENIs) are configured to connect to […]

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But I don’t think related

Gabe avatar

agreed, in our case we had used a previous version of the module so it implicitly created the egress, so when we updated terraform removed it

2019-11-06

Saichovsky avatar
Saichovsky

Greetings people

Saichovsky avatar
Saichovsky

I’m new here with a burning question on terraform <> AWS

Saichovsky avatar
Saichovsky
resource "aws_kinesis_analytics_application" "app" {
  name = var.analytics_app_name
  tags = local.tags

  // TODO: need to make inputs & outputs dynamic -- cater for cases of multiple columns, e.t.c.
  //  inputs {
  //    name_prefix = ""
  //    "schema" {
  //      "record_columns" {
  //        name = ""
  //        sql_type = ""
  //      }
  //      "record_format" {}
  //    }
  //  }
  //  outputs {
  //    name = ""
  //    "schema" {}
  //  }
}

Here’s my quandary

Saichovsky avatar
Saichovsky

number of columns is not fixed

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(perhaps try #terraform as it doesn’t seem like this is related to https://github.com/terraform-aws-modules)

Terraform AWS modules

Collection of Terraform AWS modules supported by the community - Terraform AWS modules

1

2019-11-07

Alex Siegman avatar
Alex Siegman

Anyone have a good base module for a redshift cluster? i could have sworn cloudposse had one, but i’m not seeing it in their github

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

no we did not have one

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

but easy to implement using the rds-cluster module

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

should be exactly the same except a few new resources https://www.terraform.io/docs/providers/aws/r/redshift_cluster.html

AWS: aws_redshift_cluster - Terraform by HashiCorp

Provides a Redshift Cluster resource.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We recently implemented EMR

2019-11-12

SlackBot avatar
SlackBot
09:06:25 PM

This message was deleted.

    keyboard_arrow_up