#terraform-aws-modules (2022-05)

terraform Terraform Modules

Discussions related to https://github.com/terraform-aws-modules

Archive: https://archive.sweetops.com/terraform-aws-modules/

2022-05-04

Rajesh Babu Gangula avatar
Rajesh Babu Gangula

Hi team, just checking if we have roadmap to create modules for AWS Appstream 2.0

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

I think it is fair to say we do not have that on our Roadmap. @Erik Osterman (Cloud Posse) ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Not on our roadmap at this time

2022-05-06

Jeremy avatar

Interesting situation and looking for possible solutions. Trying to utilize the dynamic-subnets module but our default gateway is going to be a transit gateway not an internet gateway as we run everything out through a security VPC that has the IGW. The way the dynamic-subnets module is written foe rhe public subnet routing table it expects an IGW ID be provided. In our case we want that to be the TGW ID. I’m really trying not to reinvent the wheel and have to maintain a whole lot of code to manage this.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy (Cloud Posse) is doing some major rewrites on this maybe it can be factored in

Jeremy avatar

I was wondering if it was something worth trying to refactor and submit. Whether it was being able to set the default route using either a gateway_id or transit_gateway_id or simply make something like enable_default_route and add to the count conditional statement

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Jeremy Look for a PR for dynamic-subnets later today. The general plan is for you to be able to suppress the creation of any resources it creates by default. In your case, you would disable the creation of public routes and create your own routes, route table(s), and associations to the public subnets. Does that sound like a reasonable path forward for you?

Jeremy avatar

The latter being the simplest solution and just have the TGW handle adding the route along with VPC attachments

Jeremy avatar

@Jeremy (Cloud Posse) yeah that would fit the bill as it would be okay to create the VPC subnets and not have a default route but have the route table created

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

No, we’re not going to create the route table if we do not create the routes, because you may want to use the VPC default route table, you may want a single route table for all public subnets, or you may want a route table per subnet. The [variables.tf](http://variables.tf) file is already 400 lines long and I really want to trim down the number of options where I can.

Jeremy avatar

Okay… I can get the default route table from the VPC module in that case. We use the separate private subnet route tables as is currently.

Jeremy avatar

The only advantage of having the extra public route table is that it actually requires explicitly associating subnets to it so if the default were left alone it would have no routes if a rogue subnet were created and jot associated to a route table

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

I understand. This is why you might want to create your separate own route table and associate it with the public subnets. It’s not that hard, but the combination of options gets very hard to manage inside the module.

Jeremy avatar

Oh yeah… I’m actually writing a local VPC module that we call that wraps the VPC and subnet modules so adding a route table and using it as needed wouldn’t be a problem. I’ve managed to convince team to use Cloudposse modules as much as possible and this is the one complexity that been left to me to resolve

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

At one point I was thinking I could accept a named default route, which would solve your use case, but then I realized I would have to accept one for all AZs as well as one per AZ and then you would have to dynamically generate the map of AZs to routes and by then it doesn’t seem like it’s helping enough to be worth the added complexity.

Jeremy avatar

I was actually happy to see the VPC change to be able to support IPAM as I’d been trying to use it myself and not much out there supported it

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

We still do not have IPAM set up in our test environment, so none of that stuff has been tested. Please let us know how it goes.

Jeremy avatar

I don’t see much of a use case for multiple public route tables. Private certainly but public subnets would got out a GW that’s usually already multi-AZ

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

The main use case for a public route table per AZ is IPv6 with NAT64, where you still need a NAT Gateway per AZ for reliability.

Jeremy avatar

Ah yeah you’re right… I hadn’t considered NAT64 as I assume IPv6 is always routable and not NAT like RFC1918 IPv4 space

Jeremy avatar

That’s partly why we have to ensure IPv6 is completely disabled

Jeremy avatar

@Jeremy (Cloud Posse) you’re not talking about changing how the private subnets are setup are you, just public?

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

OK. Our impetus for the dynamic-subnets update is adding IPv6 support, so I’ve been thinking a lot about it. With IPv6 and NAT64 you can bring up an IPv6-only public subnet and retain connectivity to IPv4 backend services. Or, more likely, use a dual-stack subnet but only assign IPv4 addresses to the load balancer and use only IPv6 internally.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

The private subnets will have IPv6 support added, and some cleanup, but should mostly be the same. I will ask you to try out the PR version once it is ready and welcome your feedback.

Jeremy avatar

Sure thing… I was just asking as you said you wouldn’t be creating route tables and resources if it was disabled as I’d need to, juat wanted to make sure private subnet resource creation would remain as is as I’m not disabling any of that other than just not creating NAT GW or Instances

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

If you do not create a NAT destination for the private subnets, then we are not going to create route tables or routes because we have nowhere to point them to.

Jeremy avatar

Okay I can work with that… I was just adding the necessary routes to the private subnet route tables to accommodate not using NAT GW

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

I’m not that familiar with provisioning route tables, maybe you can help me with the design. My assumption was that each subnet can only be associated with a single route table, which is why I did not want to create empty route tables for the subnets. If that’s wrong, and subnets can be associated with multiple route tables, then maybe I will continue to create the for the private subnets.

Jeremy avatar

Technically we are running a 3-tier subnetting scheme (public, private, data) but as we only have public in the security VPC we were making the dynamic-subnets module work and using the public subnets as our private and private subnets as our data.

Jeremy avatar

Yeah… the subnet is only associated with a single route table. When you have NAT GW or NAT Instances you would typically have each private subnet use its own route table that had thr NAT GW in the same AZ for HA redundancy. IGW and TGW are not AZ specific so using a single route table works

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Right, that’s why the current module creates a route table per AZ for private but only one for public.

Jeremy avatar

Yeah the route table setup as is isn’t the issue I’m butting heads with. It’s just the aws_route for the 0.0.0.0/0 destination that requires the IGW be defined

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Right. Hence the compromise I came up with to keep this manageable: limited options for configuring what is created, but the ability to suppress creation of specific resources and supply your own replacement.

Jeremy avatar

If the igw_id variable were optional not required and that one resource only was created of it was non-null would accomplish the same result. I can work with however you go and will definitely review for you

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Because Terraform has a plan phase, we cannot condition the creation of a resource based on the value of an input that might be generated in the same plan. Instead, there is a new input, public_network_route_enabled, that you set to false to inhibit the creation of any routing setup for the public subnets.

Jeremy avatar

That sounds like it would work. We’ll handle our route table manipulation from the calling module that bonds the VPC and subnet modules

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Jeremy I opened a draft PR (159) for you to start looking at. Additional documentation and maybe some other cleanup to come. It should serve your use case if you leave public_network_route_enabled at its default value of true but do not supply igw_id. In this case it will create the route table but no routes.

what

• Full support for IPv6

why

• Requested feature

Jeremy avatar

@Jeremy (Cloud Posse) I’ll run it through it’s paces with our build out I’m working on currently. I’ll get back with any findings on Monday. Enjoy the weekend

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Jeremy I made some changes which I hope address your concerns.

Jeremy avatar

@Jeremy (Cloud Posse) okay I’ll have a look in a little bit.

Jeremy avatar

@Jeremy (Cloud Posse) I tested with a new VPC creation and upgrading from an older version of the module and in both instances the outcome was as I’d expect. You technically would never have an “empty” route table as it will always have a local route for the VPC CIDR so not having an IGW or NAT GW or Instance just makes a VPC that only routes locally which is a valid use-case scenario. In fact one could be an HA cluster that needs a private network for heartbeat or sync communication only between instances in multiple AZs in the region. How it is operating now allows that functionality.

Now I may end up submitting a PR for the terraform-aws-vpc module as well as I’ve identified some missing functionality. I’m going to leave it for Monday though and try to enjoy the rest of my weekend. I also need to dig into the terraform-aws-transit-gateway module some to better understand it and how to utilize it. I have had some issue with the 0.9.0 version of it and have had to pin back to 0.6.1 for now.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Jeremy please tag me in a new thread in #terraform-aws-modules that explains your concerns with the terraform-aws-vpc module as I am also working on it and might be willing to add the features you need without you opening a PR.

Thank you for the feedback on dynamic-subnets.

Jeremy avatar

Sure thing @Jeremy (Cloud Posse) glad I could help. Estimate on how long that branch will remain draft so I can plan on when to change current reference method?

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

I think the code is not going to change much if at all, what is missing are the release notes and migration document. Not sure how long it is going to take. For now, you can refer to it using registry syntax as version v2.0.0-rc1

1

2022-05-07

2022-05-08

Jeremy avatar

@Jeremy (Cloud Posse) what I found with the VPC module, more to the point the VPC endpoint sub-module, was that while the interface endpoint allowed for the optional subnet IDs to be passed through the gateway endpoint did not accept route table IDs so it could associate the gateway service with the VPC route table. It makes sense having it as a sub-module as you need both the VPC and the subnets created before it can be of use. Inside the sub module makes sense to perform the aws_vpc_endpoint_route_table_association resource definition but it requires passing along the route table IDs. Also it could probably just more simply be passed as the optional property to the aws_vpc_endpoint itself. I thought it could be added as an optional variable in the existing mapped variable just like how the subnet IDs are for interface endpoints.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Jeremy Thank you. I am not familiar with how VPC Endpoints work, so I very much welcome your input on this.

As you can see in our example, we expect the user of the VPC module to perform the route table associations. It’s not clear to me that you would necessarily want all the endpoints attached to the same route table, so then you have to figure which associations to make.

Because we want to maintain backward compatibility, we cannot just add a route_table_id member to the input objects, so we would have to add new input objects that just take route table IDs and then use those to create the associations. At that point it seems just as onerous to create the inputs as it does to create the aws_vpc_endpoint_route_table_association resources themselves (which are just 2 lines), with the advantage of the latter being full control and transparency.

I agree that it would have been better to allow for an optional route table ID from the start, but at this point I do not think it is worth a breaking change or the hassle of the non-breaking change.

If you feel strongly about it and want to submit a PR anyway, I would accept (keeping in mind that I am far from the sole arbiter of such things) a PR that adds 2 new inputs: • gateway_vpc_endpoint_route_table_idsinterface_vpc_endpoint_route_table_ids which are both map(string) and take the same keys as the corresponding endpoint inputs and then create appropriate aws_vpc_endpoint_route_table_association resources. But as I said, I don’t think it is worth the trouble, just as I don’t think it is worth the trouble to create Security Groups in the module.

resource "aws_vpc_endpoint_route_table_association" "s3_gateway_vpc_endpoint_route_table_association" {
  route_table_id  = module.vpc.vpc_main_route_table_id
  vpc_endpoint_id = module.vpc_endpoints.gateway_vpc_endpoints[0].id
}

resource "aws_vpc_endpoint_route_table_association" "dynamodb_gateway_vpc_endpoint_route_table_association" {
  route_table_id  = module.vpc.vpc_main_route_table_id
  vpc_endpoint_id = module.vpc_endpoints.gateway_vpc_endpoints[1].id
}
Jeremy avatar

The route_table_ids is only valid for Gateway endpoints just as subnet_ids is only valid for Interface and GatewayLoadBalancer endpoints. The latter isn’t even supported by the vpc-endpoints sub module as far as I can tell. They are both optional and passing them to the aws_vpc_endpoint resource simply foregoes having to use the aws_vpc_endpoint_route_table_association and aws_vpc_endpoint_subnet_association respectively.

Jeremy avatar

I was attempted to add the aws_vpc_endpoint_route_table_association using both count and for_each while passing the dynamic-subnets private_route_table_ids and in both cases ran into the issue with standing up a new VPC where it was saying the value was not known until apply and hadn’t gotten it worked around which was when I dug into the sub module more.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

The roadmap for the AWS Terraform provider cals for moving away from inline resources (such as passing the route table ID directly to aws_vpc_endpoint) and toward managing as much as possible with distinct resources such as aws_vpc_endpoint_route_table_association. So we want to use that resource regardless.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Unless you supply a fixed list of AZs or AZ IDs to dynamic-subnets, the number of private subnets (and therefore route tables) created will not be known at plan time, because it will depend on the number of AZs in the region.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

So I see your dilemma. You want to stand up all of this in one root module, and cannot create the associations because they have unknown cardinality. The only solution is the inline setting.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@Jeremy So I guess you think we should support the GatewayLoadBalancer as well. What about auto_accept? Should we add that as an input? Does it apply to all endpoint types?

Jeremy avatar

I’ve not yet worked with the GatewayLoadBalancer endpoints yet myself. As I’ve read the docs on the auto_accept option it still only works with accepting services within the same AWS account not cross-account but that said I’ve not had to use it with any of the VPC endpoints I’ve used which are mostly Gateway or Interface service types. The VPC endpoints are really good for working with private only subnets and being able to reach AWS services without NAT or IGW routing available which has been my primary use case for them. It’s nice to use the S3 Gateway VPC endpoint and then be able to lock down an S3 bucket to only allow traffic from a specific VPC for DLP.

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

Do you have auto-accept turned on at the Org level via Resource Access Manager?

Jeremy avatar

not currently… Actually our Org setup is something I’m meaning to bring up with manager and higher up as I noticed we currently only have it enabled for consolidated billing and not much else so not really utilizing the Org features. In my own AWS Org I have it enabled and generally just have RAM resources shared to entire Org not just specific accounts

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

So it seems to me we should allow you to set auto_accept , shouldn’t we?

Jeremy avatar

If want the sub module to be as full featured and usable for anyone I’d agree and say yes. I may not use it but someone else might and if it’s available they would then have it. It’s kind of like the route_table_ids in that it’s optional don’t need to use as you could then add aws_vpc_endpoint_route_table_association resources but if it was an optional variable you could use it and just have it passed along to the aws_vpc_endpoint resource upon creation. Both work but currently only one is functional. Now if I were working with an Interface service the subnet_ids is already available but optional so I could use it or just create an aws_vpc_endpoint_subnet_association resource.

Jeremy avatar

Honestly, the vpc-endpoint sub module could have a case made for it being it’s own module and become more full featured

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

There’s no real reason to make it a standalone module. You can use the submodule without using the root module.

Jeremy avatar

I’d have to research when the auto_accept would actually be needed… I know with the Gateway endpoints s3 and dynamodb that it isn’t necessary

Jeremy avatar

Question for anyone who has worked with the transit-gateway module as it is… I’m currently working to codify our existing setup that has been done mostly by hand thus far. It follows a hub-spoke design with 2 route tables but we’re looking to add a 3rd RT into it. As I have been reading over the module I see it allows passing either a RT you create outside the module or creating one, but it appears like it only deals with a single RT.

    keyboard_arrow_up