#aws (2024-02)

You’d need to share your specific inputs

I don’t see in documentation which are mandatory to be passed on. In tfvars having below. vpc_id = “vpc-000xxxxxx” subnet = “subnet-xxxxxx”

instance_type = “t2.medium” ssh_key_pair = “xxxx” region = “ap-south-1”

associate_public_ip_address = true availability_zone = “ap-south-1a” security_groups = [ “sg-vvvvvvvv” ] monitoring = false

It would be referring to this documentation: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html

What AMI are you using?

btw, t2.medium is an older instance type. You may want to use a newer gen.

Using data sources - ami : ami-00f994c719fc62e9e

AMI ID exists in the region

Can you show the actual module block with all inputs or the entire file ideally..

tfvars

vpc_id = “vpc-000xxxxxx” subnet = “subnet-xxxxxx” instance_type = “t2.medium” ssh_key_pair = “xxxx” region = “ap-south-1” associate_public_ip_address = true availability_zone = “ap-south-1a” security_groups = [ “sg-vvvvvvvv” ] monitoring = false

main.tf

module “instance” { source = “cloudposse/ec2-instance/aws” # Cloud Posse recommends pinning every module to a specific version # version = “x.x.x” ssh_key_pair = var.ssh_key_pair instance_type = var.instance_type vpc_id = var.vpc_id security_groups = var.security_groups subnet = var.subnet name = “ec2” namespace = “eg” stage = “dev” monitoring = var.monitoring }

datasources

data “aws_ami” “amzn_linux” { most_recent = true owners = [“amazon”] filter { name = “name” values = [“amzn2-ami-kernel-*”] } filter { name = “root-device-type” values = [“ebs”] } filter { name = “virtualization-type” values = [“hvm”] } filter { name = “architecture” values = [“x86_64”] } }

The AMI value is not set for module "instance"

I tried with adding ami id and data sources. module “instance” { source = “cloudposse/ec2-instance/aws” # Cloud Posse recommends pinning every module to a specific version # version = “x.x.x” ssh_key_pair = var.ssh_key_pair instance_type = var.instance_type vpc_id = var.vpc_id security_groups = var.security_groups #ami = “ami-0d63de463e6604d0a” ami = data.aws_ami.amzn_linux.id subnet = var.subnet name = “ec2” namespace = “eg” stage = “dev” monitoring = var.monitoring }

module.instance.aws_instance.default[0]: Creating… 2024-02-05T1311.749+0530 [ERROR] provider.terraform-provider-aws_v5.34.0_x5.exe: Response contains error diagnostic: tf_proto_version=5.4 tf_req_id=2be98416-ed4b-a900-b440-414c8c681655 tf_rpc=ApplyResourceChange diagnostic_severity=ERROR diagnostic_summary=”creating EC2 Instance: Unsupported: The requested configuration is currently not supported. Please check the documentation for supported configurations. status code: 400, request id: e8da9f9e-5ed7-4c8a-85c9-6c7be60a129a” diagnostic_detail= tf_provider_addr=registry.terraform.io/hashicorp/aws tf_resource_type=aws_instance @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go//github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:62) @module=sdk.proto timestamp=2024-02-05T1311.749+0530 2024-02-05T1311.753+0530 [ERROR] vertex “module.instance.aws_instance.default[0]” error: creating EC2 Instance: Unsupported: The requested configuration is currently not supported. Please check the documentation for supported configurations. status code: 400, request id: e8da9f9e-5ed7-4c8a-85c9-6c7be60a129a ╷ │ Error: creating EC2 Instance: Unsupported: The requested configuration is currently not supported. Please check the documentation for supported configurations. │ status code: 400, request id: e8da9f9e-5ed7-4c8a-85c9-6c7be60a129a │ │ with module.instance.aws_instance.default[0], │ on .terraform\modules\instance[main.tf line 104, in resource “aws_instance” “default”: │ 104: resource “aws_instance” “default” {

if you run TF_LOG=trace terraform apply you may get more info from the 400 error

│ Error: creating EC2 Instance: Unsupported: The requested configuration is currently not supported. Please check the documentation for supported configurations. │ status code: 400, request id: d5719aa4-c001-4de5-a78f-319672e5cbeb │ │ with module.instance.aws_instance.default[0], │ on .terraform\modules\instance[main.tf](http://main.tf) line 104, in resource “aws_instance” “default”: │ 104: resource “aws_instance” “default” { │ ╵ 2024-02-05T1640.739+0530 [DEBUG] provider.stdio: received EOF, stopping recv loop: err=”rpc error: code = Unavailable desc = error reading from server: EOF” 2024-02-05T1640.796+0530 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.34.0/windows_386/terraform-provider-aws_v5.34.0_x5.exe pid=22176 2024-02-05T1640.797+0530 [DEBUG] provider: plugin exited

I don’t see much information with trace

After disabling ebs optimized, issue resolved.

Ah yes it is on by default and you are using t2

How did you determine it was the issue?

carefully gone through logs from trace :)

Ok good to know!

how I’ve solved this in the past is to put a jumphost in the private subnets because the RDS was in an isolated subnet, then restricted access to the bastion from a known IPs list. this comes with maintenance overhead due to routine patching and audit logging requirements, but when you have limited options it was the best I could come up with.

https://aws.amazon.com/rds/proxy/

There will be proxies involved for sure, however that doesn’t solve the external access aspect, since

“RDS Proxy can be used only within a VPC, and can’t be publicly accessible (although the DB instance can be). If you connect from outside a private network, then your connection times out” • https://repost.aws/knowledge-center/rds-proxy-connection-issues# Elaborating my scenario & my thoughts… it will likely be an NLB in a public subnet, that routes allowed traffic to the private subnet where the RDS proxy lives, which then connects to the Private RDS instance. I think thats the only option, besides a jump host but the NLB aspect seems easier. We’ll see I’m going to spin up some things in my sandbox VPC and test things out

NLB -> RDS will be easier if your RDS is in a subnet that can be routed to from the public subnet (though that’s a fairly weak network security posture). if you do that, you should make sure to lock down the security group to only known IPs. probably best to even have a SG per external client group too. ideally each rule would have a clear description too.

edit: Sorry I misread your recent message

by fairly weak, I mean all that needs to happen is one person do the wrong thing to the security group (like drop a 0.0.0.0 on it while testing) and now you have an RDS exposed to the world

totally, I’ve seen that before at previous jobs LOL

but necessary evils sometimes

oh I’ve seen orgs put their RDS in public subnets because they wanted it to be easy for data engineers to connect third party services back to them

I screamed and ran for the hills

that is pretty much the same scenario I’m dealing with

I didn’t actually run, I spelled out the security implications clearly and documented the sign-offs from the higher ups on the project that they owned the risk.

I like that risk ownership aspect a lot, I will keep that in my playbook for future things like this for sure

JFYI: https://aws.amazon.com/blogs/networking-and-content-delivery/hostname-as-target-for-network-load-balancers/

I’ve used that for ALB targets, but I don’t believe it supports RDS endpoints

I’ve also ran into the “data team needs to connect DBs to FiveTran” and the privatelink support is a $40k upsell

wow! what an upsell. DBT wanted 20k for enterprise, so I could connect DBT to OKTA for SAML No thanks

Yeah, not too happy that baseline security stuff is paywalled

Figured it out after 2 days finally. It was an issue with how lambda unpacks the layers. It expects it in python/blah I just zipped it up as such and it works

Our quick fix is to go with IngressGroups but I’m curious if others have made the IPv6 migration or are using other techniques to reduce the number of IPv4 IPs they’re using.

We use Cloudflare as our WAF so we’re also thinking of standing up Cloudflare Tunnels for all ingress (currently using security groups to restrict Ingress to Cloudflare IPs).

We host our EKS clusters in a private network then use ALB controller for ingress. So we’ve definitely been hit with running out of IP space, but not the the charge for public IPs

yeah, that’s how we do it too, it was just a surprise that if you don’t use the [alb.ingress.kubernetes.io/group.name](http://alb.ingress.kubernetes.io/group.name) annotation, the controller allocates a new ALB per Ingress (not terribly surprising in hindsight) and since we have at least one Ingress per namespace, that ballooned our IP usage fast!

oh I see. yeah that group.name annotation has got me before too, but we use the same ALB for many Ingresses across namespaces. So our use-case isnt really the same

@E-Love here’s our component that we use for that https://github.com/cloudposse/terraform-aws-components/tree/main/modules/eks/alb-controller-ingress-group

yes that is what it means

I used to do KMS before this existed and then we switched to this which is FAR easier

Thoughts on removing those flags from the module? I may upstream that change.

@Matt Gowie Yeah, your PR is more than welcome. Message me that PR link and I will handle it to get merged. Thanks

Will do.

I don’t use SQL server , but I imagine you can use the same process. You can set up a “jumpbox” which uses AWS SSM session manager to remote into the VPC without exposing it publicly. Then you can use ssm port forwarding from the jumpbox to the RDS instance (with the proper security groups configured)

So at least the portion of authentication happens via AWS SSO with MFA. Another solution to look into would be Teleport

Thanks Joe!

We’ve got a TCP proxy pod in an EKS cluster (where the application that connects to the DB lives) that we can use. We were able to use a Terraform module to kubectl port-forward to the pod, and a provider to create SQL logins, but it kind of feels like a round hole square peg kind of situation. I didn’t know if this was a common/accepted approach or if there might be a better pattern.

Also, our pipeline runs a separate Terraform plan & deploy from the output, so we’d need the tunnel to be open during both plan and apply. This is turning out to be a bit tricky due to the semantics around whether or not to execute a data source, e.g. if state shows that the data source will remain the same, it will not be executed.

I have not done it myself, but you probably want to deploy middeware such as PgBouncer, Pgpool-II, or HAProxy along with RDS read replicas in each of the regions where you have users. Have the users connect to the middleware near them.

Our number of users are not substantial enough to require a connection pooler fortunately. Also, can’t use read replicas because writes are a part of the access pattern

The point of the proxy, in your use case, is not connection pooling, it is allowing reads to be served by the local read replica, while forwarding writes and DDL to the distant master.

Can you provide more detail about your access pattern?

The latency from Australia to Oregon is approximately 150ms.

If you have multiple sequential reads in order to render the UX, that can create a multiple second delay that might be the primary source of complaints. Using read replicas as Jeremy suggested would resolve that issue.

Writes are usually not done sequentially so 150ms to talk with the master seems very reasonable.

Active-active (or edge postgres) won’t speed up writes without giving up consistency guarantees that could leave your data in a borked state. If you look at the docs for pgEdge, you can see that they don’t give up the consistency guarantee; it propagates the writes to all of the masters, which will likely make it slower than your current setup.

Overall, active-active is significantly more operational overhead to maintain than read replicas, and this isn’t really the primary use case for using it (primary use is fault tolerance). Running a connection pooler is straightforward compared to an active-active setup.

I misspoke. After looking at pgEdge in more detail (personal curiosity), it looks like they do async write replication via a conflict resolution algorithm documented here.

Ultimately, if you went that route, you are guaranteed only eventual consistency (different databases might have different data) coupled with reduced durability (committed writes can be dropped if they cause conflicts). Those are two important changes from your current operating model that you should ensure the underlying application can handle before you choose that path.

Using active-active masters is an enormous headache and IMHO not nearly worth it for performance improvements in an internal tool that is primarily reading. Read-replicas plus a smart proxy is a much better solution in this case. I think the proxies specifically designed for this kind of work will even take care of things like preventing reads of stale data immediately after a write.

At the very least, I recommend you give it a try before trying any kind of multi-master solution.

The point of the proxy, in your use case, is not connection pooling, it is allowing reads to be served by the local read replica, while forwarding writes and DDL to the distant master Ah okay that’s quite cool! I thought that sort of pattern would only be possible on the application (qgis) side if say qgis expose different connection types (read / write).

This is potentially super interesting because I would much rather run AWS Read Replicas. We didn’t think this would work in the past because the workload involved writes also.
Ultimately, if you went that route, you are guaranteed only eventual consistency (different databases might have different data) coupled with reduced durability (committed writes can be dropped if they cause conflicts). Those are two important changes from your current operating model that you should ensure the underlying application can handle before you choose that path. Yeah. And this should be within the acceptable parameters for us at the moment. Our company really only has internal users so there’s really only 60 or so daily users making reads and a smaller portion of those users making writes.

Thanks all

How would the connection pooler know how to differentiate reads and writes? Does it look at the underlying SQL Query?

I’m potentially misinterpreting things in that the connection poolers likely are unable to automatically route reads to replicas and writes to the main node, but, this has given me another idea where we configure different connections in qgis based on if the workflow is read-only or not.

These are more proxies than connection poolers, and they route requests based on the underlying SQL command.

Yes. I have in the past. As the title of the blog hints, it requires private VPC endpoints (aka PrivateLink endpoints).