#aws (2024-09)

@Jeremy White (Cloud Posse)

you likely have session stickiness on on the load balancer, which means each unique session goes to the same instance, to allow for things like safe deploys (once you hit app version 2.0, you no longer hit 1.0). Your load tester is 1 session, so it always hits 1 server. You could temporarily turn off session stickiness during the load test, or else you might have to lookup how it determines that stickiness, and randomize that on the load tester side. AWS Load balancer sets a cookie for that stickiness, so you’d need to clear cookies. Cookie name is AWSALB at least on mine

Thanks @Michael Galey Currently stickiness if turned off. The problem is that one ecs task takes 87% cpu while the remaining four tasks have 0% utilization. So this has an impact on the scalability of the system. Do you know how I can resolve this ?

usually I think about python applications as needing something to share requests. There are a few ways to do this, but a common couple to try first are gunicorn and uwsgi . Do you have any application that’s sharing the listener port with your server threads/PIDs?

Here’s a site on uWSGI: https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html

if you are using one of those tool already, could you share your config? Doesn’t have to be all the gory details, but at least some notion of how it decides to spawn your application on a request

the above would be per server, he’s already at full utilization on one server. His load balancer is not balancing the load, at least not from a single source.

if you do 2-3 load tests of smaller size from a few diff ips, does it go to that number of ecs tasks? I’d suggest trying least-request if that’s an option for the load balancer, otherwise maybe just start simple, follow some tutorial for a basic load balancer + hello world, and compare the load balancer config / target group config against yours. I don’t see how it’s the app’s fault here, I think it’s the load balancer config + single origin ip

I better understand now. I’m not sure what’s up, but are you using target groups? Do all the tasks show as healthy?

You’re more likely to get help if you ask questions.

Codify each arn in aws auth config

Now days the aws auth config file is deprecated

Deprecated method https://docs.aws.amazon.com/eks/latest/userguide/auth-configmap.html

Access entries is the new way https://docs.aws.amazon.com/eks/latest/userguide/access-entries.html

Migrating https://docs.aws.amazon.com/eks/latest/userguide/grant-k8s-access.html#authentication-modes

You can’t use wildcards/globs in aws auth config or in eks access entries

One way to do it, if youre using terraform, whether with aws auth config or access entries, you can use the data source aws iam roles, specify a wildcard to retrieve all the arns, and then populate the arns in your config map or access entries

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_roles

data "aws_iam_roles" "default" {
  name_regex = ".*project.*"
}

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_access_entry

resource "aws_eks_access_entry" "default" {
  for_each = data.aws_iam_roles.default

  cluster_name      = aws_eks_cluster.default.name
  principal_arn     = each.value.arn
  kubernetes_groups = ["group-1", "group-2"]
  type              = "STANDARD"
}

yeah, i saw that workaround in the thread

but its so hacky

i also can see it breaking since we use terragrunt, and it’s hard to pass in data calls into module inputs

so we’d have to just rely on “in-module” access entries, and ignore the terragrunt layer, i think (maybe not - haven’t thought abt it enough)

Yes, we’ve run into this as well. It’s one of the reasons we also implement the aws-teams and aws-team-roles architecture in our reference architecture and allow permission sets to assume them. This allows us to have consistent roles for both programmatic/machine access (e.g. GitHub OIDC) as well as for developers.

We talk more about our approach here https://docs.cloudposse.com/layers/identity/

thanks! i’ve been considering going with this approach, but i’m hesitant because of the extra assume role hop (for human users)

especially non-technical users

Yes, the extra assume role hop is really for technical users

But sounds like you’re using EKS. You have technical users.

yeah, but we have a fair amount of non technical users (clickops in the console), and i’m not sure what we’d do for them. i think the extra assume role in the console would make dem go nuts but i guess a hybrid approach could work too…

@Jeremy G (Cloud Posse) might have some updated ideas on this we haven’t yet tried. He’s OoO though.

this could be useful https://www.youtube.com/watch?v=z1WQ-YSAsVY

here is the text version with links https://fivexl.io/blog/ecs-service-connect-encryption/

Why don’t you let the ECS service manage registration with the target group for you? It’s allowed to use both service discovery AND allow the ECS service to register task IPs with the target group

@andrey.a.devyatkin Thank you so much for the video! @Fizz Yup, this is what i chose to go with, during the creation process when defining the ECS service, i choose the load balancer and then i was prompted to create the Target Group & add it to the listener. All the ports of which i used use HTTPS now and it’s working perfectly, am just adding a route 53 entry and then we will go public!

This is super neat to see, and do share your experience during and after the process! I’m keen to implement ECS Service Connect within Fargate target groups using Terraform provisioning, as soon as I’m able wrap my mind around it.

@Rishav checkout blog post and video above - they go into details of ECS Service Connect implementation for ECS/Fargate

I think you’re looking for https://github.com/aws/amazon-ecs-cli

will check on this.

thank you

Yea, unfortunately they deprecated docker compose deployments to ECS in the docker-compose CLI

We were really bummed about that

wow, @Darren Cunningham i didn’t know about this new ECS cli

And this amazon-ecs-cli is different from the other ECS cli by AWS https://aws.github.io/copilot-cli/