#aws (2024-04)

@Jeremy G (Cloud Posse)

We do not use Pod Security Groups with EKS, and I have no experience with them, so I can’t be of much help.

Do you usually use NetworkPolicies then? Perhaps with the ALB Ingress?

@Jeremy G (Cloud Posse) @Andriy Knysh (Cloud Posse)

We use EC2 Security Groups. By default, security groups block all inbound traffic. That includes traffic from within the security group. You need to add a rule to allow inbound traffic, which is what I hope you mean by “a self-referencing rule”.

We generally allow everything in the EKS cluster to talk to everything else in the EKS cluster. People wanting finer-grained access control typically use a service mesh, although they are very difficult to manage, so using the newly available Kubernetes controls is probably a better plan.

Beyond that, I can’t help you.

@Sean Turner did you see this doc https://docs.aws.amazon.com/eks/latest/userguide/security-groups-for-pods.html?

it says that the Pod SG must have an ingress rule to allow traffic from the Node SG on all ports that your Pods are using

it also needs an egress rule on port 53 for DNS resolution (if you are restricting egress, need to open the port)

Thanks for the responses.

We’re ending up on the same approach as well.

The issue where two pods with the same security group were unable to communicate I believe was due to the VPC CNI plugin configuring POD_SECURITY_GROUP_ENFORCING_MODE as strict rather than standard. Once that was changed everything worked.

Previously we had been opening ports on the security group pod for liveness probes and the like, but now we’ve got a prefix list on the security group ingress rule which specifies all of the EKS Private Subnet CIDRs.

Thank you!

We use something of a combination. Team-focused roles, plus additional roles where maybe the team doesn’t quite align right with the desired accesses. Like readonly access is available to more folks, and parts of the billing console is available to fewer

But we also do account per team, so it’s sorta an admin-ish role for the team/account

then sometimes we have teams that build services for the wider environment, crossing accounts. they get their own permission set mapped to accounts where they provide service

Ah interesting.

Thanks Loren. Thats helpful. I was creating a plan cause at the moment, there are too many admins. Let me know if this is a good summary.

• Team based accounts (data accounts, security account, etc)

• Team focused roles (dataengineer, ops)

• Generic roles (dev, readonly, admin, billing) Anything else i could add? A lot of this is going to have iam perms created by sleuthing through cloudtrail logs

One drawback wirh being admin of a team account is being able to change account settings that could expose things like s3 buckets, so i want to build a limitedadmin role if possible (or just gate everything with root scps)

i think in the end, the most important part is to realize what you don’t know, and just have a good pipeline for building groups, permission sets, and mapping them to accounts lol

then sometimes we have teams that build services for the wider environment, crossing accounts. they get their own permission set mapped to accounts where they provide service
Naming these seem challenging.

Have you settled on a naming convention?

and yes, totally, we gate everything with scps. well, i mean, if any role has iam admin-ish permissions, then we put a lot of effort into scps, since that role could create other roles that bypass protections built into their own role

no particular naming convention at this time. naming is too hard. we sorta use something that feels like the team/role/service/access that the permission set has. readonly, billing-admin, project-admin, etc

Thanks. It seems like it’s not a one and done type of management. Currently the least privileged roles i help maintain, slowly expand until i think of a new named pattern for a grouping. I wish there was a more scalable approach to this.

yep, in the end, people need to get their job done, and they hate switching between roles

Ugh that’s why i created a generic role Dev so i can kick people off admin but still give them enough high privileges to get work done

But now that needs least privileging too lol

i sorta would like to see something of a more dynamic “user-oriented” role. like, this user is a member of X groups, which all combined have Y permissions, and when the user “signs in” they get a dynamically created role specific to them that includes just those permissions

I don’t think aws has this functionality out of the box. At the moment, it’s all okta groups with a matching role per group which unfortunately means switching roles in the same account to get things done

There is a way to use okta attributes and using attribute based permissions

yep same

I’ve been taking advantage of that for the security team

So maybe that might be the way to do it. You can add project based attributes and fine tune a large iam policy for each role

The downside is that you’re limited by the iam policy document size for sso

yeah, you can have another service in between that does the kind of mapping i’m talking about, and slaps the Policy attribute on the assume-role call to filter down the role permissions for the user

You mean with an intermediary role ?

no, Policy is one of the attributes for the api call… https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithSAML.html

but that policy can’t exceed 2048 characters

Wow… So you can pass in an additional inline policy from the idp ?

if it supports it, yeah

I wonder if okta supports that… Hmm ok another potential tool

it can’t be used to grant more permissions than the role, but it can filter out permissions you don’t want the session to have

I dont think it supports it because i cannot figure out how to change the request parameters from aws sso identity center or from okta.

However, if i could, this can also be defined PolicyArns.member.N
The Amazon Resource Names (ARNs) of the IAM managed policies that you want to use as managed session policies. The policies must exist in the same account as the role.
… You can provide up to 10 managed policy ARNs. However, the plaintext that you use for both inline and managed session policies can’t exceed 2,048 characters….

10 policies at 2048 chars would be more than enough

Yeah, I’m struggling with the way that limit is written in the doc. Is it 2048 chars per managed policy, plus the inline policy, so 11*2048 in total? Or is it 2048 across all of them?

I think per policy

Worth trying tho

If i can populate those policy members with unique attributes then that would be a game changer

there’s also JIT authorization, using things like common fate… https://docs.commonfate.io/introduction

@Jeremy White (Cloud Posse) @Matt Calhoun

Side note – I realize this is dead simple application code, but we just hate writing any custom code and this feels like it should be a solved, plug-and-play problem.

cc @kevcube as he’s working on this on my team.

Seems like it could be a solved problem

But mabye since it’s so simple, no reason to package it

Yeah I’m afraid of that. And that’s one of my big gripes with AWS: They don’t do a good job with making their services have easy plug-and-play solutions that are package-able so no-one does it. No one building open-source solutions for ECS is a perfect example.

Yea, right?

What DockerHub (or registries) are to Docker images, Helm packages and Helm registries are to Kubernetes.

Oh, and then they don’t need to implement that in 3 different products by 3 different disjoint teams, that don’t even know about each other

I’m so surprised there’s no defacto way of distributing apps for ECS like Helm.

Yeah exactly. If they had built a registry (or marketplace) for Lambdas or ECS then they really would have hit on something. But it’s not in the MO and they miss out on it entirely.

For all that sucks about helm, it still is a big reason in my book why K8s is successful. That said, we’re seeing 5:1 interest in ECS:EKS

Lol, I think they would take offense to that. That’s the AWS Marketplace

Yeah, definitely agree. And seeing similar in regards to ECS vs EKS – People don’t want the complexity unless they have the org + people to manage K8s. And most don’t. And I don’t blame them one bit.

Are you seeing more interest in ECS? (oh jinx)

Wow, interesting

I thought it was maybe just us

But the part that still sucks with ECS, is there’s no Helm. Maybe AWS Copilot can become that?

Haha I see the AWS Marketplace as a “shitty Helm for EC2” and “Use your credits to buy software vendors elsewhere”, but it’s not for everything.

That said… I know nothing about it and have no experience with it.

Even with k8s, there’s this awkward relationship with terraform, cloud dependencies, and kubernetes - that isn’t solved by Helm

Huh I didn’t see copilot solving that problem. Maybe I need to give it another look…

Generally, I wouldn’t trust in AWS to solve the “Registry for ECS” problem. I think it should be open-source.

(haha, yea, marketplace is no substitute. was definitely trolling on that one)

I wouldn’t say it’s “solving” it as described, but rather it’s positioned to solve it.

copilot should have a registry of solutions

I should be able to just “import” or vendor (atmos style!) with copilot from a registry of services

I would love to see that.

But… unless we build it, I have strong doubts we ever will see it.

Related, but not related. Thought they were working on this, but still no movement. https://github.com/aws/copilot-cli/issues/1612#top

We were finally going to try using docker-compose for preview environments with ECS, and then Docker just deprecated the integration.

Ah what a shame.

Are you folks using Copilot today?

No, just eye’ing it

We’re using ecspresso + GHA + Terraform

Terraform = ECS Cluster and tasks (and baseline task def), GHA for CI/CD, and ecspresso for application side deployment and management of task definitions. Basically keep task defs close to the apps, use ecspress to redeploy tasks, and some GHA glue to store task defs in S3 so terraform can redeploy tasks without disruption of the tasks and not ignoring lifecycles. Ignoring lifecycles is how most seem to do it, which is bad. Relying on Terraform for everything is slow.

But still looking for silver bullet.

We use some jq foo to merge terraform task def and ecspresso task def. Not ideal, but the best we could come up with so far to fuse terraform and GHA.

https://github.com/kayac/ecspresso

We built something we’re not really liking right now. We have an GitOps repo like workflow with Atmos config via GHA that is delivered by auto-deploy Spacelift Stacks.

I pushed for it and built part of it… and now I’m realizing that it won’t scale and that we’ll need to rethink it eventually.

We haven’t used ecspresso though. We’ve used ecs-deploy

yea, we started with something like that. Just too slow.

Like terraform apply times were too slow? Even with small root modules?

Terraform applies of ECS tasks in “smallish” root modules.

https://github.com/cloudposse/terraform-aws-components/tree/main/modules/ecs-service

A lot is needed to jigger it up with IAM roles, ALB target groups, Cloud Map, SSM, security groups, etc

Under the hood it also calls our upstream child modules

Just from that list, you can see an ECS task is never “just an ECS task”

…want to autoscale it? Deploy a bunch more resources. Want isolated logs? More resources. Need to wire it up to load balancer? More resources. Need DNS? more resources.

If you can control for what is the exemplar ECS task, not use open source modules, and just write something bare minimum to meet that use-case you might get away with less.

Yeah, good points. Definitely hear you.

cc @Veronika Gnilitska – Check out the above. Relevant to our discussion in 1-1 this morning.

Would love to hear what you come up with… so we can continue to evolve our strategy

@managedkaos I feel like your question today pretty similar to what we were discussing here. I wish copilot or similar tool supported helm-like deployments for ECS

Yes indeed.

I was already following this thread.

you can have a single Lambda that handles both, but it might be better to separate them. Here are a few factors to consider:

• does the daily run require more resources to run efficiently?

• any need to track usage/spending for the event and daily invocations independently?

• would it be easier to manage two code bases? ◦ if you have isolation where one group of devs who care about the event based and another group cares about the daily run, separating might be better I’m sure there are more, but those are usually the key factors I consider when choosing to break things up.

it’s pretty easy to have different lambda handlers (entry points) for the different functions, even in the same code base

got it. Thanks for the suggestion. Basically this for access key management across all environments, when ever we create Iam user with terraform, we have python code that creates access keys for that user and store the keys in ops account secretmanager, so this will be event based trigger, so other one is for automatic rotations of secrets, that requires daily checks

try without /chart

Hi @Adi, I did helm pull public.ecr.aws/aws-controllers-k8s/chart:apigatewayv2-v0.0.2 but I get this error:

Error: repo public.ecr.aws not found

this one worked for me

@Adi bro thank you so much!!!!!!!!! what’s the oci:// for? does every helm command now requires oci://?

for your specific tag which is way old version

@Adi thank you so much again!!!

oci usually on AWS repos what i have saw

https://aws.amazon.com/blogs/containers/oci-artifact-support-in-amazon-ecr/

Dang I would have not found this answer without your help, what did you Google to get this result?

tbh, i had similar issues so i recall what i did

is AWS EKS easy now for you? it’s been so frustrating on my part so far XD

yeah for now its fine, have configured couple of prod EKS, we can sync up if you have any questions

@Adi man that would be cool to sync up, my setup is almost done, I’ve deployed apache kafka, postgres, and some restful apis but I’m hard stuck on API Gateway, I’m so confused on how to integrate this with my running EKS test env.

I’m really noob so I deployed apache kafka without helm (I wish I did because it would have saved me a lot of time but I’m still confused on using Helm)–pure k8 and using bash script in my Dockerfile for them to form a cluster.

Even though I don’t use terraform, I’m currently checking terraform to give me different angles because I’m so confused. I am able to understand it better due to terraform but I don’t know the fine grain details/technicality to execute the things that I want.

use of them, either helm or eksctl, start with uninstalling release and try again with helm

Hi @Adi, dumb question, what is release that needs to be uninstalled? I can install ack controller api gateway via eksctl? Let me check that

@Adi I’ve somehow done it by using terraform, I followed a YouTube tutorial and I was deploy API Gateway with proper EKS integration through VPC

Are you using terraform?

something like this via Deny and NotAction ?

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "sts:*"
      ],
      "Resource": "*"
    }
  ]
}

@Jeremy G (Cloud Posse)

Sorry, I have no recommendation. I’m sure your proposed SCP it too restrictive, but we have not developed an SCP that is highly restrictive and recommended.

I created the following SCP for the identity account which allows me to manage iam roles, assume iam roles, and blocks all other access in this account.

data "aws_iam_policy_document" "default" {
  statement {
    sid       = "DenyAllExcept"
    effect    = "Deny"
    resources = ["*"]
    not_actions = [
      # The identity account should be able to create and manage IAM roles
      # policy_sentry query action-table --service iam | grep -i role
      # policy_sentry query action-table --service iam --access-level read
      "iam:AttachRolePolicy",
      "iam:CreateRole",
      "iam:CreateServiceLinkedRole",
      "iam:DeleteRole",
      "iam:DeleteRolePermissionsBoundary",
      "iam:DeleteRolePolicy",
      "iam:DeleteServiceLinkedRole",
      "iam:DetachRolePolicy",
      "iam:Generate*",
      "iam:Get*",
      "iam:List*",
      "iam:PassRole",
      "iam:PutRolePermissionsBoundary",
      "iam:PutRolePolicy",
      "iam:Simulate*",
      "iam:TagRole",
      "iam:UntagRole",
      "iam:UpdateAssumeRolePolicy",
      "iam:UpdateRole",
      "iam:UpdateRoleDescription",
      # Also need to be able to assume roles into this account as this will be the primary ingress
      "sts:AssumeRole",
      "sts:SetSourceIdentity",
      "sts:TagSession",
    ]

    condition {
      test     = "StringNotLike"
      variable = "aws:PrincipalArn"
      values = [
        # "arn:aws:iam::*:role/this-is-an-exempt-role",
        "arn:aws:iam::*:root",
      ]
    }
  }
}

Hope that helps anyone else secure this account. It was way too easy to accidentally create resources here when incorrectly assuming a secondary role.

Authorization API is an external API? If so, then Lambda Authorizer would be the way.

@dropinboy I use Lambda Authorizer and call my external service that checks the permissions?

correct

there’s a diagram - https://docs.aws.amazon.com/apigateway/latest/developerguide/apigateway-use-lambda-authorizer.html

your Authorizer returns an IAM-like policy back to API Gateway, which then evaluates

@dropinboy ooow! this might be challenging because we have our own custom roles and permissions, is not possible to just return true or false or something?

no, it’s not how it works, it was confusing at first when I used it but

the "Effect": "Allow" or "|Deny" serves the same purpose as true/false

@dropinboy ooow, so I could perhaps just return an okaish allow/deny and good to go since the logic is externally performed?

yes

Oooh I see Isee, thank you so much man! Btw do you have a Lambda Authorizer that calls an external API code I could refer to? :O

hahahahah

nah sorry

it’s just like any Lambda that calls external API

but you configure API Gateway to call the Lambda

AWS has a sample - https://github.com/aws-samples/api-gateway-auth/tree/main/auth0-custom-auth-lambda

@dropinboy thanks bro!

I typically see vendors use an assume-role approach. I create the role in every account, with a trust policy for the vendor account and external id condition

Some vendors create a whole account dedicated to each customer, to provide stronger guarantees about separation of Identity and data between customers

I saw that AWS SSO has an “External AWS account” option but this requires setting up an identity provider and a role in each account..

The upside of that would be that it would be click-and-go instead of assume role (or via browser extensions) but is probably a bit annoying to set up several identity providers + configuring them

Plus there is no Terraform integration to create SAML 2.0 apps so its clickops

Every approach I know of requires some bit of initial bootstrapping for each account to allow the vendor to access it

That part is always on the customer to implement and manage

In this case we have created and solely manage this particular AWS Org for our customer so can’t ask the customer to take care of it

We used to have IAM users in our org and used roles to switch to all accounts, both in the org and in customer orgs but IAM Identity Center seemed like a more userfriendly/polished approach for our developers

hmm. same deal i think. just create the role in the customer account, assume-role into it, after authenticating to your own AWS IdC. and then for the customer to access their own org, they can use their own AWS IdC instance.

Yeah, that sounds like the easiest approach

the nice part of doing it this way is that you can get cli credentials very easily, just aws sso login, and an aws profile that uses role_arn and source_profile to switch roles to the customer account. the “external aws account” approach that aws writes up doesn’t offer any builtin way to do the same, it only offers console/browser/interactive auth

https://sweetops.slack.com/archives/CCT1E7JJY/p1713272331129729?thread_ts=1713249101.578009&cid=CCT1E7JJY

We do something similar to this. In our case, we reserve AWS SSO for the customer and their IDP since it presently only supports one.

Then we implemented aws-teams and aws-team-roles which we use for Cloud Posse (and customer if they want it). It takes the Federated IAM with SAML approach, so it can be connected with any number of IdPs.

This let’s us self-direct access during an engagement

Here’s our implementation in our reference architecture: https://github.com/cloudposse/terraform-aws-components/tree/main/modules/aws-teams

@Erik Osterman (Cloud Posse) Thanks! Do you generally only provision admin / viewonly and spacelift roles to client accounts?

To elaborate on my answer, we deploy the AWS teams and team roles for two purposes. The first purpose is for granular access controls for machine systems like GitHub Action that need OIDC access to one or more accounts (e.g. to run terraform), the other purpose is to connect our IDP into the customer managed organization that was also provision by CloudPosse using our reference architecture. This connection is what gets shut down at the end of an engagement with the customer, and allows us during an engagement to manage our own teams access to that organization, providing continuity. AWS SSO is what the customer is using to authenticate. From time to time the customer also decides they want to use the federated IAM approach with SAML, for example if they have multiple IDPs (think enterprise).

Now to answer your question, we use the approach where each customer vendors in these components. Then they can define additional custom roles, but placed in files that do not conflict with the vendored files. We use atmos to do the vendoring and manage the configuration which gets passed to terraform via tf vars.

There’s too many plausible reasons. If you share the planned output, it would point to the reason

thanks erik, let me see if I can give the plan masking the info

Solution:

When MongoDB creates the peering connection it defaults to the default subnet, since I was using two other subnets: private/public is I had to manually define in the routes table the routes for my private subnet by adding the MongoDB instance IP & the generated peering connection ID in the “Peering connections” tab

It can really depend on how many pods are moving and the size of their docker images

Also your apps tolerance for downtime and how many pods in total you are running to support your apps

In reality spot termination is not that bad, depending on instance types in use and region. Check the spot weather maps.

The bigger risk is consolidation

thanks for the reply. Yeah, I like the spot instances and I like to have aws termination handler, but because management cannot allow enough overprovisioned nodes per zone, sometimes I see a lot of pods in pending because we lost almost all node in one zone.

awesome job about weather map. thanks will implement into our cluster and will try to proactively do some things before “storm”

Are you using the built-in termination handling in Karpenter or the separate controller

sorry for late reply. I am using a separate one https://github.com/aws/aws-node-termination-handler with cluster autoscaler

Please see this https://karpenter.sh/docs/faq/#should-i-use-karpenter-interruption-handling-alongside-node-termination-handler

rtb is a route table. Routes are added to route tables, not the transit gateway directly

yes I know, but if you check the id of the route table it contains tgw-rtb<id>… for me it contains

oh my bad… used a wrong aws cli command