#aws (2024-04)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS)

Archive: https://archive.sweetops.com/aws/

2024-04-01

Sean Turner avatar
Sean Turner

Do yall use Pod Security Groups with EKS? Have you ever noticed and issue where two Pods with the same Security Group are unable to communicate?

I encountered this previously I believe with a Jupyterhub deployment on EKS, and now seem to be encountering it with a Retool deployment.

I have a self referencing rule in place. Perhaps I need to also open HTTP from the EKS Node Security Group? Perhaps any sort of traffic needs to be opened on the Node Security Group.

Now that I think about it a hybrid Pod Security Group + Network Policy would probably be best. Use the Pod Security Group to secure access to RDS, use Network Policies to control everything else

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Jeremy G (Cloud Posse)

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

We do not use Pod Security Groups with EKS, and I have no experience with them, so I can’t be of much help.

2
Sean Turner avatar
Sean Turner

Do you usually use NetworkPolicies then? Perhaps with the ALB Ingress?

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Jeremy G (Cloud Posse) @Andriy Knysh (Cloud Posse)

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

We use EC2 Security Groups. By default, security groups block all inbound traffic. That includes traffic from within the security group. You need to add a rule to allow inbound traffic, which is what I hope you mean by “a self-referencing rule”.

We generally allow everything in the EKS cluster to talk to everything else in the EKS cluster. People wanting finer-grained access control typically use a service mesh, although they are very difficult to manage, so using the newly available Kubernetes controls is probably a better plan.

Beyond that, I can’t help you.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Security groups for Pods - Amazon EKS

Learn about how to assign unique security groups to individual Pods.

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Important

The security group or groups that you specify for your Pods must meet the following criteria:


They must allow inbound communication from the security group applied to your nodes (for kubelet) over any ports that you've configured probes for.

They must allow outbound communication over TCP and UDP ports 53 to a security group assigned to the Pods (or nodes that the Pods run on) running CoreDNS. The security group for your CoreDNS Pods must allow inbound TCP and UDP port 53 traffic from the security group that you specify.

They must have necessary inbound and outbound rules to communicate with other Pods that they need to communicate with.
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it says that the Pod SG must have an ingress rule to allow traffic from the Node SG on all ports that your Pods are using

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

it also needs an egress rule on port 53 for DNS resolution (if you are restricting egress, need to open the port)

Sean Turner avatar
Sean Turner

Thanks for the responses.

We’re ending up on the same approach as well.

The issue where two pods with the same security group were unable to communicate I believe was due to the VPC CNI plugin configuring POD_SECURITY_GROUP_ENFORCING_MODE as strict rather than standard. Once that was changed everything worked.

Previously we had been opening ports on the security group pod for liveness probes and the like, but now we’ve got a prefix list on the security group ingress rule which specifies all of the EKS Private Subnet CIDRs.

1

2024-04-03

2024-04-05

maarten avatar
maarten

Just came accoss this: https://github.com/basti-app/basti , nice one. Instant SSM Forwarding.

Basti (from Bastion Host) is a CLI tool for securely accessing your DB instances and other AWS resources in private networks at almost no cost.

basti-app/basti

Securely connect to RDS, Elasticache, and other AWS resources in VPCs with no idle cost

5

2024-04-06

Ryan avatar

man I did not know you could backend cloudwatch dashboards with html and lambda, TIL

2024-04-07

RB avatar

What is the best way to define aws sso role names and permissions/responsibilities?

Should it be team based, role based, account-based, or a mixture? Looking for some prior art on this and having trouble finding good blog posts.

RB avatar

Thank you!

loren avatar

We use something of a combination. Team-focused roles, plus additional roles where maybe the team doesn’t quite align right with the desired accesses. Like readonly access is available to more folks, and parts of the billing console is available to fewer

1
loren avatar

But we also do account per team, so it’s sorta an admin-ish role for the team/account

1
loren avatar

then sometimes we have teams that build services for the wider environment, crossing accounts. they get their own permission set mapped to accounts where they provide service

RB avatar

Ah interesting.

Thanks Loren. Thats helpful. I was creating a plan cause at the moment, there are too many admins. Let me know if this is a good summary.

• Team based accounts (data accounts, security account, etc)

• Team focused roles (dataengineer, ops)

• Generic roles (dev, readonly, admin, billing) Anything else i could add? A lot of this is going to have iam perms created by sleuthing through cloudtrail logs

RB avatar

One drawback wirh being admin of a team account is being able to change account settings that could expose things like s3 buckets, so i want to build a limitedadmin role if possible (or just gate everything with root scps)

loren avatar

i think in the end, the most important part is to realize what you don’t know, and just have a good pipeline for building groups, permission sets, and mapping them to accounts lol

1
RB avatar


then sometimes we have teams that build services for the wider environment, crossing accounts. they get their own permission set mapped to accounts where they provide service
Naming these seem challenging.

Have you settled on a naming convention?

loren avatar

and yes, totally, we gate everything with scps. well, i mean, if any role has iam admin-ish permissions, then we put a lot of effort into scps, since that role could create other roles that bypass protections built into their own role

loren avatar

no particular naming convention at this time. naming is too hard. we sorta use something that feels like the team/role/service/access that the permission set has. readonly, billing-admin, project-admin, etc

RB avatar

Thanks. It seems like it’s not a one and done type of management. Currently the least privileged roles i help maintain, slowly expand until i think of a new named pattern for a grouping. I wish there was a more scalable approach to this.

1
loren avatar

yep, in the end, people need to get their job done, and they hate switching between roles

RB avatar

Ugh that’s why i created a generic role Dev so i can kick people off admin but still give them enough high privileges to get work done

RB avatar

But now that needs least privileging too lol

loren avatar

i sorta would like to see something of a more dynamic “user-oriented” role. like, this user is a member of X groups, which all combined have Y permissions, and when the user “signs in” they get a dynamically created role specific to them that includes just those permissions

1
RB avatar

I don’t think aws has this functionality out of the box. At the moment, it’s all okta groups with a matching role per group which unfortunately means switching roles in the same account to get things done

RB avatar

There is a way to use okta attributes and using attribute based permissions

loren avatar

yep same

RB avatar

I’ve been taking advantage of that for the security team

RB avatar

So maybe that might be the way to do it. You can add project based attributes and fine tune a large iam policy for each role

RB avatar

The downside is that you’re limited by the iam policy document size for sso

loren avatar

yeah, you can have another service in between that does the kind of mapping i’m talking about, and slaps the Policy attribute on the assume-role call to filter down the role permissions for the user

RB avatar

You mean with an intermediary role ?

loren avatar

no, Policy is one of the attributes for the api call… https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithSAML.html

AssumeRoleWithSAML - AWS Security Token Service

Returns a set of temporary security credentials for users who have been authenticated via a SAML authentication response. This operation provides a mechanism for tying an enterprise identity store or directory to role-based AWS access without user-specific credentials or configuration. For a comparison of

loren avatar

but that policy can’t exceed 2048 characters

RB avatar

Wow… So you can pass in an additional inline policy from the idp ?

loren avatar

if it supports it, yeah

RB avatar

I wonder if okta supports that… Hmm ok another potential tool

loren avatar

it can’t be used to grant more permissions than the role, but it can filter out permissions you don’t want the session to have

RB avatar

I dont think it supports it because i cannot figure out how to change the request parameters from aws sso identity center or from okta.

However, if i could, this can also be defined PolicyArns.member.N
The Amazon Resource Names (ARNs) of the IAM managed policies that you want to use as managed session policies. The policies must exist in the same account as the role.
… You can provide up to 10 managed policy ARNs. However, the plaintext that you use for both inline and managed session policies can’t exceed 2,048 characters….

RB avatar

10 policies at 2048 chars would be more than enough

loren avatar

Yeah, I’m struggling with the way that limit is written in the doc. Is it 2048 chars per managed policy, plus the inline policy, so 11*2048 in total? Or is it 2048 across all of them?

RB avatar

I think per policy

RB avatar

Worth trying tho

RB avatar

If i can populate those policy members with unique attributes then that would be a game changer

1
loren avatar

there’s also JIT authorization, using things like common fate… https://docs.commonfate.io/introduction

Introduction - Common Fateattachment image

An overview of the Common Fate platform.

2024-04-09

Matt Gowie avatar
Matt Gowie

Does anyone here know of any open-source (or paid even) connector Lambda project to take items off of an SQS queue and put them into a bucket on S3?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy White (Cloud Posse) @Matt Calhoun

Matt Gowie avatar
Matt Gowie

Side note – I realize this is dead simple application code, but we just hate writing any custom code and this feels like it should be a solved, plug-and-play problem.

this1
Matt Gowie avatar
Matt Gowie

cc @kevcube as he’s working on this on my team.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Seems like it could be a solved problem

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But mabye since it’s so simple, no reason to package it

Matt Gowie avatar
Matt Gowie

Yeah I’m afraid of that. And that’s one of my big gripes with AWS: They don’t do a good job with making their services have easy plug-and-play solutions that are package-able so no-one does it. No one building open-source solutions for ECS is a perfect example.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, right?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What DockerHub (or registries) are to Docker images, Helm packages and Helm registries are to Kubernetes.

1
this1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh, and then they don’t need to implement that in 3 different products by 3 different disjoint teams, that don’t even know about each other

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m so surprised there’s no defacto way of distributing apps for ECS like Helm.

Matt Gowie avatar
Matt Gowie

Yeah exactly. If they had built a registry (or marketplace) for Lambdas or ECS then they really would have hit on something. But it’s not in the MO and they miss out on it entirely.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For all that sucks about helm, it still is a big reason in my book why K8s is successful. That said, we’re seeing 5:1 interest in ECS:EKS

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Lol, I think they would take offense to that. That’s the AWS Marketplace

Matt Gowie avatar
Matt Gowie

Yeah, definitely agree. And seeing similar in regards to ECS vs EKS – People don’t want the complexity unless they have the org + people to manage K8s. And most don’t. And I don’t blame them one bit.

this1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you seeing more interest in ECS? (oh jinx)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Wow, interesting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I thought it was maybe just us

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But the part that still sucks with ECS, is there’s no Helm. Maybe AWS Copilot can become that?

Matt Gowie avatar
Matt Gowie

Haha I see the AWS Marketplace as a “shitty Helm for EC2” and “Use your credits to buy software vendors elsewhere”, but it’s not for everything.

That said… I know nothing about it and have no experience with it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Even with k8s, there’s this awkward relationship with terraform, cloud dependencies, and kubernetes - that isn’t solved by Helm

Matt Gowie avatar
Matt Gowie

Huh I didn’t see copilot solving that problem. Maybe I need to give it another look…

Generally, I wouldn’t trust in AWS to solve the “Registry for ECS” problem. I think it should be open-source.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(haha, yea, marketplace is no substitute. was definitely trolling on that one)

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I wouldn’t say it’s “solving” it as described, but rather it’s positioned to solve it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

copilot should have a registry of solutions

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I should be able to just “import” or vendor (atmos style!) with copilot from a registry of services

Matt Gowie avatar
Matt Gowie

I would love to see that.

Matt Gowie avatar
Matt Gowie

But… unless we build it, I have strong doubts we ever will see it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Related, but not related. Thought they were working on this, but still no movement. https://github.com/aws/copilot-cli/issues/1612#top

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We were finally going to try using docker-compose for preview environments with ECS, and then Docker just deprecated the integration.

2
Matt Gowie avatar
Matt Gowie

Ah what a shame.

Matt Gowie avatar
Matt Gowie

Are you folks using Copilot today?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No, just eye’ing it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re using ecspresso + GHA + Terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Terraform = ECS Cluster and tasks (and baseline task def), GHA for CI/CD, and ecspresso for application side deployment and management of task definitions. Basically keep task defs close to the apps, use ecspress to redeploy tasks, and some GHA glue to store task defs in S3 so terraform can redeploy tasks without disruption of the tasks and not ignoring lifecycles. Ignoring lifecycles is how most seem to do it, which is bad. Relying on Terraform for everything is slow.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But still looking for silver bullet.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We use some jq foo to merge terraform task def and ecspresso task def. Not ideal, but the best we could come up with so far to fuse terraform and GHA.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
kayac/ecspresso

ecspresso is a deployment tool for Amazon ECS

Matt Gowie avatar
Matt Gowie

We built something we’re not really liking right now. We have an GitOps repo like workflow with Atmos config via GHA that is delivered by auto-deploy Spacelift Stacks.

I pushed for it and built part of it… and now I’m realizing that it won’t scale and that we’ll need to rethink it eventually.

Matt Gowie avatar
Matt Gowie

We haven’t used ecspresso though. We’ve used ecs-deploy

fabfuel/ecs-deploy

Powerful CLI tool to simplify Amazon ECS deployments, rollbacks & scaling

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, we started with something like that. Just too slow.

Matt Gowie avatar
Matt Gowie

Like terraform apply times were too slow? Even with small root modules?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Terraform applies of ECS tasks in “smallish” root modules.

https://github.com/cloudposse/terraform-aws-components/tree/main/modules/ecs-service

A lot is needed to jigger it up with IAM roles, ALB target groups, Cloud Map, SSM, security groups, etc

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Under the hood it also calls our upstream child modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/ecs-cloudwatch-autoscaling/aws
cloudposse/route53-alias/aws
cloudposse/alb-ingress/aws
cloudposse/ecs-alb-service-task/aws
cloudposse/ecs-container-definition/aws
cloudposse/cloudwatch-logs/aws
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just from that list, you can see an ECS task is never “just an ECS task”

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…want to autoscale it? Deploy a bunch more resources. Want isolated logs? More resources. Need to wire it up to load balancer? More resources. Need DNS? more resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you can control for what is the exemplar ECS task, not use open source modules, and just write something bare minimum to meet that use-case you might get away with less.

Matt Gowie avatar
Matt Gowie

Yeah, good points. Definitely hear you.

Matt Gowie avatar
Matt Gowie

cc @Veronika Gnilitska – Check out the above. Relevant to our discussion in 1-1 this morning.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Would love to hear what you come up with… so we can continue to evolve our strategy

2
kevcube avatar
kevcube

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@managedkaos I feel like your question today pretty similar to what we were discussing here. I wish copilot or similar tool supported helm-like deployments for ECS

managedkaos avatar
managedkaos

Yes indeed.

managedkaos avatar
managedkaos

I was already following this thread.

1
Hari avatar

Hi, I have Lambda function with python code, which takes an event and runs, but there are couple of funtions that needs to run everyday, can i still use one lambda to do both tasks or split to two different lambdas, one for event based and other for daily run?

Darren Cunningham avatar
Darren Cunningham

you can have a single Lambda that handles both, but it might be better to separate them. Here are a few factors to consider:

• does the daily run require more resources to run efficiently?

• any need to track usage/spending for the event and daily invocations independently?

• would it be easier to manage two code bases? ◦ if you have isolation where one group of devs who care about the event based and another group cares about the daily run, separating might be better I’m sure there are more, but those are usually the key factors I consider when choosing to break things up.

loren avatar

it’s pretty easy to have different lambda handlers (entry points) for the different functions, even in the same code base

Hari avatar

got it. Thanks for the suggestion. Basically this for access key management across all environments, when ever we create Iam user with terraform, we have python code that creates access keys for that user and store the keys in ops account secretmanager, so this will be event based trigger, so other one is for automatic rotations of secrets, that requires daily checks

2024-04-10

miko avatar

(Resolved, thanks @Adi)

He guys, I am following this tutorial API Gateway + AWS EKS but helm chart pull is deprecated so I went with helm pull public.ecr.aws/aws-controllers-k8s/chart:apigatewayv2-v0.0.2 but I am getting this error Error: public.ecr.aws/aws-controllers-k8s/chart:v0.0.2: not found , I have been searching for answers online but can’t find anything, my apologies–I am really new to all of this and AWS EKS making my life hell at the moment

Adi avatar

try without /chart

miko avatar

Hi @Adi, I did helm pull public.ecr.aws/aws-controllers-k8s/chart:apigatewayv2-v0.0.2 but I get this error:

Error: repo public.ecr.aws not found

Adi avatar
helm pull <oci://public.ecr.aws/aws-controllers-k8s/apigatewayv2-chart>
1
Adi avatar

this one worked for me

miko avatar

@Adi bro thank you so much!!!!!!!!! what’s the oci:// for? does every helm command now requires oci://?

Adi avatar

for your specific tag which is way old version

Adi avatar
helm pull <oci://public.ecr.aws/aws-controllers-k8s/apigatewayv2-chart> --version v0.0.2
1
miko avatar

@Adi thank you so much again!!!

Adi avatar

oci usually on AWS repos what i have saw

miko avatar

Dang I would have not found this answer without your help, what did you Google to get this result?

Adi avatar

tbh, i had similar issues so i recall what i did

miko avatar

is AWS EKS easy now for you? it’s been so frustrating on my part so far XD

Adi avatar

yeah for now its fine, have configured couple of prod EKS, we can sync up if you have any questions

miko avatar

@Adi man that would be cool to sync up, my setup is almost done, I’ve deployed apache kafka, postgres, and some restful apis but I’m hard stuck on API Gateway, I’m so confused on how to integrate this with my running EKS test env.

I’m really noob so I deployed apache kafka without helm (I wish I did because it would have saved me a lot of time but I’m still confused on using Helm)–pure k8 and using bash script in my Dockerfile for them to form a cluster.

Even though I don’t use terraform, I’m currently checking terraform to give me different angles because I’m so confused. I am able to understand it better due to terraform but I don’t know the fine grain details/technicality to execute the things that I want.

2024-04-11

miko avatar

Hey guys, I am following this tutorial API Gateway + AWS EKS but I get error trying to run this command (I’ve been searching the internet for days and I’m still far from finishing this :cry:):

$ helm install \
$  --namespace kube-system \
$  ack-$SERVICE-controller \
$  $CHART_REPO/$SERVICE-chart \
$  --version=$RELEASE_VERSION \
$  --set=aws.region=$AGW_AWS_REGION

I get this error:

Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "ack-apigatewayv2-controller" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: key "app.kubernetes.io/managed-by" must equal "Helm": current value is "eksctl"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "ack-apigatewayv2-controller"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"

I’ve tried modifying the ServiceAccount to use Helm instead of eksctl but to no avail:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::************:role/*****-addon-iamserviceaccoun-Role1-************
  creationTimestamp: "2024-04-11T03:59:39Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ack-apigatewayv2-controller
  namespace: kube-system
  resourceVersion: "4654570"
  uid: ********-****-****-****-************

But I’m still getting error, am not really sure what I’m doing wrong, I’m really new to this and super noob at it.

Adi avatar

use of them, either helm or eksctl, start with uninstalling release and try again with helm

miko avatar

Hi @Adi, dumb question, what is release that needs to be uninstalled? I can install ack controller api gateway via eksctl? Let me check that

miko avatar

@Adi I’ve somehow done it by using terraform, I followed a YouTube tutorial and I was deploy API Gateway with proper EKS integration through VPC

Are you using terraform?

2024-04-12

RB avatar

What is a good SCP for the identity account ? Is it only iam roles that belong here so perhaps ?

RB avatar

something like this via Deny and NotAction ?

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "sts:*"
      ],
      "Resource": "*"
    }
  ]
}
Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Jeremy G (Cloud Posse)

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

Sorry, I have no recommendation. I’m sure your proposed SCP it too restrictive, but we have not developed an SCP that is highly restrictive and recommended.

RB avatar

I created the following SCP for the identity account which allows me to manage iam roles, assume iam roles, and blocks all other access in this account.

data "aws_iam_policy_document" "default" {
  statement {
    sid       = "DenyAllExcept"
    effect    = "Deny"
    resources = ["*"]
    not_actions = [
      # The identity account should be able to create and manage IAM roles
      # policy_sentry query action-table --service iam | grep -i role
      # policy_sentry query action-table --service iam --access-level read
      "iam:AttachRolePolicy",
      "iam:CreateRole",
      "iam:CreateServiceLinkedRole",
      "iam:DeleteRole",
      "iam:DeleteRolePermissionsBoundary",
      "iam:DeleteRolePolicy",
      "iam:DeleteServiceLinkedRole",
      "iam:DetachRolePolicy",
      "iam:Generate*",
      "iam:Get*",
      "iam:List*",
      "iam:PassRole",
      "iam:PutRolePermissionsBoundary",
      "iam:PutRolePolicy",
      "iam:Simulate*",
      "iam:TagRole",
      "iam:UntagRole",
      "iam:UpdateAssumeRolePolicy",
      "iam:UpdateRole",
      "iam:UpdateRoleDescription",
      # Also need to be able to assume roles into this account as this will be the primary ingress
      "sts:AssumeRole",
      "sts:SetSourceIdentity",
      "sts:TagSession",
    ]

    condition {
      test     = "StringNotLike"
      variable = "aws:PrincipalArn"
      values = [
        # "arn:aws:iam::*:role/this-is-an-exempt-role",
        "arn:aws:iam::*:root",
      ]
    }
  }
}

Hope that helps anyone else secure this account. It was way too easy to accidentally create resources here when incorrectly assuming a secondary role.

2024-04-14

miko avatar

Hey guys, I’m currently using AWS API Gateway and need to check if user have enough permission (JWT scope), all the tutorials that I’ve seen so far was using either Lambda or AWS Cognito but in my case I have an Authorization API that check for user permission. Now I’m thinking of using Lambda and from there call my Authorization API to check if user have enough permission, is there other optimal way of doing this?

dropinboy avatar
dropinboy

Authorization API is an external API? If so, then Lambda Authorizer would be the way.

miko avatar

@dropinboy I use Lambda Authorizer and call my external service that checks the permissions?

dropinboy avatar
dropinboy

correct

dropinboy avatar
dropinboy

your Authorizer returns an IAM-like policy back to API Gateway, which then evaluates

miko avatar

@dropinboy ooow! this might be challenging because we have our own custom roles and permissions, is not possible to just return true or false or something?

dropinboy avatar
dropinboy

no, it’s not how it works, it was confusing at first when I used it but

dropinboy avatar
dropinboy

the "Effect": "Allow" or "|Deny" serves the same purpose as true/false

miko avatar

@dropinboy ooow, so I could perhaps just return an okaish allow/deny and good to go since the logic is externally performed?

dropinboy avatar
dropinboy

yes

miko avatar

Oooh I see Isee, thank you so much man! Btw do you have a Lambda Authorizer that calls an external API code I could refer to? :O

dropinboy avatar
dropinboy

hahahahah

dropinboy avatar
dropinboy

nah sorry

dropinboy avatar
dropinboy

it’s just like any Lambda that calls external API

dropinboy avatar
dropinboy

but you configure API Gateway to call the Lambda

miko avatar

@dropinboy thanks bro!

1

2024-04-15

Frank avatar

For those using AWS SSO (Identity Center) and manage multiple AWS Orgs (e.g. from customers): How do you arrange access to those? Just by assuming IAM roles in those accounts or is there a better method?

loren avatar

I typically see vendors use an assume-role approach. I create the role in every account, with a trust policy for the vendor account and external id condition

loren avatar

Some vendors create a whole account dedicated to each customer, to provide stronger guarantees about separation of Identity and data between customers

Frank avatar

I saw that AWS SSO has an “External AWS account” option but this requires setting up an identity provider and a role in each account..

Frank avatar

The upside of that would be that it would be click-and-go instead of assume role (or via browser extensions) but is probably a bit annoying to set up several identity providers + configuring them

Frank avatar

Plus there is no Terraform integration to create SAML 2.0 apps so its clickops

loren avatar

Every approach I know of requires some bit of initial bootstrapping for each account to allow the vendor to access it

loren avatar

That part is always on the customer to implement and manage

Frank avatar

In this case we have created and solely manage this particular AWS Org for our customer so can’t ask the customer to take care of it

Frank avatar

We used to have IAM users in our org and used roles to switch to all accounts, both in the org and in customer orgs but IAM Identity Center seemed like a more userfriendly/polished approach for our developers

loren avatar

hmm. same deal i think. just create the role in the customer account, assume-role into it, after authenticating to your own AWS IdC. and then for the customer to access their own org, they can use their own AWS IdC instance.

Frank avatar

Yeah, that sounds like the easiest approach

loren avatar

the nice part of doing it this way is that you can get cli credentials very easily, just aws sso login, and an aws profile that uses role_arn and source_profile to switch roles to the customer account. the “external aws account” approach that aws writes up doesn’t offer any builtin way to do the same, it only offers console/browser/interactive auth

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

https://sweetops.slack.com/archives/CCT1E7JJY/p1713272331129729?thread_ts=1713249101.578009&cid=CCT1E7JJY

We do something similar to this. In our case, we reserve AWS SSO for the customer and their IDP since it presently only supports one.

In this case we have created and solely manage this particular AWS Org for our customer so can’t ask the customer to take care of it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then we implemented aws-teams and aws-team-roles which we use for Cloud Posse (and customer if they want it). It takes the Federated IAM with SAML approach, so it can be connected with any number of IdPs.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

This let’s us self-direct access during an engagement

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Here’s our implementation in our reference architecture: https://github.com/cloudposse/terraform-aws-components/tree/main/modules/aws-teams

Frank avatar

@Erik Osterman (Cloud Posse) Thanks! Do you generally only provision admin / viewonly and spacelift roles to client accounts?

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

To elaborate on my answer, we deploy the AWS teams and team roles for two purposes. The first purpose is for granular access controls for machine systems like GitHub Action that need OIDC access to one or more accounts (e.g. to run terraform), the other purpose is to connect our IDP into the customer managed organization that was also provision by CloudPosse using our reference architecture. This connection is what gets shut down at the end of an engagement with the customer, and allows us during an engagement to manage our own teams access to that organization, providing continuity. AWS SSO is what the customer is using to authenticate. From time to time the customer also decides they want to use the federated IAM approach with SAML, for example if they have multiple IDPs (think enterprise).

Now to answer your question, we use the approach where each customer vendors in these components. Then they can define additional custom roles, but placed in files that do not conflict with the vendored files. We use atmos to do the vendoring and manage the configuration which gets passed to terraform via tf vars.

2

2024-04-16

TechHippie avatar
TechHippie

Hello All - Is there a way to get the entire OU path for a aws account using boto3?

2024-04-17

jonjitsu avatar
jonjitsu

In regards to using amazonMQ with lambdas is it better to allow it to drive the lambdas or would it be better to forward the messages to SQS that can better drive lambdas (at least there would be concurrency). Also anyone have experience using amazonMQ as a way to get messages from an on prem rabbitmq (either federated with it or by configuring on prem to use shovel plugin to send messages to amazonMQ).

2024-04-23

Prasanth avatar
Prasanth

wave Hello, team!

Prasanth avatar
Prasanth

Hi All, I am unable to publish the code version using AWS CodeArtifact. Can you assist me with this?

Juan Pablo Lorier avatar
Juan Pablo Lorier

Hi, I’ve being using the ecs modules for a while but lately, something changed and if I add a permission or modify a cluster, terraform is redeploying all the services. Is there a way I can track the reason behind this? It’s pointing that the json task definition changed, while it didn’t. Also modifying the permissions created by the modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

There’s too many plausible reasons. If you share the planned output, it would point to the reason

Juan Pablo Lorier avatar
Juan Pablo Lorier

thanks erik, let me see if I can give the plan masking the info

2024-04-24

miko avatar

Status: CLOSED

Has anyone here ever used peering to establish MongoDB connection within their AWS EKS cluster before? I’ve created an AWS EKS cluster with VPC private/public subnets, I have contained my RESTful APIs in the private subnet and exposed my frontend clients in the public subnet and now I have to allow my RESTful APIs contained inside the private subnet to establish connection with MongoDB via peering connections but I can’t get it to work even though I’ve already supplied my VPC’s id and region and region of my MongoDB instance. However when I use my default VPC as peering I was able to establish connection inside of my EC2 that is using the default VPC.

miko avatar

Solution:

When MongoDB creates the peering connection it defaults to the default subnet, since I was using two other subnets: private/public is I had to manually define in the routes table the routes for my private subnet by adding the MongoDB instance IP & the generated peering connection ID in the “Peering connections” tab

1
Balazs Varga avatar
Balazs Varga

hello all, we use spot instances in our kube cluster… we have termination handler that checks the rebalance recommendation signal. Is that safe if I wait only for the 2 minutes signal ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

It can really depend on how many pods are moving and the size of their docker images

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also your apps tolerance for downtime and how many pods in total you are running to support your apps

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

In reality spot termination is not that bad, depending on instance types in use and region. Check the spot weather maps.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The bigger risk is consolidation

Balazs Varga avatar
Balazs Varga

thanks for the reply. Yeah, I like the spot instances and I like to have aws termination handler, but because management cannot allow enough overprovisioned nodes per zone, sometimes I see a lot of pods in pending because we lost almost all node in one zone.

Balazs Varga avatar
Balazs Varga

awesome job about weather map. thanks will implement into our cluster and will try to proactively do some things before “storm”

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you using the built-in termination handling in Karpenter or the separate controller

Balazs Varga avatar
Balazs Varga

sorry for late reply. I am using a separate one https://github.com/aws/aws-node-termination-handler with cluster autoscaler

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
FAQsattachment image

Review Karpenter Frequently Asked Questions

2024-04-26

2024-04-30

Balazs Varga avatar
Balazs Varga

my transit gateway ID contains tgw- when I would like to add a route do I need to remove the tgw part? the create-route accepts only rtb…

Nat Williams avatar
Nat Williams

rtb is a route table. Routes are added to route tables, not the transit gateway directly

Balazs Varga avatar
Balazs Varga

yes I know, but if you check the id of the route table it contains tgw-rtb<id>… for me it contains

Balazs Varga avatar
Balazs Varga

oh my bad… used a wrong aws cli command

    keyboard_arrow_up