#aws (2024-04)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS)

Archive: https://archive.sweetops.com/aws/

2024-04-01

Sean Turner avatar
Sean Turner

Do yall use Pod Security Groups with EKS? Have you ever noticed and issue where two Pods with the same Security Group are unable to communicate?

I encountered this previously I believe with a Jupyterhub deployment on EKS, and now seem to be encountering it with a Retool deployment.

I have a self referencing rule in place. Perhaps I need to also open HTTP from the EKS Node Security Group? Perhaps any sort of traffic needs to be opened on the Node Security Group.

Now that I think about it a hybrid Pod Security Group + Network Policy would probably be best. Use the Pod Security Group to secure access to RDS, use Network Policies to control everything else

Gabriela Campana (Cloud Posse) avatar
Gabriela Campana (Cloud Posse)

@Jeremy G (Cloud Posse)

Jeremy G (Cloud Posse) avatar
Jeremy G (Cloud Posse)

We do not use Pod Security Groups with EKS, and I have no experience with them, so I can’t be of much help.

2

2024-04-03

2024-04-05

maarten avatar
maarten

Just came accoss this: https://github.com/basti-app/basti , nice one. Instant SSM Forwarding.

Basti (from Bastion Host) is a CLI tool for securely accessing your DB instances and other AWS resources in private networks at almost no cost.

basti-app/basti

Securely connect to RDS, Elasticache, and other AWS resources in VPCs with no idle cost

5

2024-04-06

Ryan avatar

man I did not know you could backend cloudwatch dashboards with html and lambda, TIL

2024-04-07

RB avatar

What is the best way to define aws sso role names and permissions/responsibilities?

Should it be team based, role based, account-based, or a mixture? Looking for some prior art on this and having trouble finding good blog posts.

RB avatar

Thank you!

loren avatar

We use something of a combination. Team-focused roles, plus additional roles where maybe the team doesn’t quite align right with the desired accesses. Like readonly access is available to more folks, and parts of the billing console is available to fewer

1
loren avatar

But we also do account per team, so it’s sorta an admin-ish role for the team/account

1
loren avatar

then sometimes we have teams that build services for the wider environment, crossing accounts. they get their own permission set mapped to accounts where they provide service

RB avatar

Ah interesting.

Thanks Loren. Thats helpful. I was creating a plan cause at the moment, there are too many admins. Let me know if this is a good summary.

• Team based accounts (data accounts, security account, etc)

• Team focused roles (dataengineer, ops)

• Generic roles (dev, readonly, admin, billing) Anything else i could add? A lot of this is going to have iam perms created by sleuthing through cloudtrail logs

RB avatar

One drawback wirh being admin of a team account is being able to change account settings that could expose things like s3 buckets, so i want to build a limitedadmin role if possible (or just gate everything with root scps)

loren avatar

i think in the end, the most important part is to realize what you don’t know, and just have a good pipeline for building groups, permission sets, and mapping them to accounts lol

1
RB avatar


then sometimes we have teams that build services for the wider environment, crossing accounts. they get their own permission set mapped to accounts where they provide service
Naming these seem challenging.

Have you settled on a naming convention?

loren avatar

and yes, totally, we gate everything with scps. well, i mean, if any role has iam admin-ish permissions, then we put a lot of effort into scps, since that role could create other roles that bypass protections built into their own role

loren avatar

no particular naming convention at this time. naming is too hard. we sorta use something that feels like the team/role/service/access that the permission set has. readonly, billing-admin, project-admin, etc

RB avatar

Thanks. It seems like it’s not a one and done type of management. Currently the least privileged roles i help maintain, slowly expand until i think of a new named pattern for a grouping. I wish there was a more scalable approach to this.

1
loren avatar

yep, in the end, people need to get their job done, and they hate switching between roles

RB avatar

Ugh that’s why i created a generic role Dev so i can kick people off admin but still give them enough high privileges to get work done

RB avatar

But now that needs least privileging too lol

loren avatar

i sorta would like to see something of a more dynamic “user-oriented” role. like, this user is a member of X groups, which all combined have Y permissions, and when the user “signs in” they get a dynamically created role specific to them that includes just those permissions

1
RB avatar

I don’t think aws has this functionality out of the box. At the moment, it’s all okta groups with a matching role per group which unfortunately means switching roles in the same account to get things done

RB avatar

There is a way to use okta attributes and using attribute based permissions

loren avatar

yep same

RB avatar

I’ve been taking advantage of that for the security team

RB avatar

So maybe that might be the way to do it. You can add project based attributes and fine tune a large iam policy for each role

RB avatar

The downside is that you’re limited by the iam policy document size for sso

loren avatar

yeah, you can have another service in between that does the kind of mapping i’m talking about, and slaps the Policy attribute on the assume-role call to filter down the role permissions for the user

RB avatar

You mean with an intermediary role ?

loren avatar

no, Policy is one of the attributes for the api call… https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRoleWithSAML.html

AssumeRoleWithSAML - AWS Security Token Service

Returns a set of temporary security credentials for users who have been authenticated via a SAML authentication response. This operation provides a mechanism for tying an enterprise identity store or directory to role-based AWS access without user-specific credentials or configuration. For a comparison of

loren avatar

but that policy can’t exceed 2048 characters

RB avatar

Wow… So you can pass in an additional inline policy from the idp ?

loren avatar

if it supports it, yeah

RB avatar

I wonder if okta supports that… Hmm ok another potential tool

loren avatar

it can’t be used to grant more permissions than the role, but it can filter out permissions you don’t want the session to have

RB avatar

I dont think it supports it because i cannot figure out how to change the request parameters from aws sso identity center or from okta.

However, if i could, this can also be defined PolicyArns.member.N
The Amazon Resource Names (ARNs) of the IAM managed policies that you want to use as managed session policies. The policies must exist in the same account as the role.
… You can provide up to 10 managed policy ARNs. However, the plaintext that you use for both inline and managed session policies can’t exceed 2,048 characters….

RB avatar

10 policies at 2048 chars would be more than enough

loren avatar

Yeah, I’m struggling with the way that limit is written in the doc. Is it 2048 chars per managed policy, plus the inline policy, so 11*2048 in total? Or is it 2048 across all of them?

RB avatar

I think per policy

RB avatar

Worth trying tho

RB avatar

If i can populate those policy members with unique attributes then that would be a game changer

1
loren avatar

there’s also JIT authorization, using things like common fate… https://docs.commonfate.io/introduction

Introduction - Common Fateattachment image

An overview of the Common Fate platform.

2024-04-09

Matt Gowie avatar
Matt Gowie

Does anyone here know of any open-source (or paid even) connector Lambda project to take items off of an SQS queue and put them into a bucket on S3?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy White (Cloud Posse) @Matt Calhoun

Matt Gowie avatar
Matt Gowie

Side note – I realize this is dead simple application code, but we just hate writing any custom code and this feels like it should be a solved, plug-and-play problem.

this1
Matt Gowie avatar
Matt Gowie

cc @kevcube as he’s working on this on my team.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Seems like it could be a solved problem

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But mabye since it’s so simple, no reason to package it

Matt Gowie avatar
Matt Gowie

Yeah I’m afraid of that. And that’s one of my big gripes with AWS: They don’t do a good job with making their services have easy plug-and-play solutions that are package-able so no-one does it. No one building open-source solutions for ECS is a perfect example.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Yea, right?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

What DockerHub (or registries) are to Docker images, Helm packages and Helm registries are to Kubernetes.

1
this1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Oh, and then they don’t need to implement that in 3 different products by 3 different disjoint teams, that don’t even know about each other

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I’m so surprised there’s no defacto way of distributing apps for ECS like Helm.

Matt Gowie avatar
Matt Gowie

Yeah exactly. If they had built a registry (or marketplace) for Lambdas or ECS then they really would have hit on something. But it’s not in the MO and they miss out on it entirely.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

For all that sucks about helm, it still is a big reason in my book why K8s is successful. That said, we’re seeing 5:1 interest in ECS:EKS

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Lol, I think they would take offense to that. That’s the AWS Marketplace

Matt Gowie avatar
Matt Gowie

Yeah, definitely agree. And seeing similar in regards to ECS vs EKS – People don’t want the complexity unless they have the org + people to manage K8s. And most don’t. And I don’t blame them one bit.

this1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Are you seeing more interest in ECS? (oh jinx)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Wow, interesting

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I thought it was maybe just us

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But the part that still sucks with ECS, is there’s no Helm. Maybe AWS Copilot can become that?

Matt Gowie avatar
Matt Gowie

Haha I see the AWS Marketplace as a “shitty Helm for EC2” and “Use your credits to buy software vendors elsewhere”, but it’s not for everything.

That said… I know nothing about it and have no experience with it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Even with k8s, there’s this awkward relationship with terraform, cloud dependencies, and kubernetes - that isn’t solved by Helm

Matt Gowie avatar
Matt Gowie

Huh I didn’t see copilot solving that problem. Maybe I need to give it another look…

Generally, I wouldn’t trust in AWS to solve the “Registry for ECS” problem. I think it should be open-source.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

(haha, yea, marketplace is no substitute. was definitely trolling on that one)

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I wouldn’t say it’s “solving” it as described, but rather it’s positioned to solve it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

copilot should have a registry of solutions

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I should be able to just “import” or vendor (atmos style!) with copilot from a registry of services

Matt Gowie avatar
Matt Gowie

I would love to see that.

Matt Gowie avatar
Matt Gowie

But… unless we build it, I have strong doubts we ever will see it.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Related, but not related. Thought they were working on this, but still no movement. https://github.com/aws/copilot-cli/issues/1612#top

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We were finally going to try using docker-compose for preview environments with ECS, and then Docker just deprecated the integration.

2
Matt Gowie avatar
Matt Gowie

Ah what a shame.

Matt Gowie avatar
Matt Gowie

Are you folks using Copilot today?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

No, just eye’ing it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We’re using ecspresso + GHA + Terraform

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Terraform = ECS Cluster and tasks (and baseline task def), GHA for CI/CD, and ecspresso for application side deployment and management of task definitions. Basically keep task defs close to the apps, use ecspress to redeploy tasks, and some GHA glue to store task defs in S3 so terraform can redeploy tasks without disruption of the tasks and not ignoring lifecycles. Ignoring lifecycles is how most seem to do it, which is bad. Relying on Terraform for everything is slow.

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

But still looking for silver bullet.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We use some jq foo to merge terraform task def and ecspresso task def. Not ideal, but the best we could come up with so far to fuse terraform and GHA.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
kayac/ecspresso

ecspresso is a deployment tool for Amazon ECS

Matt Gowie avatar
Matt Gowie

We built something we’re not really liking right now. We have an GitOps repo like workflow with Atmos config via GHA that is delivered by auto-deploy Spacelift Stacks.

I pushed for it and built part of it… and now I’m realizing that it won’t scale and that we’ll need to rethink it eventually.

Matt Gowie avatar
Matt Gowie

We haven’t used ecspresso though. We’ve used ecs-deploy

fabfuel/ecs-deploy

Powerful CLI tool to simplify Amazon ECS deployments, rollbacks & scaling

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yea, we started with something like that. Just too slow.

Matt Gowie avatar
Matt Gowie

Like terraform apply times were too slow? Even with small root modules?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Terraform applies of ECS tasks in “smallish” root modules.

https://github.com/cloudposse/terraform-aws-components/tree/main/modules/ecs-service

A lot is needed to jigger it up with IAM roles, ALB target groups, Cloud Map, SSM, security groups, etc

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Under the hood it also calls our upstream child modules

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
cloudposse/ecs-cloudwatch-autoscaling/aws
cloudposse/route53-alias/aws
cloudposse/alb-ingress/aws
cloudposse/ecs-alb-service-task/aws
cloudposse/ecs-container-definition/aws
cloudposse/cloudwatch-logs/aws
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Just from that list, you can see an ECS task is never “just an ECS task”

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

…want to autoscale it? Deploy a bunch more resources. Want isolated logs? More resources. Need to wire it up to load balancer? More resources. Need DNS? more resources.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

If you can control for what is the exemplar ECS task, not use open source modules, and just write something bare minimum to meet that use-case you might get away with less.

Matt Gowie avatar
Matt Gowie

Yeah, good points. Definitely hear you.

Matt Gowie avatar
Matt Gowie

cc @Veronika Gnilitska – Check out the above. Relevant to our discussion in 1-1 this morning.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Would love to hear what you come up with… so we can continue to evolve our strategy

2
kevcube avatar
kevcube

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@managedkaos I feel like your question today pretty similar to what we were discussing here. I wish copilot or similar tool supported helm-like deployments for ECS

managedkaos avatar
managedkaos

Yes indeed.

managedkaos avatar
managedkaos

I was already following this thread.

1
Hari avatar

Hi, I have Lambda function with python code, which takes an event and runs, but there are couple of funtions that needs to run everyday, can i still use one lambda to do both tasks or split to two different lambdas, one for event based and other for daily run?

Darren Cunningham avatar
Darren Cunningham

you can have a single Lambda that handles both, but it might be better to separate them. Here are a few factors to consider:

• does the daily run require more resources to run efficiently?

• any need to track usage/spending for the event and daily invocations independently?

• would it be easier to manage two code bases? ◦ if you have isolation where one group of devs who care about the event based and another group cares about the daily run, separating might be better I’m sure there are more, but those are usually the key factors I consider when choosing to break things up.

loren avatar

it’s pretty easy to have different lambda handlers (entry points) for the different functions, even in the same code base

Hari avatar

got it. Thanks for the suggestion. Basically this for access key management across all environments, when ever we create Iam user with terraform, we have python code that creates access keys for that user and store the keys in ops account secretmanager, so this will be event based trigger, so other one is for automatic rotations of secrets, that requires daily checks

2024-04-10

miko avatar

(Resolved, thanks @Adi)

He guys, I am following this tutorial API Gateway + AWS EKS but helm chart pull is deprecated so I went with helm pull public.ecr.aws/aws-controllers-k8s/chart:apigatewayv2-v0.0.2 but I am getting this error Error: public.ecr.aws/aws-controllers-k8s/chart:v0.0.2: not found , I have been searching for answers online but can’t find anything, my apologies–I am really new to all of this and AWS EKS making my life hell at the moment

Adi avatar

try without /chart

miko avatar

Hi @Adi, I did helm pull public.ecr.aws/aws-controllers-k8s/chart:apigatewayv2-v0.0.2 but I get this error:

Error: repo public.ecr.aws not found

Adi avatar
helm pull <oci://public.ecr.aws/aws-controllers-k8s/apigatewayv2-chart>
1
Adi avatar

this one worked for me

miko avatar

@Adi bro thank you so much!!!!!!!!! what’s the oci:// for? does every helm command now requires oci://?

Adi avatar

for your specific tag which is way old version

Adi avatar
helm pull <oci://public.ecr.aws/aws-controllers-k8s/apigatewayv2-chart> --version v0.0.2
1
miko avatar

@Adi thank you so much again!!!

Adi avatar

oci usually on AWS repos what i have saw

miko avatar

Dang I would have not found this answer without your help, what did you Google to get this result?

Adi avatar

tbh, i had similar issues so i recall what i did

miko avatar

is AWS EKS easy now for you? it’s been so frustrating on my part so far XD

Adi avatar

yeah for now its fine, have configured couple of prod EKS, we can sync up if you have any questions

miko avatar

@Adi man that would be cool to sync up, my setup is almost done, I’ve deployed apache kafka, postgres, and some restful apis but I’m hard stuck on API Gateway, I’m so confused on how to integrate this with my running EKS test env.

I’m really noob so I deployed apache kafka without helm (I wish I did because it would have saved me a lot of time but I’m still confused on using Helm)–pure k8 and using bash script in my Dockerfile for them to form a cluster.

Even though I don’t use terraform, I’m currently checking terraform to give me different angles because I’m so confused. I am able to understand it better due to terraform but I don’t know the fine grain details/technicality to execute the things that I want.

2024-04-11

miko avatar

Hey guys, I am following this tutorial API Gateway + AWS EKS but I get error trying to run this command (I’ve been searching the internet for days and I’m still far from finishing this :cry:):

$ helm install \
$  --namespace kube-system \
$  ack-$SERVICE-controller \
$  $CHART_REPO/$SERVICE-chart \
$  --version=$RELEASE_VERSION \
$  --set=aws.region=$AGW_AWS_REGION

I get this error:

Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "ack-apigatewayv2-controller" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: key "app.kubernetes.io/managed-by" must equal "Helm": current value is "eksctl"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "ack-apigatewayv2-controller"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kube-system"

I’ve tried modifying the ServiceAccount to use Helm instead of eksctl but to no avail:

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::************:role/*****-addon-iamserviceaccoun-Role1-************
  creationTimestamp: "2024-04-11T03:59:39Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: ack-apigatewayv2-controller
  namespace: kube-system
  resourceVersion: "4654570"
  uid: ********-****-****-****-************

But I’m still getting error, am not really sure what I’m doing wrong, I’m really new to this and super noob at it.

Adi avatar

use of them, either helm or eksctl, start with uninstalling release and try again with helm

miko avatar

Hi @Adi, dumb question, what is release that needs to be uninstalled? I can install ack controller api gateway via eksctl? Let me check that

miko avatar

@Adi I’ve somehow done it by using terraform, I followed a YouTube tutorial and I was deploy API Gateway with proper EKS integration through VPC

Are you using terraform?

2024-04-12

RB avatar

What is a good SCP for the identity account ? Is it only iam roles that belong here so perhaps ?

RB avatar

something like this via Deny and NotAction ?

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "NotAction": "iam:*",
      "Resource": "*"
    }
  ]
}

2024-04-14

miko avatar

Hey guys, I’m currently using AWS API Gateway and need to check if user have enough permission (JWT scope), all the tutorials that I’ve seen so far was using either Lambda or AWS Cognito but in my case I have an Authorization API that check for user permission. Now I’m thinking of using Lambda and from there call my Authorization API to check if user have enough permission, is there other optimal way of doing this?

dropinboy avatar
dropinboy

Authorization API is an external API? If so, then Lambda Authorizer would be the way.

    keyboard_arrow_up