#aws (2022-09)

first customer managed policies and permission boundaries, now user and group management! hurrah! now if they’d just separate it from the org and make it a standalone service, i’d be ecstatic!

It would be possible but it sounds like a bad idea

Since buckets have a global namespace, there’s no guarantee you will always get the bucket name you want.

But more importantly, complex IAM policies are a special circle of hell all by themselves. Why would you change something that works for something that’s clever?

Because we’re hitting the hard caps

X policies * y customers is approaching 5k, so we’re trying to figure how to cut that down while keeping least access

Makes sense. If you ask AWS support, they will write policies like this for you

Conditions and abac are hard

Could I suggest instead an AWS credentials vending machine, which lambda uses to get credentials that are scoped directly to the relevant bucket via a role that has the customer account imbedded in it?

It might also help for me to understand what actions you are taking with the bucket in question to give a recommendation

It’s really just mostly get/put operations. Wanting to make sure that regardless of what code gets uploaded to a lambda, make sure from a policy perspective. That the trigger can only operate on the bucket that it was triggered from

I’d probably create an intermediary that can do the validation and generate a presigned get/post to pass to the lambda to trigger. Then it doesn’t even need any credentials

@Andriy Knysh (Cloud Posse)

we did not use it (looks like a new feature), but it looks like it requires two steps to enable this:

https://www.eksworkshop.com/beginner/115_sg-per-pod/30_cni_config/

which can be done here w/o modifying the module https://github.com/cloudposse/terraform-aws-eks-node-group/blob/master/variables.tf#L98

or could be added here as another policy attachment (requires module modifications) https://github.com/cloudposse/terraform-aws-eks-node-group/blob/master/iam.tf#L39

steps #2 to execute

kubectl -n kube-system set env daemonset aws-node ENABLE_POD_ENI=true

Yeah I saw there qas an additional IAM policy needed to the role which I didn’t see as hard to accomplish, as you said it could be an additional policy attached to the role not needed to be done in the module per se. I was however not seeing anything apparent to set the necessary env variable to ‘true’ as I can see node groups deployed via the module have it set to ‘false’ bit that seems like just default values

This was more an exploratory inquiry but I have been asked to deploy out a Windows node group to our EKS cluster and preferably via TF

you mean you want to set ENABLE_POD_ENI via TF and not calling kubectl -n kube-system set env daemonset aws-node ENABLE_POD_ENI=true ?

@Andriy Knysh (Cloud Posse) Yes, I was curious if the TF module already supported a way to set this or if otherwise possible to set using the TF as if we went with using it would like to deploy out with the TF not execute additional CLI commands. Right now without it you end up with node level security groups which are fine if you trust all pods running in the cluster on those nodes, I’m just looking into LOE to enable pod level security groups with existing deployment method that could reduce the effective blast radius.

k8s resources can be provisioned using terraform kubernetes provider, but I’m not sure what can be used to set env

Not entirely sure but you will need to provide the license secret key in the AWS marketplace under the product listing.

https://www.buymeacoffee.com/nyshawnb

try A alias instead of CNAME

Cannot cause the original record (app.x.io) is a CNAME and A alias is looking for A recored

This is a really oddball solution but, if you have the stomach for it:

1. create an S3 bucket website with 0 content and a rule to redirect requests to app.x.com
2. create a route 53 entry for app.x.io and add the S3 bucket as the target.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/how-to-page-redirect.html

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/RoutingToS3Bucket.html

Note that this solution is the most cost effective (compared to running a webserver on EC2/ECS or using an ALB).

Before you create the bucket, keep these points in mind (since this is the only way it will work)
Value/Route traffic to
Choose Alias to S3 website endpoint, then choose the Region that the endpoint is from.
Choose the bucket that has the same name that you specified for Record name.
The list includes a bucket only if the bucket meets the following requirements:
• The name of the bucket is the same as the name of the record that you’re creating.
• The bucket is configured as a website endpoint.
• The bucket was created by the current AWS account.

This one it the most important:
• The name of the bucket is the same as the name of the record that you’re creating.

CPU value Memory value (MiB) 256 (.25 vCPU) 512 (0.5GB), 1024 (1GB), 2048 (2GB) 512 (.5 vCPU) 1024 (1GB), 2048 (2GB), 3072 (3GB), 4096 (4GB) 1024 (1 vCPU) 2048 (2GB), 3072 (3GB), 4096 (4GB), 5120 (5GB), 6144 (6GB), 7168 (7GB), 8192 (8GB) 2048 (2 vCPU) Between 4096 (4GB) and 16384 (16GB) in increments of 1024 (1GB) 4096 (4 vCPU) Between 8192 (8GB) and 30720 (30GB) in increments of 1024 (1GB)

that are the allowed combinations (copied from the documentation)

Thank you

so, in your case: 2048 (2 vCPU) Between 4096 (4GB) and 16384 (16GB) in increments of 1024 (1GB)

We have tags that specify:

• Owner (business unit, service name)

• Source (source repo and path)

• Environment

we called CostCenter what you have as chargeback I guess. I would use camelCase or similar naming for all tags but not mixed dash or underscore. we had additional info on classification e.g data classification for s3 bucket also service tier could be a good addition imo. plus I see you have product owner but I would add product as well just for grouping, tech contact also missing… e.g. LauchedBy / OwnerTeam etc

This is expected behavior, and it’s based on the number of subnets. For example, we’re deployed in us-east-2, so there are 3 subnets, and our MNG is set to 1/1/1, so it spins up 2 new nodes in each AZ, before settling on one.

Hi you likely just need to add the text record to route53

Basically go into route53 and create the record they tell you with the value they provided you with

^ not sure if that helps or not. The txt records allows them to verify that they can give you a cert for the domain. Otherwise you could request a cert for any domain and easily get a cert

Maybe list all active connections and verify if its the same IP for attack and block them with a security_group rule

doesn’t sound like IP restrictions would help if the attack is SQL injection, you’ll want to kill the processes that are eating the CPU then patch the application(s) ASAP. If this is going to take “too long” you might choose to make your application connection to the DB read-only and/or potentially take an outage. but these are all considerations for the business team.

I don’t think I can help you on this front but I am genuinely curious about your use case here. What kind of an EKS service is it that you need the CloudFront to deliver? I’ve only used CF for static websites and presenting static large media files, so that’s why I’m asking.

running a nodejs app, it has some static elements but a bunch of dynamic stuff as well. using cache-headers per endpoint to dictate to CF when stuff should be cached or not but it gives me a single endpoint for all the content.

If they are public hosted zones, then yes it’s easy. In the zone hosting example.com, create ns records for subdomain.example.com and you’re golden

you can have [example.com](http://example.com) set up in Account A and [subdomain.example.com](http://subdomain.example.com) in Account B — you would just setup the NS from Account A to point the Nameservers for Account B

If they are private hosted zones, you need to do magic with route53 resolver rules, since private zones do not honor ns records

but yeah great callout @loren this assumes publicly hosted zones

Ah no — Sorry misunderstanding. I’ve done a hosted zone delegation like example.com in one account and then *.subdomain.example.com in another account.

What I’m trying to do is:

Account One (Legacy) — Existing Hosted Zone for example.com Account Two (New) — New Hosted Zone for example.com

I want records that are created in both Hosted Zones to work. And then I’ll be creating delegated (e.g. *.subdomain.example.com) hosted zones in other accounts.

I don’t think the account boundaries actually matter, but it’s just to illustrate the point: This is because I’m working with a client who has all of their resources in one account right now and we’re building out a proper account hierarchy for them now.

I’m re-reading my initial question and I see how I made that confusing, my bad.

No, I don’t think you can do that? I’m trying to think how the ns records would look… You might be able to create the zones and records, but at some point you have to transfer the public ns records so public name servers resolve from the new zone… It’s basically a zone transfer

Ah this is from the AWS Route53 FAQs:
Q. Can I create multiple hosted zones for the same domain name?

Yes. Creating multiple hosted zones allows you to verify your DNS setting in a “test” environment, and then replicate those settings on a “production” hosted zone. For example, hosted zone Z1234 might be your test version of example.com, hosted on name servers ns-1, ns-2, ns-3, ns-4, ns-5, ns-6. Similarly, hosted zone Z5678 might be your production version of example.com, hosted on ns-7, ns-8, ns-9, ns-10, ns-11 and ns-12. Since each hosted zone has a virtual set of name servers associated with that zone, Route 53 will answer DNS queries for example.com differently depending on which name server you send the DNS query to.

But that doesn’t sound like what I would want…

what’s the goal of having both zones handling queries? not doubting you, just making sure I’m not recommending something that breaks the goal

I don’t want to touch the client’s existing Hosted Zone in their legacy all-in-one account. I’d rather leave that alone as it is and then manage a new hosted zone for all new records and delegated zones.

you can setup a delegated administrator account from your org settings and within that account you can configure security hub to automatically enroll all member accounts

@Darren Cunningham from security hubs side, everything looks fine. I can see the accounts in my organization. I believe my problem is with aws config. I am not sure on how to enable it in member accounts. Does delegated administrator account handle enabling aws config?

ah sorry, IIRC AWS config has “delegated admin” but the rollout of enabling AWS Config in all accounts/regions is not something that’s integrated into the product but there is a CF StackSet that’s provided: https://docs.aws.amazon.com/securityhub/latest/userguide/securityhub-prereq-config.html#config-how-to-enable

I only have 3 accounts per environment(qa, prod, staging, management) what do you think is the difference between enabling the app config manually vs deploying the stackset?

well AWS Config also needs to be deployed per region so it’s accounts x regions which doesn’t sound fun to do manually

thank you after enabling config in the regions that my resources are in and after 24 hours, I was able to see the security scores in the management account’s security hub dashboard.

I looked at the worker nodes (ec2) launch template / user data. The kubelet root path was not the standard /var/lib/kubelet. Instead it was a different one. I fixed the missing CSINode driver information by updating the volumes host paths with the correct kubelet root path.

it’s a lot easier now that they introduced https://docs.aws.amazon.com/cli/latest/reference/organizations/close-account.html

but still has it’s limits

you mean something like this? https://github.com/aws-samples/aws-securityhub-remediations/tree/main/aws-ecr-continuouscompliance

good find @Maciek Strömich - it’s what I was looking for

Correct, only saml

https://docs.aws.amazon.com/singlesignon/latest/userguide/other-idps.html and it this reqs still valid? 1.1 saml ?

• IAM Identity Center requires a SAML nameID format of email address (that is, urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress).

solved.

I am using ECS fargate and I think I can integrate it with S3 and could point S3 inside task definitation instead of ECR?

because there is lot of cost with ECR private endpoints. ECR use S3 internally then why not use S3 instead of ECR with free s3 gateway endpoint.

could you please elaborate on why isn’t worth it?

I don’t have all the data points about your situation, so I could be wrong. but, in general (IMO) the more off script you go, the more complexity you have. complexity has operational costs (more difficult to onboard other team members to home grown solution), takes longer to modify your solution in the event that it breaks or needs upgraded as service change and you end up finding yourself on the outside looking in when improvements to integrations are made by AWS.

Agree… increasing complexity to save a buck is a bad idea, at a minimum use an EC2 with S3 backed storage and create a private registry via docker. Again costs may be a wash in that scenario

Thanks guys

Could you self sign a cert and then any certs could then just be imported into ACM ?

The drawback with that is that you’d have to renew your certs manually I believe

We need to be able to easily create/revoke SSL certificates, since these need to be deployed on IoT devices

any reason you need to use a private ca instead of amazon’s ca ?

We need to create and deploy Private certificates, we will use these to connect to an MQTT broker

Hashicorp vault can work as a Private CA, I think. Not sure how much cheaper it would be, especially if you need HA

How does the broker know the cert is valid?

You can revoke public ACM certificates. It requires a support request so the viability of this depends on how many certs you revoke

Private CA - look at easyrsa (this is used by openVPN for their certificates) https://github.com/OpenVPN/easy-rsa

AWS doesn’t natively support step up authorization for multiparty, you would need a dedicated solution for that. Or find a provider that offers off the shelf support

Do you know any?

I think I would need to know more about your specific use case to make a suggestion. Would you be able to add more color to it?

We just would like to add another layer to security. So on base step. Everybody would get read only access to aws console and/or progmatic access, and few person ( admins) can get admin access if there is any issue in the system, but for the access they would need to get an approvement. Using IAM roles

Usually I’ve found if you need this, something organizationally has gone wrong. Like more than ~8 people using one AWS account. The ROI on segregating AWS accounts at the team boundary is sooo high compared to implementing something like this. Also it won’t work with the console, you’d need something custom outside and it would only support API interactions.

what do you meant ~8 people using one aws account. you mean 8 admin in the account? we will implement organization and separate deployments and infra part to separate UO and subaccount. This 4eyes is a big dream from one of the manager and that’s why I am trying to find something to implement.

No I mean 8 people per account

yeah that is correct currently. My idea and goal is at the end to create account per customer. ok for dev part maybe we will have more than 8 users, but per customer hope it will be 1 / account

I’m not sure what “customer” is here

How does “8 people per account” work with EKS?

we don’t use eks. we create cluster with kops and using spot instances. the clusters are not too big. we did not have any issue with that currently. customer = companies that bought our product. we have an isolation by design request.

Don’t use EKS. I’ll let you know when I encounter a problem at scale that requires its usage

And what is your advice to fullfill manager’s request?:)

I tell them to read Turn The Ship Around

let me add my 2 cents to the conversation above. I would grant STS AssumeRole access to the users. so when you add a user to a specific role you can get approval for that… But when a user uses the role all you can do is log the activities… so no approval in this granularity is possible and this would make approver’s life a nightmare… logging should be more than enough…

AWS has several products that can do this

AWS Systems Manager Inventory is specifically for EC2

Hmm, I’d not looked at it that way,

that’s useful to think about

AWS Config may also be appropriate

config can get me the data, but it’s not great at displaying it

Why do you want to display it? A wall dashboard?

something like that yes

see the status of a specific deployed fleet

It might be reasonable to get data off the AWS event bus and push it to your own system then (like datadog)

I can ‘get’ the data with ease, visualising it in a human format that’s simple to read is where I’m looking for inspiriation

cli or web interface?

Might be an easier question to ask during #office-hours

ECS fargate is probably the lowest complexity approach. Use the same compute type for all your apps for simplicity

I think EKS will increase cost and complexity. ECS Fargate would be best option.

Scouting around here an in other slack orgs, so far I’ve gathered (in this order of preference):

• https://github.com/godaddy/aws-okta-processor

• https://github.com/Nike-Inc/gimme-aws-creds

• https://github.com/Versent/saml2aws

I use the first one and like it a lot. It has the best model for understanding that there are two tokens/credentials with different expirations (one for okta, one or more per aws role) and managing them separately

The other option I’ve used is AWS SSO, with okta as the external IDP for AWS SSO, and SCIM syncing all users and groups… Then you can use anything that understands AWS SSO

When doing that, I like granted a lot, https://github.com/common-fate/granted

Or leaap, but leaap has problems with govcloud and other non-aws partitions

I’m looking into AWS SSO, thanks

Looks like granted is geared towards browser access?

I’m looking for a pure CLI tool (if I can login to the browser via CLI that’s a bonus but not the main goal)

Not necessarily, with granted, assume will export the creds to your env, and assume -c will open the console in a browser container-tab, and assume exec will just run the command with the credential

They’re also working on a credential_process version that won’t muck with the env and will support a refreshable credential

Nice

Versent/saml2aws is neat and supports Okta pretty well.

Another vote for saml2aws

I tried out aws-okta-processor. So far so good. I just don’t see a way to switch between roles. I have to run rm -rf ~/.aws/boto/cache/ every time and have to do another eval $(aws-okta-processor authenticate --environment)

Oh I use it with credential_process, to avoid polluting my env and get a refreshable credential for free. So I have a different aws-cli profile for every role

I think for your use case, there is a cli option to disable the cache

–no-aws-cache

Cheers, gonna keep exploring

https://docs.leapp.cloud/0.14.3/configuring-session/configure-aws-iam-role-federated/

you can add your okta federated role in Leapp

for AWS SSO it integrates automatically,

what are the issues with Leapp and cloudgov @loren ? I would love to solve those issues.

Let me know if this docs are solving your problem @yegorski

https://github.com/Noovolari/leapp/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+author%3Alorengordon

Thank you so much, I will look to solve those bug asap

Leapp definitely the “wow factor” looks great

let me know if you have any question on this, or need any help

I see, I need a “valid SSO portal URL” so to use this tool I need to set up AWS SSO. This requires changing how I currently have our 6 AWS accounts connected, via federated login

What do you mean? Your accounts are connected via SAML with Okta?

Yeah right all accounts are connected with Okta SAML, going through the main AWS org account.

Old topic but I’ve used https://github.com/dowjones/tokendito in the past.

Thanks! Yep, that’s on my radar

We decided to stick with aws-okta

Does it has activated deletion protection in his configs? Unless this should not stop you from modified it.

thanks, I’ll check it

Nope, even deletion protection has been removed, when you try to delete an instance it asks to start cluster first

I’ve seen that, for both serverless v2 and provisioned RDS instances

I can’t tell you why, maybe it is to run the last snapshot? I do have last snapshot turned on and haven’t tried it without

Hi! im not an expert, i used the service once or twice but with other porpouse, i think DMS have to work for that use case (that is the original use case i guess). You can test it setting the source endpoint the RDS N°1 and for the target endpoint the RDS N°2. Be carefoul if you are using secrets manager for the passwords and make a diferent secret for each RDS or you can accidentally replicate the data in the original RDS

@Aritra Banerjee: DMS indeed does support RDS like any other source or destination. When you choose ‘rds’ as a source, you can now select it from a drop-down list instead of having to manually enter host/port/service details, so that’s a bit nicer. in the docs, you’ll notice that both source and targets both list RDS as a potential.

• https://medium.com/team-pratilipi/how-to-migrate-rds-to-rds-via-dms-b8f9b86f23c

• https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Sources.html

• https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Introduction.Targets.html

wow, great thank you

What do you mean, open search “wants” the latter format? The two forms are functionally identical.

Perhaps you could specify the statement as raw JSON rather than using the Terraform data source

Seems to be an AWS quirk, https://github.com/hashicorp/terraform/issues/6438

If you have a statement with a wildcard and add a second statement AWS will barf if you don’t listify the principals in the second statement

Even trying to adding it through the AWS Console will fail until you add the []s

try using just jsonencode instead of iam_policy_document?

difficult to answer without knowing what the problem is

I have account under infra ou and account under dev ou. I have s3 bucket in infra that ai would like to access from account under dev ou. But only one bucket and only from that account. I get 403 when I try to download a file from that bucket

When you’re not using a wildcard, pretty sure you should be using the “ForAnyValue:StringEquals” operator

got same error, when I set it to GetObject and try to download file

if I add a ‘*/ after r-stuff… it works… I limited the access to getobject and if I add the condition like this:

 o-funny/r-stuff/*/ou-path/*

can somebody explain why I need the * in the condition? second i think I know, but the 1st ?

solved

port 53 is the DNS port. Why are you telnetting a host of port 53?

If so how can I reach the instance via RDP on the private subnet?
This depends on where are you connecting from? Are you connecting from another Windows instance in one of your private subnets?

If yes, then simply, go into your hosts server manager > IPV4 > Go into the respective network interface setting > Advanced > DNS

set your DNS entries there.

Yes that’s right it’s a windows vm on a private subnet

change your DNS on that Windows server. If you change AWS DHCP options, then you will have wider issues

The engineer sent me a screenshot of what I should see when doing telnet (53), he claims I should be able to connect

Inside the vm? Alright thank you so much!

Once I change the dns on the windows server, what would be the easiest way to rdp into it?

Simply launch RDP from your machine and connect to the other machine

I am a bit confused, the vm is inside a private subnet and it only has a private ip, can I still access it just with rdp without any vpn or bastion host?

ok, tell me this…

From where are you trying to access the VM? From your laptop? Or from another Windows host in your VPC?

From my laptop

Are you conencting your laptop to a VPN?

Not at the moment, I used to use SSM Sessions and RDP into it but changing the dns inside the vm makes the host unreachable

in all, your laptop and the vm you are trying to connect must be in the network that have a route between each other. Currently how is this routing established?

Via ssm, it opens a port and it allows me to access the vm on that given port in localhost

if you are able to access the VM via SSM what is the issue that you want resolved?

you can also use FLEET MANAGER

to directly RDP from AWS console

I think I get you

you log into that VM using SSM, and on the VM, launch server manager > > IPV4 > Go into the respective network interface setting > Advanced > DNS and set the DNS on the VM

after changing DNS on the VM as described above, you can still connect to it via SSM

Fleet manager? That sounds good I will definitely give it a go, thank you so much!

Hi did you find a solution for this?

Not one I particularly like… I manually upgraded the cluster and node group through console and then updated terraform version to match. I found if I changed the version in the terraform then the plan would fail.