#aws (2022-08)

I’m thinking a role per device is not the best approach. Maybe you can have fewer roles that are based on the device class/location/etc. That way you can have one role for “Ohio” devices and another for “Utah” devices (just giving examples). The roles would be tightly locked down to only allow writes to CloudWatch and S3 based on their class/location/etc. I would also limit access to CloudWatch and S3 by the class as well…no need to give any more permissions if you are worried about compromise of the device.

If we used a role per location like “Ohio” for example and a single device is compromised how would we just deny access to that device instead of all of Ohio since they’ll be using the same role?

Isn’t this what you want? https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_revoke-sessions.html

I guess I’m not sure what sort of compromise you’re trying to mitigate.

Indeed if you had long-lived AWS cred sitting on disk for each device, some gets those and you have to reset everything everywhere.

With IAM creds they expire regularly and are regenerated. So if someone grabs those creds (and the associated session token) they are only good for a short time.

However, if someone is camped on your device, yes, they could likely use the role creds.

Is it the case that someone could sit on your device undetected? How would they connect? If its via SSH, have you considered removing SSH access altogether and using an SSM agent?

I have lots of questions!

These devices sit on utility poles and have been stolen in the past.

If these are IOT devices (vs server devices) perhaps there are managed IOT services that you can use in AWS for the same purpose (CloudWatch logging and uploads).

The idea is to brick them, not necessarily to brick the OS but the running application

got it

Does each device have its own log group?

This is a good one for #office-hours

@Joe Niland - Yes they do. The cloudwatch agent is sending logs to their respective log groups

@Erik Osterman (Cloud Posse) - Thanks for posting the link. It looks like the only option is to revoke ALL sessions for a particular role. I’m looking to revoke or deny a device session without affecting the rest of the devices.

I’m wondering if the IoT services have any faculty for helping with that. I’m thinking of the TF plans you’ll have to run to manage all those roles and its giving me chills

But seriously, at that scale, I’m thinking there should be a better approach than having to manage 10k roles across multiple accounts

@Patrick McDonald and they each have their own IAM user right?

I’ve just recently taken a look at IAM Roles Anywhere… From what I’ve understood so far, you need a CA to serve as trust anchor. In my case I’m using Hashicorp Vault for my POC testing. If you issued unique certificates for each device under the CA then if one were to become compromised you should be able to revoke the certificate of that device. The CRL URL should be part of the CA certificate and I would assume it would be queried to check if the certificate was validate besides just being signed by the CA. This would allow to use the same role if the devices didn’t require unique IAM permissions but still have unique device authentication.

Jeremy, that’s an interesting approach which makes total sense. I’m going to look into that! Thank you.

IAM isn’t really designed to be supported at scale in an account for your users like that. The solution that we usually help our customers implement is using our platform to generate private/public keys per device, and then verifying those keys on your service side. Exposing all of these clients direct access to your cloudwatch and S3, is a huge risk. Throwing a CF + lambda@edge plus service client authentication goes a long way. If you really want to allow direct access, because you consider the huge risk to be worth the small decrease in cost, you could proxy the requests through CF/APIGW directly to the service API.

Alternatively, you could take the private/public key signed JWT that you have (either custom built or using our platform) and use cognito identity pool to vend temporary AWS tokens.

Another question i have is the network access. That is, is the device using a private network when in service? That could be another way to lock down access by limiting access to devices on the private network. That assumes if the device is compromised/stolen and then connects to a different network, access to AWS resources would be blocked.

check out today’s office hours recording for some good suggestions

is the recording already published?

@Patrick McDonald https://www.youtube.com/watch?v=zANiSr_PzcQ&ab_channel=CloudPosse

Yes

i ahve added nlb and lab both but when i add nlb target group to the ecs service it says InvalidParameterException: loadBalancerName and targetGroupArn cannot both be specified. You must specify either a loadBalancerName or a targetGroupArn.

i am using cloudposse ecs-alb-service-task ecs-container-definition`

You have to set the load balancer name to null and give it multiple target groups that are attached as listener rules to the same load balancer

@RB Thanks man I did that it worked thanks but the target container are not getting registered. Registration and deregistration in loop

I’d look into why that’s happening. Are the health checks failing?

yes i think with helath checks 401 in logs

Sharing an approach I used for a similar situation with ECS serving two LBs (internal ALB and external NLB)…

Instead of having two sets of target groups, I configured the ECS to use the target group associated with the internal ALB. That way, deployments are only updated in one place.

Then I created an externally available NLB that uses the internal ALB as its target.

This has been working great.

But in my infra the container has two ports one for internal use and one for external which will be connected to NLB and internal to alb so two separate target groups are needed

That’s interesting! I’m curious why you would need two ports on the container if its providing the same service internally or externally… does the app/service/container do any sort of processing based on where the client is connecting from?

In my experience, the only need for internal vs external access if the DNS. on VPN, clients use an internal address. On internet, they use the external address. Once they hit the application, the app forwards it to an IAM service for authentication, then they get redirected back to the app.

if possible, is there a way for you to only use one target group? Asking because, if i recall correctly, i started to go down the multiple target group route and it looked like I would need two deployments of the same application, one for the internal ALB TGs and one for what would have been an external ALB TG.

If all else fails, and you have to keep the applicaiton configured as is (with two ports), it might just be easier to consider it as two seperate services in the same cluster. One service for internal ALB with its own TG and another server for external ALB and TG (i would def use ALB with this approach and not NLB). When you deploy, just deploy to both services at the same time.

Sounds like the db is stopping, as it does. You can try v2

yes, i see it is stopping and starting, but why only on few clusters and not on all with same usage?

it started few days ago and happens in every 4 hours

I cannot try the v2 because of the price and it is production traffic. Is there any zero downtime upgrade ?

https://sweetops.slack.com/archives/CCT1E7JJY/p1657761753049409

Have you tried adding www.example.com as an allowed origin for the S3 bucket?

“I have a bucket with CORS configuration allowing only certain domains as origins.” Al ready mentioned that

example.com is one of the certain domains

Maybe open a support ticket with Cloudflare? This seems like a fairly common task, surprised there’s not more documentation (after doing a quick google)

Yes, I am also thinking it might be something simple in the end. There is this https://developers.cloudflare.com/images/image-resizing/control-origin-access/ I hope that will not be solution to have to have cloudflare workers

You can use Cloudflare Image Resizing with either a pre-defined URL format or, for advanced use cases, with Cloudflare Workers. Which of those resizing option you’re using ?

as mentioned in the original question

<https://www.example.com/cdn-cgi/image/https://my-bucket.s3-eu-central-1.amazonaws.com/images/myimage.jpg>

We use Lambdas, we use SSM Docs, we use terraform local-exec, we use kubernetes jobs. So, my answer would be “it depends”

can you elaborate on what the script will do ? that will make answering your question easier

You need to share the exact output in order for anyone to be able to help

makes sense @Erik Osterman (Cloud Posse)

So we assume you have curl installed

ah

im running on windows and installed a vm of debian

ill install that now

any other packages?

make: *** /mnt/c/Users/S903935/OneDrive: Is a directory. Stop

im running make Makefile @Erik Osterman (Cloud Posse)

is there a guide somewhere @Erik Osterman (Cloud Posse).

https://askubuntu.com/questions/1240219/make-command-does-not-work-in-directories-containing-space

it seems like i am running into this

im just moving my github directory, that will solve this

The code is stored in Lambda, not S3. S3 is just used as a mechanism to upload large code bundles to AWS, and then it is copied from S3 into the Lambda environment.

This is evident as you can edit the code in the Lambda console, using the editor in the web UI, and then if you checked some previously uploaded code in S3, the changes will only exist in the Lambda environment

Thank you so much

Let’s not ping randow people. If you are stuck with something specific post a code example with the error you are getting. It’s the quickest way to get help.

Submit a pull request to fix it

• Yes, I will, if I cannot get an answer. And I’d rather it was just a configuration error.

use the old provider

an file the issue

I wsa able to get around this issue by deleting all of the users, and re-creating them with the restricted_home variable set to false. Should the restricted_home variable be set per_user as an input?

@Chandler Forrest please create an issue with a minimal viable reproducible example and we can then see what the error is. The additional inputs off the sftp_users should be optional so if they are not, then this may be a bug but it’s hard to debug without all the necessary context

Yes. But your question is too generic, please be more specific

The bucket access logs need to sync with cloudwatch logs

this logs @Alex Jurkiewicz

S3 access logs can only be written to another S3 bucket. But you can process log files as they come in and write to cloudwatch logs using custom compute. For example, a Lambda function. However, I’m not sure why you would need this. What’s wrong with having the access logs in S3?

just for info i have config on same bucket

we have some file on S3 we need know access count for each file

Neither the mainline AWS or the AWSCC providers seem to have a “support” related resource, so I think that’s a no.

Support plans can only be changed by the root account creds -> which aren’t often used with IAC.

In AWS org, you only need to turn on support at the org root. Not each individual account

This can only be done by submitting a support ticket under Account management > General Account Question ;and more specifically you need to specify each sub account you want to enable under the root, as the plan is technically billed per account

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/consolidatedbilling-support.html

Thanks all

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/emr_cluster#auto_termination_policy

Make sure youre using the latest aws provider version

I see it in latest 3.x provider version too so you must be using an older version of the aws provider

https://registry.terraform.io/providers/hashicorp/aws/3.75.2/docs/resources/emr_cluster#auto_termination_policy

What confused me when I was looking at it earlier is that there is no reference in my code to auto_termination_policy which is marked as optional Also currently at 3.63.0 AWS provider and requirement in our docs says requirement is aws >= 3.5.0

Perhaps the requirement needs to be bumped

for now, you can remove your pin, or remove the .terraform.lock.hcl file locally and run another terraform init to see if it will resolve the issue

thanks for suggestions

did it work?

sorry but would not be able to provide you with answer since I am not trying it atm

no worries, let us know when you can

Looking for that same underlying technology like tailscale if possible. I love this tool and would love to recommend it to more people but this is a big annoyance

Lots of competitors in the market — What I’d suggest looking into is these tools in the following order: StrongDM >= Teleport > CloudFlare Access > HashiCorp Boundary > AWS Client VPN. Search for any of those tools in this Slack and you’ll get plenty of hits. BeyondCorp is one of the most talked about topics.

Tailscale is the best though and for a very good price. I would suggest just accepting the Gmail account cost — You’ll save more of your own money + time not switching away to reduce a hundred or two hundred in yearly spend?

Thank you Matt! I actually found a good workaround bc I really wanted to stick with tailscale. Since they are developer contractors, I was able to use GitHub as the Identity Provider.

Also a nice note to everyone here, tailscale does support several OpenID Connect identity providers. Found here: https://tailscale.com/kb/1119/sso-saml-oidc/

This will need to be a new task for me to provide a better identity provider

That works too. And definitely best to stick with tailscale — it’s a great tool.

Might be a good question for #office-hours

Thought the same thing. I had missed it when I posted this

what do you mean “operate at”?

It’s probably most correct to say subnets operate at layer 3, since they primarily group traffic by IP. But “operate at” is a very broad phrase…

yep its layer 3

thanks Alex

How certain are you the restriction is for abc/tmp/* and not abc/tmp*?

Pretty straightforward like Effect:Allow, ActionPutObject, Resourceaws::abc/tmp, arn:aws::abc/tmp/*

That policy is on the iam user/role/group? Is there more than one policy, maybe allowing more s3 actions than you thought? Also, what is the s3 bucket policy? Does it have an allow statement that might match?

You might need to be a little more specific as to the type of attack

My developer said, it is an just bot attack, i dont know the more details.

Preliminary Security group check needs to do verify whether any protocol has anywhere access [ 0.0.0.0/0 ]

No @sai kumar

cloudflare has bot protection out of the box, just check if you have it enabled

@Max Thank you.