#aws (2023-06)

Check the name of the bucket. IIRC, the name of the bucket has to match the domain name pretty much exactly. Its been a while and I haven’t looked at the docs yet but that’s one thing to consider.

I hope I understand the question correctly, but I use permission-sets for cross account access that is focused around users and groups. When it comes to services, and automation I use self-managed rules.

With permissions sets, it’s primarily a mix of inline-policies and AWS managed policies. So far haven’t really used the customer managed policies option of permission sets.

Yes, that’s what I am interested in. Do you run EKS clusters?

I do not, the org I’m working in is primarily ECS.

With permission sets, AWS sets up those SSO roles with random numbers in the ARN’s across accounts. Do you use those anywhere to configure access or allow something else to assume the role?

We use sso roles with random numbers in terraform plans.The random numbers are appeneded to the role names so role names can still be used applying some basic regex

Thanks @jimplet me do that

I described the cluster and got two clusters. Then I was trying to import both individually. But they did not succeed at terraform plan stage.

I’ll now follow what you have suggested.

May Terraformer help?

Thanks let me check that.

Are you updating the Task Definition before pressing “update service”?

to reference the new docker image

@Nat Williams no, i don’t. But i have set some tag like …[us-east-1.amazonaws.com/test-classifier-fargate//us-east-1.amazonaws.com/test-classifier-fargate:arm-fp16)

the tag is always the same

hmm. It sounds like the default ECS behaviour should be to check the repo for a newer version every time

you’re on ECS, right?

I just sort of assumed that from the “Update service” and “force deploy” verbiage

@Nat Williams ECS Fargate

normally need to use a different image tag

Yeah, ideally I guess you’d be updating the task definition with a specific version each time

@Hao Wang why? :(

Is there any way to use the same without having to set up awful CodeDeploy?

Sure, just create a new revision of the Task Definition and updating the service to use it

@Nat Williams so it’s either i set up CodeDeploy or manually create another task revision? (or maybe via CLI somehow so it just duplicates it)

Using different image tags is a best practice especially in prod env

I mean, you’re already manually forcing the deploy, so it’s not that big a change

@Hao Wang but i’d still need to update the task definition, right?

hmm ECS should update image if forcing deploy

yeah, it should. There is something weird going on

yeah, if using a different tag

AWS support would know more

have run into some other issues and finally AWS support worked them out, it is a black box for us

@Hao Wang thank you very much

Jut reached out to support with this issue :)

ec2 hosted ecs might use the locally cached image of that tag to avoid the network cost. Fargate would pull fresh on a redeploy of the same tag. You might be able to disable image caching if using ec2, dunno how.

@Michael Galey not using ECS + EC2, but ECS + Fargate

then it’d work. Your ecr repo might have tag immutability checked? if so, your 2nd push wouldn’t overwrite the first one. Or theres something else that’s more user-side, if your force deployed tasks failed to pass health checks for instance, so checked the date of the task start, and view the logs of any recently stopped instances

but in general i’d highly recommend using unique ids for deploys to solve possible confusion issues like this

you could also pull the image locally and do docker inspect or connect to it to see if your recent change is there

@Michael Galey could you please tell me where i can find that immutability tag? :)

and how to use those unique ids :(

how do you build images? codepipeline?

@Michael Galey nope. I build locally, push locally and then Fargate takes my image from ECR and deploys

add something like this to deploy script ? it assumes code is committed

      - COMMIT_HASH=`git rev-parse --short HEAD` 
      - IMAGE_TAG_VER=v-${COMMIT_HASH:=latest}
- docker build ... -t <repo url>:$IMAGE_TAG_VER
- docker push 
- <deploy command> <repo_url>:$IMAGE_TAG_VER

@Michael Galey without <deploy command> <repo_url>:$IMAGE_TAG_VER

have no idea what the deploy command should be

whatever it is now, you’d just be looking for a parameter for the image, i haven’t used this stuff but quick googles show things like https://github.com/awslabs/fargatecli

@Michael Galey but if it doesn’t work in UI, how can it even work in CLI and even some third-party apps?

I tried deploying with AWS cli. Does not work

Something like

aws ecs update-service --cluster ${{ inputs.ecs_cluster_name }} --service ${{ inputs.ecs_service_name }} --force-new-deployment

did you check the actual tasks are successfully starting?

@Michael Galey they are up and running now

“Fargate does not cache images, and therefore the whole image is pulled from the registry when a task runs.”

everything works, LB handles requests to them just fine

did you inspect the latest image?

@Michael Galey locally?

pull the image from ecr, and see if your intended change is in there

and see if theres a date via docker inspect

it’s there

you’d have to clear your local cache

and the code isn’t working in the deployed version?

code is working. AWS is not redeploying on “force redeploy”

do the tasks start date line up with the redeploy command?

@Michael Galey Nope. Old date. They do not redeploy

oh ok, not a cache thing then, not sure

command looks right to me

@Michael Galey thanks anyways

@Michael Galey which command?

aws ecs update-service --cluster <<cluster-name>> --service <<service-name>> --force-new-deployment --region <<region>>

@Michael Galey i would rather make sure it works in UI, then try to get the command working

oh it is common to have such restrictions, AWS should have sent some email to root account

@Hao Wang but they’re refusing to tell me the restrictions…

hard to know the details for this case which is not related to security

great to know this project, just took a look, it is CFN wrapper with JS/TS

feeling much better than CT for some insights into the black box of AWS

Fwiw, we use #terraform https://github.com/cloudposse/terraform-aws-components/tree/main/modules/account

https://docs.cloudposse.com/components/library/aws/account/

I use socat to make some private things available to different clouds and it’s worked well so far, been running 6 months or so with no issues.

mmm I think I have the same issue than nginx container, that I can’t pass multiple lines to the entypoint to create multiple configurations

would ssm port forwarding work for your use case?

no, this users do not have AWS access, that is my first option

use NLB. NACL management is less than management of an entire proxy

that is true

https://docs.aws.amazon.com/apprunner/latest/dg/what-is-apprunner.html

You can also deploy docker images to lambda

Take a look at copilot, it still starts ecs cluster and other resources but in a simpler way.

You no longer need to provision by yourself.

Lambda? I don’t want to configure api gateway so I could just make requests

@tommy thanks

@Fizz thanks

Lambda can be accessed via http now

https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html

@Fizz it’s still not the best option. Lambda requires specific Docker requirements, whereas EC2 does not

@Fizz App Runner costs $0.064 per hour? It’s crazy. EC2 costs around $10 a month whereas this would cost almost $50.

@tommy

Copilot such an awful tool.

Takes a lot of time to deploy just one small Docker image.

It takes less time to deploy a Fargate task manually.

You could use AWS Lightsail for this. 7$ for the nano sized per month. (0,25 vCPU, 0,5GB RAM)

@Stoor awful tool as well. Don’t remember why tho

@Stoor but thank you very much

Awful tool? What do you mean? It’s doing exactly what it needs to be doing.

@Stoor the UX is awful in the first place…

right now I’m using this module to create 1 cluster then bring in the global cluster resources with ordinary aws terraform resources

hi John , my guess is people do not use much global clusters in Redis li it was in the rds module a while ago

but that can be implemented

PRs are welcome

Just making sure I’m going down the right path with using the module and resources for the rest. The secondary inherits most of the primary’s settings so it basically works.

I think taking the approach of the RDS module and do a for_each for the replication group , or different count logic and add

resource "aws_elasticache_global_replication_group" "example" {
  global_replication_group_id_suffix = "example"
  primary_replication_group_id       = aws_elasticache_replication_group.primary.id
}

should be sufficient

Thanks Pepe!

It’s maybe because the image is built for ARM but it deploys on x86.

Amazing, what an awful UX, Amazon.

Lightsail tho

Blaming incorrect architecture on the platform is something though. Maybe try running a windows executable

Cause of what exactly? :)

cannot know the cause for now with the shared information, do you have source codes or doc?

@Hao Wang source code of what?)

source codes or instructions you followed

I did not have any instructions :(

figured out. You have to use that exact URL:

http://localhost:9000/2015-03-31/functions/function/invocations

And what is even this None???

woosha

I think you’re suffering from the same thing that I suffered from as I was getting started, slow down and approach the problem a little more pragmatically. If you’re attempting to implement something that has been done before then if it’s not working the problem is not the language/framework/platform, it’s your understanding thereof. That’s not a judgement statement, but an encouragement to slow down and breakdown the problem into digestible bits.

I’m not trying to be crude/discouraging but just trying to help break things down in a meaningful way. If the technology was truly as lacking or ass backwards as you’re making it out to be, then AWS wouldn’t be nearly as successful as they are.

@Darren Cunningham don’t you agree that making different environment in dev and prod is such a stupid thing of Lambda?

are you able to provide some more detail?

• “too long” — what’s the limit you want to stay under?

• what is the response time Lambda vs EC2?

• what runtime? ◦ what memory allocation are you giving it?

• what is the access pattern? meaning, how are you invoking the Lambda vs EC2 ◦ is this a local invocation or remote?

• what does the lambda do? aka, does it access resources within the VPC? if so, are you deploying it into the VPC?

Generally it’s not surprising that an EC2 is going to process a single or a few hundred requests faster than a Lambda. Lambda shines in it’s ability to “scale infinitely” (besides rate limits, etc).

EC2 is like owning a car, you can get it in and go wherever you want/whenever you want without waiting. But you own the maintenance thereof. Lambda is a taxi, it’s not always faster (sometimes is) but it’s worry free…outside of the cost and the more you know how to work the system, the more value you get out of it.

sticking with my car analogy, you can get a 1000 taxis to show up at your door step a lot easier, faster and cheaper than if you tried to rent 1000 cars for a night

@Darren Cunningham seems like i figured it. I had to increase the memory so it implicitly increases CPU and makes responses faster. But thank you (:

Generally i want to stay under 2s while not paying so much for 2 CPU lambda

Lambda responds in 2s on 2048 memory.

EC2 responds in under 1s on 1 vCPU. But only on pure flask multi-threaded without any WSGI thing like gunicorn (makes responses longer)

Runtime: public.ecr.aws/lambda/python:3.10-arm64

gave it 512 memory. On 2048 it’s faster, but i don’t need that much, i only need more CPU, not more memory

access pattern: Function URL

remote invocation

does not access resources. Makes AI CPU inference (yolov8 ONNX)

If your lambda is called infrequently, increased latency is result of slow lambda start-up, which is kinda expected for lambdas is general. You might want to play with provisioned concurrency and see if it helps for your case https://aws.amazon.com/blogs/compute/new-for-aws-lambda-predictable-start-up-times-with-provisioned-concurrency/

It’s still to expensive, but thanks

Isn’t it a case of updating the task definition to increase or decrease counts via CloudWatch alarms?

@james192 don’t know. Which counts?

to be frank, stop learning AWS would be a good option for you for now. picking it up in a few months would be a good strategy

@Hao Wang why? I need to deploy my app to it now, not in a few months

did you read my message?

the purpose of an auto-scaling group is not to manipulate instances manually

if you need more instances launched, increase the min size

if you want to refresh all running instances (for any reason), there is a menu item in the AWS console “Refresh Instances” (if you click on it, the ASG will replace all old instances with new ones making sure the min desired size is always running)

This reminds me of the termination policy of AWS:

I don’t use the policy for a while but still remember the pain lol

If investing too much into ASG, the beauty of k8s will be missed

Learn k8s directly

@Andriy Knysh (Cloud Posse) not manipulate maually? But how to update instances with new code? I have a launch script in the “user data” which clones github repo. How can i update all my instances automatically?

if you want to refresh all running instances (for any reason), there is a menu item in the AWS console “Refresh Instances” I did exactly this and accessing my load balancing was giving me http errors for a while, meaning it drop an existing instance from the group while another instance was not ready. Why?

@Hao Wang what’s the ASG?)

@Hao Wang does k8s launches Docker inside EC2 instances? Or Creates EC2 instances already as a Docker image without inner Docker overhead?

It sounds like you could learn a lot from some introduction tutorials to auto scaling. AWS have lots of good video introductions to resources. If you feel masochistic you can also try to read the docs

But how to update instances with new code? I have a launch script in the “user data” which clones github repo. How can i update all my instances automatically?

you use a Launch template, update any parameter in it, and the ASG will update all instances to use the new version of the Launch template

if you know terraform, this is how it’s done https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/blob/main/main.tf

https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/blob/main/main.tf#L82

https://www.google.com/search?q=aws+launch+template+tutorial

@Oleh Kopyl ASG = AutoScaling Group, yeah you can understand docker in EC2 for now, but under the hood it is much more complex

Docker is just a middle man(run time) which can be replaced by others, like containerd

@Alex Jurkiewicz docs? no, thank you very much.

If you could share a couple of good videos which actually helped you personally, i would appreciate it

@Andriy Knysh (Cloud Posse) so it does the update automatically upon each template version creation?

@Andriy Knysh (Cloud Posse) i don’t know Terraform. But should I?

@Hao Wang i mean that i don’t want to Have Docker in Docker. So if messing around with user data of a launch template is the only option – i better do this than pay for 1 CPU i don’t use (which is used under the hood to have Docker inside Docker))

It is not DnD

@Hao Wang k8s?

yeah

lol better to dig in more, before I didn’t have ChatGPT…

@Hao Wang thanks :)

My pleasure, keep learning

@Oleh Kopyl If you just want to refresh the instances in the ASG , this could be helpful. https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html

if you are using terraform, you can add below setting into your asg configuration : https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/autoscaling_group#instance_refresh

@Satish Tripathi thank you

can be found under /var/log, it should be managed by systemd otherwise cloud-init will never finishes running

i am not very sure but i remember i had a ubuntu server running with similar setup as yours and was trying to find stderr and stdout logs but i ended up finding nothing until you write those logs to a file. This article says the same. https://askubuntu.com/questions/1030938/where-do-i-find-stderr-logs#<i class="em em-~"</i>text=Command%20output%2C%20i.e.%20stdout%20and,writing%20it%20to%20a%20file> This could be helpful you can send your user data logs: https://repost.aws/knowledge-center/ec2-linux-log-user-data

Another option is to look at the cloudinit logs.

@Hao Wang yeah, but there are multiple files. So i was wondering what is the one for the logs i need…

I could as well check all the files, since there are not so many and avoid posting here.

Thank you very much any was

Is checking all the files fun? Lol starting with cloud-init*

if python3 launch-app.py run, cloud-init will hang forever

@Hao Wang not fun, true

if python3 launch-app.py run, cloud-init will hang forever @Hao Wang what do you mean by “hang forever”?

yeah, forever

cloud init is not meant to run long running services. it’s meant for doing EC2 first run bootstrapping. It won’t even run on reboot. Your cloud init script should create a systemd service file, enable the service and start the service.

if you did run the app at the end of the launch script then all the stdout/err will be in the /var/logs/cloud* files.

cloud init is not meant to run long running services Sure, but why can’t it?

@jonjitsu

cloud init is an agent which runs setup which eventually ends. It is not a service manager. For example it won’t run on reboot so if you try to run a service at the end of a cloud-init script it won’t run. All linuxes come with service managers. Ubuntu’s is systemd, so in your cloud init you can create the systemd service file and systemctl enable then systemctl start it and your service will be properly managed including in the case of reboots.

I don’t know what happens when you try to run a never ending process at the end of a cloud init script or if that is even valid according to the cloud init specs.

@jonjitsu just yesterday rewrote it to systemd since i needed restarts when an app fails (OOM kill)

#!/bin/bash
git clone https://[email protected]/kopyl/crossdot-yolo.git && \
mv crossdot-yolo/onnx /home/ubuntu/onnx && \
rm -r crossdot-yolo && \
mv /home/ubuntu/onnx/* /home/ubuntu/ && \
rm -r /home/ubuntu/onnx/ && \
echo "wget <https://bootstrap.pypa.io/get-pip.py> && \
python3 get-pip.py && \
python3 -m pip install --ignore-installed flask && \
python3 -m pip install psycopg2-binary && \
python3 -m pip install onnxruntime==1.13.1 && \
python3 -m pip install opencv-python-headless==4.7.0.72 " >> i.sh && \
chmod 777 i.sh && \
./i.sh && \
echo "[Unit]
Description=Flask app
After=multi-user.target
[Service]
Environment=AWS_POSTGRES_DB_HOST=.
Environment=AWS_POSTGRES_DB_USER=.
Environment=AWS_POSTGRES_DB_PASSWORD=.
Environment=ONNX_MODEL_PATH=/home/ubuntu/model.onnx
Environment=PYTHONUNBUFFERED=1
Type=simple
Restart=always
ExecStart=/usr/bin/python3 /home/ubuntu/flask-server-postgres.py
[Install]
WantedBy=multi-user.target" >> /etc/systemd/system/crossdot-flask-inference.service && \
systemctl daemon-reload && \
systemctl enable crossdot-flask-inference.service && \
systemctl start crossdot-flask-inference.service && \
systemctl status crossdot-flask-inference.service

@jonjitsu
agent which runs setup which eventually ends Do you have any proofs? Not trying to be rude, just really want to read it from some official source

@jonjitsu by the way, is it going to start on its own on reboot or you need some additional configs?

I don’t really have any proof. It might be out there though, you’ll have to check the docs.

It should start on it’s own after reboot because you ran systemctl enable ...

@jonjitsu was not able to find it in docs. How can you know then if you have no proofs?

Good post. Straightforward and brings receipts

if you change the service desired count to 1 you are saying it is taking 59 mins to remove all the extra containers?

I’m not changing it to 1

It changes to 1 on its own

Reducing amount of “desired” by 1 per minute on its own

I need it to reduce to one in 1 minute, not in 59

why does it go from 1 to 60 back down to 1 in one minute? Is this a batch job or something? what mechanism are you using to scale it from 1 to 60?

when a container dies it’s not getting restarted by fargate?

@jonjitsu what do you mean by “dies” exactly?

@jonjitsu i was getting 5xx HTTP error, so restarts wasn’t working obviously

You can try fore deployment option in ecs servcie.

With CloudWatch metrics

Seems to be the case

https://aws.amazon.com/getting-started/hands-on/machine-learning-tutorial-deploy-model-to-real-time-inference-endpoint/

I think “light and quick” correlates more with using the technologies you know well, rather than using specific tools that have buzz

I think that’s some wisdom I needed to hear

when a container fails healthcheck, the entire thing is replaced

containers aren’t meant to be touched beyond the initial process being started. If you want the app to automatically restart, add a supervisor.

Systemd is one approach, but it’s pretty heavyweight. You could also use a shell script which contains something like

while true; do python main.py ; done

What is supervisor? This thing? https://dev.to/rabeeaali/install-supervisor-on-aws-eb-with-laravel-5g8a

Is my python app going to be relaunched in a case of OOM kill if i launch it with the shell script?

when a container fails healthcheck, the entire thing is replaced Damn.. Not something i need…

no, supervisor is “init system”. Generally, containers run without one. init systems are designed for long-running servers, but that’s not how containers work. As you see, they follow the philosophy “if it stops working, kill it and get a new one”

What do you mean by backup if not a snapshot?

In rds I see snapshots and backups. I wanted to restore from backup. Guess that is good only for backup point in time

@Balazs Varga do you still need to restore from backup? Maybe @Alex Jurkiewicz can help

yes, to a different location. I think I can use the aws backup to copy over the backups to different region and then restore from that backup

Ok, lmk if you need help

thanks

We have significant experience, that sounds like there is an issue with your trust policy of the role

I gave the role admin access, i know its not good practice but i just wanted to make sure it was working

that’s not the trust policy

what’s the trust policy

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “arniam:oidc-provider/token.actions.githubusercontent.com” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringLike”: { “http://token.actions.githubusercontent.com<i class="em em-aud|token.actions.githubusercontent.com"</i>aud>”: “sts.amazonaws.com”, “http://token.actions.githubusercontent.com<i class="em em-sub|token.actions.githubusercontent.com"</i>sub>”: “repo:dandiggas/firstwebapp” } } } ] }

I think you are missing a *

repo:dandiggas/firstwebapp:*

ok ill try that now

Yeah still not working. keep getting this error message Run aws-actions/configure-aws-credentials@v2 https://github.com/Dandiggas/FirstWebApp/actions/runs/5412372323/jobs/9836381905#step<i class="em em-3"6 (node:1742) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. https://github.com/Dandiggas/FirstWebApp/actions/runs/5412372323/jobs/9836381905#step<i class="em em-3"7

https://github.com/Dandiggas/FirstWebApp/actions/runs/5412372323/jobs/9836381905#step<i class="em em-3"8 Please migrate your code to use AWS SDK for JavaScript (v3). https://github.com/Dandiggas/FirstWebApp/actions/runs/5412372323/jobs/9836381905#step<i class="em em-3"9 For more information, check the migration guide at https://a.co/7PzMCcy https://github.com/Dandiggas/FirstWebApp/actions/runs/5412372323/jobs/9836381905#step<i class="em em-3"10 (Use node --trace-warnings ... to show where the warning was created) https://github.com/Dandiggas/FirstWebApp/actions/runs/5412372323/jobs/9836381905#step<i class="em em-3"11 Error: Not authorized to perform sts:AssumeRoleWithWebIdentity

What happened when you tried removing the condition altogether?

It worked thats what happened lol

Thanks a lot! I’m quite new to this so making a lot of errors

so you need some sort of condition, but that means there is a mismatch between your condition and the reality

Yeah i just fixed it with the condition

what’s the new condition?

{ “Version”: “2012-10-17”, “Statement”: [ { “Effect”: “Allow”, “Principal”: { “Federated”: “arniam:oidc-provider/token.actions.githubusercontent.com” }, “Action”: “sts:AssumeRoleWithWebIdentity”, “Condition”: { “StringLike”: { “http://token.actions.githubusercontent.com<i class="em em-aud|token.actions.githubusercontent.com"</i>aud>”: “sts.amazonaws.com”, “http://token.actions.githubusercontent.com<i class="em em-sub|token.actions.githubusercontent.com"</i>sub>”: “repo:Dandiggas/FirstWebApp:*” } } } ] }

I didn’t realise that it was case sensitive

Thanks Warren

no problem

I suggest cross posting link to this message in #terraform

Will do. Thanks Eric.

@Simon Weil you clearly know what you’re doing since it’s your PR and the code in your example is super clean. Any ideas on how to fix this?

@Ben Smith (Cloud Posse)

1. Todo this you’d have to use the aws organizations delegation to delegate Organization management to a delegated role. So you could have your AWS Org delegated to an aws-team . By default your management account is the root account, and contains your state bucket and is where your SuperAdmin role deploys the accounts component, which creates the member accounts.
2. The aws organizations delegated role can be limited to an OU, meaning you just need to specify in the roles permissions
3. I’ve never done this, as this is where we deploy our TFState and manage organization security policies. It appears from the aws docs that it would delete the organization, resulting in your member accounts becoming standalone accounts