#aws (2023-02)

NOTE: while using fargate i’m able to pull the image

Thanks for reaching out about the Nginx 504 error you’re experiencing. A 504 error typically indicates that the server didn’t receive a timely response from the upstream server, which in this case would be the proxy server.

In order to troubleshoot the issue, here are a few steps you can take:

Check the logs of the upstream server to see if there are any errors or issues that may be causing the delay in responses.

Monitor the network traffic between the Nginx server and the upstream server to see if there are any connectivity issues or bottlenecks.

Increase the buffer sizes in your Nginx configuration to allow for larger payloads to be sent between the server and the proxy.

If you’re using load balancing, consider using a different load balancing method, such as round-robin, to distribute the requests more evenly.

Try increasing the proxy_connection_timeout value to a higher number to see if that resolves the issue.

I hope these suggestions help resolve the issue, but if you’re still having trouble, don’t hesitate to reach out for further assistance.

Sorry for the delay lee, we had nginx 504 error, since we trying to pull huge amount of data

We’ve got mac users on https://cyberduck.io/ and it works well for them. They do have a windows version as well, but I’ve never used it personally.

FileZilla was another option

Cool, didn’t know filezilla could do that now.

oh I guess it’s only Pro, sorry

Still good to know

Commander One on Mac works ok for me

https://ftp-mac.com/amazon-mac.html

I missed an “s” I meant “Windows” desktop tool but I’ll try cyberduck

There is also a VM image for enterprise that connects to S3 and allows you to expose that as a SMB network share.

https://docs.aws.amazon.com/filegateway/latest/files3/what-is-file-s3.html

for Window(s), there are a bunch of those tools, e.g. https://s3browser.com/

S3 Browser is the one I use. Petty simple actually.

+1 for cyberduck. I set it up for a team that uses mac and windows. non-tech folks that needed up transfer files from local storage to s3. it worked great.

Cloudberry

blog post: https://aws.amazon.com/blogs/aws/new-amazon-cloudwatch-cross-account-observability/

Poking around in the console it appears that CloudWatch events are EventBridge events. Guess the cli docs are just out of date.

set up replication https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html and switch over at a certain time. the tricky bit is really how to switch over as thats the bit where you may end up with eventual consistency issues. if you can live with a bit of downtime, (minutes?) this can easily become a non-problem however. depends on your usecase.

Yeah but if you want to move a bucket, just setting up replication is not enough as that will only handle new objects created after replication has been setup. It will not replicate existing objects in the bucket.

That’s why AWS created S3 Batch Replication (ga August 2022, very recent), but it’s not clear whether there are cheaper (and likely slower) ways. Batch replic on a bucket having 200M objects will cost at least $200. Quite affordable, but is there cheaper

1. create a new bucket
2. aws s3 sync <s3://orig> <s3://tmp>
3. delete old bucket
4. create orig bucket in new region
5. sync back
6. delete tmp bucket

It’s the best…. Step 0 should be stop create/update/deletes to bucket.

if you have the time to wait for the sync and include downtime where there are no writes/reads, yes this is the easiest solution. its also the cheapest option usually

when you don’t want to be offline and/or its a huge amount of data and offline would be too long or you want to minimize it, replication is the way and its a combination of batch replication and replication itself. as you noted batch replication is for the old things and replication itself for anything, that changes. batch also fixes things if there are failed objects (that didnt replicate). in any case, both are described there on that page and they will cost more than an s3 sync

so really if you can live with that downtime of waiting for the s3 sync, that is the cheapest available option. Alex is right about that and I should have mentioned it too

Yes, there are a few different options for moving an S3 bucket from one region to another, and each has its pros and cons in terms of cost and speed. Here are a few methods you could consider:

S3 Batch Replication: This is the method you’ve already used, and it’s a good option if you need to replicate a large number of objects or have specific requirements around replication time and data consistency. However, it can be relatively expensive compared to other options.

S3 Transfer Acceleration: This feature allows you to transfer data to S3 over the Amazon CloudFront content delivery network, which can speed up transfers by up to several times compared to standard internet transfer speeds. While this option is faster than batch replication, it’s still not the cheapest option.

S3 Cross-Region Replication: This is another option that you can set up to automatically replicate objects from one bucket to another in a different region. Unlike batch replication, this feature is designed for continuous replication and provides immediate and eventual consistency for updates. It’s a more cost-effective option compared to batch replication, but the transfer speed may be slower.

S3 Transfer Manager: You can use the AWS S3 Transfer Manager to move objects between S3 buckets in different regions. This is a good option if you have a large number of smaller files, as it provides a higher level of parallelism and can be faster than batch replication or cross-region replication. However, it’s not as cost-effective as cross-region replication.

Ultimately, the best option for you will depend on your specific requirements, including transfer speed, cost, and the number and size of objects you need to move.

Thanks guys I appreciate all the input.

Note for others reading this: s3 cross-region replication will not copy existing objects, so cloning a non-empty bucket cannot be done with it alone.

To force replication in the past I wrote a script to update metadata in the source objects

was written in golang

and it will search for the instance in a specific region

this one

https://github.com/claranet/sshm

mmmm not updated lately

You can use the ssm start-session so not sure why you’d need a 3rd party tool. https://awscli.amazonaws.com/v2/documentation/api/latest/reference/ssm/start-session.html#examples

The sshm utility is pretty handy. It still works for me even tho its old

this 3rd party tool searches for all the instance running and registered with ssm in your region and you can select them from a menu, it is very handy

infracost

That looks good, but I have not created AWS Backups through Terraform. Can you or someone else recommend a local tool that uses aws cli or something similar to find costs?

look at the office-hours videos, there has been some mention of many tools before

If you want to use cost explorer CLI to get costs by service, you can use the ce module.

for example, i use the following to get monthly costs by service by team from four accounts:

for profile in acct1-{prod,nonprod} acct2-{prod,nonprod};
do
    aws --profile=${profile} ce get-cost-and-usage \
        --time-period Start=2023-01-01,End=2023-01-31 \
        --granularity=MONTHLY \
        --filter <file://filters.json> \
        --group-by Type=DIMENSION,Key=SERVICE \
        --metrics=UnblendedCost| tee UnblendedCost-${profile}.json
done

the file filters.json looks like this:

{
  "Tags": {
    "Key": "TEAM_ID",
    "Values": ["abcdefghijklmnopqrstuvwxyz0123456789"]
  }
}

Output looks like this:

{
    "GroupDefinitions": [
        {
            "Type": "DIMENSION",
            "Key": "SERVICE"
        }
    ],
    "ResultsByTime": [
        {
            "TimePeriod": {
                "Start": "2023-01-01",
                "End": "2023-01-31"
            },
            "Total": {},
            "Groups": [
                {
                    "Keys": [
                        "AWS Direct Connect"
                    ],
                    "Metrics": {
                        "UnblendedCost": {
                            "Amount": "0.0024723419",
                            "Unit": "USD"
                        }
                    }
                },
                {
                    "Keys": [
                        "Amazon Elastic Load Balancing"
                    ],
                    "Metrics": {
                        "UnblendedCost": {
                            "Amount": "11278.9931034615",
                            "Unit": "USD"
                        }
                    }
                }
            ],
            "Estimated": false
        }
    ],
    "DimensionValueAttributes": []
}

Thanks! but, this is all so cryptic. I have credits on my account so when running this command I get negative costs. Is there a metric that tells me what would AWS charge for the usage?

Cost estimator gives actual and expected costs for the account.

Are you trying to get an estimate?

in that case you will have to use the pricing module and do some calculations. but if you think the ce output is cryptic, the pricing output is even more so…

i have the following call (but not the contents of filters.json:

aws pricing get-products --service-code AmazonEC2 --filters <file://filters.json> --format-version aws_v1 --max-results 1 --region us-east-1

you can check out the man page docs for pricing to get more details.

good luck!

What I’m trying to do and failing is to understand how fast will my credits burn. What does a negative amount in Cost even mean? Maybe that’s what AWS is crediting… Anyway, thanks

Try the console? Click the bucket and select the option to Empty it

it won’t let me delete from here

One level higher. Select the bucket and at the top right you should see empty

Hi!

There is a faster way to delete all the objects in an S3 bucket, which is by using the AWS CLI command aws s3 rm. This command will recursively remove all objects and directories in the specified S3 bucket.

Here is an example:

aws s3 rm s3://my-bucket –recursive

Note that this command will permanently delete all the objects in the specified bucket and it cannot be undone, so make sure you have a backup of the data if needed. Also, keep in mind that when you delete a large number of objects in a bucket, it may take a while to complete the operation.

I tried this but its still super slow

because the bucket is versioned

i am doing it like @Charles Rey proposed and its on 20800 objects already and counting

Sorry Steve, don’t have any more ideas. Good luck

set a lifecycle rule to delete objects older than one day, and wait a day

if you have a big bucket this will also save non-trivial $ as you aren’t charged for the API calls to list the bucket and delete individual objects

if you google this exact phrase you can find out in the first result

@Alex Jurkiewicz you do realize that the results you get from google are personalized? I know my reply is not helping too

At this year re:invent it has been a chalk talk about AWS SSO and this use-case is still not one of the well supported by AWS SSO. if you need to access external AWS accounts, probably AWS SSO is not the right choice.

Under IAM Identiy Center you can add external accounts under Applications and “External AWS Account” under Preintegrated applications.

Need one each for every type of access.

This access type is to me not so useful. It’s not easy to implement programmatic access as other accounts

SNS doesn’t charge per-topic. Do whatever is easier for you

@Dan Miller (Cloud Posse)

AFAIK pod capacity is recorded in launch template userdata, which is managed by Managed Node Group, which re-reads state of vpc-cni addon on changes to MNG configuration.

In short you need to recreate MNG or trigger its update, like changing AMI version or allowed instance types.

Yeah, thank you, @z0rc3r! As you suggested, updating the node group to the latest AMI version did the trick . Now my new nodes show the 98 pods limit, which is the expected number.

the latter. Good luck

i suppose you could use an SCP to deny cloudwatch:PutMetric*

Looks like no… https://github.com/aws/aws-appsync-community/issues/83

Found https://developer.okta.com/blog/2021/10/08/secure-access-to-aws-eks, interesting

Although this is based on okta, so it’s not a small change for a team … something in AWS would be so much better.

Actually this seems like the same as IRSA, except that it ties in to SSO if that is used by the org

What is the CI/CD system?

For GitHub Actions, this is why we deploy self-hosted runners

It is bitbucket pipelines (not my choice!)

hah, first thing to “Fix”!

Haha, probably will not happen, but I think bb supports local runners

(although I will bring up the possibliity, as it would be nice)

The CI runners can connect into a machine you do control, eg via SSH. This can be a bastion server to the same VPC as the EKS cluster

Thanks @Alex Jurkiewicz, I have actually used that in another project. But unless the ssh key is temporary (like obtained at the time the pipeline is run, via OIDC from AWS EC2 Instance Connect), having a permanent key in a CI/CD secret does not seem like added security since kubectl comms is over https. Also best practice is to not leave sensitive data on bastion (such as .kube file), only use it as tunnel, in case bastion is hacked.

Or maybe if you use a bastion 100% dedicated to the CI/CD (so have separate bastion(s) if other tasks are needed, like for other devops tasks for team), such that bastion can only do what the CI/CD needs and nothing more… then yeah that might be better, as I could see that temp SSH key via OIDC in pipeline + .kube on bastion would be better than having the .kube in a CI/CD secret.

It’s added security because only your projects can authenticate with the bation. IP-based whitelisting allows anyone’s pipelines to access your EKS endpoints directly. Right?

If you mean that anyone using same cicd deployment engine as us could access the EKS endpoint, yes. In that sense, ssh does provided added security.

In the end I found a way to setup ssh socks5 proxy bastion that is used by the cicd alone (ie non-public EKS endpoint), and let devs still use kubectl via IP-filtered public endpoint. This way using kubectl from laptop can continue as before. It will be trivial to switch everyone over to using ssh tunnel if a security concern warrants that later.