#aws (2021-09)

What EC2 instance type?

Error message indicated you are not using a nitro based instance

https://aws.amazon.com/ec2/instance-types/

Thank you Conor, if i understand well this nitro system is for a little bit more expensive ec2’s, as we use the smallest possible t2, i don’t see it in the table

T3 instances are supported so you could possibly change the instance type. Also https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html is a great alternative option

Thank you Conor, the reason that i am asking all these things is that every month as we install some python related packages our server crashes, and none of the 4 methods works EC2 Instance Connect and SSH client should work out of the box (they are unreachable, as server crashed). Session Manager should be configured by me and EC2 Serial Console says that our instance is not compatible (wants nitro)

T3 should be the same price as T2. You can even use t3a which is even cheaper

If the server is crashing that easily its likely undersized

the day of crash it reached 34% cpu usage but i dont see anything else weird about utilization

You can see your CPU credit balance is getting very low (at that point the performance will be throttled) this is not the cause of your issue in this particular case but worth watching

Its likely memory exhaustion

What instance type is it exactly?

t2.micro

Not really suitable for most production workloads unless they are incredibly small / bursty

This is an “insert credit card” fix

Try a t2.medium for a while and see how it goes

what if i change to t3a.micro like alex suggested above? btw how is it possible to have double cpu but be even cheaper? is it because of the switch to AMD?

I doubt it will make any difference if memory consumption is the problem

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html

aha, now i noticed that RAM status is not showing in any graphs, right?

by the way, the serial console didnt help either, only black screen appears

there is a bar going up to the right of the screen

You might need to patch the module. Why is it a problem to use a custom parameter group?

If you use the default one, applying parameter changes in future will require you to first apply a custom parameter group, which will cause downtime

Hello Alex, These are clusters that will almost never need a parameter change. My boss is a kinda OCD about having N amount of parameter groups (and any other unnecesary resource) laying around when all the clusters (more than 30) all use the same params.

that’s too bad you have an irrational boss

It’s helpful to have a custom param group if you ever need a custom param in the future…

Did you want to use an existing param group instead? Or simply not use a param group at all? I think expanding the module to use an existing param group and existing subnet group would be a nice feature

If you want to put in a pr, you can start here

https://github.com/cloudposse/terraform-aws-documentdb-cluster/blob/5c900d9a2eaf89457ecf86a7b96960044c5856f4/main.tf#L88

Other thread in #kubernetes

https://sweetops.slack.com/archives/CBW699XE0/p1630659219013100?thread_ts=1630659219.013100&cid=CBW699XE0

@Almondovar yeah, the ec2 serial console is serial access, not ssh. A handful of ec2 AMIs come preconfigured forit (e.g. amazon linux and i think ubuntu 20). You also need to turn the service at the AWS account level and use an IAM role/user permissioned to use the service.

Hi Carlos, do i understand well that the steps i performed are not necessary to enable cloud console connnection? tbh once i followed them they instantly allowed access to the console connection

@Almondovar hey, missed your IM, yes, that is my understanding. But if the changes made worked, then even better

Have you checked out using Leapp instead?

https://www.leapp.cloud/

We used to use all kinds of scripts, hacks, and tools but leapp has replaced them for us

It’s an open source electron app distributed as a single binary

Cc @Andrea Cavagna

interesting @Erik Osterman (Cloud Posse) you just use the free one?

Yup

Leapp is free for anyone, it’s an open source projects, we are going to close the federation with AzureAD and AWS pull request this week and having a release, for any further question, @Steve Wade (swade1987) feel free to text me :)

The only paid solution by now is the enterprise support to the opensource project, made by the maintainers of the app

Btw @Erik Osterman (Cloud Posse) I grant you that in the next weeks I will partecipate to an office hour so we can respond to any question about Leapp!

We should chat tomorrow, I have some code in ConsoleMe that parses the nested json config returns

Github.com/Netflix/consoleme

wild, I’m doing that right now.

mysql? postgres? sqlserver? oracle?

rds snapshot or a db-native (like pg_dump or mysqldump)?

@Matthew Bonig mysql and rds snapshot

gotcha. So what I ended up doing was writing a lambda that did the dump (using pg_dump) and then streamed that to a bucket. I then have another process that reads that object from the bucket and restores it in another database. Nothing is packaged up nicely for distribution yet, but so far it seems to be working ok.

The plan is to have that Lambda be part of a state machine that will backup and restore any database requested from one env to another.

nice thats pretty cool

the dump into S3 makes sense as a lambda

what you write it in?

only concern is that you can’t get the entire database in 15 minutes =-/

nodejs

you can’t get it?

yes, I can in my case. I was just saying my only concern about using a lambda is that the database is so big that it couldn’t be dumped and uploaded to s3 within 15 minutes.

generally the runs I’ve been doing in a fairly small database were getting done in just a few minutes (with an 800mb dump file) so we’ll probably be fine. But if you’re trying to do this for some 3 tb database, you’re going to have a bad time with Lambdas

Did you notice much size difference between a MySQL dump to S3 and just taking a snapshot @Matthew Bonig ? My boss seems to think a dump is over engineering it for some reason

I’ll paste you his points here when I’m at my laptop in about 15 mins if you’re still around

We wont be retaining the shapshots. The RDS snapshot we should use is the automated ones, which we have to keep for contractual and compliance reasons anyway, so there is no additional cost.

Moving this into a sqldump would be over engineering in my view, RDS has the ability to export a snapshot directly to S3 so why reinvent the wheel? Lets face it AWS are pretty good at this stuff, so we should leverage their backup tooling where possible.

The snapshots moved into S3 will need to be retained indefinitely due to the contractual wording…this is being worked on, but wont change any time soon.

I also dont want to have a split between HelmReleases and TF. If we can manage this all in one place (which we can) it feels better than splitting it out. As a consumer having to deploy the infra, then also deploy a HelmRelease feels clunky. Where as deploying just the RDS instance and its backup strategy as a single unit would be more intuitive.

I proposed using a CronJob in our EKS clusters to facilitate the backup

In my base, postgres, so pg_dump. But, that was pretty large since totally uncompressed SQL.

I had looked into using snapshot sharing, but since the db was encrypted with a KMS I couldn’t share with the other account, I couldn’t ship it that way.

Should have looked more for the native s3 integration, but didn’t. Will look now. I don’t know how the encryption would work though. I would assume shipping it to s3 keeps the data encrypted (and needing the same key as the RDS instance)

I use cronjobs in a cluster to backup a mysql and mongodb. Works great.

oh man, totally should have done this s3 export.

Actually, scratch that. I haven’t completed open sourcing it, apologies about the false start.

Make another change and tell it to apply immediately

thanks @Alex Jurkiewicz , aws support pretty much told me the same thing. they said to do it via CLI, not console.

aws rds modify-db-instance  \
  --db-instance-identifier my-db-instance-01  \
  --allocated-storage 200  \
  --max-allocated-storage 500  \
  --apply-immediately;

maybe this will help https://stackoverflow.com/questions/53386811/terraform-the-db-instance-and-ec2-security-group-are-in-different-vpcs

looks like one of them is using the default VPC

The only change I can think of, that can cause this is docker deprecation: https://kubernetes.io/blog/2020/12/02/dockershim-faq/

But that’s not included to the 1.20 by default, you should do it separately in a node group. So if you followed the release notes and did it - must be it.

Other than that the k8s version itself (the control plane) has no effect on workload resource consumption, it’s involved only during CRUD of the yamls.

It must be something else - the AMI version of a worker, the runtime, instance type of a worker, and so on

easier in regards to what, you mean like for a public bucket?

Yes! I need to release AWS S3 to people in the marketing department

These are people who have no technical knowledge and they need to have the option to download a full AWS s3 folder

theres this which lets you browse a bucket https://github.com/awslabs/aws-js-s3-explorer

I recently tested AWS s3 explorer, but it doesn’t have the option to download a full folder.

https://github.com/awslabs/aws-js-s3-explorer

what about using a app like cyberduck or something like that

this guy seems to have a fork where you can select multiple files https://github.com/awslabs/aws-js-s3-explorer/pull/86

It’s working perfectly. Thanks a lot for the help!

output? what do you mean?

I mean when I run aws ec2 describe instance command, I would like to get result

do you have a firewall or something that could be blocking connections?

I think the problem is that it’s private ec2 instance right? Doesn’t have public IP address. Instance metadata received

<http://169.254.169.254/latest/meta-data/>

By using curl command got result

the instance should have internet and it should be able to hit the api

it has nothing to do with the public ip

but usually to get metadata from within an instance you use this address http://169.254.169.254/latest/meta-data/

no need to run the cli for that

We have an open enhancement in Leapp. maybe it can help you: https://github.com/Noovolari/leapp/issues/170

Odd, I’m using kubectl and leapp just fine right now oh, this is to have kubectl ask leapp directly. Huh.

Hi @Steve Wade (swade1987)! May I ask you how you have federated Azure AD to AWS?

@Eric Villa I have a blogpost on it https://medium.com/p/how-to-configure-azure-ad-as-an-oidc-identity-provider-for-eks-53337203e5cd?source=social.tw&_referrer=twitter&_branch_match_id=967466935918883996

Ok thank you! I’ll check it out

CloudWatch Events?

Lambda functions to send sns notification to communication channel like teams or slack?

You checking the whole log group?

You are creating a lot stream but Lambda won’t use that

Are the health checks passing?

Have you spot checked the health checks as being valid/working?

currently what it looks like

it was failing earlier.

now it is stuck on registering

I would suggest reaching out to their support if you haven’t already. Likely they will be able to spot the problem easily. Could very well be on their end too.

does your exec role have permissions to pull the image?

@Darren Cunningham sorry what exec role?

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_execution_IAM_role.html

additionally, if you’re pulling from a private ECR – double check the policy on the ECR

hrmmm not following. master should be compatible too

Sorry it was not about master/main

You are correct

I used to use jq with awscli a lot but then i switched to their native jmespath using the --query method.

See if something like this works for you

https://github.com/aws/aws-cli/issues/621#issuecomment-36314975

i saw that syntax and thought about it, but the awsconfig advanced query uses simplified SQL syntax and HAS to dump an entire tags object. I have to then filter using jq. thank you for the link tho! super useful in other ways.

Yes what’s the issue your running into

Ecr has two endpoints you need to add, api and dkr once you create the endpoints you just need to add a security group for 443 and make sure fargate can access it

hmm ive done exactly that but it seems the iamges are still being pulled through my public NAT

You’ll probably also need to add s3 that can be a gateway endpoint

so ive got 2 interfaces and a gateway

Did you add the private subnets on the ecr interface endpoints

yeah, all my private subnets are on both of the ecr interface endpoints

same SG’s too

Did you enable private hosted zone on the interface endpoint

used the same policy as recommended by the docs, and my private DNS looks to be correct too (*.dkr.ecr.ap-southeast-2.amazonaws.com)

Is this a dev env

yeah

Like can you temporarily remove the route to the nat and see if the ecr image still pulls for container

I’ll give that a go now

Yea I’m curious, it should be working from what you’ve setup

One of these blog posts may or may not help

https://www.easydeploy.io/blog/how-to-create-private-link-for-ecr-to-ecs-containers-to-save-nat-gatewayec2-other-charges/

https://aws.amazon.com/blogs/compute/setting-up-aws-privatelink-for-amazon-ecs-and-amazon-ecr/

What platform version are you using for fargate

3 or 4

1.4

ive checked out both of those blog links haha, there must be a misconfiguration somewhere else that im not seeing

Yea I’m trying to think of any gotchas, I’m not at my computer right now so can’t visually run through my setups

sitting here waiting for terraform cloud to see the commit, not realising i didnt push yet

alroghty, did some stuff

turned off route for private subnets, added the cloudwatch private endpoint,and got it to seemingly pull from the private endpoint

Nice

Cluster AutoScaler can do that!

EKS-specific docs are at https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html and https://www.eksworkshop.com/beginner/080_scaling/

Thanks @Vlad Ionescu (he/him)!

Reading https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html, it only lists MySQL an postgres support, not Mariadb. Im sure aws support can confirm it, but that’s likely why. Don’t think the public IP has to do w it

Thank you for confirming Michael

You don’t need separate Lambdas, but I would typically encourage it. Being that the intent of Lambda is Function as a Service, it’s ideal when a Lambda does not have too much baked into it. The more you pack into a single Lambda (1) it’s inherently going to run slower, which then has cost implications as you scale (2) increases code complexity, which makes testing all scenarios harder (3) depending on the runtime, can make bundling the Lambda a PITA.

If I was going to implement something like this, I would create two Lambdas: 1 for PagerDuty & 1 for MS Teams and then determine the best way to filter SNS Messages accordingly.

However, your use case sounds like one that others should have solved so I’m betting there is a tool or OSS project out there. Typically the teams that I’ve worked on have solved similar use cases with Datadog or New Relic. Those platforms are great, however they come with a significant investment…both time and money. Each of the platforms will have “Get started in 10 minutes” but that’s about as honest as 6 minute abs.

Thanks. I AM now bundling 2 teams lambda’s (multiple webhook Environment variables, and the core function doing the notification for each). thanks, so my function names basically should be

• cloudwatch-alarms-to-sns-to-teams

• cloudwatch-alarms-to-sns-to-pagerduty

We also have prometheus, but it’s a bit of black box to me as a practitioner (ie. how to use requires involvement from my platform/monitoring team) , I need a point solution and honestly want to know the “AWS way” of doing things.

Pagerduty does not require json payload handling, I have usually recommended customer (who have PagerDuty) to funnel all alerts from PagerDuty, you can make rules and push the data to MS Teams or Slack.

MS Teams and Slack use web hooks so the data needs to be structured which is what the lambda does.

Whereas, Pagerduty has CloudWatch integrations, so it can injest the json output from CloudWatch as is.

ah – so what you’re saying is

1. Cloudwatch alarms to SNS topics
2. SNS topics to PAGERDUTY (easier integration)
3. PAGERDUTY to Teams (integration guide)

Yup, spot on. So there’s no needs for lambdas to manage and you can create event rules in PagerDuty.

how big is it?

inserts are not in parallel in Postgres AFAIK

and there is no S3 import in RDS for postgres

to give you an idea 500 gb will take like 10 hours or more

but depends on size , IO etc

@Fabian: trying to dissect your statement. yyou’ve already restored (manual) snapshots successfully, but here you’re trying to restore automatic snapshots (ie. those that are taken by AWS nightly)? there typically should be no difference. what i’ve seen is that you’re supposed toopen up a ticket to aws support and they can probably see what’s going on in the backgrund using their API calls

I feel like the lambda would need to do the error handling… Or maybe, create a metric from a log filter… https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/MonitoringLogData.html

DLQ -> SNS -> SQS & Lambda. Lambda handles the notification and SQS becomes the “repo” of messages that need to be manually remediated. As you identify automatic remediation opportunities you could add a Lambda process the messages from the SQS queue. You should also then go and add a filter to the notifications Lambda since you don’t need to get alerted for events you can automatically remediate.

1. Create a VPC Call it “VPN Client VPC” - Can be a small VPC /28 - Do not deploy any workloads in this VPC.
2. Deploy AWS Client VPN into o this VPC (1)
3. Attach the VPN Client VPC to the TGW as a VPC Spoke - Set appliance mode enabled for symetric routing (Attach the Subnets (Multi AZ) where the client vpn is deployed) , Add 0.0.0.0/0 to the TGW-id in the VPC RTs
4. Attach VPC B from another Region to the TGW - Modify its VPC RT to 0.0.0.0/0 TGW-id
5. If youre only using Default TGW Route Domain with auto prop and auto association then any client originating from the Client VPN should be able to ping resources in another region Spoke attachment

Typing this top off my head

Thanks @msharma24 !

Sorry dont have a working example

@Eric Steen I have recently built a AWS network Firewall with Transit gateway - you could easily fork and replace the Firewall VPC with the VPN Client VPC and it should work https://github.com/msharma24/terraform-aws-network-firewall-deployment-models/tree/main/centralized

Thanks @msharma24

like the node termination handler?

Yes

https://github.com/aws/aws-node-termination-handler

we deploy this fwiw

Yes i was looking for this ;)

Cause getting unready nodes on aws nowadays

My buildspec.yml looks like so, its failing in the install phase:

version: 0.2

phases:
  pre_build:
    commands:
      - echo prebuild commands
  install:
    commands:
      - nohup /usr/bin/dockerd --host=unix:///var/run/docker.sock --host=<tcp://127.0.0.1:2375> --storage-driver=overlay2 &
      - timeout 60 sh -c "until docker info; do echo .; sleep 1; done"
  build:
    commands:
      - ls
      - pwd
      - docker login -u $DOCKERHUB_USER -p $DOCKERHUB_TOKEN
      - git clone {repository_url} code
      - cd code
      - git checkout {branch}
      - dg project generate --name {project_name}

I’m using custom image and I confirmed its running “previlliged” mode

huh, never seen that nohup call before. why you do that?

It’s based on https://docs.aws.amazon.com/codebuild/latest/userguide/sample-docker-custom-image.html

UPD: So it turns out that I had to add VOLUME /var/lib/docker to the dockerfile because I am using a custom image and codebuild has moved from Alinux v1 to Alinux v2

Potentially because they’re managed services, with the user having no/not much control over these services.

I think you can tap into aws.health as a provider in CloudTrail to get notifications for other services.

it depends on the service. Check the EventBridge AWS schema registry and you can figure out what sort of events are published

thanks. it was more that Iwas expecting to find a ton of other tutorials for “ec2 event subscription” “fargate event subscription” “eks event subscription” , only to find that it’s really just RDS, DMS, an DynamoDB