#aws (2021-05)

You can change instance types, families, regions!

You don’t actually get reserved capacity tho. Buuut you can do capacity reservations so

Savings Plans apply to EC2, Lambda and Fargate

They’re way more flexible

The better question: is there any good reason to use RIs over savings plans?

If you do the Compute Plan where you lock into a region and I think a family of instances, it should be 1:1 with the RIs (while still allowing slightly more flexibility)

the Savings Plan is the more broad one - any region, any compute type. You get less overall savings, but more freedom to change your operation on the fly

Before savings plans, we were locked into instance types which resulted in putting off upgrading to modern instance types for years… Glad that’s over since the ability to use the newer instance types sooner (via savings plans) has saved us more money than upgrading after our RIs expire

I’m with @RB. The money saved from RIs is offset by the loss in flexibility and added management time. Savings Plan is a better tradeoff for us.

RDS is only compatible with RI’s, but I’m pretty sure you implied that by specifically mentioning fargate/lambda/ec2. =] seems like savings plans is the recommended way to go from the ‘common wisdom’ im reading

I imagine they’ll add RDS and elasticache savings plans at some point

Only a matter of time

The RDS RIs make me extremely sad

At this point, RDS being included in Savings Plans is the only announcement I am hanging on the edge of a chair for.

interesting, do you guys believe RIs take more time to manage (@Alex Jurkiewicz? News to me that people are so much more excited by Savings plans than RIs. Personally, we havent personally had that much of an issue w/ locking in instance types but I was def curious if it was easier to manage. (We also don’t use fargate and our lambda cost is miniscule)

Starts getting interesting when you’re managing RIs across regions and sub-accounts. However, a Savings Plan can be applied to shared accounts and across regions, so it’s much simpler at scale.

right, we’re managing our RIs in a single region and in a single account (although we buy RIs for all accounts there) I can see where savings plans can be easier to manage at scale when it comes to multi-region etc.

we have nonprod in 1 region and prod in 2 region, all with a jumble of different RDS db classes, so we have to pre-buy

• us-west-1 db.m5.*

• us-west-1 db.t3.*

• us-west-1 db…

• us-east-2 db.m5.*

• us-east-2 db-t3.*

• us-east-2 db… and so forth. savings plans we could have went for a single block and not worry about splitting between regions.

Managing RIs was hours of work each week for one person at our company

and then someone wants to a resize an RDS the day after you purchased

https://github.com/duo-labs/cloudtracker

this is a “feature” that we need to actually fix

in cases like that, just add some attributes to ne of the module

it will add -2 to all names in that module

not pretty, but will work

and you can select whatever attribute you want, not necessary the “2” in this example

thank you, I will try this

this worked, but how do i add the security group from the database to the elastic beanstalk, so the application ca connect to the database?

Elastic beanstalk has the SG ID output https://github.com/cloudposse/terraform-aws-elastic-beanstalk-environment/blob/master/outputs.tf#L16

DocumentDB has allowed_security_groups input https://github.com/cloudposse/terraform-aws-documentdb-cluster/blob/master/variables.tf#L7

alright, thanks

It’s an issue that stems from the Terraform Kubernetes provider. See https://github.com/cloudposse/terraform-aws-eks-cluster/issues/104 and https://github.com/terraform-aws-modules/terraform-aws-eks/issues/911

Thanks! It seems to be affecting a lot of people…

I’m seeing that problem when I set enabled=false in order to destroy the cluster.

-parallelism=1 doesn’t help

null_resource.wait_for_cluster is also causing the issue

I tried removing wait_for_cluster_and_kubernetes_configmap from the terraform state, but it still causes the aws-auth configmap to be read, leading to the error: Error: Get "<http://localhost/api/v1/namespaces/kube-system/configmaps/aws-auth>": dial tcp [::1]:80: connect: connection refused

I think that resource should be tied to context.enabled

OK a workaround turned out to be removing the aws-auth configmap from the terraform state

/25 VPC CIDR is way too small

I recommend a big CIDR

would a /24 suffice in prod too?

or something bigger?

21

we have /21 Subnets for EKS

Remember that every single Pod gets an IP addres

/21 for CIDR, what do you suggest for the subnets? Private needs to be bigger

Well, it depends on how many unique IPs you think you will need. Try to estimate it, then add a big safety margin

In our case, we will able able to spin up around 5K pods per EKS cluster

Which is way more than required

Yeah, you have to do the math. Don’t forget to account for ALBs/RDSs and whatever else you may have in the VPC.

For 5k pods I think you may have to use a /16 and at least 1 extra /16 range.

I have some logic for this that I should open source around splitting a CIDR block into regions, then VPCs in that region and then finally subnets in that VPc

You need it ASAP? I’m away from my laptop right now, how soon you need it?

I could wait till you’re back, no problem

I believe it sets it but doesn’t pass it through if set upstream

Their documentation is very ambiguous on whether its set. But we have an app seeing that header as ‘null’ on the other side so we’re a bit confused.

Personally I’d say always have at least two of anything - if you have a dev-tier and prod-tier (TGWs in this case) then you have somewhere lower risk to apply changes before you start affecting production, but you still have the headache of routes/config being specific to combinations of environments

Having multiple of anything also forces everyone to cater for a different reality, which is useful for DR, other regions, additional environments etc.

I do like separation as @bazbremner already mentioned. Depends what you want to achieve, and how your architecture looks like, and in same time to be cost effective . For example if you want to connect your own network with AWS over site-to-site vpn tunnel you would need to do it for both tgws, and that add additional costs…

If more detail and code samples are useful I’ll do a write-up

I use the cloudposse module for tagging and leave it at that. Makes it easy!

Only advice is start simple

have a specific goal, then determine if tagging is what you need to achieve that goal. then tag accordingly.

If there isn’t an absolutely must be exactly this way … then I just changed my approach to fit the module cloudposse wrote as it’s so good.

Then all my other tools just get tags = module.label.tags or module.label.id. Or I paste in the [context.tf](http://context.tf) file and add support for the naming tools they built.

As a result naming is consistent, but takes no more mental effort.

We mostly want it for costing/billing purposes

We rely mainly on Application and Environment tags. If you’re trying to allocate to a team that could be another one or you could configure cost categories to map apps to teams.

(This really depends on your company and how you want to allocate costs though)

The terraform provider has the ability to set default tags for almost all resources which is nice.

Consistency is key. From the billing requirements, some of the required tags should be fairly obvious. It’s more challenging for shared resources, in terms of cross-charging, etc.

We have this standard set of tags, which allows us to allocate AWS usage by client, environment, and service . Datadog uses the env tag to identify a “unique environment”, which for us is basically client-environment (i.e., clientX-prod). We also keep client and environment as discrete tags for searching.

This also sent me towards looking at: https://platform.sh/

Always interested in tooling that reduces complexity when possible and allows me to focus on core business needs. Love to hear any thoughts (and I plan on looking at archives too)

Sigh. I’m so tired of ‘contact us’ pricing

Just … tell me what your numbers are. If we have to negotiate from there fine

Just curious, are you objecting to platform.sh pricing transparency?

I’m not sure what @Zach meant on your pricing page, but I’ll say asking for a call to just get info on product is frustrating with all the tooling devops work requires, it’s just a major annoyance to me. I’ll most likely move on.

On your pricing page I see some transparency on pricing, and the contact us seems to be for highest tier which probably makes sense. I’d like to know what @Zach noticed specifically as well.

Just what you said

I ask because I’m an engineer and CEO. Certainly agree with you . But curious because platform.sh does have quite a bit of pricing albeit complicated and hard to extrapolate.

Nothing specific to you guys, I just clicked to check out the link the other day and I always look at the pricing. Sort of an off the the cuff thought about these things in general

To be clear, I have nothing to do with platform.sh.

Ah! misunderstood your stmt above

I’m just interested in pricing plans that don’t suck.

The problem generally is they’ll put a plan out at like $5/month for a single dev working on a personal project and then next tier is like “3 people for $50/month” and then “all others contact sales”

Yeah, exactly. You want more visibility to the point that the product is an entrenched success.

and then I’ll have to sit through a bunch of back and forth email, schedule a meeting, have an hour call to find out that the enterprise starts at like $50/person plus some other scaling factors.

Even if its complicated pricing I’d like to see something like ’starts $x based on A B and C factors”

Aside from AWS, any really awesome examples you have of those that don’t suck. Off the top of your head. CI usually pretty clear of course.

haha no, I think just about everyone hides their ‘enterprise’ prices

GitHub, GitLab, atlassian.

Its sort of like if you walk by a fancy clothing store in the mall, when they don’t have prices out you figure “out of my league” and just walk past

Yeah, I guess the counter to this is that pricing that seems inconsequential at small scale can appear asinine at large scale.

plus the usual “please don’t put SSO on your enterprise only plan”

Here’s a couple that show pricing very clearly on the enterprise level (though they may have addons) https://spacelift.io/pricing.html vs https://www.env0.com/pricing (no detail on enterprise)

I think that enterprise level at some of these places is such a small number that it’s probable that they want hands on with with sales/engineering to ensure it’s successful. I get that. I guess I’d still like an approx range from someone offering enterprise support like “starting at” vs “contact us.

I didn’t. What error are you getting?

NLB can have static IP

If you mean static IP for outbound traffic, use a NAT gateway

ok, thanks! NAT gateway sounds like a good solution

Global accelerator is another option, depending on the use case

multi-payer/multi-org environments is the main reason for me

we have some accounts that need to strictly separate billing, but everything else (cloudtrail, securityhub, guardduty, etc) can be linked to a single management account

Thank you.

Why would an S3 bucket be down? What do you mean there?

okay why i asked this question is, s3 is a object storage and behind the scenes its just a storage devices what if that goes down and my s3 buckets are down,.. at that time is there any why to notify users

I think s3 is resilient to nine nines

And I only remember 1 time on passed 10yrs of it going down

okay thank you @pjaudiomv

For lambda you can use a cloudwatch metric alarm to alert on failed invocations

You can use the health dashboard to alert on actual aws service loss

Yea you can use sns to send an email notification

I think if S3 goes down, all of us will have bigger problems.

But if you truly want to monitor your content and access to S3, I would suggest setting up a resource outside of AWS (perhaps in GCP or Azure) that has permission to access your bucket in the way you want to monitor: HTTPS or direct API access. And then have that service report into your monitoring system that the access is still good. So if indeed S3 goes offline for some reason (which would likely be a larger problem in AWS which might also affect their ability to notify you) you would still be able to get your monitor and alert that your bucket is inaccessible.

Github actions, Github packages with self hosted Github runner

there is no best CICD, test a group of them and use the one that match your use case.

Have to agree with Mohammed. There are literally hundreds of CICD tools out there and the best one is the one that fits your needs and use cases.

• If you are a GitHub shop, you might consider GitHub actions

• If you are a GitLab shop, you might consider GitLab CI

• If you use Docker with your PHP, you might rely on Docker Hub to build and publish your images i’m sure there are other combinations I can come up with but you get the idea. Start by taking a look at a few tools that are already close or tangential to the way you are already working. Don’t belabor your analysis but quickly pick one and go with it. If it works, you just found the best tool.

At the same time, don’t feel stuck. If you realize what you thought was the best tool is no longer working, start the process again with the next tool until you find the right fit.

for flexibility and ease of use i’d recommend checking out Buildkite.

@managedkaos @Mohammed Yahya I agree. Asked because I’m looking for a few to vet

@Mo Battah can you give a high level description of your application and/or runtime environment? that might help a bit with recommendations. also if you are using repo deploys, building containers, kubernetes, etc..

For resource heavy CI/CD pipelines, which include dependencies and project specific build/deploy workflows, perhaps Jenkins or TeamCity.

For CI/CD on single repos or with fewer dependencies, and less complicated workflows I’d go for Gitlab CI or Github actions, Travis, CircleCI. Mainly due to the reduced set up overhead and simpler to plug-and-play.

There’s also the world of GitOps/k8s-oriented with tools like ArgoCD. I’m unfamiliar with this though, but it’s anyways interesting

Buildkite is really nice. It’s very simple, flexible and easy to use. It doesn’t really do anything that GitHub Actions doesn’t, though.

Does it have a native roll your own solution? Actions doesn’t have anything official for that although there are 2 really good open source solutions for that

“It” being BuildKite?

yarp, just curious

They supply the SaaS control plane and a golang runner/agent. That’s really it. Think of it like GiLab CI unbundled. Or GitHub Actions unbundled.

If I understand right, you can use Github Action with your own runners ( for example in ECS service for AWS use case) for critical workload, same as buildkite

I see a lot of hype following hybrid approach for their CICD even for Terraform

yeh but there isn’t any kind of native scaling options, they provide a self-hosted capability which is mainly designed for static machines. If you want it to be elastic you need to do quite a lot of work yourself

https://github.com/philips-labs/terraform-aws-github-runner this project lets you provide self-hosted machines using the serverless tooling on AWS and https://github.com/actions-runner-controller/actions-runner-controller this project does it for the k8s platform

Thanks for sharing, good to know

I personally feel that a hosted control plane and self hosted agents (with pull connectivity) is the sweet spot of SaaS for CICD. It gives a good amount of flexibility without the overhead of managing control planes.

Yes, @Chris Fowles. The first time I tried Buildkite (2017 or so) my reaction was “OMG, this is how CI was supposed to be. Why did I suffer for so long?” I’m sure there were others that got it right before them. But it’s still a great standalone experience. Now lots of choices with this model.