#aws (2019-11)

you can avoid this by not enabling ebs encryption for existing instances ? There is no such thing as live-encryption of existing ebs volumes hence it will try to recreate one.

Thanks, I have encrypted the ebs root volume manually , how to refresh the terraform remote state?

you should use the key id, you can use a datasource to lookup the id with an alias. https://www.terraform.io/docs/providers/aws/d/kms_alias.html

it’s not perfect, and some boilerplate code is required to make it usable.

make sure the ami supports enhanced networking

I use EKS optimized-ami

https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html

It doesn’t say but I guess it supports enhanced networking

I think it’s hard to answer generally speaking

Also, you can look into spotinst for managing efficient pools of autoscaling spot, OnDemand and reserved instances

Depending on your workloads it might give you even more bang for the buck

They have a Kubernetes autoscaler called ocean that takes care of all the heavy lifting

Spotinst looks very great. I have a task for it in my backlog

https://aws.amazon.com/quickstart/architecture/spotinst-ocean-eks/

Cool

I’m running the fleet on T3 medium instances at the moment, with nothing reserved yet, until I figure out what we really need

9 nodes per cluster, 3 environments

It’s getting expensive lol

We use the t3 series too

Medium for playing around

I see in grafana our nodes CPU is pretty much idle, around 5%, but memory is high

SO that’s why I was asking about R5 instances

xlarge or above for anything serious

I have the K8s Cluster autoscaler, and hpa for each pods, so for now it “does the job”

But yeah, I plan to upgrade node eventually as we grow

T3 are more expensive but you trade that for the ability to burst which is may be cheaper depending on your usage

Also, #kubecost is nice to see where your money is going if you’re running a lot of micro services and namespaces

t3.medium = 0.0416 per Hour (2vcpu 4gb ram) let’s say I need a cluster of at least 8gb so 2 nodes for 0.83$

Compare to a m5.large (2vcpu 8gb) : 0.096$ per hour.

It seems to me t3 are cheaper, can burst, and gives me 2 additionnal vCPU

Correct ?

R5 seems less expensive, (if you can run on very low cpu consumption)

With r5.large (2vcpu 16gb) $0.126 per Hour

So I would need 4 t3.medium to match the 16gb of ram. Resulting in 0.1666$ per hour, or a single t3.xlarge (4vcpu 16gb) at $0.1664 per Hour.

But you need to max out the ram on the servers to be cost effectivive, or have a large fleet

Well, that’s how I see it. But I might be missing something

@kskewes how do you do version change and how do you test it?

We have extended the 2 worker groups example in eks-workers to use some locals as an easy way to update image and toggle which/both worker pools.

locals {
  # Fetch image AMI using Makefile
  # worker pool variable map shared across AZ's
  workers_01_blue = {
    enabled       = false
    image_id      = "ami-082bdeda2726e4fff" # 1.14
    instance_type = "t2.small"
  }
  workers_01_green = {
    enabled       = true
    image_id      = "ami-0f4f8678ca910061a" # 1.13
    instance_type = "t2.small"
  }

When we toggle or update image_id/etc it does update the launch_template, as can see new version (eg: v2) in AWS Console. However the default version is not changed to v2 (for example). There are also no change to the ec2 instances (rollout or otherwise), which could be my misunderstanding of how ASG’s work.

In any case, we can blue/green worker pools for now. I saw in a medium post that the ASG rolling upgrade API wasn’t exposed for use by Terraform/API directly, only CloudFormation can use it. This makes the launch template stuff above moot. Have attached snippet of what we have working right now, albeit with a nasty local hack. Don’t worry we’ll move to git based module location too.

did you try to add new instance and sea if it gets new AMI?

sorry, yes I tried to delete an instance but it didn’t create replacement with new k8s node AMI

i mean a completely new one (it’s calls launch template for a reason)

no sorry

Yeah, we think the same because sometime images do not “arrive” to the second and/or third registry for some reasons So, our way is one registry for all regions

Cool!!!

yuuuuuge!

btw, see this https://github.com/hashicorp/terraform-github-actions?

awesome

just created #github-actions channel

aha! i was just looking for that today

do they have one ?

I’ve been stuck on calls. Haven’t gotten back to it.

Basically, I’m curious about recreating the atlantis workflows

…jusing github actions

exactly my point too

we are about to buy TF Cloud

but so I think we will go that route

are you 100% terraform?

but this open the idea of having real test on instances etc

no we are not

we have TF, CF, CDK

and people clicking in the console

Haven’t yet booked it. A lot going on - not sure if can make it

no booth or anything like that?

That is good question and I am looking for such tool for our project. If you find it, please let me know

First of all Cloudformation StackSets can be applied to accounts, OUs and regions. The preparation could be done based on a python script while utilizing the code from https://github.com/awslabs/aws-securityhub-multiaccount-scripts/blob/master/enablesecurityhub.py. This should help creating a new OSS repo.

I did on a test DB – Aurora MySQL 5.7 RDS instance, no issues

@Darren Cunningham did you do have to do step #2 where you update your application? I don’t feel like most applications need to?

nope

will give it a shot on my test db — RDS Postgres

my case may not be yours –

"The methods for updating applications for new SSL/TLS certificates depend on your specific applications. Work with your application developers to update the SSL/TLS certificates for your applications."

Did it last week with a PHP app. We download the RDS CA bundle on every deploy so all I had to do was change the setting in the RDS instance.

You can use vpc flow logging to have a better look what is going on, sounds very weird indeed.

To me it sounds like the NLB Target Group is actually not marked healthy or some strange routing thing. Also make sure EGRESS Security Group rules, DNS resolving and Network ACL can’t be a cause.

I’m hoping the NLB is marked as internal to rule out strange routing problems.

it is not a case - i checked flow logs already everything is accepted. I am using dns name of NLB and it seems that it does not work if dns responds with ip of NLB from subnet where there is no healthy target. I have 3 subnets and 1 instance - to allow it to migrate to another one etc. But then only one of the NLB ip addresses is working

Do you have cross zone balancing enabled ?

enabling/disabling doesn’t change this behavior

AWS Support

AWS Config provides an inventory of your AWS resources and a history of configuration changes to these resources. You can use AWS Config to define rules that evaluate these configurations for compliance.

We’ve just about ready to migrate to EKS and this comes out. hah. Always the way. We don’t think we’ll move to managed node groups just yet. Maybe next year.

It feels like we are always as fast as we can but AWS is moving so fast it’s hard to keep up

yeah, and to be fair the other clouds have had this for a while so it was inevitable. We’re super happy so far and starting to do more PR’s that we hope are useful.

What he said; GCE has had this stuff for years; and they dont charge for the control plain (grumble 150 a month eks) Good stuff though I like having the option to fob off uninteresting details

“our vision has always been that Amazon Elastic Kubernetes Service (EKS) should provide the best managed Kubernetes experience in the cloud”

lol they’re not even the second best K8 in the cloud

https://github.com/aws/containers-roadmap/issues/530 “coming soon” in the roadmap

To clarify, my point is that AWS in aggregate moves so fast. Yes, managed node pools were late to the party, but that neglects to account for the 100x other services/features AWS has released since EKS first came on the scene.

It’s more to lament that as a company trying to codify their services and best practices in the form of terraform modules, it’s hard to keep up.

Makes TF hard work

Hey @Steve Boardwell while I haven’t personally tried this out.. what you are trying to achieve may soon become one of my use cases too. I was hoping to solve this via IAM Permissions to K8s service accounts in EKS : https://aws.amazon.com/about-aws/whats-new/2019/09/amazon-eks-adds-support-to-assign-iam-permissions-to-kubernetes-service-accounts/. Basically I would setup my pod’s SA’s role to be restricted to specific private buckets. Would that work in your case ?

Hey, thanks for the link. I’m not sure if it completely covers my use case.

For example, what about public buckets in the same account? If access is restricted to specific buckets in that account, would the ‘public’ buckets still be accessible?

We actually have a separate role for each eks clusters set of worker nodes, meaning each cluster has their own permissions pointing to a specific bucket.

However, these roles can still see the other private buckets (which do not belong to the cluster) in the same account.

Maybe https://aws.amazon.com/blogs/security/how-to-restrict-amazon-s3-bucket-access-to-a-specific-iam-role/ would solve it ? Basically allow the eks cluster node role to be able to access all buckets (IAM role Policy for nodes) but use resource -based policy on their respective private buckets to only allow the respective nodes access (hence restricting other roles automatically)?

That might work, thanks. I’ll give it a shot

great ! .. let me know how it goes

Brilliant. Thanks a lot @curious deviant - works like a charm.

Hi @Saichovsky, are you trying to put-config-rule instead ? https://docs.aws.amazon.com/cli/latest/reference/configservice/put-config-rule.html

Haven’t tried that yet. My calls ebs-get-encryption-by-default to get status and if disabled, calls enable-ebs-encryption-by-default ; I’d then like to call AWS Config’s put-evaluations which takes in resource params. This is where I get stuck.

Let me check out the link you shared for more info

sure..let me know how it goes.

For me in was a long journey, did at least 3 refactor on the job

In the end, ended up with applications policy,roles,tokens generation etc in terraform instead of packer, and kept packer image script only for the vault base image not bounded to any environment and re-usable.

Going all in with packer in the beginning was a mistake I did

Also, if you run Vault with multiple instance, when you launch them with terraform, you’ll eventually have to wait for every nodes to get correctly into the cluster.

I had trouble “configuring” the vault too soon, because some vault slave was still in “Could not find elected master” state or something like that.

So make sure you wait properly for the cluster to be fully up and running before you start configuring it

I remember setting up the certificate correctly for SSL was a bit of a pain too lol

But in the end I think it was worth

Thanks!

Haha this I can relate to!