#aws

:aws: Discussion related to Amazon Web Services (AWS) Archive: https://archive.sweetops.com/aws/

2019-11-18

AWS Config ?

loren

Terraform plan?

Lewis

Config would be your best bet - it will inform you of changes from your BASELINE report created at the start, any changes from that such as s3 bucket permissions changed it will notify you so that you can resolve.

Lewis

AWS Config provides an inventory of your AWS resources and a history of configuration changes to these resources. You can use AWS Config to define rules that evaluate these configurations for compliance.

Erik Osterman
Extending the EKS API: Managed Node Groups | Amazon Web Services

By Raghav Tripathi, Michael Hausenblas, and Nathan Taber From our first conversations with customers, our vision has always been that Amazon Elastic Kubernetes Service (EKS) should provide the best managed Kubernetes experience in the cloud. When we launched EKS, our first step was to provide a managed Kubernetes control plane, but we never intended to stop […]

kskewes

We’ve just about ready to migrate to EKS and this comes out. hah. Always the way. We don’t think we’ll move to managed node groups just yet. Maybe next year.

Extending the EKS API: Managed Node Groups | Amazon Web Services

By Raghav Tripathi, Michael Hausenblas, and Nathan Taber From our first conversations with customers, our vision has always been that Amazon Elastic Kubernetes Service (EKS) should provide the best managed Kubernetes experience in the cloud. When we launched EKS, our first step was to provide a managed Kubernetes control plane, but we never intended to stop […]

Erik Osterman

It feels like we are always as fast as we can but AWS is moving so fast it’s hard to keep up

kskewes

yeah, and to be fair the other clouds have had this for a while so it was inevitable. We’re super happy so far and starting to do more PR’s that we hope are useful.

2019-11-17

has anyone been able to terraform out inspector in an automated fashion?

inspector ?

aws inspector

im having issues finding good examples

I have not used, just guard duty and config

I have those both working well, was wondering about inspector. its cool

Hi

I’m looking for a solution to monitor changes in AWS

Anyone knows a solution? preferable open source one

2019-11-16

2019-11-15

Mateusz Kamiński

Hem, does AWS network load balancing works between instances? I have insance A (part of asgA) and instance B(part of asgB). I need to connect from A to non http endpoint in B. I am trying to do it with NLB but it does not work. I am starting to wonder if i am missing something - maybe it cannot be done that way? (Security groups on both A and B allows whole 10/8 network and all servers are in it.)

Mateusz Kamiński

This looks really strange - when i try to telnet from A to B it stays around 3-5 minutes in Trying.. and then it connects. Till that time i can see only Syn packages being sent from source host and no packages on target host. Once connection is established i can see first packages reaching target

You can use vpc flow logging to have a better look what is going on, sounds very weird indeed.

To me it sounds like the NLB Target Group is actually not marked healthy or some strange routing thing. Also make sure EGRESS Security Group rules, DNS resolving and Network ACL can’t be a cause.

I’m hoping the NLB is marked as internal to rule out strange routing problems.

Mateusz Kamiński

it is not a case - i checked flow logs already everything is accepted. I am using dns name of NLB and it seems that it does not work if dns responds with ip of NLB from subnet where there is no healthy target. I have 3 subnets and 1 instance - to allow it to migrate to another one etc. But then only one of the NLB ip addresses is working

Do you have cross zone balancing enabled ?

Mateusz Kamiński

enabling/disabling doesn’t change this behavior

AWS Support

2019-11-14

Unang Val

This can be used for instances only http://answersforaws.com/code/graffiti-monkey/

Unang Val

@

Andrew Jeffree

That looks like it only transfers existing tags from the instance to volumes/snapshots?

2019-11-13

loren
Detecting Manual AWS Console Actions

Practice infrastructure-as-code in your organization and learn how to detect when engineers make manual changes in your AWS Console

3
1
Erik Osterman

Amazing! Thanks for sharing. This is a must have.

loren

yeah, would be a cool project around the idea, tracking/matching the event patterns and managing the cloudwatch event rule

Erik Osterman

Would also like to see more meta data

Erik Osterman

In the slack alert

Looking for a solution to automatically tag AWS resources that were created with the creator and time . I came across this project https://github.com/GorillaStack/auto-tag .

Anyone else have recommendations?

GorillaStack/auto-tag

Automatically tag AWS resources on creation, for cost assignment - GorillaStack/auto-tag

2019-11-11

2019-11-10

2019-11-08

Anybody knows a tool, shell or GitHub (python) project which helps me working within AWS organizations to rollout a bunch of settings across all accounts and regions? Especially for those services which aren’t integrated into AWS orgs. I’d like to use assume-role into the target account.

That is good question and I am looking for such tool for our project. If you find it, please let me know

First of all Cloudformation StackSets can be applied to accounts, OUs and regions. The preparation could be done based on a python script while utilizing the code from https://github.com/awslabs/aws-securityhub-multiaccount-scripts/blob/master/enablesecurityhub.py. This should help creating a new OSS repo.

awslabs/aws-securityhub-multiaccount-scripts

This script automates the process of running the Security Hub multi-account workflow across a group of accounts that are in your control - awslabs/aws-securityhub-multiaccount-scripts

joshmyers

Have you looked at organisation group policies? SCP

SCPs don’t configure services.

well, only a few

Rotating Your SSL/TLS Certificate - Amazon Relational Database Service

Rotate your SSL/TLS certificate as a security best practice.

Darren Cunningham

I did on a test DB – Aurora MySQL 5.7 RDS instance, no issues

Rotating Your SSL/TLS Certificate - Amazon Relational Database Service

Rotate your SSL/TLS certificate as a security best practice.

@Darren Cunningham did you do have to do step #2 where you update your application? I don’t feel like most applications need to?

Darren Cunningham

nope

will give it a shot on my test db — RDS Postgres

Darren Cunningham

my case may not be yours

"The methods for updating applications for new SSL/TLS certificates depend on your specific applications. Work with your application developers to update the SSL/TLS certificates for your applications."
Joe Niland

Did it last week with a PHP app. We download the RDS CA bundle on every deploy so all I had to do was change the setting in the RDS instance.

Darren Cunningham

If anybody has recommendations as to how to best implement DLQs w/Lambdas I’d appreciate input – trying to figure out if I should have a single target by Lambda, by region, by account or have a single target for my org. I’m thinking that I want to “default” to a SNS topic by region.

2019-11-07

Nikola Velkovski

Nikola Velkovski

so you need GH actions to compile GH actions

lol

Question : is Cloudposse going to be at re:Invent ?

Erik Osterman

Haven’t yet booked it. A lot going on - not sure if can make it

no booth or anything like that?

2019-11-06

Hugo Lesta

Hello there, how are you? Could you please tell me if u have any experience using cloudwatch anomaly detection? I couldn’t find this resource in terraform docs.

Jean-Michael Cyr

I kept my images in gitlab

New – Savings Plans for AWS Compute Services | Amazon Web Services

I first wrote about EC2 Reserved Instances a decade ago! Since I wrote that post, our customers have saved billions of dollars by using Reserved Instances to commit to usage of a specific instance type and operating system within an AWS region. Over the years we have enhanced the Reserved Instance model to make it […]

4

Cool!!!

New – Savings Plans for AWS Compute Services | Amazon Web Services

I first wrote about EC2 Reserved Instances a decade ago! Since I wrote that post, our customers have saved billions of dollars by using Reserved Instances to commit to usage of a specific instance type and operating system within an AWS region. Over the years we have enhanced the Reserved Instance model to make it […]

4

yuuuuuge!

4

that is running inside one my ECS instances in AWS

I did not have to do anything access wise, it is done trough a tunnel to github

no user input needed just the command to run the installer ( it is in beta)

but the fact I can run this means I could start a container with the software needed, spin it up upon action in git wait until the hosted runner service is up and then run terraform apply that will be using the credentials given by IAM profile to specific resources, limiting what the runner can do. This is a pretty cool new feature of githug actions self hosted runners

Awesom

Erik Osterman
hashicorp/terraform-github-actions

GitHub Actions For Terraform. Contribute to hashicorp/terraform-github-actions development by creating an account on GitHub.

awesome

Erik Osterman

just created #github-actions channel

Nikola Velkovski

@PePe I guess it can be run in Fargate as well ?

as long as you run the executable ( compiled from source) then yes

as the entrypoint

my guess is they will have a docker image soon

Erik Osterman

aha! i was just looking for that today

do they have one ?

Erik Osterman

I’ve been stuck on calls. Haven’t gotten back to it.

Erik Osterman

Basically, I’m curious about recreating the atlantis workflows

Erik Osterman

…jusing github actions

exactly my point too

we are about to buy TF Cloud

but so I think we will go that route

Erik Osterman

are you 100% terraform?

but this open the idea of having real test on instances etc

no we are not

we have TF, CF, CDK

and people clicking in the console

2019-11-05

kskewes

Hey team, looking at the autoscale-group module as called from eks-workers module. I see in Console that launch_template_version default isn’t being set to the latest. I have v1 (default) and v2. $Latest is the default in the autoscale-group module though: https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/blob/master/variables.tf#L78 eks-workers module doesn’t supply launch_template_version value to autoscale-group so figure it should use default (of $Latest) I’m trying to work out how nodes will rotate with an update to launch template and validating blue/green worker pools. $Latest seems to be correct per terraform provider docs: https://www.terraform.io/docs/providers/aws/r/autoscaling_group.html#with-latest-version-of-launch-template

cloudposse/terraform-aws-ec2-autoscale-group

Terraform module to provision Auto Scaling Group and Launch Template on AWS - cloudposse/terraform-aws-ec2-autoscale-group

aknysh

@kskewes how do you do version change and how do you test it?

cloudposse/terraform-aws-ec2-autoscale-group

Terraform module to provision Auto Scaling Group and Launch Template on AWS - cloudposse/terraform-aws-ec2-autoscale-group

kskewes

We have extended the 2 worker groups example in eks-workers to use some locals as an easy way to update image and toggle which/both worker pools.

locals {
  # Fetch image AMI using Makefile
  # worker pool variable map shared across AZ's
  workers_01_blue = {
    enabled       = false
    image_id      = "ami-082bdeda2726e4fff" # 1.14
    instance_type = "t2.small"
  }
  workers_01_green = {
    enabled       = true
    image_id      = "ami-0f4f8678ca910061a" # 1.13
    instance_type = "t2.small"
  }

When we toggle or update image_id/etc it does update the launch_template, as can see new version (eg: v2) in AWS Console. However the default version is not changed to v2 (for example). There are also no change to the ec2 instances (rollout or otherwise), which could be my misunderstanding of how ASG’s work.

kskewes

In any case, we can blue/green worker pools for now. I saw in a medium post that the ASG rolling upgrade API wasn’t exposed for use by Terraform/API directly, only CloudFormation can use it. This makes the launch template stuff above moot. Have attached snippet of what we have working right now, albeit with a nasty local hack. Don’t worry we’ll move to git based module location too.

aknysh

did you try to add new instance and sea if it gets new AMI?

kskewes

sorry, yes I tried to delete an instance but it didn’t create replacement with new k8s node AMI

aknysh

i mean a completely new one (it’s calls launch template for a reason)

kskewes

no sorry

Chris Fowles

people using multi account and ecr + eks: how are you distributing images between accounts? i.e. are you a) allowing accounts to pull from a central repository or b) pushing images to multiple accounts — follow up question: what hurts about the option you’ve gone with?

Erik Osterman

Good question. On a recent engagement, we went with option (a), but considered both. It was just so much easier to have a centralized repo rather than worry about promotion.

Erik Osterman

@Igor Rodionov can answer technical questions.

Chris Fowles

how painful was management of cross account repo policies?

are you doing multiregion ?

Chris Fowles

single region

so multi aws account one ECR ?

Chris Fowles

yeah - that’s option a

Chris Fowles

basically ecr hosted within a shared ‘infra’ account and different clusters pull from that

Chris Fowles

option b is that images are pushed to different accounts based on processes

Igor Rodionov

@Chris Fowles we use A

We are in multi region but we use the “regional account ecr” as a central repo

Yeah, we think the same because sometime images do not “arrive” to the second and/or third registry for some reasons So, our way is one registry for all regions

so same as having option A

Chris Fowles

yeh cool

Chris Fowles

cheers

it is such a pain to push images to many repos

4
Chris Fowles

I was leaning towards A for that reason

Chris Fowles

thanks folks

2019-11-04

Jean-Michael Cyr

Anyone here runs R5 AWS instance type for kubernetes ? Our fleet runs pretty high on memory and low on CPU, and I wonder if switching to R5 would be a good cost effective move. Is R5 cpu usually good enough to support a kubernetes with multiple web services (with low to mid CPU consumption) ?

Karoline Pauls

make sure the ami supports enhanced networking

Jean-Michael Cyr

I use EKS optimized-ami

https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html

It doesn’t say but I guess it supports enhanced networking

Amazon EKS-Optimized Linux AMI - Amazon EKS

The Amazon EKS-optimized Linux AMI is built on top of Amazon Linux 2, and is configured to serve as the base image for Amazon EKS worker nodes. The AMI is configured to work with Amazon EKS out of the box, and it includes Docker, kubelet , and the AWS IAM Authenticator.

Erik Osterman

I think it’s hard to answer generally speaking

Erik Osterman

Also, you can look into spotinst for managing efficient pools of autoscaling spot, OnDemand and reserved instances

Erik Osterman

Depending on your workloads it might give you even more bang for the buck

Erik Osterman

They have a Kubernetes autoscaler called ocean that takes care of all the heavy lifting

Jean-Michael Cyr

Spotinst looks very great. I have a task for it in my backlog

Erik Osterman
Spotinst Ocean for Amazon EKS Nodes on AWS - Quick Start

Learn about the Quick Start architecture and details for deploying Spotinst Ocean for Amazon EKS Nodes in the AWS Cloud.

Erik Osterman

Cool

Jean-Michael Cyr

I’m running the fleet on T3 medium instances at the moment, with nothing reserved yet, until I figure out what we really need

Jean-Michael Cyr

9 nodes per cluster, 3 environments

Jean-Michael Cyr

It’s getting expensive lol

Erik Osterman

We use the t3 series too

Erik Osterman

Medium for playing around

Jean-Michael Cyr

I see in grafana our nodes CPU is pretty much idle, around 5%, but memory is high

Jean-Michael Cyr

SO that’s why I was asking about R5 instances

Erik Osterman

xlarge or above for anything serious

Jean-Michael Cyr

I have the K8s Cluster autoscaler, and hpa for each pods, so for now it “does the job”

Jean-Michael Cyr

But yeah, I plan to upgrade node eventually as we grow

Erik Osterman

T3 are more expensive but you trade that for the ability to burst which is may be cheaper depending on your usage

Erik Osterman

Also, #kubecost is nice to see where your money is going if you’re running a lot of micro services and namespaces

Jean-Michael Cyr

t3.medium = 0.0416 per Hour (2vcpu 4gb ram) let’s say I need a cluster of at least 8gb so 2 nodes for 0.83$

Compare to a m5.large (2vcpu 8gb) : 0.096$ per hour.

It seems to me t3 are cheaper, can burst, and gives me 2 additionnal vCPU

Jean-Michael Cyr

Correct ?

Jean-Michael Cyr

R5 seems less expensive, (if you can run on very low cpu consumption)

With r5.large (2vcpu 16gb) $0.126 per Hour

So I would need 4 t3.medium to match the 16gb of ram. Resulting in 0.1666$ per hour, or a single t3.xlarge (4vcpu 16gb) at $0.1664 per Hour.

Jean-Michael Cyr

But you need to max out the ram on the servers to be cost effectivive, or have a large fleet

Jean-Michael Cyr

Well, that’s how I see it. But I might be missing something

2019-11-03

Erik Osterman
localstack/localstack

A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline! - localstack/localstack

1
Maciek Strömich

it’s not perfect, and some boilerplate code is required to make it usable.

localstack/localstack

A fully functional local AWS cloud stack. Develop and test your cloud & Serverless apps offline! - localstack/localstack

1

2019-11-02

winter

terraform is replacing the instance while enabling ebs encryption after creation of the instance. Any way to avoid this?

you can avoid this by not enabling ebs encryption for existing instances ? There is no such thing as live-encryption of existing ebs volumes hence it will try to recreate one.

winter

Thanks, I have encrypted the ebs root volume manually , how to refresh the terraform remote state?

winter

Should we use kms_key_id (arn) or the kms_alias as a value for the kms_key_id while encrypting root volume? If I use the alias, terraform is recreating the instance saying the kms key id is different from the last time.

you should use the key id, you can use a datasource to lookup the id with an alias. https://www.terraform.io/docs/providers/aws/d/kms_alias.html

AWS: aws_kms_alias - Terraform by HashiCorp

Get information on a AWS Key Management Service (KMS) Alias

    keyboard_arrow_up