#aws (2020-05)

If I remember correctly, RDS doesn’t actually give real superuser privileges. It’s a big drawback. Let me look up a link for you.

https://serverfault.com/questions/661661/why-cant-i-create-a-superuser-in-aws-postgresql-instance

lol, yeah

I ended up needing/creating a logical replication slot

Cool — Glad you figured it out. Similar thing tripped me up a couple months back and I was pretty surprised by that one.

It’s funny, I’ve been using AWS for years. . . so many rabbit holes

Yeah — Learn something new all the time haha.

Why do they need to be on Github ?

where do you store them

i need best practices

SSM or Vault ?

• You can store them in k8s secrets directly.

• You can store them in your pipeline directly and have k8s secrets populated from there

• You can use any other secret manager like vault & ssm together with the pipeline to populate the secrets

• You can use ssm or vault directly from the container or container helper

i am doing the second one but i need to secure it more and also i need to make it versioned so i need some sort of SCM capability

are you only using it for passwords or all kinds of config items ?

only password

Do you really need SCM on a password which you probably have stored somewhere else already ?

Hi, do you use Helm to deploy? Helm secrets plugin can help to encrypt secrect information and you can commit encrypted file into git. And also Decrypt action will be invoke when you deply via helm

Maarteen for GitOps stuff

https://github.com/zendesk/helm-secrets

https://www.weave.works/technologies/gitops-frequently-asked-questions/#how-do-you-manage-secrets

Sealed-secrets solves the following problem: “I can manage all of my Kubernetes configs in Git, except Secrets.” Sealed Secrets is a Kubernetes Custom Resource Definition Controller that lets you to store sensitive information (aka secrets) in Git.

I never use Sealed-secrets, but I believe the concept is very similar as other secrets management in version control

However, the key point is how to store your private key.

In AWS, you can store private key in KMS

Currently i use KMS to encrypt my passwords etc and put the output string in terraform.tfvars file

I want something better

I randomly generate a pass, encrpyt it via KMS put it on terraform.tfvars (as encrypted output of KMS) and re-read and decrypt it to create kubernetes Secrets (which is not encrypted but only base64 decoded)

Maybe i can talk with developers so they can develop the code to use injected variables with AWS KMS encrpyted instead of using it plaintext

A pattern/workflow I used is this; have Terraform create a random password. Use that password to set the DB pw, store the password in SSM for later use by application or deployment.

+1 for Sealed Secrets. Great tool that is completely cloud agnostic

Implemented SealedSecret it is very easy to install/maintain also AWS recently published a blog for that https://aws.amazon.com/blogs/opensource/managing-secrets-deployment-in-kubernetes-using-sealed-secrets/

is your app running OK from the Beanstalk (not CloudFront) URL (e.g. `myapp.us-east-1.elasticbeanstalk.com) ?

I managed to get it working, I had to create a ACM cert in Virginia in order for it to allow me to use my own domain. My load balancer was rejecting the Https request because the referrer was not correct I think so the Https negotiation was failing.

Now I just need to figure out how to terraform it

I take it these are applications that don’t serve web traffic then? They’re some sort of data/job processors?

yes for the most part if I understand those services correctly

having an ELB do nothing but perform healthchecks is a bit expensive for what its doing, granted its like $20/month with no traffic but still. (And a bit absurd)

agreed!

Probably best to have a conversation with them and figure what they’re trying to do and why …

@RB I use the healthcheck configuration in ECS task definitions on most projects and wire it up to a script that either checks the PID of the primary process or if it’s a web server then executes a curl against that process.

ECS docs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_healthcheck

That’s if you need customization beyond Docker’s HEALTHCHECK. I personally haven’t used the Docker healthcheck before.

isnt it the same thing ?
The container health check command and associated configuration parameters for the container. This parameter maps to HealthCheck in the Create a container section of the Docker Remote API and the HEALTHCHECK parameter of docker run

@RB Ah it is the same. The Task Def just provides a wrapper around those same configuration arguments. I hadn’t looked deep enough at the Dockerfile HEALTHCHECK ref or the Docker run ref. Didn’t know it provided that level of granularity, so I though AWS was providing added configuration as a benefit.

TIL

We tried it out for a few days and then immediately moved everything to it

@Zach would you say its pretty cheap?

yah, if you aren’t pushing boatloads of data out

its negligible

we used it so that we could have a static IP on our entry ALB, and to move them private at the same time

we’re getting charged roughly ~400$/mo for ALB bandwidth

i am excited about moving the ALB to private and we will benefit from the static ip as well

Ah, you are in a different tier of traffic than we are then haha

i assume global accelerator is a subset of that?

so we’ll probably pay maybe $600/mo

oh, yah its on top of the ALB costs.

@Zach do u also use a CDN?

I’m waiting on some IP whitelisting for using it. But loving the idea.

one!

You can do single value secret or you can store multiple. For instance I did RDP users secret and setup multiple users on local systems, while my rds logic uses 1 for admin. It is a little clunky getting the values out as ends up being key = username sorta from what I remember. Basically was like: $secret.MyName = $secret.MyName.Value or something like that. Found it confusing at first and later I tried to avoid embedded multiple and instead tried to stick with single value when possible

He makes some decent points against due to his own experience, but it seems in a number of those cases he should have RTFM and done some service comparison before jumping head first into those services.

The lambda one is pretty funny — sounds like the worst nightmare of them all if that’s the implementation path their team went down.

Can’t take the list seriously as Elastic Beanstalk is not even mentioned

Did not seem like anyone was going to pick up aws-okta maintenance, so we moved to gimme-aws-creds (maintained by Nike)

i’ve really been impressed by aws-okta-processor, https://github.com/godaddy/aws-okta-processor/

how do these methods work for delivering aws creds ?

you configure Okta as a SAML provider to AWS, and these allow you to assume a role in the account. It then passes back a temorary credential pair

and if they do it right, they refresh the aws temporary cred automatically (until your Okta session expires, then it re-prompts)

^ we haven’t gotten that part to work correctly

try aws-okta-processor

I’ve only just managed to get everyone switched to gimme-aws … I might get tarred and feathered if I make anyone swap again

it’s the only tool i’ve used that gets caching of both aws and okta sessions right, and works with mfa, and works with credential_process, and manages stdout/stderr well enough to work with pretty much any tool built for any sdk

plus the maintainers have so far been super responsive and open to prs

well, it’s your box, authenticate however you like if they keep using gimme-aws-creds that’s fine. you can enjoy the goodness of aws-okta-processor

I wonder how effective it is? there seems to be quite a bit of config + lambda to just enable this feature that when I compare this to Hashicorp Vault seems more convoluted

@jose.amengual what will you be using it for ?

The AWS counterpart of Vault would be SSM by the way, not the secret manager. For Aurora you can use this: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/UsingWithRDS.IAMDBAuth.html

I’m talking about this : https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-rds.html

I want to use the automatic secret rotation feature for programatic access

~I think IAM Auth is perfect as alternative there, unless you expect 200 new connections per second.~

we are going to use IAM auth for users

but for apps we want to rotate the secret

I don’t know if IAM auth is recommended for apps

I’m thinking it might be a tad slower with connection creation, so I’m not sure. I take it back.

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html#UsingWithRDS.IAMDBAuth.ConnectionsPerSecond

MySQL Limitations for IAM Database Authentication

so for users is fine

but apps maybe not

@raghu What’s the goal you’re trying to accomplish?

Typically IAM Policies do not involve network whitelisting / blacklist. You can restrict certain actions only be allowed if they’re coming from a certain IP using the SourceIP Condition, but that is pretty uncommon.

If you’re trying to do network related whitelisting / blacklisting for requests in-coming to your application then there are typically 3 places you should look (in order of preference likely):

1. Security Group
2. VPC NACL
3. WAF Rules

Oops, I mean, In bucket policy

This is likely what you’re looking for then: https://aws.amazon.com/premiumsupport/knowledge-center/block-s3-traffic-vpc-ip/

Absolutely, but i have to add one more condition on top of that.Thankyou @Matt Gowie

I am interested if you are able to do this… I have been wondering the same thing. I am not setting things up yet for a new project, but was wondering the same things as I was laying out the architecture

Still no luck

The only thing comes to my mind is seperating namespaces

And installing 1 ingress for each ns

And seperate ns communication with networkpolicy

Hopefully someone has a simpler answer.

Also that means a seperate external-dns in each ns as external-dns can only wathc 1 ns for each deployment

@chris so far nothing

The only answer is istio

Thanks for the update

found my answer @ https://docs.aws.amazon.com/vpc/latest/userguide/vpce-interface.html

it depends on the target of your subdomain – are you hosting your content in S3, on an EC2, in Fargate?

@Darren Cunningham thanks for responding! I’m using EC2

I was able to get my subdomain to work on a public hosted zone, but I want to restrict access to it. So I thought that moving to a private hosted zone was the move, and idk where to go from here

private hosted zone juts means that your zone would only resolve within the VPC – that’s not what you want

@Darren Cunningham use a security group , restrict access to port 443 or 80 , whatever port the webserver is running on

For username and password, you might use ngnix as proxy , whitelist the IPs of the users in the security group

Is it possible to do this without updating the app?

Would the security group be enough?

basic auth would require updating the app

security group is “enough” if you’re ok with it

Also, how do I find the security group? Is it tied to the VPC?

yeah I’m okay with just having the security group to allow it

Your security group is tied to your EC2 instance.

it would be easier to find by going to the Ec2 and seeing which SGs are allocated specifically to it

if you go to VPC you’re going to find ALL of them in your VPC/Account

And @Darren Cunningham is correct — you don’t want a private hosted zone. That means your DNS entry for your site will only be visible inside your VPC, which means the public won’t be able to access it. You want a public hosted zone.

@Matt Gowie okay, but is there a way for me to just choose who accesses the VPC?

The VPC as a whole? You can do that via your VPC’s NACL (Network Access Control Lists) but you likely just want to do that via Security Groups.

Yeah right now the NACL says “All Traffic” but I still can’t load the domain on my computer

so maybe it is best that I just make it a public hosted zone with a security group, thanks

Yeah, go with a public hosted zone and check your security group’s inbound rules. Likely your NACL is not the issue. The default rules are typically user friendly.

When I do this, would I be able to have the security groups recognize a username/password? Or do I just simply whitelist specific IP addresses?

Security Groups purely deal with the network so they cannot be configured to help with username/pass. Username / password needs to be a layer 7 level so on a proxy or application layer.

@Matt Gowie so I’ve been trying a lot of combinations of security groups the past 2 hours.

This works:

Type - All Traffic
Protocol - All
Port range - All
Destination - 0.0.0.0/0

This doesn’t work:

Type - All Traffic
Protocol - All
Port range - All
Destination - <my ip address>/32

& I don’t know why

(for outbound rules)

I’m ready to give up and just create security in the app. Do you have any idea why inserting my personal IP address doesn’t seem to work though?

The second one should work. How are you finding your IP?

I select “My IP” from the dropdown

Ah. Interesting. And your request times out?

granted, this is an ELB security group created from Kubernetes

Try http://icanhazip.com/

if that at all makes a difference

Ah if you’re in K8s land then I’m unsure.

gotcha I’ll post on that channel

thanks for the help!

What about AWS Config? It supports auto-remediation.

Pray tell. That sounds like an interesting story

¯_(ツ)_/¯

We don’t need any of that newfangled pooled connections

We were looking into RDS password rotation and the IAM Auth looked appealing, I heard good things about it, but was cautioned “be careful not to get throttled” because, supposedly, they will deny connections for 10-15minutes

read the docs, tested, warned my teams “make sure you don’t blow through the connection rate”

Now we have a couple of cases where we suspect we are being throttled, and I have a support ticket open … but AWS is incredibly vague about this

the postgres logs, once we enabled IAM Auth, are filled with all sorts of nonsense that’s hard to decipher. There’s no single error that says “stop connecting so fast” so we don’t know what we’re looking for

Is there any RDS metric you can pull that might shed light?

Nope

The docs on it just give you some very rough guidelines. It works great though - until you hit the wall. At least we think we’re hitting the wall, we can’t tell for sure.

omg, I was thinking about enabling that in the near future. This interests me.

same, please update the thread when you figure out what’s going on

I strongly suspect it was one application in particular causing the issue because it was not using pooled connections, and not even using persistent connections I’m just waiting on AWS support to tell me that yes, those errors in the PG logs are “iam throttling”

it’s silly that there’s not a metric that you could be monitoring

I agree … and I will be commenting about that to our support rep. I suspect they’re obfuscating it for “security”

There is a postgresql query to show active connections

https://stackoverflow.com/questions/5267715/right-query-to-get-the-current-number-of-connections-in-a-postgresql-db

Yah I can see the current connections no problem - but those are the ones already established. RDS exposes that as a metric directly.

The issue that I cannot see how many connections are being attempted through RDS IAM - which has a hidden max rate of connections per second

I could be well under the max connections the DB will allow but if I try to create those too fast, IAM will throttle the entire RDS instance - which is bad enough for a single application, but if you’re being economical and several apps with databases on a single instance, its really bad

also interested in this

@Zach you can maybe use vpc flow for that. You can query Tcp Syn packets coming to RDS 5 min interval or 1 min interval etc.

https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html#flow-logs-basics tcp-flags 2 means SYN

“ACCEPT 2 OK “ 2 means SYN here.

SELECT SUM(numpackets) AS packetcount, destinationaddress FROM vpc_flow_logs WHERE destinationport = 5432 AND protocol = 6 AND tcp-flags = 2 AND date > current_date - interval ‘5’ min GROUP BY destinationaddress ORDER BY packetcount DESC

You may need to create table like https://github.com/awsdocs/amazon-athena-user-guide/blob/master/doc_source/vpc-flow-logs.md including tcp-flags

in TGW you still need subnets and IPs AFAIK

the only case when this is not true is when you use Shared VPCs ( which we are using)

then you use SGids

Got it. I’ve set it up with CIDR.. does the trick, just a little less secure

I wonder how much you need to know about Amazons internals to understand why they descided to do it this way

well I guess their mistakes keep me employed

and keep the AWS wheels turning

lol

my company has just decided to hard pivot towards Azure

no good reason, but lots of work to do

It’s much cheaper.

if you have an instance with an EIP associated to a route53 record, the instance was killed, how do you make sure the route53 record is then killed so no one gets a hold of the EIP and takes over the subdomain ?

You need to see the EIP as reserved for the AWS account, no-one else can use it unless you explicitly release the EIP.

The lifecycle of an EIPs is not dependent on the instance is associate to AFAIK. You retain that EIP even if the associated instance dies.

Oh yes — You gotta reserve it though

we release our EIPs

sounds like we gotta write up a script for this

@RB Just don’t release your EIP’s, enough IPV4 for everyone

thats true. hmm. i thought we had to release the eips due to cost.

from https://aws.amazon.com/ec2/pricing/on-demand/, i suppose the cost is pretty low

Yes, I mean, if you have 10 unused EIP’s the whole time, you might want to clean up

otherwise it’s not the hassle

reserved EIPs are not very expensive. Much cheaper than a LoadBalancer. If you only have 1 instance that is sitting behind the R53 record, then pointing at the EIP is a reasonable approach

@James Woolfenden AWS Amplify.

Really great option. Comes with CI / CD, ENV var support, building pipelines, Git integration, etc etc etc. It’s the best.

yeah that could do it, it was supposed to be typescript serverless congnito with app lambdas and api gateway

type of thing

doesn’t amplify use s3 under the hood?

(if not, where does it store html files?)

yeah that could do it, it was supposed to be typescript serverless congnito with app lambdas and api gateway Haha this doesn’t sound static.

@randomy Under the hood I believe it is using S3. But it isn’t using S3 directly

that’s cheating

you could use cloudfront + lambda@edge to serve static files. you’ll probably cry more though.

not allowed to use an s3 bucket this is crazy

Yeah — I feel like trying to host static sites without S3 in some way is going to be painful. Maybe the right question to ask is: Where is the requirement against / opposition to S3 coming from?

You could use Dyanmo DB to store the contents of each file in a separate key and then create a docker container and host it on ECS. Then for each page request you pull the corresponding dynamo DB HTML content and then serve that.

You could do that

The above is obviously just a poor joke. Not trying to poke fun too much.

You could chuck the site in a container and run it on Fargate

Sheehs, If I had too… probably the API Gateway with Lambda@Edge with static files..not in S3? Its all gonna bit more than a bit hacky though

amplify might work, customer suggest using azure

https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-static-website

also fargate

thanks @roth.andy i was hoping not to go multi cloud on one small website

what would i know…

github pages?

would be easy enough, aye

Where is the requirement against / opposition to S3 coming from? well, if this webpage were the landing page for a facebook app, for example, s3 hosting doesnt work because even for a nice flat static page it has to answer a POST request on /…..

(no idea if that still applies, I haven’t tried to burn that bridge since the last facebook game I worked on deployed in ‘16..)

god no, this the enterprise. Security says no to s3 buckets, yes to Azure Blobs.

That doesn’t make no sense.

Fire your security team

(Just kidding of course, but that seems very strange and I can’t imagine why a company would be OK with AWS but not S3)

@randomy quite. some eejit made some blanket policy about s3 that buckets cant be public and now that policy is baked into every account.

Ah that connects the dots. Still not a smart policy, but makes sense.

Amplify would get around that for you. The bucket and everything underneath Amplify are hidden from everyone, so I don’t believe you would run into that account restriction AFAIK.

I’d just put cloudfront in front of a private s3 bucket

Netlify!

I will say I’m really impressed with netlify just getting up and running. They can really take you from zero all the way up to running and minutes for a full static website solution. If you’re using a static website generator like I am with Hugo for example for blogging you can even add forestry into the mix and have a nice web editor that is all CICD. To do most of what netlify would do would be just an exorbitant amount of effort in my opinion. You can add serverless, CICD, and more for free. Only need to pay at more commercial grade tier.

And you can redirect your GitHub pages to netlify with a simple file in the repo too. Definitely take a look at it. They even have deploy templates with the full stack ready if you take a look.

@Kevin Doveton you cant match a private bucket with cloudfront

you can use an origin access identity https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/private-content-restricting-access-to-s3.html

ill have to stomach a static sie in azure, its not ideal but its “ok” with security

i cant actually make the bucket to do that

i know in effect it will make it private enough, besides this is supposed to be a website so the content is actually public

thanks for all your input, hoping for someone to see sense today

Still worth checking out netlify if possible. It makes static site stuff a breeze. It’s kinda like jumping into a first class product for that. Only reason I wouldn’t use if I had to do some internal to the box static site or something.

terraform-ecs-alb-service-task is a good module. There are also good modules from Airship - https://airship.tf/getting_started/airship.html Regarding deletion of task definitions - it doesn’t delete them. You can’t change the task definition itself, if you want to make changes - you create a new revision of this task definitions with the changes applied and then update your ECS service to this new revision of task definition. So terraform’s aws_ecs_task_definition resource does exactly that - creates a new revision and deactivates the previous if you make changes. In the plan you’ll see that it will destroy (deactivate) the existing task def resource (revision) and create a new one (revision).

There is also a flag ignore_changes_task_definition in terraform-aws-ecs-alb-service-task module that allows to ignore external changes to task def applied to the service in case you modify task definition externally (e.g. changing image tag in CI/CD pipelines) and don’t want terraform to revert changes to task def on each apply

Maybe delete was the wrong word. But from my understanding inactive tasks are not allowed to referenced by new services, so it’s almost as useless in my mind. Keeping a history of active task definition allowed me to “roll back” bad deployments quickly, but updating a container version and updating the task again works as well. I guess I’m just a bit confused on how to use terraform for ecs.

How to do you deploy new images to ecs? From CI/CD pipelines or manually? Do you use any CLI tool for deployment?

In the past I queried the CFT, updates the version number using sed, and applied the new change set.

You could use https://github.com/fabfuel/ecs-deploy for example to deploy from CI/CD pipelines. There is an option to prevent task def de-registration when you do a deployment if that works for you.

Coo will check it out. Was hoping there was a pure tf solution, but poking around google doesn’t provide the solution I’m used to in cft world.

So you just use ecs to provision the infra and just do changes to app versions using tools?

Yes. This is just one way of doing deployments to ECS.

Yes, pretty much. At least that’s what we mostly see. Agree that a pure terraform approach would be nice, but the experience is more pleasant with some of the other tools for deployment.

Thanks I’ll look in to the tools

terraform is not really a deployment tool

for code/containers

in my opinion

@Igor I think my suggestion to you would be to steer clear of managing env vars / secrets in the Task Definition itself. It’s painful and causes issues like this.

I suggest storing env vars + secrets in SSM Parameter Store and then using Chamber in your Dockerfile to pull those in at runtime. That combo is really nice.

https://aws.amazon.com/blogs/mt/the-right-way-to-store-secrets-using-parameter-store/

I like the idea of using chamber to manage SSM Param Store, but I don’t like the idea of embedding chamber into the image.

It’s actually ultra light weight since it’s just a go binary and with Docker multi-stage builds it makes it super easy to pull in without many changes.

Also had huge benefit that it works with the ECS task role to pull the data from SSM / decrypt with KMS.

ECS does the same with secrets, but adding/removing/renaming secrets does require a new task to be created

playing with that now. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-blue-green.html It beats completely over writing my task definition and not having a trail in the family, even tho inactive tasks count against you on the qoutas - https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-quotas.html

Most of my deploys with ecs have been update the task def and then tell the service to use the latest. Monitor the performance (which took time), and then roll back if needed. having a blue green on code deploy now seems amazing.

I remember seeing a detailed article of the issues someone ran into with codedeploy blue/green

That scared me away from it at the time

I think it was this one: https://medium.com/@jaclynejimenez/what-i-learned-using-aws-ecs-fargate-blue-green-deployment-with-codedeploy-65dde6781fcc

I used in in past for ec2 instances and worked pretty well. Only thing that I didn’t like was missing option to specify script arguments in appspec.yaml. So I ended in making wrappers of deploy scripts which were fully functional by itself already.

and last agent release is from Dec 6, 2018

I use ansible to create their accounts and grab their public keys from github. Then they can login using the same keys they use for pushing code. Just add .keys to a user’s github profile URL and you get their public keys.

Definitely not aws-centric, but it’s easy.

ah cool. we dont use ansible, we use puppet and trying to migrate off. we were considering doing something similar with ssm or use ec2 instance connect.

If I was starting now I’d probably look at ssm / ec2 instance connect first, but we’re pretty happy with ansible. Also, at this point we don’t have much that’s not in k8s so our ansible usage is pretty limited these days.

I should also mention, gitlab also gives you user public keys if you add .keys to their profile page.

Migrated away from bastions / ssh for client projects and only use ssm-agent / Session Manager. Got tired of managing client SSH keys. Session manager + gossm is a really nice combo.

https://github.com/masterpointio/terraform-aws-ssm-agent

https://github.com/gjbae1212/gossm

What @Matt Gowie showed is a really slick implementation. I like it!

using that setup @Matt Gowie what does the ssm agent accomplish ?

seems interesting but unsure how it all works

It’s a single instance in your ~account~PC that is dedicated to being your session manager instance. I have a couple Fargate only projects so that small instances acts as the “bastion” into the private network since I don’t have others.

That SSM Agent instance is also setup for session logging to S3 / CloudWatch, so sessions are auditable.

I wrote a short post on that setup: https://medium.com/masterpoint/time-to-ditch-your-bastion-host-f22cc782de71

https://userify.com/

Speaking of gossm - we were moving everyone over to that, with anticipation of disabling ssh access once done. But now a bunch of people have complained that their terminals are all sorts of f’d up when connected to the remote hosts. And I’ve noticed it too - you get weird behavior if you try to cycle through your shell history, or even using page up/down in vi or less when looking at a file. Has anyone else seen that, and is there a solution? We can’ tell if the issue is SSM or goSSM

(we just disabled logging in dev and the behavior went away so maybe this is AWS’ fault)

Huh — I haven’t seen that. I’ll keep an eye out… that doesn’t sound great.

It may be our config, we’re opening a support ticket to see if we missed something

Post back here if you find out more… interested if I’m going to have clients run into that issue.

I started using this https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-connect-set-up.html

i was wondering about that. what are your thoughts on it ?

Its pretty good so far. I don’t have to deal with passing aroudn ssh

@Matt Gowie my colleague found that the issue goes away if you resize the terminal window, so he wrote a little script that we’re adding to the default bash profile on the AMIs we build

resize() {
  old=$(stty -g)
  stty -echo
  printf '\033[18t'
  IFS=';' read -d t _ rows cols _
  stty "$old"
  stty cols "$cols" rows "$rows"
}
resize

@Zach — Ah interesting… I may add that as a note to that small TF module I’ve got to give others a heads up. Thanks for providing your resolution.

Further followup - AWS Support was able to replicate the issue and it seems to be a bug between the SSM Session logging the SSM KMS Encryption option. When both are turned on, terminal sessions get mangled when using anything that can scroll the shell. They’re looking into it to see if there’s a bug on their end. For now the resize function above is an ok workaround

Hey @Victor Grenu — This is cool stuff! I built something similar for test / experiment accounts that nukes the account using aws-nuke: https://github.com/masterpointio/terraform-aws-nuke-bomber

I’m sure the combo of the two could really ensure the user has a cheap AWS account

Great, sounds interesting, will take a look.

Your VPC has a single CIDR range. You carve out blocks within that for the subnet ranges

oh i thought you could add multiple cidr ranges to a vpc

@RB Since you know Terraform — I’d check out the CP VPC + Dyanmic Subnet modules as a good example.

https://github.com/cloudposse/terraform-aws-vpc https://github.com/cloudposse/terraform-aws-dynamic-subnets

Uhh yah I think that’s a ‘thing’ but its for a particular use-case

thanks gowiem, ill check those out

zach, i was looking at this yesterday https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Scenario2.html but was seeing errors when adding the cidrs

ill try the terraform approach. it might be easier

seems like the terraform example also only allows for a single cidr block per vpc

Right, that page is illustrating what I described though. Their example is a VPC with CIDR 10.0.0./16; the public subnet is 10.0.0.0/24 and the private subnet is 10.0.1.0/24. The subnets are formed within the VPC cidr

so it might be easier to create a new vpc for private subnets ?

I think you are misunderstanding how the subnets work

Note that in the diagrams from the AWS docs that the subnets are drawn within the vpc

What you’re talking about with multiple CIDR blocks on the VPC itself is more related to this - https://aws.amazon.com/about-aws/whats-new/2017/08/amazon-virtual-private-cloud-vpc-now-allows-customers-to-expand-their-existing-vpcs/

You should be using a private CIDR in the VPC, the only distinction between a public subnet and a private subnet is the gateway. You can use multiple CIDRs in the same vpc but it’s only for certain use-cases.

stay away from adding multiple CIDR to a VPC it is a nightmare to maintain and debug and it is in some ways not the “proper” way to do networking

ah ok, thanks everyone. it sounds like i can continue to use the same cidr but create a new subnet with a cidrmask and to make it private, just change (or remove) the gateway

use “proper” as hacky, short-cutting etc

the cloudposse module for dynamic subnets uses the cidr function to do subnetting

it does it automatically for you ( +- some inputs)

we do multiple cidrs on a vpc now. haven’t had any problems yet… was surprisingly easy. we create the vpc first with a cidr sized to just private subnets, routed to a central natgw via the transit gateway. then if the account needs public ingress we attach another cidr to the vpc, add an igw, and carve new subnets out of the new cidr, and create a route table for the new subnets that sets the default gw to the igw

I had the same issue. AFAIK, it is a limitation of that feature. Basically, you can only define a target bucket that is in the same region and account.

If you want to centralize the logs, you have to implement a custom solution.

We haven’t attempted this configuration before. I was talking to some one at a bank through that ran something similar. It was the first I heard of the pattern. @Andriy Knysh (Cloud Posse) do you think the EKS modules we have would work this way? I don’t see why not - but the e2e automation might be difficult. Might require multiple terraform runs to avoid races.

this is not possible with managed Node Groups, not in the EKS console, nor in a terraform module (e.g. https://github.com/cloudposse/terraform-aws-eks-node-group) - EKS places the node groups into the same VPC

might be possible with unmanaged workers (e.g. https://github.com/cloudposse/terraform-aws-eks-node-group) if you allow communication b/w those two VPCs. Both control plane and the workers need to communicate with each other, so you have to make sure the two VPC can communicate and the security groups of the control plane and the worker nodes have the corresponding rules (e.g. allow from a CIDR block of the other VPC)

They are also using unmanaged workers..I dont know how to achieve this using Terraform. I was relying on using cloudposse terraform modules for EKS deployment. But this scenario I am not able see any help on online or could see here in cloudposse..If you could guide me or share some lead on how to achieve it would be really great..

I don’t know all the details involved, but I believe you have to at least do these two things (more things could be involved, not sure):

1. Allow communication b/w the VPCs using any of the methods https://docs.aws.amazon.com/whitepapers/latest/aws-vpc-connectivity-options/amazon-vpc-to-amazon-vpc-connectivity-options.html . VPC peering is prob the best for the task (e.g. https://github.com/cloudposse/terraform-aws-vpc-peering)

1. Cluster and workers security groups: You have to use CIDR blocks from the other VPCs to allow communication b/w the control plane and the workers

https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/main.tf#L127

https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/main.tf#L87

Thanks @Andriy Knysh (Cloud Posse) for the links..let me go over it and try to setup something with 2 VPCs.

the “why” is a good question regardless of whether you put it into one or two VPCs, you have to allow communication b/w them anyway. I don’t see any benefits of doing two VPCs besides a more complicated setup

it’s possible. as we already discussed you have to do this:

1. Create two VPC and add VPC peering between them

1. Add the CIDR block of VPC1 to the ingress rule of the SG of VPC2

1. Add the CIDR block of VPC2 to the ingress rule of the SG of VPC1

yep looks similar request..but here was talking about worker nodes in different VPCs. But implementation looks same from your inputs..Also just for the update..Managed to bring up with Cluster and Node1 in VPC1 and Node2 in VPC2 with VPC Peering enabled. It works fine..Only things still not working is, not able to go to bash prompt using kubectl(from local machine) of a POD running in Node2 VPC2. Pod to Pod ping across nodes are working fine..only bash prompt is failing to connect. Will work through that.

did you open the entire CIDR blocks as ingresses?

and all ports?

could be that some ports are not allowed

Yep opened all port all traffic with VPC CIDR block on both Node2 n Cluster secgrps. Still no luck.

aws-vault? https://github.com/99designs/aws-vault

aws-vault

thanks I will check it out.

it’s great