#aws (2020-11)

Deny iam:CreateAccessKey

Though, is that really the right way to go? If you do that, you have to manage them, including rotating them.

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html#access-keys_required-permissions

You can allow them to create keys for themselves

right

If you create my access keys for me and give them to me, that means you know them. As the user I don’t like that. I want to be the only person on the planet that knows those keys

Do you allow users to set their own passwords (within the confines of a secure password policy)?

@kalyan M what are you trying to achieve? Or what is the risk you’re trying to mitigate?

previously one of the developer machine keys got malwared and there are c9 large clusters spinned in the Ireland region, other regions, which we are not aware. the ec2 spin up is done from china region. we got a a billing of $10K on that month. Looking for a way to remove their keys after work is done.

Encouraging use of secure development practices like pre-commit hooks that check for AWS keys before a commit, or use of tools like aws-vault which always creates a temporary key using AWS STS (and better training on protecting keys and other security practices) would be a better strategy. What you mentioned is a serious problem and one that really needs to be solved, but it should be solved in a way that increases trust in the dev culture, not locking things away

https://pre-commit.com/

https://github.com/99designs/aws-vault

https://blogs.sap.com/2018/06/19/trust-is-the-foundation-of-devops/

Okay here’s an actual strategy you can use instead of what you are proposing: You can configure IAM policies such that MFA must be used, even when running console commands with an access key. aws-vault supports it beautifully.

https://aws.amazon.com/premiumsupport/knowledge-center/mfa-iam-user-aws-cli/

You can also lock out regions that you don’t use via IAM

Or use something like CloudCustodian to monitor AWS activity

Agree with Andrew 100% here. Making things harder for devs won’t solve your problem here. Also, dev’s are very resourceful and find ways around the blocks you set up. You need to make sure you build something that’s easy for them to adopt.

We use aws-vault and AWS SSO (with G Suite IDP). The keys are never saved to the HD. Malware can try and make aws-vault calls, so it’s not fool-proof, but you’re adding more hurdles for the malware to try to overcome.

Be aware that if you increase the number of sg-per-eni then the number of rules-per-sg will decrease.

https://docs.aws.amazon.com/vpc/latest/userguide/amazon-vpc-limits.html#vpc-limits-security-groups

Thanks! I use only a few rules-per-sg. The thing is, I’ve created separate sg for every ECS cluster, and I have a lot of ECS clusters…

I’ve created everything trough Terraform. So other option is to redesign everything

You can redesign things by creating a common security group, and having each ECS cluster add rules to the common group.

If you have this many clusters you might want to consider some other form of security. You could have a single rule based on CIDR and use dedicated subnets

Thanks @Alex Jurkiewicz - at the end there was no problem at all to begin with since I’ve misunderstood how VPC endpoints work :)

Even better

i don’t necessarily want to set it at the higher threshold all the time as prolonged iops will lower our burst balance but the 2am spike is when we are at our lowest traffic and the sync jobs don’t last too long

And what about increasing the duration of elevated iops?

Also, this is something you should be able to control with your escalation platform

Yeah, if you feed these alerts into pager duty, you can have scheduled maintenance to silence alarms for a certain time period. But that brush might be a little broad.

There’s no way to do what you want natively in AWS, except the hammer of a lambda scheduled action.

I wonder how useful this alert is though. Why do you want to be notified about high write iops? Can you instead be notified about degradations in application performance directly? Or if you must alert on the database, maybe there are other metrics which are more useful. Like number of connections or CPU.

Hmm, I don’t necessarily want to increase the duration as I think it could lead to slower reaction time to a system that might be impaired soon. Also I’m not sure if we can get that granular in our escalation platform (victorops) to be in maintenance mode for just a specific alert type? I’m reading that people have set up lambdas to disable/enable alarms. So I could have an alarm with a higher threshold enabled between 2am PST and 3am PST while disabling the general alarm during that period.

@Alex Jurkiewicz we alert on many things including cpu, connections, and iops. for us, we are very read heavy and increased read iops past our specific threshold for a longer period of time can drop our burst balance (which once depleted will cause read latency and service degradation) I like to alert on read iops because it generally gives us the most time to verify if we’re likely to be in trouble soon. Alerts on low burst balance, queue depth, read latency generally would come after as they are lagging symptoms of the problem. I feel like alerting on degradation in application performance to me is reactionary and you definitely want proactive alerting like prolonged read iops spikes as it gives you time to debug and hopefully prevent an issue that hasn’t happened yet.

Thanks for the explanation. That makes a lot of sense, alerts are always application/system-specific. I agree with your point that if something is going wrong, it’s better to know before than after. But I think this situation is a great example of how difficult it is to build an alert that predicts problems in a reliable manner.

I’m reminded of a tweet from one of the original YouTube SREs, who said something like “in the early days, YouTube monitoring alerted on two metrics only: stream starts and number of uploads”. If you are a large company with a big complement of on-call staff, you can get tricky and predictive and build automated remediation based on this alert! But if not, you will end up with overly sensitive alarms and a culture of alert fatigue.

agree on alert fatigue!

I still see this as not an alert problem but an escalation problem. That alert escalation should be associated with a schedule. Very easy to do in something like opsgenie, but not sure about victorops

opsgenie

got it @Erik Osterman (Cloud Posse)

use the postgresql provisioner

and I’m sure there’s a mysql one

You mean using postgresql provider in the terraform specification?

yah

Interesting didn’t think of that thanks!

https://www.terraform.io/docs/providers/postgresql/index.html

That one right?

That’s it. I do this exact thing. The only rough part about this is that you typically want a separate root module to do this because :

1. You typically need a SSH tunnel into your VPC so you can access your private RDS instances for the postgresql provider to do its work.
2. It’s not done often, so keeping it isolated and not having to share that tunnel across multiple root modules is the right way to go.

I’ve also found that the postgres provider doesn’t seem to integrate very well with the terraform dependency graph. It tries to ‘jump the gun’ and execute before things are ready

You mean like users AWS configs, configs for the “AWS Extend Switch Roles” plugin etc…? I created a simple Jenkins job at a previous employer so that users could self-serve getting configs for their user (email). Currently, I have some janky Ruby / rake tasks that do some of those things. I haven’t found anything out in the wild that does it.

Yeah, that’s what I was referring to.

What is AWS Extend Switch Roles? I should check this out…

Oh a browser extension. Interesting.

It’s great if you work across many accounts without SSO or some sort of app portal like you get with Azure AD.

The FAQ https://aws.amazon.com/fsx/windows/faqs/ says:
Q: Can I change my file system’s storage capacity and throughput capacity?
A: Yes, you can increase the storage capacity, and increase or decrease the throughput capacity of your file system – while continuing to use it – at any time by clicking “Update storage” or “Update throughput” in the Amazon FSx Console, or by calling “update-file-system” in the AWS CLI/API and specifying the desired level.

Thx @Alex Jurkiewicz I also saw that as well. I’m guessing the only option is increasing but you cannot decrease as aws doc doesn’t mention this.

I’m sure they would mention if it’s possible. AWS prefer to say nothing than to say “you can’t do X”

Can’t decrease the storage space of a FSX. Only thing that can be decreased after creation is the throughput.

Thx

There are solutions for this but they tend to be very intrusive and enterprise-y. Not really an #aws issue. Some examples:

• Citrix makes a remote-access virtualization product. You open up your app through it, and then users log in via a special Citrix agent that prohibits copy/paste.

• On iOS and Android there are enterprise management tools (i.e., rootkits but from your employer) that can create special sandboxes that prohibit copy/paste, screenshots, etc.

• A simpler one is Windows Remote Desktop. IIRC you can set it so that copy+paste works within the RDP session but is disallowed between remote and local.

We migrated to aurora (MySQL) and very happy.

The replication is disk block based, so during failovers you lose almost no data, and they are very fast.

The biggest caveat is that Aurora is slower for writes than a traditional MySQL master. Because it waits for writes to be acked across multiple AZs. Not that much slower, but AWS don’t really mention this anywhere as it’s the only meaningful regression

Can’t update from 5.6 to 5.7 in place. Cloudposse module wants to recreate instances on minor engine version change. You can avoid by forking module and add lifecycle version ignores…

Otherwise been great for us too. Fail over quick. Odd MySQL error resulted in crash and fail over but fixed on later versions.

very heavy concurrent writes will force a failover of the cluster instances due to the things mention about the storage so as long as you can slow down your writes then you will be ok

I’m talking about millions of rows here

we have tables with more than 1B rows

I haven’t seen failover caused by high write load. We’ve hit >30k WIOPS sustained without issues

we can literally crash it at will on an aurora 5.6 but again that is usually not a limit you will get to and there is still ways to avoid it

we use aurora for many other things and no issues

One of the things I like the most is how fast the clones are since is basically “sharing” the same storage it is pretty fast

@jose.amengual We have like ~3TB of data we’re going to migrate and I do think one or more of our tables have > 1B rows

We’re also going to migrate to Aurora Postgres rather than MySQL. @jose.amengual curious if you could clue me into how many writes would cause you to fall over?

mmmm I do not know exactly how many but I can tell you how I did it

yeah, that’d be good to know, if possible

I work with @rms1000watt and we just did a sync of data using pglogical from rds pg -> aurora pg (11.8) and it went up to 100k WIOPS and stayed there without any issue. This is not live traffic just syncing data over. Really pleased with the performance as of now.

but you need to keep in mind that the size of the instances matter

so it is a combination of the two, I was using db.r5.4xl

yes we are using r5.24xlarge

lol

I do want to scale it down but not during this transition

and keep in mind too, no tuning nothing, straight aurora

we use mostly the reader for the app so we can basically write all the time

you can do multimaster too that can double the WIOPS

and does some sort of sharding capabilities

I am really interested in multimaster!! Do you have any doc or article around this?

I see this https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-multi-master.html

I will check it out.

This seems to not support PG

Currently, multi-master clusters require Aurora MySQL version 1, which is compatible with MySQL 5.6. When specifying the DB engine version in the AWS Management Console, AWS CLI, or RDS API, choose 5.6.10a.

oohhhh yes PG is always behind

You can tune things. For example we disable per commit sync for MySQL Aurora through parameter group tuning. I’m sure there are similar knobs for postgres

yes they are, but you can’t disable fsync in postgres aurora, you need to play with workmem, temp tables and such

I am trying to get IaaC right in oder to deploy the same EKS cluster with service roles and helmfiles for alb, ingress, cert-manager, external-dns, etc. And for several environments (dev, prod, stg, …) And keeping the differences between them low. At the end Gitlab CI will deploy each branch to an own namespace “deployment” to be accessed using an unique URL. VPN access too. And you?

that sounds like a beautiful endeavor! I tried to do something similar. EKS has been lowered in priority for us so we won’t get to conquer it until Q1 next year.

I’m currently trying to solve giving devs iam permissions using a permission boundary. I’m hoping I can get this out the door this week and unblock a lot of devs. Meanwhile, I’ve been exploring arm a lot in our ECS clusters so it always seems to be a balancing act.

@RB here’s how we manage the requirement for the permissions boundary…

Thanks Loren. I’m using basically this same policy from the aws blog on perm boundary

yeah, that one flips the logic, using Deny with StringNotEquals instead of Allow with StringEquals

may i ask what you use for the permission boundary policy ? we’re still trying to figure out what’s the best maximum permissions to give without giving too much. we were thinking of removing all deletion, for instance.

we actually use the same policy we assign to the role. it’s primarily to prevent privilege-escalation, for our use case

that doesn’t work of course if you apply multiple policies to the role

but say you have a DEV role, with only the DEV policy attached, you can require the DEV policy as the permissions boundary for any role/user they create to prevent privilege escalation

pain points have been places where the console will auto-create roles when the user first attempts to use the service. such as service-linked roles, and the rds-monitoring role, etc… so we pre-create those in the account as we identify them

ah so whatever access the DEV role has, it can also create roles with max permissions as the DEV role since the single DEV policy is used as the boundary

our issue is that we have multiple iam policies associated to our DEV role

so i was going to create a separate maximum iam policy to use as a perm boundary policy

yeah, so that doesn’t work so well then, and you have to come up with an actual policy for the perm boundary separately

pain points have been places where the console will auto-create roles when the user first attempts to use the service. such as service-linked roles, and the rds-monitoring role, etc… so we pre-create those in the account as we identify them yes, i can see this being an issue. we have most of these created in our accounts so hopefully this wont be an issue. we’re worried about existing iam roles exceeding the perm boundary so retroactively attaching may be an issue

yeah, so that doesn’t work so well then, and you have to come up with an actual policy for the perm boundary separately exactly. we’ll have to be crafty here. not much online about this except for the aws blog side.

yes, we also rolled through all accounts and slapped the perm boundary on pre-existing roles/users when we shipped the perm boundary requirement

awesome! how did you verify that each role did not exceed the permission boundary tho ?

it’s a fairly permissive policy for this environment, so we didn’t expect issues and encountered none

ah I see. We have a more strict environment. Perhaps that’s something we can work on.. making things looser.

yeah, the stricter the perm boundary policy, the more likely you’ll have problems

or, you may need multiple perm boundary policies for different use cases

we stick with basic stuff around security and auditing that we manage in the account. if we create it, you can’t touch it. that kind of thing

Thanks everyone this looks like will help us out with IAM permissions for devs in their sandpit account. Currently IAM:* is basically off but we need to enable so they can do lambda etc. Will do some reading.

We’re currently investigating rabbitmq service. Want to kick our k8s one out of cluster and get from 115 to 118 etc.

i have not. i dont think i would touch it because it is massive. i’d prefer if it was broken up into modules.

we dont use code commit either.

Is there a reason that this can’t be handled directly on the ALB itself? I’m a beginner too.

ALB has it’s own security group.

How did you teach yourself about SGs?

The hard way. You should go and buy Stephane Mareek course on Udemy!

not enough I need to just read some simple code and keep moving

and by testing, do you mean ensuring your safeguards for CIS compliance are working, or putting the safeguards in place?

Because if you mean the latter, that’s what you get with SecurityHub standards.

We’ll have support for security hub in the next day or so. @matt is working out the last details fixing our tests for https://github.com/cloudposse/terraform-aws-security-hub/pull/3

I really don’t know what that code is doing. I learned what a resource and datasource was last week. Time to give it the old college try.

are you using terraform at your company?

Yes

I’m trying to develop my skills as a devops engineer.

I keep on getting

Error updating CloudTrail: InsufficientEncryptionPolicyException: Insufficient permissions to access S3 bucket nf-cis-benchmark or KMS key arn:aws:kms:us-east-1:721086286010:key/af37e59b-e5fa-446c-b43c-180b425c6222.

when I try to apply my terraform with a kms_key

resource aws_kms_key "cisbenchmark" {
  description             = "cis-benchmark"
  enable_key_rotation     = true
}

I’m trying to setup my cloudtrail with a key like so

resource "aws_cloudtrail" "cisbenchmark" {
  name                          = "cis-benchmark"
  s3_bucket_name                = aws_s3_bucket.cisbenchmark.id
  enable_logging                = var.enable_logging
  enable_log_file_validation    = var.enable_log_file_validation
  is_multi_region_trail         = var.is_multi_region_trail
  include_global_service_events = var.include_global_service_events
  is_organization_trail         = var.is_organization_trail
  kms_key_id                    = aws_kms_key.cisbenchmark.arn

  # CIS Benchmark 3.1 Ensure CloudTrail is enabled in all regions
  # for a multi-regions trail, ensuring that management events configured for all type of
  # Read/Writes ensures recording of management operations that are performed on
  # all resources in an AWS account
  event_selector {
    # Specify if you want your event selector to include management events for your trail.
    include_management_events = true
    # Specify if you want your trail to log read-only events, write-only events, or all. By default, 
    # the value is All. Needed for logging management events.
  }
}

I stole some of this from your guys code.

You can do that. It would be represented as two origins, four url behaviours, and one default url behaviour

There’s a rather low limit on url behaviours by default though (I think 20) so you can’t scale it up too far if that’s a concern

It can scale using lambda@edge : https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-examples.html#lambda-examples-content-based-S3-origin-request-trigger

@Alex Jurkiewicz thanks! testing it out today.

Instead of using a bucket as origin, you can also use a bucket as “Custom origin”, this way it should be possible to change the default path.

and yes I realize that “that’s how all feedback systems work” but I feel like this is something that we collectively can ask for better options

I’m totally open to feedback too

good old “5 stars is normal service” rating as popularised by the service economy

“help keep our employees precariously employed by giving us an excuse to fire them if they get close to a benefit date”

send this to Corey Quinn

forgive me I’m socially dysfunctional …who’s that?

I can Google, but I just feel weird sending somebody I don’t know a DM…again bad at the social thing. I don’t really take part in the social medias. Slack is the only thing I stick with because it feels like less of a blackhole of dialog..and people are generally more rational.

he runs the Last Week In AWS blog

and does AWS consulting

very funny guy

if others agree with my position and want to share it out I’m all for it. I want to try to do something to change the stats quo because (1) it hurts our fellow engineers (2) it doesn’t help us at all

buuuut, my amount of trying probably equates to about 1hr of effort

you could just email that to jeff bezos and ask him “wtf”

I could, or I could just drop a dumbbell on my foot

? I mean that Bezos pretty famously deals with customer feedback like this in a very serious way

oh sorry, I thought you were being facetious

heh, no - his email at amazon is public and people will on occasion send him product complaints about amazon or aws

I can give that a shot too but just figured that the only way real change was going to happen if Amazon saw that it’s something we collectively cared about as it’s customers…you k now how typical large enterprises operate

https://www.cnbc.com/2018/05/07/why-jeff-bezos-still-reads-the-emails-amazon-customers-send-him.html

thanks @Zach I sent it off – who knows if it will go anywhere but makes me feel like I’m doing my part to try to help our fellow engineers working the support desk at AWS

do you have non-VPC Lambdas? those are usually the culprit for me

wait, you mean just from the ec2? Sorry I thought you meant account wide

Can you change the A record for mydomain.com to be the same value as the one for ourcoolcloud.com? If you are using an alias to an AWS ALB/ELB then probably not, but if you are just pointing at an IP address then you should be able to just point both A records to the same IP address

I am thinking not since ourcoolcloud.com would typically point to a CloudFront distribution and would have other record sets say ALBs etc.

Would a CNAME record suffice ?

I’m not an expert at DNS. I know just enough to be dangerous. If I was in your shoes I’d be finding someone who is an expert, and if there aren’t any that I can get, hiring a consultant.

DNS can be quite complicated, and in most cases it is a single point of failure. If you screw something up your whole site will go down. Most of the downtime for the big players like CloudFlare, AWS, etc, have been because someone screwed up a DNS setting

thank you so much for taking the time to understand the problem. I’ll try and see if I can do a spike or something to test how this would work using a test DNS maybe. I’ll share the results here just for future reference. Thank you again

Can you use Terraform instead?

https://registry.terraform.io/modules/colinwilson/traefik-v2/docker/latest

nope :(

you have two hosted zones now: test.com delegated.test.com

test.com has an entry [delegate.test.com](http://delegate.test.com) IN NS …

This means that all records for the domain delegate.test.com (including the A record for the top level domain) will be handled by the NS servers configured in the test.com IN NS record and those records are handled by delegate.test.com hostedzone (in whatever aws account this hosted zone is deployed)

Thanks Maciek!

try #terraform

thanks

Hey @Shreyank Sharma Im not sure what do u mean by backing up a lambda function for DR - I would just keep the infra configuration and the lambda application + cicd code in a git repo ?

The backup is simply your code in your repo, the ‘lambda’ in AWS is just the runtime configuration

Yes.. the simplest way is check your AWS bill

You can use CostExplorer to get the list of AWS resources/services that you are being charged for

Thanks @Santiago Campuzano

i couldn’t find a resource that aggregated all of it. it was a bit disparate. i tried to put it all together.

if anyone knows of additional lifecycle hooks for other services or whatever, please let me know

im still trying ot figure out the difference of using only cloudwatch (ecs-drain-lambda), using sqs (claranet method), and using sns

Yeah, not a fun morning.

I know better and yet one of my larger clients is running in us-east-1

Do you mean us-east-1 is the aws staging environment?

tbh for me it’s the 2nd larger outage that I’m affected by in roughly 7 years of running stuff in us-east-1. first one being s3 outage (happily we lived through it without downtime) few years back.

and currently i see longer response times from firehose endpoint (800-1000ms instead of normal 50-80ms) and cognito endpoint is completely down

We’re having troubles with ECS auto scaling and scaling out. Desired tasks threshold is not triggering new task creation and some tasks are failing to be placed due to “limited capacity”. I think it’d all tied to CloudWatch problems.

yeah and CW is failing because of kinesis streams which IMO started the domino to fall. Happily for me I’ve quite good understanding of CPU needs and cosidering those few k rpm i’m receiving on a constant basis if no instance will be replaced i should be fine

@rei I always thought that

when AWS has issues, a lot have too

still down…..shit tomorrow 3000 people from amazon fired

so now that shit hit the fan people will start migration to us-east-2 and then it will go down

You need to first create the resources with a local backend and then transition over.

Usually you should do this through having a separate root module for your tfstate-backend which bootstraps your backend

@voidSurfr did you see these? I think the above is your immediate problem.

sorry, I was attending office hours. I’ll double-check your notes against my issue.

you do something like : terraform apply -var-file='production.tfvars' -target module.terraform_state_backend

with local state defined

once the s3 bucket is created , hen you can define the backend-config on a file or main.tf and then

terraform init -var-file='production.tfvars' -backend-config=production-backend-us-east-2.tfvars

and it will ask you to move the local state over

this is all on the README

Cloudwatch events has cron support for scheduled triggers, codebuild works as a target and can run arbitrary commands

Or cloudwatch events to lambda works fine

I think ssm automation is also an option

yeah thx! I have found as well this https://github.com/blinkist/terraform-aws-airship-ecs-service

We even have an #airship channel :-)

cool!

Sounds like you want something managed? But how about Serverless Framework? It satisfies your Lambda requirement easily. Defining events is really easy in the serverless.yml file.

Not sure how well it’ll fit with ECS but found this: https://www.serverless.com/plugins/serverless-fargate-tasks

Part of the challenge here is coming up with sufficiently restrictive policies to allow developers to deploy without being able to change things they shouldn’t.

Also managing everything for cost and tracking purposes will require some thought. Serverless.com paid offering might help with that but I’ve not used it yet.

also, since you mentioned something easy one thing we use a lot of is Zapier and Integromat. Both of them have cron like functionality.

Oh yeah, ofc! @Erik Osterman (Cloud Posse) just curious - would you choose one over the other? I was looking at doing something with Xero the other day, and for some reason the Integromat connector was more full-featured.

Yea, we started with Zapier so we have ~300 zaps there. However, if we were starting from scratch would 100% use Integromat. Easy things are a little bit harder to do in integromat (compared to Zapier), but it more than makes up for it with all the advanced features. I feel like Integromat is more geared towards advanced users / programmers. It has iterators, routers, variables, etc. If their integration doesn’t provide the API endpoint you want, they usually provide a raw method to use the integration. Integromat supports handling errors, while Zapier doesn’t.

Ah thank you for the insight! Very useful.

Integromat’s API looks pretty good too

very valuable info, thx all!

Good week-ends reeding In short, use just EC2 instances and don’t use new and fancy services because everything is interconnected internally.

I deeply disagree with that. The best example against it is the experience iRobot had: https://twitter.com/ben11kehoe/status/1332391783829913600

Yes, they had a bad incident, but it was: 1) rare, and by managing EC2 themselves they would’ve definitely had more incidents + higher cost + huge opportunity cost 2) relaxed, as in they had to wait for AWS to recover. No stress, no fighting to figure out the root cause, no marketing hit of “Roombas are down” but “A bunch of IoT things are down”

… it was trolling from my side. I understand that such incidents happen all the time. Smaller or larger, and it is our job to prepare and minimize impact as much as we can.

welp, I did not catch that. Sorry!

Some people posted on linkedin during an incident that their EC2+RDS services were 100% up during that period, so they are lucky that they don’t use fancy services… and then it started a “discussion” ec2 VS serverless :)))

The discussion we’re having is: “should we replace AZ-level redundancy (within us-east-1), with the more resource intensive region-level redundancy”.

This explains the load balancer sg rules but not the ec2 instances

https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-update-security-groups.html

The security group on the EC2 instances should allow ingress and egress to the load balancer security group, and vice versa.

I would suggest adding allow all out egress rule to both groups as well though.

There are only 2 references to lifecycle hooks here https://docs.aws.amazon.com/autoscaling/ec2/userguide/asg-instance-refresh.html

It does yes

Well, I can say for sure it certainly invokes the create hook, we don’t make use of the termination one

You’re saying only the insurance refresh will invoke the lifecycle hook?

No I’m just confirming that instance-refresh definitely does. I assume it triggers in both cases that you’re asking though

i can confirm it triggers the termination hooks

we make use of it regularly

from my tests, it looks like manual termination of the ec2 instance does not invoke the lifecycle hook

ah and it looks like an instance refresh does invoke the lifecycle hook. thank you.

interesting. I guess they take the stance that if you manually asked for a termination that you want it done right now

weird. i spoke with support and they said that manual terminating should invoke the hook

perhaps the https://github.com/getsocial-rnd/ecs-drain-lambda just doesn’t work for the manual termination

ah ok it was a miscommunication. Confirmed with support that manual termination from the UI does not run the lifecycle hook but it does if the aws autoscaling terminate-instance-in-auto-scaling-group cli command is run

oh and what about asg max instance lifetime settings ? does it invoke the lifecycle hook too ?

@Steve Wade (swade1987) you need to enable DNS resolution at the VPC peering config

Let me show you

thanks man your help is much appreciated

We enabled it for some of our VPC peerings.

so i need to enable in the VPC connection in account X, right?

Yep//// you’re right !

let me try that

it doesn’t seem to be working with nslookup

Wait… I think you should do it the other way….

ok let me try that

I read your config again

trying …

Cool… let me know how that goes

will do

nope

could it be security group related?

Wait…weird…. are you having issues with DNS resolution or with TCP connectivity ?

Is that K8S by any chance ?

100% its EKS

Are you doing that from a POD ?

yes i am

I’m wondering if that could be an issue… would you be able to do that from a regular EC2 instance ?

we are using bottlerocket so going onto the node isn’t the easiest for me unfortunately

Hmmm gotcha

i might just try dns resolution from both sides

oh shoot is it because i am peering with the database subnets from the upstream VPC module which deny inbound, don’t they?

What do you mean. When you configure VPC peering, you peer the entire VPC

And then you configure SGs and Routing Tables to allow certain traffic

Have you configured both ?

let me create a gist it might be easier

https://gist.github.com/swade1987/61e43b74abd3d3147e7143fedf2173c2

i tried traceroute as well to no avail

i think my routing logic is correct

@Santiago Campuzano i have an update, i can nslookup from account X to an instance in the database subnet in account Y without a problem, the issue is just DNS resolution is not working

@Steve Wade (swade1987) What do you mean by you can nslookup but DNS resolution is not working. ?

That is like contradictory

I mean… nslookup is pretty much DNS resolution

i can nslookup to an AWS known DNS record (see above)

Ok… and you’re able to get the private IP address from that RDS instance

i can’t resolve custom private DNS addresses

e.g. [yellowfin.rds.de.qa](http://yellowfin.rds.de.qa).underwriteme.internal which is a CNAME to an RDS endpoint

Hmmm got it… you’re resolving the public IP, not the private one

currently i have dns resolution setup on both sides

maybe i need to turn off account X like you said before?

I would give it a try….

made no difference

i wonder if this has something to do with coredns configuration?

or maybe i need to create a route53 resolver?

I was checking my config, and I don’t have the R53 resolver configured

but are you peering the whole VPCs together?

or subnets in account X to others in account Y ?

The whole VPCs together

yeh i think that might be the difference

Well…. wait.. is it possible to peer certain subnets ?

AFAIR you peer the entire VPC

not certain subnets/CIDRs

well the route tables are subnets

Ok… The route tables indicate what traffic to send through the VPC peering…

yeh mine looks right

the issue is just the private dns resolution is not working

I would discard. the EKS component of this equation

I’d create a simple Ec2 instance and try from there

The DNS resolution

EKS/K8S DNS resolution is weird sometimes

https://aws.amazon.com/blogs/compute/enabling-dns-resolution-for-amazon-eks-cluster-endpoints/

Morning @Steve Wade (swade1987) !

How this ended up for you ?

The DNS resolution issue ?

after talking to AWS you can do this via the command line

see https://aws.amazon.com/premiumsupport/knowledge-center/private-hosted-zone-different-account/

either separate AWS accounts or separate VPCs is what we do (depending on scale)

@Erik Osterman (Cloud Posse) did a good overview in office hours a few weeks ago

ah thanks, do we have recording of those? Till now I have decided to go with separate vpc, not sure where shared services like logs/metrics etc should go. I have read that they should generally go into a different “shared” environment, but not sure of how their config etc would be tested. Like where should staging version of such shared services go?

https://www.youtube.com/c/CloudPosse/videos

I couldn’t find the exact video (there’s a lot there). A “Cloud Posse Explains: AWS Account/Environment Laydown” video would be helpful

Who the hell is that ugly guy @Erik Osterman (Cloud Posse)?? You really should keep out the rifraf

ah thanks for the help and the direction.

We’re actively releasing stuff every week as part of our reference architecture. It’s too early to point you to something concrete. But by end of december it will be much closer. Documentation will begin in earnest Q1 2021.

All of our top-level AWS components are here: https://github.com/cloudposse/terraform-aws-components

this is the future home of our reference-architectures https://github.com/cloudposse/reference-architectures but its documentation is stale and examples incomplete.

we’ll definitely be recording lots of videos, demos, walkthrus in addition to more step by step documentation.