#aws (2021-04)

Ye, you don’t need to respin. It automatically matches reservation. Saving plans can’t be applied to RDS.
do you guys standardize on a specific instance size (ie. m4.2xlarge) so you don’t waste reservations? This depends. I actually never did it.

Check the https://aws.amazon.com/rds/reserved-instances/ How billing works

If your RDS are all in the same family (ie, M or R, etc) then the reservation is flexible and can be applied to any instance in that family

for example if you are using entirely XLs you could actually purchase all Large RIs, and 2 Large RIs would cover 1 XL instance. Or vice versa, an XL RI could cover 2 Large RDS.

Thanks - I see people in my org have done EC2 reservations but not savings plans. I like the flexibilityoption since I doubt i’d ever move from a t- to an m-.. most likely ever just go up and down (ie. m4)

so in theory, this would be a completely separate and non-impacting change / process that doesn’t matter at ALL to my terraform/deployment setup.. I simply arrange the purchase with my cloudPlatform team/aws admins

it does say “AssumeRoleTokenProviderNotSetError: assume role with MFA enabled”. Do you have MFA enabled?

Yes I don’t think cloud nuke can handle it

We use https://github.com/rebuy-de/aws-nuke

We also have MFA

My guess though is your ~/.aws/config is missing something

Getting tired of being surprised after a deploy when an execution fails because the lambda was missing a permission…

no, but if you find something I’m very interested in it!

I’ve thought about creating a Managed Policy that was overly permissive and assigning Lambdas with that policy, then writing something that ran on a CloudWatch rule and evaluated the CloudTrail log history for a given lambda over the past 30 days and then automatically provisioned a new role with the correct level minimum required access and updated the Lambda to use that – but all that seems too magical and like it would be potentially messy when trying to later update a Lambda with more access

oh, i think that kind of thing does exist… https://github.com/duo-labs/cloudtracker

I’ve only tested this with the aws cli, but maybe it’d work if you ran the lambda code locally?

https://github.com/iann0036/iamlive

if i squint, i can kinda see how i might use iamlive to capture mocked api actions from my unit tests, but i’d still need to build something to analyze it’s output and compare against the role/policy… https://github.com/iann0036/iamlive

glad I wouldn’t have to make it, but again I don’t think it’s the right solution as you’re counting on the CloudTrail auditing to prune the permissions, doesn’t really allow you to have ephemeral IAC that you can freely move between accounts

oh haha, @bradym beat me to iamlive

great minds… or something

we’ve started writing unit tests for lambda code and mocking the api calls with moto. so that might be an avenue…

well i do use terraform to deploy the lambda and it’s execution role, so i could compare the iamlive output and the iam policy from the terraform plan/output…

hmm, puresec made a serverless plugin for exactly this, before they were bought by palo alto. unfortunately now the project seems inactive, https://github.com/puresec/serverless-puresec-cli/

seeing lots of tools analyzing roles and comparing allowed permissions against cloudtrail usage or iam access analyzer, but basically nothing i can use in a pipeline before a deployment

Palo Alto strikes again – I’m afraid Bridgecrew will fall into this category too

the same guy who wrote iamlive has a number of ‘iamfast’ repos that will inspect your code and produce a policy. Its very much “beta”

also the minute that you have code that does something like “create cloudwatch stack” I think all bets are off

right, cloudformation becomes the executing agent… i aint doing that. i’ll just have terraform do it. and i’ll see the errors on tf apply

for me this is mostly a lambda that deploys fine, but then gets an event, executes, and dies because we missed a permission. it’s kinda “async” in terms of feedback of the error

thanks for the iamfast pointer… that’s promising, though i don’t want to generate the policy, i want to compare against a policy i already assign… https://github.com/iann0036/iamfast-python

ah. I think you’d have to do the diff manually … or write something on top of that

we’ll see where this goes… https://github.com/iann0036/iamfast-python/issues/1

Never seen the term Pilot light before. Thanks for sharing.

You mean this doc ? https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-streams.html

If its this one, its only applicable if you are using aws amplify + cognito in your web-app

https://docs.aws.amazon.com/firehose/latest/dev/writing-with-sdk.html

Should have provided the link, this is the example we’re looking at.

Ugh, wrong one… too many aws tabs open

https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/kinesis-examples-capturing-page-scrolling.html

I see, this uses the federated ids which you can use. Its common pattern when you want to interact with aws services from frontend

I guess my main question is, if you setup iam creds correctly with only the exact permissions you need, what is the benefit of introducing cognito?

Well, how would you hook-up iam on your web-app? Its running on client-side(User’s browser)

Could be okay if you want to do server-side operations

What’s being suggested to me is that since the IAM user would only have very specific permissions… we could just put the IAM creds in the browser. I’m pushing back against that, but need to better understand why that is more risky than putting an identity pool id in the browser.

Yeah, not a good idea to put aws access keys in browser

Yeah… that’s what I’m saying!

id-pool is a common aws design pattern which you should be using for these kind of operations, its isolated

Any chance you can point me at some documentation that would help me understand it better?

https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/loading-browser-credentials-cognito.html

Amazon Cognito helps you manage the abstraction of identities across multiple identity providers with the AWS.CognitoIdentityCredentials object. The identity that is loaded is then exchanged for credentials in AWS STS.

https://docs.aws.amazon.com/cognito/latest/developerguide/role-based-access-control.html

What you basically wanna do is create an un-authenticated role for the cognito identity pool and have that role access the kinesis

Hope this helps to prevent setting accesskeys on your frontend app

Thank you!

Fortunately since I first posted it’s been decided we’re going to have the frontend post to a backend server that will forward data onto kinesis. Maybe not the most elegant way, but better by far than aws creds in the frontend!

I’m still gonna do some reading so I can better understand cognito for the next time something crazy like this comes up.

I think that is a bad idea, it is yet another hop in between the db and the app and something else you have to maintain

you can use RDS proxy to do some connection handling but I guess the reason you have not seen any result to your question is because you are already in a container environment where you can pass ENV variables, deploy easily etc

no reason to over complicate things

I think it’s trading complexities rather than adding and the extra hop is nominal. My application priorities are stability and security, not saving nanoseconds or even milliseconds for that matter.

to that point, I’m mostly trying to address the latter concern of DB connection details being able to just be printed. But I guess to that point I could just be using IAM Authentication and locking down the Role.

This change would exchange security for extra maintenance overhead and reduced developer autonomy.

You have to trust developers to some extent. How paranoid you need to be depends on your company industry and regulatory environment. If you can rely on existing controls, it will make your life easier. For example, your PR process will already ensure nobody adds this sort of code.

yup, which is why I’m exploring the options. I see the pros/cons, but it’s fun to talk about them.

wearing my developer hat, I think it’s cool to not have to think about db connection details and just be able to use localhost all the time and know that it’s going to be there.

but cool doesn’t necessarily mean it’s good

I don’t think that a PR process ensures that a line like this could be entered - I think it’s pretty easy to overlook a print/log statement.

and I’m not even accounting for bad actors, I think this could easily be a whoops

I don’t think there is any added convenience from the developer with this approach. If you read the hostname from an env var, that’s no harder than a hardcoded localhost

fair

You are right that the PR process is not perfect. Governance never is. But it’s a lot cheaper and often Good Enough. The PR process protects against all developer bad actors. The sidecar approach protects only a single compromise. For example, a developer could write code to dump specific data to the log file, or even POST it to a remote system.

(For example, a product I work on has functionality where our customers can add a webhook subscription for all changes. A developer could write a migration that adds their own webhook subscription for a customer and exfiltrate data that way. We trust our PR process to detect this.)

thank you

https://www.percona.com/blog/2020/03/23/setting-up-a-proxysql-sidecar-container/

was linked to this by AWS Support - I asked them the same initial question

I guess to that point I could just be using IAM Authentication and locking down the Role Are there issues for you with using this solution?

nope, I just haven’t done it before.

Looks like these are the limitations you need to watch out for https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html#UsingWithRDS.IAMDBAuth.Limitations

Mainly

• Instance type and max connections

• Are you using CNAME

• I don’t know if you mentioned rds type, but if postgres and you are using master (hopefully not) log in method may need to change for you

Using PG and it will not be the master, we’d be creating IAM Roles and DB Users accordingly

That solution give you the separation of concerns if thats what you are looking for, and removes the extra hop

and IAM auth is not recommended for production workloads only humans

you’re referring to
The maximum number of connections per second for your DB instance correct?

yes

there is a limit of 200 connections per token

right on, trying to find the aws doc/reference for that, do you have it?

https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.IAMDBAuth.html#UsingWithRDS.IAMDBAuth.Limitations

Mysql : Use IAM database authentication when your application requires fewer than 200 new IAM database authentication connections per second.

got it, and postgres is just based off instance limits then?

the recommended way to do this for apps is to use the Secret Manager autorotation

Do you have some reference to that ^ for the OP

I do not know but if Mysql has that limit I will say is the same for postgress since Mysql is always ahead that Postgres from AWS support

https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-rds.html

https://instances.vantage.sh/rds/

yeah i know, it’s a little unfortunate that i gotta resort to that but totally, thanks.

nah i was wrong. you want this.

aws rds describe-orderable-db-instance-options --engine postgres

You can also add this to only get the enging version and dbinstanceclass

--query 'OrderableDBInstanceOptions[][EngineVersion, DBInstanceClass]'

add this to only get a specific version’s instance type

--engine-version 13.1

@RB you are correct. came back with a solution before i could!

I was looking at getting all of them with a script like this:

REGION=us-east-1
aws rds describe-db-engine-versions --query="DBEngineVersions[].{Version:EngineVersion,Engine:Engine}" --output=text | while read engine version;
do
    for instance_class in $(aws rds describe-orderable-db-instance-options --engine "${engine}" --engine-version "${version}" --query "*[].{DBInstanceClass:DBInstanceClass,StorageType:StorageType}|[?StorageType=='gp2']|[].{DBInstanceClass:DBInstanceClass}" --output text --region us-east-1);
    do
        echo "${engine} ${version} ${instance_class}"
        aws rds describe-orderable-db-instance-options --engine "${engine}" --db-instance-class "${instance_class}" --output=json --query="OrderableDBInstanceOptions.AvailableProcessorFeatures"
    done
done

yowza. looks like it could work but at that point, id just use python.

that wall of text is terrifying

i haven’t run it yet but i think that will give you everything

i am getting null for the processor stuff though. might need to tweak my quuery a bit. left to the reader!

haha nice nice

you could dump all the json and convert the json to a csv. might be easier

yeah

and i agree, when you start looping over services, a pythonic approach will save your sanity

this approach works but i’m looking at this doc: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass.html#USER_ConfigureProcessor.CLI.Example3

and all the queries i’ve done come up with null for the AvailableProcessorFeatures.

Are you seeing anything different?

incredible! thanks for the input, didn’t realize about describe-orderable-db-instance-options . The max-storage & maxIOPs is really nice. Was there any other table/lookup scheme that actually blew up the actual instance-class themselves (ie. db.m4.4xlarge ) to actual core-counts/GB? Again, that was where I was trying to help witih my licensing true-up.

https://github.com/vantage-sh/ec2instances.info

And https://github.com/vantage-sh/ec2instances.info/blob/master/rds.py

Might give you a great start as well. Really cool sure and source available.

To edit EB subnets, you need to recreate the enviroment from scratch, I believe

interesting

in my case the alb with the api is public

and the api gateway is api is public too

the only reason to use the api gateway is to transition old apis to the new apis so we need to route some stuff to a lambda and such

ahh i see. so in your case, i’m not sure. but if you intend for the API to be public, i would just go for the easy win and put the VPC link in the public subnet.

so in my case I could get away no using a VPC link BUT it is required is you want to use a Load balancer endpoint

yeah yeah.

makes sense. So yeah, i see the need for VPC Link.

the weird this is this: if I send all the traffic to a external public endpoint , it works

if I send it to the alb I get 503

so I was trying to figure out why

ahh is the ALB in the public and private subnets? otherwise it won’t be able to route to anything on the back end in the private subnet.

since is a public alb there is no SG to worry but since I was using the VPC link then what will be the SG to use ?

it gets confusing

yeah yeah! :sweat_smile:

seem like the SG on your ALB should allow traffic from 0.0.0.0/0 on the ports (80 and 443, perhaps) but it should also be in the public and private subnets.

the ALB is internet facing on front of the EKS cluster

is a ALB-controller

yes it is 0.0.0.0 since is public

i see. is it also in the EKS security group? so, subnets and security groups should all match up.

well the alb works just fine without API gateway

ok i misunderstood. the ALB works but the API pointing at the ALB does not. is that correct?

api gateway pointing at the alb does not

got it

but if I add and integration to a ngrok url pointing to my computer it works just fine

your set up might be different but here’s the TF code i am using for mine:

resource "aws_apigatewayv2_vpc_link" "link" {
  name               = "${var.name}-${var.environment}"
  security_group_ids = var.security_group_ids
  subnet_ids         = var.subnet_ids

  tags = merge(var.tags, {
    "Resource" = "aws_apigatewayv2_vpc_link"
  })

  lifecycle {
    ignore_changes = [tags]
  }
}

resource "aws_apigatewayv2_integration" "api" {
  api_id             = aws_apigatewayv2_api.api.id
  description        = var.description
  connection_id      = var.vpc_link_id
  integration_type   = "HTTP_PROXY"
  connection_type    = "VPC_LINK"
  integration_method = "ANY"
  tls_config {
    server_name_to_verify = var.server_name_to_verify
  }

  # For an HTTP API private integration, specify the ARN of an Application Load Balancer listener
  integration_uri = var.integration_uri
}

resource "aws_apigatewayv2_route" "api" {
  api_id             = aws_apigatewayv2_api.api.id
  route_key          = "$default"
  target             = "integrations/${aws_apigatewayv2_integration.api.id}"
  authorization_type = "CUSTOM"
  authorizer_id      = aws_apigatewayv2_authorizer.api.id
}

There’s more but these are the pieces that use VPC Link top the ALB.

I would say, maybe check your integration_uri. Mine looks like this:

    "integration_uri" = "arn:aws:elasticloadbalancing:us-east-1:ABCDEFGHIJKLMN:listener/net/xyz-qa-api/2484cd5ecaba5155/d33d18f01a965145"

But i am also using an HTTP API not a REST API so your mileage may vary.

same url for me, is the https listener arn

did you defined this payload_format_version = "1.0"?

for the integration?

mmm default is 1.0

checking

i get the same. not defined but that is how its configured:

    "payload_format_version" = "1.0"

I’m starting to think that the ALB is meant to be internal and not internet facing

hmmm, could be. however in my config the NLB is indeed internal. but still accessible from the API GW!

yes trough the VPC link

yep

I bet if I add an integration to point directly to the ALB public url it will work

what is your config for the SG of the vpc link ?

do you have inbound rules and outbound ?

or just inbound ?

it has both inbound and outbound.

in:

ype
Protocol
Port range
Source
Description - optional
HTTP	TCP	80	10.0.0.0/8	–
HTTPS	TCP	443	10.0.0.0/8

out:

Type
Protocol
Port range
Destination
Description - optional
All traffic	All	All	0.0.0.0/0	–

what about your HTTPS hostname ?

are you using a wildcar cert?

i have that set to match the name of the site hosted at the ALB and yes, using a wild card cert

in my case I have multiples services behind…

so the HTTS Hostname = *.pepe.api.thatkeepscrashing.com

hmm i wonder if that’s the catch? meh, not sure. but wondering if the API “hostname” needs to match the ALB hostname explicity?

this is the part i am referring to:

  tls_config {
    server_name_to_verify = var.server_name_to_verify
  }

but why the ALB hostname ?

on the integration

the api gateway should pass the Host header from the request

ahh not really the ALB hostname, but the name of the site that matches the cert attached to the ALB

yes, that yes

but that will happen with a plane ALB

I mean is the same requirement for a public ALB

i would imagine you could not verify the hostname with a cert

CN = cert = host name

correct

but it should still route

so I just created a new VPC link manually

using private subnets

same SG ( after I added the outbound rules)

and now the ALB is responding

the old integration was using public subnets

ahh got it

but now I wonder if the SG was the problem all along

I’m going to test it and see

and now even works with the custom domain name too

sweet!

Thanks @managedkaos

yeah no problem! it might be a week or so before i have any updates on my situation but i’ll keep you posted

for your use case you can have VPC endpoint policies to restrict access to the private api endpoint

I have used it in the pass to restrict only certain VPCs for examples

and FYI you can’t use a custom domain name in the private API

agree, we are prepared to skip the custom domain name on a private API. but i don’t know all the VPCs that will need to access the endpoint. so setting up an endpoint with rules will be difficult. This would be an API for use in an enterprise with many (hundreds) of AWS VPCs in addition to networks in other clouds.

ahhh I see ok

we really wanted a vanity URL but have gotten past that part. we just want it to be private now

could you proxy to a LAMBDA authorizer to do authorization for the endpoint?

yes, we have an authorizer in place. so that will be the gate for now.

I see ok

but still, say the endpoint is exposed and gets DDOS’d. how much of that do we have to pick up in cost of resource use vs AWS?

i may be overthinking but i would rather have the endpoint unresolvable on the outside

DDOS? you can set Throttling in that case

indeed. like i said, forgive me for being paranoid

but those are good concerns

@managedkaos you were creating an Internal API?

with version 1 of api gateway?

correct?

hey! no, internal api but with version 2. i got it worked out eventually.

for now the endpoint is public. we have a lambda authorizer attached

ahhh I c ok

I’m building now and internanl api with vpc endpoint and such

and now I need to change from alb to NLB since it does not seem to support albs

yes. for our service we have an ALB for the web interface and an NLB for API interfaces. i suppose it can get pricey if you have both.

instead of doing the VPC endpoint, i ended up using a VPC link to the NLB. that eliminates the need for the endpoint.

but you can’t do a vpc link to an ALB, right?

for an internal REST api

hmmm not sure. i didn’t try.

don’t think you can. this doc just specifies HTTP APIs https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-vpc-links.html

but using this integration method you might be able to do REST -> ALB https://docs.aws.amazon.com/apigateway/latest/developerguide/setup-http-integrations.html

the last link is what I’m doing right now with a public api

in this case is for internal api

Holy cow

Are there errors in there?

Well we use a sort of logger for root access on this machine, a wrapper on top of sudo: rootsh

I can’t look into it because it needs space in /var/log

can you use ec2 serial to access the server ?

I feel like we need a weekly thread with these…

“Hey everybody, here are the bugs that this community wants upvoted, let’s swarm em with s!”

There is 3700 people in this Slack, so I feel like that might be useful.

the contributor’s lament

Yeah Matt, that’s a good idea. I always thumbs up such issues, even if they don’t touch my current infra.

Upvoted. and the thread idea is such a good idea.

This reminded me of a terraform PR I’ve had for almost 2 years, with tests and documentation, no amount of pings have helped. The infra it was meant for is long gone

@Erik Osterman (Cloud Posse) let’s chat about a weekly thread or announcement post for a list of Terraform or Open source issues to

I’d be happy to get together a simple spreadsheet if you could help with making it a weekly post in #announcements or something.

Success

w00000!!!!

thank you all!!!

Ok. Found the solution. Put it under filters IAMLoginProfile: - "<user-name>"

also put following to be able to switch account:

IAMRole: - "OrganizationAccountAccessRole" IAMRolePolicyAttachment: - "OrganizationAccountAccessRole -> AdministratorAccess"

I think yes, you also pay storage fees in addition to the data transfer fee

I think you didn’t understand my question

I was referring to this one https://aws.amazon.com/s3/pricing/ For storage fees, maybe I didn’t get the question

If you see data transfer fee from internet to s3 is free… So there is no fee for that . Beside initial PUT requests and storage there is no any transfer from S3 or GET requests…

At least in example that I mentioned…

My question is simple. After you initially store 20 GB and pay for that at the end of month. Will you pay next month again for that storage if its not retrieved at all…

You will have to pay for storage. If there are data transfers, there will be additional cost incurred. Try this to calculate the exact cost that will be incurred: https://calculator.aws/#/

Well actually I did

Wouldn’t be cumulative if it is how you are saying?

that part isn’t clear to me to be honest

If you could explain your question further, I can try and help answer.

lets say you store 20 GB each month

in march you will have 20GB. in April you will have 40GB and so on… Do you pay for storage that just sitting there and you don’t access it or you just pay when you initially store objects? Generally that’s what I asked in initial question as well.

Oh I see now. You pay for storage sitting there. You get charged for 20GB in March and 40 GB in April. This is further clarified here : https://aws.amazon.com/s3/pricing/ where they say You pay for storing objects in your S3 buckets. The rate you're charged depends on your objects' size, how long you stored the objects during the month, and the storage class

well that exact line doesnt seem clear to me

that exact line lead me to post question here

and in that case that calculator would be misleading if you agree?

and basically you would see doubling s3 costs each month

It would be helpful if you could share how do you think the calculations are done. Here’s how I interpret this : March :20 GB * (0.46 USD) , April: 40 GB * (0.46 USD) , assuming 20 GB storage was added in April.

amount of stored data over one month is same

To simplify question. Do you pay for data that is just sitting there without any retrieval?

by your example you say yes, is that correct?

Yes, you pay for data just sitting there, even if you don’t access it

Thanks again

Well calculator is useless in that case

Go for infrequent access if you rarely retrieve it. Better yet, turn on Intelligent Tiering and let it drift to one of the Archive Access tiers. The total cost generally consists of how much you store + the object retrieval fees for whatever storage class you are looking at

yeah, that have sense. I just wanted to preform some initial calculation based on some inputs, and I run to calculator which is miss-leading Thing is that I still don’t know how frequent data will be accessed and how it will be processed.

You definitely pay for the data sitting there on AWS servers/SANs as it costs them resources to host. They do give you 5 GB free, “As part of the AWS Free Tier, you can get started with Amazon S3 for free. Upon sign-up, new AWS customers receive 5GB of Amazon S3 storage in the S3 Standard storage class; 20,000 GET Requests; 2,000 PUT, COPY, POST, or LIST Requests; and 15GB of Data Transfer Out each month for one year.”

In your case it sounds like you have even more reason to go with Intelligent Tiering - automatic cost savings for unknown/changing access patterns

Its unknown in this point, it will not be stored before I find out how it will processed. So less worries about choosing proper tier, and data retention/life-cycle.

I would just expect that their oob calculator would take that in count when gives 12 Months Estimate

Ultimately, it is probably not worth your time to forecast the math that directly. Paying at most $23/TB and maybe just $4

What you mean?

Ya probably should have a better calculator; just know that you want it in standard if you have lots of requests/retrievals

$.023/GB/month = $23/TB/month

its not that simple

ya that’s why their calculator is less than desirable for you; cost is variable based on retrievals during the period

we have options where costs dictate design, and opposite

but isnt

that’s just one parameter

yep, exactly; the $23/TB/month is like a fixed constant, object retrievals is another parameter, etc.

and if you take in count that upload of same data could be done in few requests, or in millions requests

it can be huge difference on bill

@Milosb you’ve been answered in first reply
I think yes, you also pay storage fees in addition to the data transfer fee S3 is really cheap storage both for storing and for putting data there. Calculating how much it will cost to store 20GB is not worth the time spent on calculating

There are many nuances in fees though which you should be aware of, for example, watch out for intelligent tiering and IA because for small objects you may be paying a lot more than you can save on it.

@Marcin Brański Maybe, maybe not. I got answer on question that I didn’t ask, and that made me think that either I didn’t properly ask or question wasn’t understood well since I didn’t mention data transfer fee at all. Cheap is subjective term. I took 20GB as an example. But thanks guys for clearing it up, it was really helpful.

That’s what GitHub Codespaces or GitPod do Cloud9 is still using vanilla EC2s as far as I know.

I really like Gitpod as it’s the fastest option. You can also self-host it, if that matters to you

Yeah, I’m very aware of codespaces, gitpod, and coder. However, I’m not using GitHub at my new role, so Azure DevOps only.

Codespaces also takes forever to build some of the larger images. Was thinking just doing SSH with container would give me the same experience in a way.

Gitpod SSH is really exciting, but again not showing Azure DevOps, and SSH is beta/Q1 2021 but no release date announced.

How about https://github.com/maasglobal/ecs-event-notifications ?

Looks pretty good. Thanks!

https://sweetops.slack.com/archives/CCT1E7JJY/p1619127317235400 @managedkaos responding here for future archive/context

Basically I have some ecs tasks with 3-4 containers each. They might need to talk to another task’s container. They also might need to accept traffic from an ALB in the public subnet.

Trying to find the simple way to ensure they can talk but ideally not care about which task is running in the case of having multiple instances of the task for scaling.

I’ve seen people use something like App Mesh, and other non-AWS implementations (Consul, etc.) to achieve such requirements at scale. Not sure how big the environment you’re thinking of here.

yeah i think some sort of service discovery might be needed if you are considering inter-task communication. if all the tasks are in the same service, you might be able to achieve that pretty easily with container names similar to how docker-compose does it..not sure though!

as for the ALB, you can:

1. put the ALB in public and private subnets. The ALB will be accessible publicly while the tasks will live in the private subnet.
2. Attach a target group with the tasks added to them using the IP of the task
3. Make sure the service has an security group that allows traffic from the ALB’s security group so the tasks can receive traffic

is service discovery for tasks or the containers in the task? And is it pretty easy to get going with or adds a lot of complexity?

I was hoping for something as easy as a private dns type of entry or elastic ip perhaps

i have not added/used service discovery with ECS so i can’t speak to the complexity. I know its an option though. You can decide how much you want to invest: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

indeed if your tasks are running they have an IP depending on where they are deployed. i don’t think they will have a DNS entry associated though. I would be up to you to capture their IPs and then figure out how to access them from the other tasks. which is kind of what service discovery does for you.

I just browsed through this doc… might be helpful… https://aws.amazon.com/blogs/architecture/field-notes-integrating-http-apis-with-aws-cloud-map-and-amazon-ecs-services/

Would you all recommend external internal load balancer pattern over service discovery?

I’m learning a bit on discovery and seems to be what I need if it can handle multiple nodes of same. I’m not doing mucroservices though and want to make sure that using for a non microservices based environment still is useful to ease communication of fargate tasks internally.

Bonus points of anyone has an example in Terraform of setting that up.

Have you read this: https://docs.aws.amazon.com/AWSEC2/latest/APIReference/throttling.html

Oh, I am good it’s only limited to:

• A third-party application

• A command line tool

• The Amazon EC2 console

This is also useful… https://github.com/nccgroup/ScoutSuite/wiki/Handling-Rate-Limiting

If you did hit rate limiting during a scan, it won’t have caused a permanent issue. Other applications using the API should also respect rate limiting and retry with a backoff. If you’re running this as an automated job, run it with a reasonable –max-rate.

Thanks guys, I was just worried if I screwed some AWS services with this…

If you did… It’s too late now and the request “buckets” will have been replenished.

Great, thanks

@Tim Birkett it looks like there are separate buckets for Non-mutating actions and Unfiltered and unpaginated non-mutating actions, so basically with all describe and get actions, so I think that I’m good

very hard to give you any pointers but the client port usually is not important since it on their end so I do not know why is relevant in this case?

One task decides the port and then the other container is supposed to accept a port in a large range. Most containers have specific port mapping so unclear if it’s possible to have a container in a task accept a wide range of ports inbound. A container seems limited to 100 from the docs but not clear if any alternative

I see ok

if you had a proxy layer on front you could do it

something like haproxy, envoy

you could add it as a sidecard of the fargate task or another service

that is a very particular usecase

Any docs or examples? This is new to me so anything more specific to help me get the concepts and know where to look would help

https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/

on Envoy you will need to check the docs and see if it is possible

@sheldonh curious as to the reason the client needs to choose a random target port

It’s connectivity service stuff, tunnels etc. Not a normal web service, so it needs to accept a range. It seems like I might need to stick with an EC2 instance for this unless you think fargate task can handle a long running service with many (say 1000 at least) port selection

Let’s assume I do this with an EC2 instance instead to allow easier vertical scaling. 1 server only right now. I’m focused on the minimal “best practice” way to implement this currently, as I don’t have any clustering with this service at this time. Must be a single server for now.

Any best practice recommendations esp regarding:

• Do you normally put a LB in front of any exposed instance, even if it’s just a single and not scaling at the time?
• Do you put the instance in a private subnet and use LB to point to it, or default to public subnet to avoid NAT Gateway load if it’s going to be accepting inbound calls, and just not give it a public ip?
• Or do you just leave the server as directly accessible on the range of ports?

regarding best practices:

• Yes, even if i only have one EC2, i use an ALB for the endpoint. If I need to change the EC2 or scale up, I don’t have to change anything that points to the endpoint

• Yes, EC2 in private subnet to keep it (more) secure. and ALB in both so it can route the traffic.

• I avoid leaving the server directly accessible if at all possible.

but for your use case you will not be able to use and ALB

since the alb does not have concept of port ranges

That’s what I was trying to figure out. So port range (not traditional web server), means this changes a bit. Normal single port stuff and I’d just use an ALB to a private subnet with ecs tasks I guess. But port ranges, it will be better to just assign an elastic ip/route53 to this, it seems. Just newer to the LB side and trying to vet I’m not missing anything obvious there

For ec2 instance with a specific port only then ALB makes sense to put in front.

Is there any reason to put in private subnet if the instance security group doesn’t have any inbound traffic other than from alb?

It adds more nat load so just checking if it’s a “go to” to always try to put in private subnet when not directly accessed externally, even if no public ip assigned.

you need to build the LB layer yourself that is why I said that you will have to use something like haproxy in front of the instance(instances) that will receive the traffic

your instances stay in the private subnet

the haproxy will have a public and private subnet

I’m not very familar with haproxy. I am thinking it might help in future, but the “MVP” of getting something up I might just need to simplify this. That’s a lot to figure out with little time.

haproxy is very easy to configure to do that

maybe 5 lines of configs

early ELBs were haproxy (rumour has it)

easy is relative. Is there a terraform module that makes this easy. My background isn’t networking so this is newer concepts to me. I have to accept a range of inbound and the easiest way is to allow inbound on this range.

I don’t use k8’s or any of that, just fargate tasks so what’s easy to you might be science to me

but that is the issue, you need some sort of software that can do that, otherwise your app will have to do it for you

so you can use the cloudposse ec2 instance module and create userdata to install haproxy

The service running is designed for connection management. It’s just i’m trying to figure if best practice would be for this to be behind a LB period of i just let it do it’s work directly.

If installing haproxy in user data is all it takes then maybe I could figure out. The docs seemed pretty detailed/complicated.

but if you need to listen for a range of port you Can Not use aws load balancer

Found this article too. https://medium.com/@onefabulousginger/fully-automated-infrastructure-and-code-deployment-using-ansible-and-terraform-2e318820fe0c This is useful to know about. I’m basically replacing LB usage based on service limitations with my own self-managed LB alternative.

Is there an absolutely critical reason why I have to have it run behind haproxy instead of accept inbound directly in your mind (if the tool is a connection management tool)

That’s kinda the piece I think I need to understand. I don’t mind evaluating for improvements, but at stage 1 does this add more complexity than value for a single instance?

if you app can listen in a a range of ports then you do not need it

if you app does not , then you need a layer to do it for you

and that layer behaves like a load balancer but it has the capability to listen on a range of port and then forward to any type of endpoint ( fargate, ec2 instance, alb etc)

Perfect. For now that’s what I’ll do then. The open source project https://github.com/jpillora/chisel this is what I’m using and it accepts a wide range of ports based on configuration. I think this means for my use public subnet directly should be used.

The only “abstraction” I can think of is using route53 to simplify calling directly regardless of ip changes.

if you have a network person in your company I recommend to go and talk to them to get a better idea

thanks for the tips!

The idea of combining AWS usage data with revenue sources is great, I would love to hear more about that part

It always amazes me how difficult it always is to track costs in AWS. I agree with @Alex Jurkiewicz, tracking usage along with revenue is a good way to look at this. For Cloudrail, we track spend at the account level, per service, etc., but not at a per-customer level.

This started coming up recently - looking at how much a given customer costs us (in Lambda, RDS, etc.) so we can later plot that vs revenue from given customers. Hopefully we’ll be able to share our learnings as well.

Great writeup @Corey Gale!

Thanks @Yoni Leitersdorf (Indeni Cloudrail) & @Alex Jurkiewicz, appreciate it! We’ve been ingesting our CUR data into Snowflake via Snowpipe, which provides the ability to join against everything else in our data warehouse. Once in Snowflake, we use a custom built Looker Explore for insights that incorporate said business data (that’s where our CPM/IHMP metrics are calculated). My #1 tip would be put some time into designing a tagging system that matches your splicing needs since all billing tags are included on CUR data points.

can you add more instances and then scale down?

you want connection draining bascially

Yep, I can add, just wondering if instance refresh would do it, not overly attached to the idea

if you terminate an instance it will do connection draining

instance refresh should do the same

You need a lifecycle hook on your ecs asg in order to first drain the instances and then terminate the instances once the running task count has reached 0 per instance

See this https://github.com/nitrocode/awesome-aws-lifecycle-hooks

Figured it out, they don’t, need to use the default event bus

If you’re interested in this as a paid gig either ping me here or email me at [email protected]

are you seeing something different?

It starts to add up when you have many millions of requests though :)

Although it looks like you can use Scope down statements to only use bot control for certain endpoints.

EB CLI handles this by listing the instances when you run the command. (ref https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/eb3-ssh.html)

This is handy to use in the bastion to connect to various instances in the environment — e.g., eb ssh -n 1 . However, this won’t work if the EC2 instances have public IP addresses assigned.

If https://github.com/aws/aws-elastic-beanstalk-cli/issues/3 gets resolved, it might be possible to do this without the constraint on public IP addresses.

See also

• https://serverfault.com/questions/824409/aws-elasticbean-beanstalk-eb-ssh-using-private-dns-names

• https://stackoverflow.com/questions/39613260/ssh-in-to-eb-instance-launched-in-vpc-with-nat-gateway

Magic I hope never to have a need for

for sure!

An nginx reverse proxy is like an HTTP(S) load balancer that sits on front of the users and transparently redirects http traffic to backend servers/services

We use nginx at our company to front Tomcat servers, do SSL/TLS termination, static content caching, URL filtering/redirection, etc .

So the users connect to the nginx server, it then forwards the connection back to one the backend servers,

That’s a smart 18 year old answer. How about answer like I’m 5? I’m still wrapping my head around ssl/tls termination in this so that’s a good reminder.

In one use case it’s in front of a single server so not quite sure what benefit it offers other than tls/ssl termination.

proxies like nginx give the ability to take care of some of the request chain before actual routing traffic to your application server.

benefits:

• CPU resources saved - server running your application is only doing what it needs to do.

• flexibility - you can add/replace servers behind the proxy seamlessly

Since a load balacer would let me replace a server that doesn’t apply for this I think.

Let’s take the CPU part. If the app is a Go app it should be able to handle reasonable concurrency with it’s http router/gorillamux etc. What load would you consider it helps with in a scenario like that? Any simple examples of a reverse proxy in front of a normal web server that might help give me context?

https://www.nginx.com/resources/glossary/reverse-proxy-vs-load-balancer/

https://stackoverflow.com/questions/17776584/what-are-the-benefits-of-using-nginx-in-front-of-a-webserver-for-go

Nice! Got me started on the right track . Making more sense now. Appreciate it

Also looks like gorilla web kit offers much of that functionality, albeit with a bit more work on middleware and all. Good info!

“Let’s take the CPU part. If the app is a Go app it should be able to handle reasonable concurrency with it’s http router/gorillamux etc.” it sounds like you’re only considering valid requests, something to remember is that when you have an application, if it’s exposed to the world it’s going to get crawled and bad actors are going to try to poke holes in it. by having layers (load balancers, reverse proxies, web application firewalls) in the request chain that can filter out erroneous or potentially malicious requests, you’re saving your application servers from having to spend the CPU cycles to discard such requests.

Sounds like you’ll have a mess if you just move accounts over, no? The legacy accounts will be using legacy configurations, and so you second org will be a “salad” of things.

They hired me for this job lol

Any chance you can move the accounts one by one, but making sure they are cleaned up and meet the shiny org’s structure and settings?

The new CT org has only dev workloads and the legacy LZ is using production workloads but is not managed well

@Yoni Leitersdorf (Indeni Cloudrail) thats what Im getting to , I trying to find the least used AWS account in the Legacy Org and will move it across the Shiny CT ORG

Can you do this without converting legacy org to use CT?

Any chance all of this is built using IaC?

They hire us when shit has hit the fan

The Legacy org is all hand jammed

define the configs in terraform, import everything, and use atlantis or TFC

I think that wil be alot of work

Im thinking Lean , and trying to find a way to first make them in one AWS ORG with CT

It depends on what they want to achieve - a consistent org, or a mess?

CT is all cloudformation under the covers, right? does it even support the features needed to import or otherwise move resources between stacks?

and to update the existing resources to match new/different configs in the CT org?

moving them into one org is not all that hard. it’s a well defined process. iirc, you “leave” the org with the root user of the member account to make it a standalone account, send the invite from the new org, and accept the invite from the (now) standalone account

@loren just read the AWS KB on that too

but i have no idea on how an invited account would be managed by CT…

So I think my best option would be to start with the least productive account in the Legacy org ad migrate it to CT Org , may be in an “unregistered” OU and park the account , then apply the CT guardrails , rinse-repeat

in terraform, that’s pretty well defined also. cumbersome, but well defined. hence, my suggestion

could you maybe create a new account in the legacy org to test the workflow, instead of relying on the least productive?

Thanks for the suggestion.

it’s an interesting problem. i’d love to hear more on how it works out for you. this type of workflow was one of our main considerations for why we’ve avoided both LZ and CT (and CFN), and put all our focus on terraform… but if CT/CFN can actually do it……

Yes definitely

It’s a fun problem.