#aws (2021-02)

I’m trying to see if there’s any general best-practice regarding the whole auditing part of a fully-cloud architecture

And any other comments regarding that phase / recommendations would be very beneficial as well

Same org, dedicated audit account, all dev-stage-prod-sharedServices accounts send logs to audit account, like cloudtrail - access logs - anything with logs only sec/audit team has access to this account, you enable guardduty and securityhub for Centralized sec/compliance dashboards there

Do you create compliance dashboard using cloudwatch?

sechub

thanks @Mohammed Yahya!

any chance there’s some reference guide regarding all the auditing aspects? I can read on each of them separately but it feels that there might be some overlaps between them

I think you can start with sending cloudtrail logs, enable guarduty and security hub as a start. also check https://aws.amazon.com/blogs/aws/aws-audit-manager-simplifies-audit-preparation/ which is a new service

What about using the AWS CLI and or API to grab the metrics ?

Yup, that’s doable. I was hoping there was a way to overlay the graphs somehow

did you check AWS Backup?

https://aws.amazon.com/about-aws/whats-new/2020/01/aws-backup-supports-cross-region-backup/

Oh, very nice!

Will look into it, thanks

but more than old s3 backup replication

I’ll have to check their pricing, might be worth it if it’s less overhead to manage

It’s being actively worked on and is expected “soon” according to AWS Support

I have no clue, I switched to the old UI

I’d assume it’s because the majority of people don’t leave feedback

I think feedback in AWS goes to /dev/null

I’ve seen them respond to quite a bit of feedback overtime… but I agree, some of the new UIs are a regression

feature regression but the UI is overall better. I guess they hope for better long term results

I agree that the direction is good, but regression of features that make it harder to get to information we need to be successful is

Can you paster your aws-vault config, while masking values of course

yes -

[profile root]
region=us-east-1

[profile dev]
region=us-east-1
role_arn=arn:aws:iam::yyyyyy:role/OrganizationAccountAccessRole
source_profile=root

i’m using the root credentials in order to access all other accounts(dev, staging etc) in the organization

hmm, that uses a role, not a user nor group, here is my config for aws-vault with SSO ( permission set on SSO):

   1   │ [default]
   2   │ output=yaml-stream
   3   │ cli_pager=
   4   │ 
  11   │ [profile XYZ-master]
  12   │ region=eu-west-1
  13   │ 
  14   │ [profile XYZ-shared]
  15   │ sso_start_url=<https://XYZ.awsapps.com/start>
  16   │ sso_region=eu-west-1
  17   │ sso_account_id=123456789012
  18   │ sso_role_name=SuperAdmins
  19   │ region=eu-west-1
  20   │ 
  21   │ [profile XYZ-dev]
  22   │ sso_start_url=<https://XYZ.awsapps.com/start>
  23   │ sso_region=eu-west-1
  24   │ sso_account_id=123456789012
  25   │ sso_role_name=SuperAdmins
  26   │ region=eu-west-1
  27   │ 
  28   │ [profile XYZ-prod]
  29   │ sso_start_url=<https://XYZ.awsapps.com/start>
  30   │ sso_region=eu-west-1
  31   │ sso_account_id=123456789012
  32   │ sso_role_name=SuperAdmins
  33   │ region=eu-west-1

This allows us to use SSO single source of truth for access from users, and usernames appear on cloudtrail logs

thanks! very helpful

We ran into a similar issue (also via DataDog). So we set it to alert only if it fails in a few consecutive attempts.

Yeah we are rolling out that same change… but this is making me wonder if client connections are occasionally experiencing long DNS lookup times as well, which is the concerning bit.

There are many possible causes here… could be Route53, could be the DNS servers located closer to DataDog, etc. You can use tools like this: https://www.dnsperf.com/dns-speed-benchmark

That will help isolate if it’s a DataDog thing or a DNS thing.

Hm for some reason that tool doesn’t seem to work with my client’s domain. Odd.

Yeah it seems to only work with “www.“’s. Maybe if you go premium ?

Oh, it also works with the top level, like so: https://www.dnsperf.com/dns-speed-benchmark?id=1okq398kkppk1cu

Haha it looks like a great tool — shame that it has that limitation.

I’m running into the same issue with Rabbit MQ behind NLB - Route53 is delegated in dev account, look like a common issue

most of the time we lost the first tcp packet we send to NLB

so in the sender we add a check for the NLB and then send the message

We had the same problem so we did the same as @Yoni Leitersdorf (Indeni Cloudrail)

This is very similar to the new approach I’m seeing to memory leaks in containers: Just kill the container every 10 minutes (instead of finding out why it’s happening…)

Datadog edge synthetics servers are no so reliable

we have a lots of false positives from different locations

FYI we’re planning on using Terraform for our long lived infrastructure.

Some of the teams here are starting to use Serveless framework, which do a lot of heavy lifting regarding zip/upload the code. It also allows for you to setup a lot of the lambda’s integration

Sorry to interject here after two weeks but did anyone of you go with terraform + docker approach for AWS lambda deployments?

Talking about this new feature: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/

Yeah, that’s available via AWS SAM which I used for a POC. I thought it was neat to use but haven’t spent a lot of time with it.

if you are using this module

<https://github.com/cloudposse/terraform-aws-eks-cluster>

then the variable to add additional roles is

variable "map_additional_iam_roles" {
  description = "Additional IAM roles to add to `config-map-aws-auth` ConfigMap"

  type = list(object({
    rolearn  = string
    username = string
    groups   = list(string)
  }))

  default = []
}

@Thomas Hoefkens ^

Hi @Andriy Knysh (Cloud Posse), thanks - that is exactly what I used if you look at the question I posted… is it maybe because of the groups? Are there any default groups that I can assign to the role such as the one the cluster owner AWS access key may have?

I used this: map_roles = [     {       rolearn  = “arniam:role/my-role”       username = “my-role”       groups   = [“system:masters”]     }   ]

This is the setup:

1. EKS cluster is created with access key 1
2. I can only access the cluster using access key 2 and an assume role command to turn me into identity arnstsassumed-role/my-assumed-role/my-session-name
3. I would now like to access “kubectl get pods”

does the username have to exist as an iam user?

(nit: use codeblocks )

@Thomas Hoefkens here is what we used

my question about the variable map_additional_iam_roles was if you actually assigned your map to this variable when you instantiate the module

you can do it step by step:

1. Assume the role that created the cluster

1. Get kubeconfig from the cluster by, for example, executing these commands:

where --profile <AWS PROFILE WITH THE ROLE ARN THAT CREATED THE CLUSTER> is the AWS profile in ~/.aws/config for the IAM role that created the cluster

if the above works, then you can add additional IAM roles to the config map to allow them to access the cluster

then create an AWS profile with that role

and try to execute the same commands with the new profile

aws --profile <MY AWS PROFILE> eks update-kubeconfig --name=my-eks-cluster --region=<MY REGION> --kubeconfig=/dev/shm/my-kubecfg
chmod 600 /dev/shm/my-kubecfg
export KUBECONFIG=/dev/shm/my-kubecfg
kubectl get pods --all-namespaces
kubectl get nodes

@Andriy Knysh (Cloud Posse) thanks a lot for that, I will try that tomorrow and update the thread

Hi @Andriy Knysh (Cloud Posse)I have tested this I assume a role and create the cluster - then I can update kubeconfig and actually see the pods In the AWS console, my logged-on user assumes another role which I have added in map_additional_iam_roles. However, in the console now,when looking at EKS, the console says “your current user or role does not have access to Kubernetes object on this cluster”. Perhaps this is only a console issue..?

after you update kubeconfig and it’s on your computer, look inside - you should see all other IAM roles that can access the cluster listed there (and only those, plus the role that created the cluster, will be able to access the cluster)

I just had a bunch of ECS deployments issues

task in Pending state for ever

same

ohhhhhh

First it couldn’t pull a container from docker hub

do you have a capacity provider?

no

ok, we thought was a problem with the capacity provider

this ecs task has been running for 6 months with no issue

we just changes instance types and then it died

we have done that before and never had an issue

we were on version 1.3.0 of fargate and when I redeployed i started getting

“CannotPullContainerError: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)Read only root file systemfalse”

we are on ECS+EC2

well, turns out one of the teams deleted all the nacls

ohhhhhhhh

ON FRIDAY!!!!

yah it took me a while to track it down, went through their terraform repo and saw they added aws_default_network_acl to their terraform without defining rules

… almost as much fun as someone manually adding a vpc endpoint but not configuring it properly

That is cool, would love to see the resulting policy incorporated into the docs of each terraform resource and data source

check out tailscale? uses wireguard, very actively developed…

Pritunl

(Also search slack - some recent discussions on this)

I 2nd Pritunl

Thx ! but Printunl requires mongodb…which is a cons

+1 for Pritunl

two options:

• HashiCorp boundary

• AWS Client VPN ( tested with TLS auth or AWS SSO auth or Okta Auth) my preferred

fwiw, Pritunl will probably work with DocumentDB (as an alternative to Mongo)

+1 for tailscale - have good experiences tried to get boundary running, but it felt a bit cumbersome OpenVPN can have custom authorization via plugins / scripts - but be careful - it has serious impact on the forwarding engine (single blocking threat)

Might be a possibility for sending the data directly to redshift and firehose supports this by dumping first to s3 and then to redshift.

yes. read the docs on kinesis shards and data partitioning

I read the docs… just wanted to find out whether e.g. the lambda event trigger will assure that there will be no parallel processing of a single shard…

basically does AWS make sure when a lambda is triggered that this will always be a single lambda invocation… but as I understand it, a batch of records will always be passed into the lambda.. @Alex Jurkiewicz just wanted to make sure no 2 batches ever get fed to 2 different lambda instances so that records being written may overtake each other…

yes, the docs are a little ambiguous. We have a similar requirement and confirmed with AWS support that things work the expected way. I think there is some risk when resharding a stream but not in normal operation

ok, great, thanks for the info

Is the Fargate task running in ECS or in EKS? Haven’t tried it myself but in EKS it might be possible to use kubectl exec for fargate tasks - couldn’t find a result in a quick search.

https://docs.aws.amazon.com/toolkit-for-jetbrains/latest/userguide/ecs-debug.html seems quite interesting - so it must be possible to connect somehow to fargate tasks

If normal SSH is not an option - how about the reverse principle? Running a sidecar container that connects to a service where you can tunnel to it - I’ve seen somewhere that someone used AWS SSM (with custom instance registration) for having a “serverless bastion”. Can’t remember though if he only used it to tunnel into the network, or if he used it to access aspects of the Fargate task

Hi @Patrick Jahns, the Fargate task is running in ECS - I will have a look on what you’ve suggested and maybe try to run the SSM agent as a sidecar or as a Fargate task. Thanks for the hint!

Take a look at https://github.com/aws/aws-cli/issues/4685

Hopefully that helps, i use https://github.com/aws/aws-cli/issues/4685#issuecomment-700923581 looks like a user made more improvements, I havent personally tested those

Thanks! @Tom Dugan

@Zach oof thats a bit worrisome, we ran the upgrades for all our environments over the span of 2 months (we have many - qa, demo, trial, and 6 or so other environments) where either internal users or external users (customers) are accessing our many sites there and never got any complaints. We have 2 production RDS servers, one which we upgraded 2 weekends ago and one which we upgraded this past weekend.

What problems did you have?

horrible performance on indexed operations

we initially just ran renidexing, but then later found some documentation saying that computed indices during a major version upgrade really need to have Analyze run

we caught this issue when I finally found that RDS does a pgupgrade, which doesn’t catch all the recommended actions from postgres on major version upgrades. They actually recommend doing a pgdump and restore … obviously not so great on RDS but we didn’t realize initially that RDS didn’t conform to that

Interesting, other than the re-indexing and analyze and perfomance hits, you didn’t run into any other issues? My worry is more-so with data loss or data corruption. we did plan on doing the reindexing as mentioned here and will definitely run an analyze as well now: https://info.crunchydata.com/blog/just-upgrade-how-postgresql-12-can-improve-your-performance

that was the only issue, and we had problems with just about every database on 3 different RDS until we did the analyze

queries were taking forever to complete

hmm, we have roughly 1000 customers (big and small) and have not heard any complaints in regards to application performance. we migrated roughly 500 of them over 10 days ago.

well thats good then. You seem to be seeing the perf increases we had hoped we’d see from pg12

I’ve sanity checked many of their sites but nonetheless I’m still EXTREMELY anxious about it.

was the re-indexing resource intensive? I know pg12 allows for concurrent re-indexing now which is nice

yes big cpu/load spikes even with the concurrent

also make sure you are nowhere near low storage, because it has to make copies of the indices

thanks for answering my many questions @Zach. how much storage are you using and what instance type?

we actually are WAY overprovisioned on storage as we had to scale up to handle upwards of 12,000 iops. We’re currently using 500GB out of 4TB of storage.

we’re pretty small, these are like m5.2xl with 500gb

we tend to be cpu bound more than memory or iops for some reason

does ANALYZE do anything in regards to improving performance?

Allows the query planner to be smarter?

ANALYZE collects statistics about the contents of tables in the database, and stores the results in the pg_statistic system catalog. Subsequently, the query planner uses these statistics to help determine the most efficient execution plans for queries.

if the indices are using ‘computed’ values, it regenerates those

our problem seemed to be upgrade + reindex leaving lots of stale values in the indexes, so the planner decided to just ignore it

I’m not a DB guy so thats just what I learned on the fly as we put the fire out

neither am I. I’m wondering if your issue was specific to indicies w/ computed values and if my team doesn’t use computed values

I’m asking as we speak

I think our issue was computed values + really bloated partition tables

how did you verify analyze + reindex worked? I assumed since it was a fire, just by having the application re-run its queries?

db load shot back down

it was operating at like 40-50x load for the # of vcpus

hmm. my db load is low

yah your useage seems very different

or alternatively, my teams write bad sql

yeah knock on wood but I might be lucky with this one. I’m definitely going to keep digging though. much appreciated @Zach

for the record, our cpu load is hanging around 20-40%

vcpu sessions in perf insights between 1~2 out of 8vcpus

sorry last question @Zach, if you dont mind roughly how many databases are you running in your server? im curious because we have many databases in a single server, my index tables could be small compared to yours even though we run basically the same size DB Server?

~10 services on one, which is where the fire came from when some of the unanalyzed indexes caused everything else to slam to a halt

the others are 1-2 services per rds

looking forward to hearing the final conclusion

its been over 14 days, still nothing has come up from this upgrade in regards to performance or data issues. We went from peaks of 12k iops to <1k iops. Soo much healthier. We’re just way overprovisioned on storage space now because of the iops balance being tied to storage (pretty cheap so not doing anything about it right now)

I don’t know why the drop in iops was so drastic, but I assume it was fixed somewhere between postgres 9.5 and postgres 12

ClamAV

Yeah, that’s the one tool that I’ve seen used in this space. Question for you: Do you actually find it useful that you install that or is it just a checked box?

I guess it goes along with the saying “it is better to have it and not need it, then need it and not have it”

there is a bunch of Enterprise where this type of softwares are mandatory, we installed some at EA what were bakes on the images

EA as in Electronic Arts? Is that where you work Pepe?

I used to work there

the only commercial platform i’d come close to recommending is trend micro’s deep security - but i’d only recommend that if something like ClamAV doesn’t tick the boxes required - and that recommendation is quite a few years old now

I’m either going to with ClamAV OR I have a spike to investigate AWS Inspector and that has an “Inspector Agent” model which you can run on your instances, which I’m now interested in as well.

Thanks for weighing in gents.

I use inspector, you can then do auto remediation and such based off findings using ssm

Inspector is an aws thing?

Yea

I have a lambda that auto triggers an inspector assessment on all newly created ec2s and if they don’t meet a certain threshold it gets terminated or marked for termination as well as sns notifications etc

That Is pretty cool

we investigated a few vendors (paloalto twistlock, aquasec, stackrox) and went with Stackrox as it was the cheapest of the three yet they still checked all our boxes for container security monitors for certain benchmarks (CIS, SOC2, HIPAA, etc.), kubernetes security best practice checks on running containers and intrusion detection. None of the vendors in this space are cheap though.

Turns out Spinnaker only supports SNS Lifecycle Hooks so we going down that path with lifecycled. :)

just guessing…any chance you just need to update to a newer version of boto3?

thank you @Darren Cunningham just updated the boto3 still getting same error

that does suck. though i thought you only had to login to a given sso instance (e.g. the start_url) once per session, then every sso profile that used that same endpoint would be available… no? i.e…

aws sso login --profile account1.readonly

aws s3 ls --profile account2.readonly

… I just tested it out, and YES you’re right!! I just have to go through the whole rigamarole once , then I point to the profiles. The work is setting up the ~/.aws/config file with all hundred accounts.role’s I have.

and my SSO login should work for the default 12 hours.

whew! it is a lot of setup for all the profiles, but at least once that’s done, the login is only needed once per session!

there’s some discussion about simplifying the sso config using “defaults” or some kind of “base” profile…

• https://github.com/aws/aws-cli/issues/5727

• https://github.com/aws/aws-cli/issues/4800

If you have a standard set of roles in each account, then you can easily write a script to generate the aws config . That’s what I did

Morning Denis, what do you mean by AWS resources/assets ? Accounts ?

sorry @Santiago Campuzano, I was referring to any type of AWS resource, especially EC2 and RDS instances for example. One of the use cases is to track the lifecycle of an instance, for audit purposes for example. Back in the day, when everything was done manually, someone would create a VM and then create an asset record in a CMDB tool. When the VM was deleted, the record would have to be deleted as well.

The most easy and powerful way of doing that is to use proper tagging. We have 4-5 tags on average per resource.

If you properly tag your resources you can the use 3rd party or in-house tools to inventory and lifecycle your resources

I’ve used ServiceNow in the past, where our pipelines would call the ServiceNow API to create/update/delete records. But I’m trying to stay away from ServiceNow, but I’m also trying to cover scenarios where we don’t necessarily have full control of the deployment process. For example, a sandbox environment where users may create their instances manually.

To give you an example, we have: Billing, Owner, CreationTime, Team, etc

I completely agree with that, Tag is certainly key to the success here. But you mentioned 3rd party inventory tools and this is what I’m interested on. Any recommendations on that?

TBH, we have worked with in-house custom solutions… Python applications creating/updating Google Spreadsheets

yeah, that’s always an option. However, I’m trying to stay away from in-house solutions, given that we are a really small team and the overhead to maintain such a system can be significant, right?

Yep, you’re right. In our case we need pretty much basic information about the resources, our account is not that large (~2.000 ec2 instances)

well, that’s a pretty decent size. We are probably half of that size today, but my biggest problem is the size of the team. If I can find a SaaS solution with reasonable cost, I’d definitely go for that

You’re right… if the price is reasonable, go for it !

And let us know what your decision was about it

sounds good. Thanks for the feedback btw. Let’s see if anyone else has had experience with any other tools, it’d be nice to hear what others are doing in this space.

have you tried starting with aws config? it has an inventory feature for aws resources

I’ve looked into AWS config in the past, but I don’t think it had support for multi-account/Organization at the time. I’ll have another look, thanks !

multi-account is a bit harder, no matter what solution you go with. but it does now at least have organizations-level support, if all your accounts are at least part of one aws organization

and if not you can still use the config aggregator, but managing it and the account setup more directly

yeah, I think they are moving towards that, by adding support for most services at the org-level.

unfortunately i need multi-org management

I’ll give that a go and see how that plays out, thanks again. At this point we are only managing a single org

yeah, that’s a bigger problem for sure and I know I’ll have that same problem with multi-org and multi-vendor in the near future

if nothing else, it may at least give you one integration point to point some other tool at

agreed. Our Security Team is using Prisma Cloud, which has inventory capabilities. But it’s not always easy to convince security teams to give us API access, etc

I came across this project yesterday, but I haven’t had a chance to give it a go just yet. I thought I’d still share it here, in case anyone is interested: https://github.com/turnerlabs/antiope

Have you looked at CloudAware? I know some F500 companies that use it as their Cloud CMDB for the big 3 providers

Thanks @Sean Holmes ! I haven’t heard of them before, but will certainly have a look now. Thanks for sharing!

maybe you can use a cloudwatch syntetic?

This seems possible. No overlap of CIDRs. Did you try?

We did 1:1 NAT on VPC B side, because you cannot route traffic to different network CIDRs than the peered one

when we add 10.115.0.0/24 to VPC A route table to go thru peering nothing happened

I think thru peering you can push only traffic to peered VPC B CIDR

Oh that makes sense. Peering would forward traffic that it knows the other side is publishing. What I don’t fully understand is how the other side has a subnet with a CIDR that is not included in the VPC’s CIDR

There is custom side to side VPN solution.

Ahhh OK

2. Other option is to use Cloud Front, which we can be used for https
our problem is with both options the s3 bucket will be public, we dont want everyone to access our repository. You can configure a S3 bucket without public access to be the origin for a CloudFront distribution, using a “Cloudfront access origin identity”. It sounds like that is what you want to do.

You could also use a service like https://cloudsmith.com and we’ll take care of it for you. Including handling authentication, and things like per-customer access keys for private distribution. See the cloudsmith channel for some more information (we power hosting for Cloud Posse behind the scenes). Happy to help with questions.

thanks guys

Ya, we decided to stop managing our own package repositories at cloudposse and moved to cloudsmith

Gonna chime in and say this is something you don’t want to roll on your own if it’s not your core business. Too much work . Use a service or at least something flexible.

Ran across this as a random resource that’s github driven. Not saying use it… But just sharing as I found it interesting.

https://github.com/rpatterson/github-apt-repos

If you want to make it available via custom domain you can migrate to fips endpoint which by the end of the month will support only tls1.2+ (https://aws.amazon.com/security/security-bulletins/AWS-2020-001/). Bear in mind that the standard endpoint will still be available as you can’t disable it and it will expose tls/1.0 and tls/1.1 on the cognito domain.

AFAIR you can put your own cloudfront distribution (with no caching) in front of cognito. The default custom domain setup is actually a cloudfront distribution but it’s not exposing tls configuration.

FIPS is only for US + Canada

Did you manage to resolve this ?

My call with aws support seemed to find that there’s no way to find trusted-advisor-recommendations per resource, only by the trusted-advisor-check. Ultimately, I’m trying to setup a database health/scorecard. So presumably, each DB would have either all green TA checks (100% happy), or some yellow/red results against it (meaning <100%, or lower score).

weighted TGs work well but I haven’t done it specifically with ECS. Also curious as i’ll be exploring this in the next month and would like a solution without codedeploy as well

I imagine you’d want to just create a new taskset with your staged code and gradually shift over using weighted TGs. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-create-loadbalancer-bluegreen.html