#aws (2022-07)

If you are using AWS, why not use AWS SSO with permission set in combination with AWS Session Manager?

I’m not sure I understand you

https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html

With System Manager Agent, you don’t need to manage SSH Keys or open inbound ports to your machines

That sounds good! Thanks, I’m gonna have a look

And if you want to leverage with SSO, you can use AWS SSO

https://docs.aws.amazon.com/singlesignon/latest/userguide/what-is.html

In my company we use Session Manager + AWS SSO + Okta (as IdP)

works like a charm

the downside: only works at AWS

glad to hear

I mean that should be OK

We have around 100 servers, and we’re done with copying ssh keys back and forth. So we need something simple

yep, I hear your pain

you basically need to install AWS Session Manager, create some policies to start using it

it works!

thanks Rodrigo

@Rodrigo Rech: slightly piggybacking on this thread, can you assign AWS SSO to AD groups? We use AWS SSO with Azure AD, but i’m being told by my platform administrator that we cant do nested groups, and thus we have to enter AWS account/role access per user .

Hi @mikesew not sure if I got your use case

On Okta I assign AWS SSO to specific users groups

On AWS side I grant a Permission Set base on the user group

I didn’t get it what do you mean by nested groups

If need more granular control, you could look for ABAC (Attribute-based access control) https://docs.aws.amazon.com/singlesignon/latest/userguide/abac.html

Org Had a set of basic roles ie. AWS administrator access , AWSReadOnlyAccess. We could assign human ad users to thise roles but apparently not ad groups. Puzzling.

From what I understand @mikesew, it’s possible

What happen on my company:

• We manage users and groups at Okta side.

• On the AWS side, we have different Permission Sets, which includes different IAM Policies

• Based on the AWS Account and User Group, we assign this permission set. You can also assign multiple permission sets to a single group within same account if you wish.

thank you. Okta doesn’t have a free tier, does it? I want to try for my own learning, to setup an aws org, setup some free directory service like.. azure AD, setup AWS SSO to use that azure AD SAML, then see if i can add nested groups to those roles.. not just users.

• Q: are there any other free tier directory services i can try in a sandbox? my AWS free tier account expired.. I could try jumpcloud, but was hoping to use something closer to my company (azure AD)

I like cloudflare tunnels a lot if you use cloudflare.

Thanks Andrey! I’m gonna have a look

https://aws.amazon.com/blogs/aws/new-customer-carbon-footprint-tool/ is a first step in that direction!

Thanks, I was sure I had seen an article about it somewhere..

I use Code Deploy but on a small scale. Works really well and very happy with it - nothing to maintain and pretty much free. My use case is to get it to pull repos from Github and trigger ansible runs so you can get it to do pretty much anything that Tower does but in a proper way - securely, pulling rather than pushing etc. It’s pretty sweet

End of support for 1.20 is 03 Oct 2022

https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html#kubernetes-release-calendar

thanks

Here’s the response I got from AWS support with regards to my concerns of not being able to upgrade our EKS cluster in time, before the End of support date, hope it helps people to plan the upgrade accordingly:

Query 1 : What is the impact of running our EKS with K8s 1.19 after June 2022 ? According to the docs 1.19 will be unsupported past that date.


* From [1] it is stated that On the end of support date, you can no longer create new Amazon EKS clusters with the unsupported version.
* Existing control planes are automatically updated by Amazon EKS to the earliest supported version through a gradual deployment process after the end of support date.
* After the automatic control plane update, make sure to manually update cluster add-ons and Amazon EC2 nodes.
* In this regard you might have a query of when the control plane takes place exactly ?
* In the same document [1] Amazon EKS can't provide specific timeframes. Automatic updates can happen at any time after the end of support date.
* We recommend that you proactively update your control plane without relying on the Amazon EKS automatic update process.


==================

==================
Query 2 : Will the cluster be shut down after that date?


* No your cluster won't be shut down from the AWS end.
* However as mentioned above the control plane of the EKS cluster will be upgraded any time after the end of support date.

==================

==================
Query 3 : Won't we be able to add new nodes?

* After the end of support date you won't be able to add new nodes with the expired version since the support for that version will be expired.
* Also you won't be able to create new EKS cluster with the version for which the support has ended.

==================

==================
Query 4 : What is the actual date for end of support, 1st of june or 30th of June ?

* The date for the Amazon EKS end of support for Kubernetes version of 1.19 is June 30, 2022
* You can check the Amazon EKS Kubernetes release calendar [2] for the same

==================

As you have mentioned that the upgrade might not be completed within the end of support date for EKS 1.19 ( June 30th 2022 ) I recommend you to provide the EKScluster,region along with the business justification as to why the support should be extended so that I can reach out to the EKS Service Team and create a request for extension of support on your behalf for your EKS cluster for 1.19 version.

This sort of question is great for AWS support. It’s a simple closed technical query

You could use a WAF in front of the ALB to check for the API GW http header and then allow access based on that header response

but a malicious user can also manually set that http header right?

what’s the use case of having both actually? dont they do the same job?

@tor best bet is the code for this

https://github.com/cloudposse/terraform-aws-s3-bucket/blob/caf2af9a03f947ae51348e675146f6475ca228a3/main.tf#L180-L278

id, priority, status, etc are keys

basically anything in aws_s3_bucket_replication_configuration’s rule

reason for list(any) is because terraform hasn’t upgraded to optional map keys as a thing until terraform 1.3 which is not out of beta yet

Thanks for the pointer. I appreciate it.

but it could be still be documented in the respective variable’s description. perhaps we need to get better at that

this is not an official AWS suport channel

given “panic: could not get number of attached ENIs”, I suspect the permissions you’ve attached to the role are incomplete

I’d have an architectural preference for private link, personally, I think. Not sure what would be more “common” though

VPC peering was more common as private link wasn’t available. Private link is much the preferred method as VPC peering is much more intended for using across your orgs crosx account comms

consider:

1. Moving the original database. This will be cheaper than running two copies of the DB
2. Peer the Azure VPC with a VPN

For performance (and security) reasons I would prefer having a separate instance for them though

I don’t disagree with having a separate instance, but would still advise haviit vpc locked, and accessible over a vpn.

Agreed. But the only way to properly automate migrating data (without having it change IP’s every time with a snapshot restore) would be using DMS? Or is there an easier approach?

well… is a provisioned replica supported by aurora serverless?

https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-serverless.html#aurora-serverless.limitations “Aurora Replicas” are not supported by Aurora Serverless v1 it seems

I don’t have the full context here, but it seems you are trying to fix an architecture issue with an infrastructure solution. From an application/architecture perspective, this Azure application should directly access an API, not the DB. Each service/app should have its own database and expose its data using some API/integration mechanism. Sharing it across multiple services/apps will make your data governance, security, compliance, and operations chaotic in the long run.

truly a fulsome endorsement

love that pricing model

Very interesting

have you configured a hosted UI for the app client? it’s a requirement for Oauth flow

https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-assign-domain-prefix.html

yes I have

when i click the hosted ui link it says An error was encountered with the requested page.

“unauthorized client”

If I set the Cognito user pool client allowed OAuth flow to “Authorization code grant” and create the load balancer, it will get created. I can then after change it to “client credentials”.

Not sure I understand your ask… you have a public domain cert that you want to renew via a private CA server?

If that is the case, no that’s not possible. Highly recommend using Let’s Encrypt, it provides auto renewing free public domain Certificates. https://letsencrypt.org/

https://eksctl.io/usage/cluster-upgrade/

Thank you! That is great.

are you using RBAC or ABAC?

I have a mix of

• IAM users mapped to groups ( custom/ inline/ managed policies)

• IAM users with direct attached managed policies

• AWS SSO users mapped to permissions sets

However the q is not what i have as i know is rubbish, the q is how other folks managed to adhere to least privilege principle and managed in a “sane” way w/o too much operational overhead . Especially when in prod accounts not everyone has the same privileges

I would say the most “typical” or common approach you will see is user management via Active Directory/Azure AD –> AWS SSO ( or other Identity provider ala: centrify) –> Control Tower –> IAM Roles –> Services. To that end, you have your users in AD, and you create groups for those users, you then map those groups to Roles in IAM, and the Roles define access to services. Using Control tower to centrally manage the Roles/Policies for your account access. More into the Grit: What I have done for simplicities sake is to have basically 3 roles per service that your org will be using, Admin Role, User Role, Read Only Role.

Then it becomes easy to assign Matching groups of users from AD to one of the 3 roles in any given service. The advantage is that you keep all your user management inside AD, and it makes it easy to assign users or groups of users to services, and take away that access just as easily, without having to go in and check every AWS service and Role you have created otherwise.

Also get the advantage of using nested grouping in AD to assign a single person to multiple roles, could be an admin for one service, and RO for another

Also a really good idea to leverage Service catalog along with this, so based on Group/Role Assignment the users are presented with a list of services they have access to, and nothing they dont.

thanks for detailed info @jsreed !
Using Control tower to centrally manage the Roles/Policies for your account access. is this version controlled somehow or is UI driven ?

For audit reasons folks where i currently work had to log every change via Jira and then i came and added TF but that is getting out of control as you end up providing a “managed” policy or a role which covers “a lot” to not spend tons of time on this requests . And the whole process is also odd as after Jira ticket raised and approved, folks or “a monkey” needs to raise a PR to update the tfvars file , not so smooth process

inter-AZ latency (~1-2ms) is higher than intra-AZ latency (<1ms). Is that your question?

if one service talks to another service, that will be faster if they are colocated in the same AZ

To be more specific. In the AZ where Redis is colocated, the app responds to a request 50%+ faster than in the other AZ’s

example:

eu-central-1b

0m0.546s
0m0.537s
0m0.567s

other az's

0m1.141s
0m1.312s
0m1.299s

You’re going to need some more specific numbers than overall latency from an internal app

with aws_elasticsearch_domain, you can specify opensearch versions https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/elasticsearch_domain

i did the same, should work fine if you change tf to:

  elasticsearch_version = "OpenSearch_1.2"

and then just refresh state

yup it works! thank you guys!!!

Could it be broken out as two independent concerns? For example, you could have private service discovery using Cloud Map and App Mesh, which is in effect private DNS. On the backend, whatever services present themselves with the appropriate envoy container get the traffic. On the frontend, regional load balancers can represent their geographic regions using geolocation routing in route53.

I think I saw some lambda functions that did this; also, you can use the EC2 Lifecycle Manager. I am not sure if it will only clean up going forward or if it’s retro. Maybe something like this https://medium.com/nerd-for-tech/ebs-snapshot-management-using-aws-lambda-and-cloudwatch-d961fdbe3772

Ya, the life cycle manager doesn’t seem to be useful for the existing snapshots.

Were these snapshots created manually?

- name: "Delete snapshots for {{ domain }} cluster"
  block:
    - name: Set the retry count
      set_fact:
        retry_count: "{{ 0 if retry_count is undefined else retry_count|int + 1 }}"

    - name: "Get all remaining snapshots"
      raw: "aws ec2 describe-snapshots --cli-read-timeout 300 --filters Name=tag:kubernetes.io/cluster/{{ domain }},Values=owned --region {{ region }} | jq -r '.Snapshots | .[] | .SnapshotId'"
      register: snapshots_to_delete

    - name: Delete snapshots
      raw: "aws ec2 delete-snapshot --snapshot-id {{ item }} --region {{ region }}"
      loop: "{{ snapshots_to_delete.stdout_lines }}"
  rescue:
    - fail:
        msg: Ended after 5 retries
      when: retry_count|int == 5

    - include_tasks: delete_snapshots.yaml
  tags:
    - delete-cluster
    - delete-snapshots
    - skip-delete-cluster

I use this with ansible…

@Denis Unfortunately Yes, it seems to be. Not sure if they were using any other tool. But, certainly they are not through AWS Backup (or) AWS DLM.

if you have a tag based on which you can filter which exactly to delete, then a simple aws cli call or a small sdk app can do the trick.

@Denis Now, that’s another challenge as none of them have tags. I am getting them added and segregate a list (which needs to be cleaned off and which is not) using AWS-Tagger.

But, I don’t see a way of deleting snapshots by passing a file as input, read and delete in bulk using CLI??

no you’ll need a for cycle, like this one for example

or you can run one command that list the ebs IDs based on the tag or whatever, and pipe that output into that for cycle

@Denis @Balazs Varga Great. Thanks for your inputs. Let me try that and will let you know how it goes.