#aws (2021-07)

you could probably create cloudformation templates to manage the resources, and if you want put a terraform wrapper around the cfn templates… https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-quicksight-analysis.html

@loren thanks for that. I also found out that troposphere supports quicksight, https://github.com/cloudtools/troposphere/blob/master/troposphere/quicksight.py

Might end up going that route

same deal, troposphere is cloudformation

if you like that, you could also do cdk

Haha, you’re quick! I was just looking at that too

Figured if I had to package things into containers for re-use, using the cdk might be worth my time. Never used it before.

personally i like terraform a lot, so i’d use my first suggestion

but totally just personal preference

yes, but they may vary

@Shreyank Sharma you could ping the DNS endpoints. Those will not likely be static IP’s though.

@Erik Osterman (Cloud Posse) @Cody Halovich Thank you, where i can get that info? just a nslookup to the DNS endpoint

it was under EC2->Network Interfaces. thanks

aha, i was thikning you wanted a terraform way

Nice. I sort of fear it’s eight years too late. Everyone’s tooling deals with security groups not having IDs. Why bother changing

yeah, and it’s going to suck if the terraform aws provider switches to this new attribute and support for it is not implemented evenly across all partitions (govcloud, iso, etc)

you can add a step to catch errors and run another step function, right?

sounds very straight forward. Obviously the downtime will be quite large, but for extremely rare work like this I think that’s often a good tradeoff

this approach also has the benefit you can completely test it beforehand. Have you done so? How long did it take?

Someone on the team will be running the test today

We did smth similar, but we were able to switch the site into readonly mode, with an appropriate maintenance message on it. Then you don’t go offline 100%, just readonly until you test the new instance of a site on the new DB, and switch the traffic.

your plan is the arguably the fastest in regards to effort/orchestration

fastest in regards to least amount of time where your application is in either a read-only or offline state – would be to deploy the new RDS cluster and deploy a new application stack pointed to said new cluster in parallel, shutdown (either read-only on actual shutdown) “old”, do a data migration to new stack and cutover DNS

For our lower environments, we created new DB instances and did a dump+restore from the old DB.

For the staging and production environments, we used Database Migration Service to keep the new DBs in sync with the old DBs (kind of a pain, honestly) in real time. No down time since we just deployed the app with the new DB settings and it cut over seamlessly.

However, I agree if you can take the time to go offline softly and do the cut that is a good way to do it.

aws ssm add-tags-to-resource

Thanks man!

i checked the repo, it’s shell code

There was just a discussion about this over in hangops, and the people trying it out kind of threw up their hands in frustration

it absolutely hammers the k8s api and the aws api
it’s too complicated for something that should be simple

and someone else mentioned that that it was nuking their cluster etcd

@curious deviant that is how Fargate works - it will not launch any nodes until you provision something to the cluster (e.g. some k8s deployment)

check out this example https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/examples/complete/main.tf

and this test https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/test/src/examples_complete_test.go#L173

which provisions a k8s deployment (after which EKS will add a node)

Thank you for your response. I am going to bother you with another :slightly_smiling_face:. How about the coredns deployment? I did furthermore digging, looks like this is a known issue with fargate wherein the default coredns deployment that AWS does to the cluster, make it look for an ec2. The resolution seems to be to patch the deployment to have it run on fargate. In your experience, is it suggested to use a worker group alongside fargate profiles ?

We don’t use fargate in production for exactly the same reasons: it’s has a lot of limitations and issues

But yes, you can use a managed node group alongside fargate profiles

CloudPosse has modules for managed and unmanaged node groups

https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/examples/complete/main.tf#L70

https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/examples/complete/main.tf#L86

Awesome !! Thank you so much for sharing. This was very helpful

https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.html#fargate-gs-coredns

use cross-account IAM roles

https://medium.com/adobetech/working-with-cross-account-roles-in-aws-b4f21dd72f4a

https://docs.aws.amazon.com/IAM/latest/UserGuide/tutorial_cross-account-with-roles.html

thank you

what do you mean by API key?

i would also ask “which secrets?” do you mean AWS Secrets Manager?

If you are using secrets manager, the most straight forward way to do this (provided you know the API key reference (name and/or value), you can do this:

for i in $(aws secretsmanager list-secrets --query="SecretList[].Name" --output=text); 
do 
    echo $i; 
    aws secretsmanager get-secret-value \
       --secret-id "${i}" \
       --query=SecretString \
       --output=text | jq .| grep -E "(YOUR_SECRET_NAME_HERE|YOUR_SECRET_VALUE_HERE)"; 
done

Note that this requires the system to have jq installed

Yes its aws secret manager and @Darren Cunningham its some value stored in AWS secret manager.

ok, so just to be clear you’re really asking How do I find all AWS Secrets Manager Secrets that contain <value>? – sorry the “API Key” bit confused me as I thought you were asking how to find resources associated to your IAM Access Key which was causing my head to explode – looks like @managedkaos has you sorted…bash will always be scrutinized as to “a better way” but that looks like it will do the trick

So, @managedkaos how will the above script will search for a particular string. For example, I have 100 aws secrets in my account and some of them have a value stored same value in it. And how will we retrive all of the secrets which have same value??

Yep, API key i was thinking IAM Secret Key+Secret Value. but then opted for the low hanging fruit of AWS CLI with secrets manager

YES I am only talking about AWS SECRET MANAGER

You probably want to implement the kube-downscaler to scale workloads to 0 on a schedule. That will then allow the cluster-autoscaler to scale down instances.

We use spot instances in an ASG with a min of 0 and use cluster auto scaler to scale up and down when needed. Works like a dream

how do you use the cluster autoscaler to scale down?

Take a look at https://github.com/hjacobs/kube-downscaler

ah yes that in combination with the cluster-autoscaler ? nice

Indeed

nice I’l take a look at it

Keep me posted

I’ve compared the LSNs on the master and replica and they seem to be closely in sync.

• MASTER:

select * from pg_stat_replication;

• REPLICA:

select pg_is_in_recovery(),pg_is_wal_replay_paused(),pg_last_wal_receive_lsn(),pg_last_wal_replay_lsn(),pg_last_xact_replay_timestamp();

Does anyone know what might be causing the high lag in the monitoring?

The timings on the graph between peaks and troughs:

• low latency -> high latency ~ 4 mins

• high latency -> low latency ~ 1 min

“If no user transactions are occurring on the source DB instance, a PostgreSQL read replica reports a replication lag of up to five minutes. ” https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.Replication.ReadReplicas.html#USER_PostgreSQL.Replic[…]adReplicas.Limitations

Are you referring to the domain policy? https://github.com/cloudposse/terraform-aws-elasticsearch/blob/a216b2797b190ac0b996918c4c93cf16bf2e0425/main.tf#L100

The identity policy can be added using iam policy role attachments

by running this module, the private and public key are stored in SSM. the contents of the public key are also in the terraform outputs.

you could use this to automatically store that key in ec2

https://registry.terraform.io/modules/terraform-aws-modules/key-pair/aws/latest

i think i understand, i just created a keypair with terraform and set its public key to the ssm param for that module

seems kind of obvious now

it’s been a minute since I’ve done this, but I thought you weren’t able to delete the account without first removing all the resources anyhow…and there’s no (approved) way to delete an account automatically

but you can clean up the account with aws-nuke

aha, thank you very much!

if you’re using an ALB then your health check needs to be HTTP/S – looks like in this case you should be using a NLB with a TCP health check

the only pattern is either it’s instant (ish), or starts over an hour later. maybe a ntp / time issue?