#aws (2021-07)
Discussion related to Amazon Web Services (AWS)
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
2021-07-01

Anyone here working on AWS Quicksight? To my knowledge only the API is available, but no terraform modules. The customer I am working with wants to be able to build quicksight dashboards in various environments with CI/CD pipelines (jenkins). Wondering if anyone has done something like this/worked on anything like this? Any help is very much appreciated.

you could probably create cloudformation templates to manage the resources, and if you want put a terraform wrapper around the cfn templates… https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-quicksight-analysis.html
Creates an analysis in Amazon QuickSight.

@loren thanks for that. I also found out that troposphere supports quicksight, https://github.com/cloudtools/troposphere/blob/master/troposphere/quicksight.py
troposphere - Python library to create AWS CloudFormation descriptions - cloudtools/troposphere

Might end up going that route

same deal, troposphere is cloudformation

if you like that, you could also do cdk

Haha, you’re quick! I was just looking at that too

Figured if I had to package things into containers for re-use, using the cdk might be worth my time. Never used it before.

personally i like terraform a lot, so i’d use my first suggestion

but totally just personal preference

Hi all, is it possible to get the ip address of my elasticache nodes.

yes, but they may vary

@Shreyank Sharma you could ping the DNS endpoints. Those will not likely be static IP’s though.

@Erik Osterman (Cloud Posse) @Cody Halovich Thank you, where i can get that info? just a nslookup to the DNS endpoint

it was under EC2->Network Interfaces. thanks

aha, i was thikning you wanted a terraform way
2021-07-02

Hi, we are migrating from Redis in ec2 to Elasticache, and we have a lot of applications accessing that using that Redis with a password. (i.e lambda and inside Kubernetes etc….(code written in java, c#, pythons)) now if I have to Elasticache with the password I have to enable, Encryption in transit -> Redis AUTH default user. which means all connections happen with TLS (am thinking we have to make lot of changes to code just to connect to redis)
is it possible to add a basic password without encryption in transit feature.. Thanks

2021-07-03
2021-07-05

Do we have a community solution to rotate the aws org wide IAM Keys ? so far I have found this reference https://awsfeed.com/whats-new/apn/automating-rotation-of-iam-user-access-and-secret-keys-with-aws-secrets-manager

By Biswajeet Rakshit, AWS Solution Architect at TCS By Sigit Priyanggoro, Sr. Partner Solutions Architect at AWS By Will Horn, Manager – Partner Solutions
2021-07-07

this is quite nice. makes the api easier for interacting with security group rules. could lead to a number of improvements for the terraform resource/data source implementations also… https://aws.amazon.com/blogs/aws/easily-manage-security-group-rules-with-the-new-security-group-rule-id

At AWS, we tirelessly innovate to allow you to focus on your business, not its underlying IT infrastructure. Sometimes we launch a new service or a major capability. Sometimes we focus on details that make your professional life easier. Today, I’m happy to announce one of these small details that makes a difference: VPC security […]

Nice. I sort of fear it’s eight years too late. Everyone’s tooling deals with security groups not having IDs. Why bother changing

At AWS, we tirelessly innovate to allow you to focus on your business, not its underlying IT infrastructure. Sometimes we launch a new service or a major capability. Sometimes we focus on details that make your professional life easier. Today, I’m happy to announce one of these small details that makes a difference: VPC security […]

yeah, and it’s going to suck if the terraform aws provider switches to this new attribute and support for it is not implemented evenly across all partitions (govcloud, iso, etc)
2021-07-08

I need to create some kind of automation, maybe a runbook, whenever a step function fails. Any ideas on how to handle this?

you can add a step to catch errors and run another step function, right?
2021-07-12

Hi - I’m using Cognito for authentication flow. For a demo account (only) I want to have a passwordless login or atleast have no complex password. Has anyone done anything like this? Any pointers greatly appreciated. Thank you.

Free AWS training and 50% off the Exam for AWS Certified Solutions Architect - Associate https://pages.awscloud.com/GLOBAL_TRAINCERT_takethechallenge.html
Amazon Web Services offers reliable, scalable, and inexpensive cloud computing services. Free to join, pay only for what you use.
2021-07-13

Hi all, I’ve got two EKS related questions:
- has anyone managed to enable ASG metrics for managed node groups?
- Has anyone been able to use the cluster-autoscaler to scale down to 1/0 nodes at a given time? (ie; at night)
2021-07-14

Hi all, has anyone upgraded their AWS PostgreSQL 9.6 dbs yet? We have a master with 4 replicas that we’re looking to upgrade. We’d also like to switch to using encrypted volumes at the same time. The approach we’d use is:
- Take site offline
- Take snapshot of master
- Create encrypted snapshot from previous snapshot
- Create a new master RDS instance from the encrypted snapshot
- Create 4 replicas for the new master
- Migrate the master (RDS will then migrate the replicas in turn)
- Bring site back online Does that approach sound sensible? (and fastest)

sounds very straight forward. Obviously the downtime will be quite large, but for extremely rare work like this I think that’s often a good tradeoff

this approach also has the benefit you can completely test it beforehand. Have you done so? How long did it take?


We did smth similar, but we were able to switch the site into readonly mode, with an appropriate maintenance message on it. Then you don’t go offline 100%, just readonly until you test the new instance of a site on the new DB, and switch the traffic.

your plan is the arguably the fastest in regards to effort/orchestration
fastest in regards to least amount of time where your application is in either a read-only or offline state – would be to deploy the new RDS cluster and deploy a new application stack pointed to said new cluster in parallel, shutdown (either read-only on actual shutdown) “old”, do a data migration to new stack and cutover DNS

For our lower environments, we created new DB instances and did a dump+restore from the old DB.
For the staging and production environments, we used Database Migration Service to keep the new DBs in sync with the old DBs (kind of a pain, honestly) in real time. No down time since we just deployed the app with the new DB settings and it cut over seamlessly.
However, I agree if you can take the time to go offline softly and do the cut that is a good way to do it.

is there a way to only update the tags on an SSM parameter via the CLI ?

aws ssm add-tags-to-resource

--resource-type Parameter
--resource-id <value>
--tags <value>


tailscale is blowing up… https://www.lastweekinaws.com/blog/corey-writes-open-source-code-for-lambda-and-tailscale/

I’m afraid I come to you this morning with terrible news: I’ve been writing code again.

i checked the repo, it’s shell code

I’m afraid I come to you this morning with terrible news: I’ve been writing code again.
2021-07-15

Has anyone had first-hand experience with crossplane in AWS EKS, seems awesome on paper, just wondering in practice:
• documentation: seems ok, but when the rubber hits the metal, is it adequate?
• community: active, responsive? (maintainers, users)
• robustness: should I consider it experimental or prod-level? not just for the AWS resources it manages, but for the controllers themselves (eg is crossplane easy to upgrade? what if upgrade fails partially, is it easy to rollback? are error messages adequate to troubleshoot issues with custom resources?)
• AWS resource coverage: looks minimal, eg there’s RDS and S3 but no SQS, SNS, documentDB, etc and for RDS there is no paramgroup so some things still definitely need to be provisioned outside of cluster

There was just a discussion about this over in hangops, and the people trying it out kind of threw up their hands in frustration

it absolutely hammers the k8s api and the aws api
it’s too complicated for something that should be simple

and someone else mentioned that that it was nuking their cluster etcd
2021-07-16

Hello,
I am new to EKS Fargate and I am trying to setup a fargate cluster using the AWS TF registry module. Upon creation I observed that the coredns
pods stay in pending state looking for a node to run on. Do fargate only clusters need worker nodes to run coredns
( and other system pods) ?

@curious deviant that is how Fargate works - it will not launch any nodes until you provision something to the cluster (e.g. some k8s deployment)

check out this example https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/examples/complete/main.tf
Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

and this test https://github.com/cloudposse/terraform-aws-eks-fargate-profile/blob/master/test/src/examples_complete_test.go#L173
Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

which provisions a k8s deployment (after which EKS will add a node)

Thank you for your response. I am going to bother you with another :slightly_smiling_face:. How about the coredns
deployment? I did furthermore digging, looks like this is a known issue with fargate wherein the default coredns
deployment that AWS does to the cluster, make it look for an ec2. The resolution seems to be to patch the deployment to have it run on fargate. In your experience, is it suggested to use a worker group alongside fargate profiles ?

We don’t use fargate in production for exactly the same reasons: it’s has a lot of limitations and issues

But yes, you can use a managed node group alongside fargate profiles

CloudPosse has modules for managed and unmanaged node groups

Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

Awesome !! Thank you so much for sharing. This was very helpful

This topic helps you to get started running pods on AWS Fargate with your Amazon EKS cluster.

Hi, We have 2 AWS account, for some reason resource inside Account B has to access a resource which is inside Account A, is it possible to do that?? other than Access key and secrets.
is it possible by using IAM assume role?

use cross-account IAM roles


What to do when you need to provide a cross account access to the objects in your AWS account.

Learn the steps for delegating API access in your AWS account to an AWS Identity and Access Management (IAM) user in another account. (First of four).

thank you
2021-07-18

If anyone uses Amplify Console, I just wrote up a short piece on how to do fast rollbacks using a multi-branch approach: https://link.medium.com/YMubiWOq0hb
I’m a big fan of AWS Amplify Console for hosting static applications on AWS without having to manage your own pipelines, S3 buckets…
2021-07-20

I have a requirement to find what all AWS secrets are using a particular API key. Is there a way to find all secret keys which are using a particular API key like can we find using regx or something??

what do you mean by API key?

i would also ask “which secrets?” do you mean AWS Secrets Manager?

If you are using secrets manager, the most straight forward way to do this (provided you know the API key reference (name and/or value), you can do this:
for i in $(aws secretsmanager list-secrets --query="SecretList[].Name" --output=text);
do
echo $i;
aws secretsmanager get-secret-value \
--secret-id "${i}" \
--query=SecretString \
--output=text | jq .| grep -E "(YOUR_SECRET_NAME_HERE|YOUR_SECRET_VALUE_HERE)";
done

Note that this requires the system to have jq
installed

Yes its aws secret manager and @Darren Cunningham its some value stored in AWS secret manager.

ok, so just to be clear you’re really asking How do I find all AWS Secrets Manager Secrets that contain <value>?
– sorry the “API Key” bit confused me as I thought you were asking how to find resources associated to your IAM Access Key which was causing my head to explode – looks like @managedkaos has you sorted…bash will always be scrutinized as to “a better way” but that looks like it will do the trick

So, @managedkaos how will the above script will search for a particular string. For example, I have 100 aws secrets in my account and some of them have a value stored same value in it. And how will we retrive all of the secrets which have same value??

Yep, API key i was thinking IAM Secret Key+Secret Value. but then opted for the low hanging fruit of AWS CLI with secrets manager

YES I am only talking about AWS SECRET MANAGER

Hi all , has anyone been able to scale down managed nodes for EKS to 0 or 1 based on time? ie; Id like to scale the ASG down to 1 node if possible in the evenings. Is this possible? We use the cluster-autoscaler for scaling up but all my searches come up empty on if its possible to use the autoscaler to scale down.

You probably want to implement the kube-downscaler to scale workloads to 0 on a schedule. That will then allow the cluster-autoscaler to scale down instances.

We use spot instances in an ASG with a min of 0 and use cluster auto scaler to scale up and down when needed. Works like a dream

how do you use the cluster autoscaler to scale down?

Take a look at https://github.com/hjacobs/kube-downscaler
Scale down Kubernetes deployments after work hours - GitHub - hjacobs/kube-downscaler: Scale down Kubernetes deployments after work hours

ah yes that in combination with the cluster-autoscaler
? nice

Indeed

nice I’l take a look at it

2021-07-21

2021-07-22

I’ve set up a psql 13 master and replica in AWS, and am seeing some strange ReplicaLag on the replica. Currently there is no load on either the master or the replica.

I’ve compared the LSN
s on the master and replica and they seem to be closely in sync.
• MASTER:
select * from pg_stat_replication;
• REPLICA:
select pg_is_in_recovery(),pg_is_wal_replay_paused(),pg_last_wal_receive_lsn(),pg_last_wal_replay_lsn(),pg_last_xact_replay_timestamp();

Does anyone know what might be causing the high lag in the monitoring?

The timings on the graph between peaks and troughs:
• low latency -> high latency ~ 4 mins
• high latency -> low latency ~ 1 min

“If no user transactions are occurring on the source DB instance, a PostgreSQL read replica reports a replication lag of up to five minutes. ” https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.Replication.ReadReplicas.html#USER_PostgreSQL.Replic[…]adReplicas.Limitations
Replicate with read replicas and external sources when you use PostgreSQL with Amazon RDS.
2021-07-23

Hi there, im new to terraform, i used your elasticsearch module to create an es instance, but i cant work out how to apply the access policy using your api?

Are you referring to the domain policy? https://github.com/cloudposse/terraform-aws-elasticsearch/blob/a216b2797b190ac0b996918c4c93cf16bf2e0425/main.tf#L100
Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - terraform-aws-elasticsearch/main.tf at a216b2797b190ac0b996918c4c93cf16bf2e0425 · cloud…

The identity policy can be added using iam policy role attachments
Amazon Elasticsearch Service (Amazon ES) offers several ways to control access to your domains. This topic covers the various policy types, how they interact with each other, and how to create your own custom policies.
2021-07-25

[Blog Post] AWS Config is the AWS Configuration auditor. It is the foundation of cloud assets inventory, change management, cost management and security. But does it fulfils the promise? https://blog.cloudyali.io/aws-config-know-before-you-take-a-plunge

“Knowing is the half the battle won”, a wise soul once said. History is testimony to it. It worked brilliantly in favour of those, who knew exact state of troops, rations and weapons. Not just that, any weaknesses that adversaries may pick on. If you…
2021-07-27

I am trying to use this https://github.com/cloudposse/terraform-aws-ssm-tls-ssh-key-pair for an ec2 instance, im a little confused as to how to actually create the keypair with these credentials in ec2
Terraform module that provisions an SSH TLS Key pair and writes it to SSM Parameter Store - GitHub - cloudposse/terraform-aws-ssm-tls-ssh-key-pair: Terraform module that provisions an SSH TLS Key p…

by running this module, the private and public key are stored in SSM. the contents of the public key are also in the terraform outputs.
Terraform module that provisions an SSH TLS Key pair and writes it to SSM Parameter Store - GitHub - cloudposse/terraform-aws-ssm-tls-ssh-key-pair: Terraform module that provisions an SSH TLS Key p…

you could use this to automatically store that key in ec2
https://registry.terraform.io/modules/terraform-aws-modules/key-pair/aws/latest

i think i understand, i just created a keypair with terraform and set its public key to the ssm param for that module

2021-07-28

hi guys ,we have a testing aws account with several pieces of infra that we dont use anymore, isnt it better to delete the aws account that is a part of an org instead of going manually to delete everything one by one?

it’s been a minute since I’ve done this, but I thought you weren’t able to delete the account without first removing all the resources anyhow…and there’s no (approved) way to delete an account automatically
but you can clean up the account with aws-nuke
Nuke a whole AWS account and delete all its resources. - GitHub - rebuy-de/aws-nuke: Nuke a whole AWS account and delete all its resources.

2021-07-29

Hi all, can someone point me to the proper direction of how to “write” health checks for load balancer target groups? we have servers running fine but we cant figure out how to create health checks for port 1883 (mqtt). we know that servers listen to this port because they write telemetry that are coming from the sensors to the database successfully, thanks!

if you’re using an ALB then your health check needs to be HTTP/S – looks like in this case you should be using a NLB with a TCP health check
2021-07-30

We are changing the way that asynchronous invocations of AWS Lambda functions work when the function has reserved concurrency set to zero. Previously, if the reserved concurrency was set to zero for such a function, the events sent to that function were retried for up to six hours, or a customer configured maximum number of attempts or event age, before being sent to the dead letter queue (DLQ) or on-failure event destination configured for that function. As of August 16, 2021, for functions with reserved concurrency set to zero, all events will be automatically sent to the configured DLQ or the on-failure event destination immediately, instead of being retried. Customers who wish to process events that were sent while reserved concurrency was set to zero will need to consume the events from the DLQ or on-failure event destination. This behavior will be enabled in all regions. Please refer to the AWS Lambda User Guide for information on how to configure a DLQ[1] or an on-failure event destination[2]. [1] https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#dlq [2] https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#invocation-async-destinations
When you invoke a Lambda function asynchronously, Lambda places the request in a queue and returns a success response without additional information. A separate process dequeues requests and invokes your function synchronously.
When you invoke a Lambda function asynchronously, Lambda places the request in a queue and returns a success response without additional information. A separate process dequeues requests and invokes your function synchronously.

Is anyone great at EC2
userdata (cloud-init
) ? - I have a userdata script that sometimes will be done in seconds and sometimes it takes over an hour.
• cat /var/log/cloud-init-output.log
has logs in it, but only after above has started
• is there something I am unaware of that can prevent or slow down cloud-init ? It does not seem like there is a pattern Thanks!

the only pattern is either it’s instant (ish), or starts over an hour later. maybe a ntp
/ time issue?