#aws (2021-07)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS) Archive: https://archive.sweetops.com/aws/

2021-07-30

Michael Warkentin avatar
Michael Warkentin

We are changing the way that asynchronous invocations of AWS Lambda functions work when the function has reserved concurrency set to zero. Previously, if the reserved concurrency was set to zero for such a function, the events sent to that function were retried for up to six hours, or a customer configured maximum number of attempts or event age, before being sent to the dead letter queue (DLQ) or on-failure event destination configured for that function. As of August 16, 2021, for functions with reserved concurrency set to zero, all events will be automatically sent to the configured DLQ or the on-failure event destination immediately, instead of being retried. Customers who wish to process events that were sent while reserved concurrency was set to zero will need to consume the events from the DLQ or on-failure event destination. This behavior will be enabled in all regions. Please refer to the AWS Lambda User Guide for information on how to configure a DLQ[1] or an on-failure event destination[2]. [1] https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#dlq [2] https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#invocation-async-destinations

Asynchronous invocation - AWS Lambda

When you invoke a Lambda function asynchronously, Lambda places the request in a queue and returns a success response without additional information. A separate process dequeues requests and invokes your function synchronously.

Asynchronous invocation - AWS Lambda

When you invoke a Lambda function asynchronously, Lambda places the request in a queue and returns a success response without additional information. A separate process dequeues requests and invokes your function synchronously.

jack fenton avatar
jack fenton

Is anyone great at EC2 userdata (cloud-init) ? - I have a userdata script that sometimes will be done in seconds and sometimes it takes over an hour.

cat /var/log/cloud-init-output.log has logs in it, but only after above has started

• is there something I am unaware of that can prevent or slow down cloud-init ? It does not seem like there is a pattern Thanks!

jack fenton avatar
jack fenton

the only pattern is either it’s instant (ish), or starts over an hour later. maybe a ntp / time issue?

2021-07-29

Almondovar avatar
Almondovar
11:29:38 AM

Hi all, can someone point me to the proper direction of how to “write” health checks for load balancer target groups? we have servers running fine but we cant figure out how to create health checks for port 1883 (mqtt). we know that servers listen to this port because they write telemetry that are coming from the sensors to the database successfully, thanks!

Darren Cunningham avatar
Darren Cunningham

if you’re using an ALB then your health check needs to be HTTP/S – looks like in this case you should be using a NLB with a TCP health check

2021-07-28

Almondovar avatar
Almondovar

hi guys ,we have a testing aws account with several pieces of infra that we dont use anymore, isnt it better to delete the aws account that is a part of an org instead of going manually to delete everything one by one?

Darren Cunningham avatar
Darren Cunningham

it’s been a minute since I’ve done this, but I thought you weren’t able to delete the account without first removing all the resources anyhow…and there’s no (approved) way to delete an account automatically

but you can clean up the account with aws-nuke

GitHub - rebuy-de/aws-nuke: Nuke a whole AWS account and delete all its resources. attachment image

Nuke a whole AWS account and delete all its resources. - GitHub - rebuy-de/aws-nuke: Nuke a whole AWS account and delete all its resources.

1
Almondovar avatar
Almondovar

aha, thank you very much!

1

2021-07-27

Pavel avatar
Pavel

I am trying to use this https://github.com/cloudposse/terraform-aws-ssm-tls-ssh-key-pair for an ec2 instance, im a little confused as to how to actually create the keypair with these credentials in ec2

GitHub - cloudposse/terraform-aws-ssm-tls-ssh-key-pair: Terraform module that provisions an SSH TLS Key pair and writes it to SSM Parameter Store attachment image

Terraform module that provisions an SSH TLS Key pair and writes it to SSM Parameter Store - GitHub - cloudposse/terraform-aws-ssm-tls-ssh-key-pair: Terraform module that provisions an SSH TLS Key p…

Cody Halovich avatar
Cody Halovich

by running this module, the private and public key are stored in SSM. the contents of the public key are also in the terraform outputs.

GitHub - cloudposse/terraform-aws-ssm-tls-ssh-key-pair: Terraform module that provisions an SSH TLS Key pair and writes it to SSM Parameter Store attachment image

Terraform module that provisions an SSH TLS Key pair and writes it to SSM Parameter Store - GitHub - cloudposse/terraform-aws-ssm-tls-ssh-key-pair: Terraform module that provisions an SSH TLS Key p…

Cody Halovich avatar
Cody Halovich

you could use this to automatically store that key in ec2

https://registry.terraform.io/modules/terraform-aws-modules/key-pair/aws/latest

Pavel avatar
Pavel

i think i understand, i just created a keypair with terraform and set its public key to the ssm param for that module

Pavel avatar
Pavel

seems kind of obvious now

1
1

2021-07-25

Nishant avatar
Nishant

[Blog Post] AWS Config is the AWS Configuration auditor. It is the foundation of cloud assets inventory, change management, cost management and security. But does it fulfils the promise? https://blog.cloudyali.io/aws-config-know-before-you-take-a-plunge

Understand perils of AWS Config. attachment image

“Knowing is the half the battle won”, a wise soul once said. History is testimony to it. It worked brilliantly in favour of those, who knew exact state of troops, rations and weapons. Not just that, any weaknesses that adversaries may pick on. If you…

2021-07-23

Dan Stein avatar
Dan Stein

Hi there, im new to terraform, i used your elasticsearch module to create an es instance, but i cant work out how to apply the access policy using your api?

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)
terraform-aws-elasticsearch/main.tf at a216b2797b190ac0b996918c4c93cf16bf2e0425 · cloudposse/terraform-aws-elasticsearch attachment image

Terraform module to provision an Elasticsearch cluster with built-in integrations with Kibana and Logstash. - terraform-aws-elasticsearch/main.tf at a216b2797b190ac0b996918c4c93cf16bf2e0425 · cloud…

RB (Ronak) (Cloud Posse) avatar
RB (Ronak) (Cloud Posse)

The identity policy can be added using iam policy role attachments

Identity and Access Management in Amazon Elasticsearch Service - Amazon Elasticsearch Service

Amazon Elasticsearch Service (Amazon ES) offers several ways to control access to your domains. This topic covers the various policy types, how they interact with each other, and how to create your own custom policies.

2021-07-22

Andy avatar
Andy
06:06:19 PM

I’ve set up a psql 13 master and replica in AWS, and am seeing some strange ReplicaLag on the replica. Currently there is no load on either the master or the replica.

Andy avatar

I’ve compared the LSNs on the master and replica and they seem to be closely in sync.

• MASTER:

select * from pg_stat_replication;

• REPLICA:

select pg_is_in_recovery(),pg_is_wal_replay_paused(),pg_last_wal_receive_lsn(),pg_last_wal_replay_lsn(),pg_last_xact_replay_timestamp();
Andy avatar

Does anyone know what might be causing the high lag in the monitoring?

Andy avatar

The timings on the graph between peaks and troughs:

• low latency -> high latency ~ 4 mins

• high latency -> low latency ~ 1 min

Andy avatar

“If no user transactions are occurring on the source DB instance, a PostgreSQL read replica reports a replication lag of up to five minutes. ” https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_PostgreSQL.Replication.ReadReplicas.html#USER_PostgreSQL.Replic[…]adReplicas.Limitations

Working with PostgreSQL read replicas in Amazon RDS - Amazon Relational Database Service

Replicate with read replicas and external sources when you use PostgreSQL with Amazon RDS.

2
1

2021-07-21

Bschaatsbergen avatar
Bschaatsbergen

Heya

1
2

2021-07-20

DevOpsGuy avatar
DevOpsGuy

I have a requirement to find what all AWS secrets are using a particular API key. Is there a way to find all secret keys which are using a particular API key like can we find using regx or something??

Darren Cunningham avatar
Darren Cunningham

what do you mean by API key?

managedkaos avatar
managedkaos

i would also ask “which secrets?” do you mean AWS Secrets Manager?

managedkaos avatar
managedkaos

If you are using secrets manager, the most straight forward way to do this (provided you know the API key reference (name and/or value), you can do this:

for i in $(aws secretsmanager list-secrets --query="SecretList[].Name" --output=text); 
do 
    echo $i; 
    aws secretsmanager get-secret-value \
       --secret-id "${i}" \
       --query=SecretString \
       --output=text | jq .| grep -E "(YOUR_SECRET_NAME_HERE|YOUR_SECRET_VALUE_HERE)"; 
done
managedkaos avatar
managedkaos

Note that this requires the system to have jq installed

DevOpsGuy avatar
DevOpsGuy

Yes its aws secret manager and @Darren Cunningham its some value stored in AWS secret manager.

Darren Cunningham avatar
Darren Cunningham

ok, so just to be clear you’re really asking How do I find all AWS Secrets Manager Secrets that contain <value>? – sorry the “API Key” bit confused me as I thought you were asking how to find resources associated to your IAM Access Key which was causing my head to explode – looks like @ has you sorted…bash will always be scrutinized as to “a better way” but that looks like it will do the trick

1
DevOpsGuy avatar
DevOpsGuy

So, @ how will the above script will search for a particular string. For example, I have 100 aws secrets in my account and some of them have a value stored same value in it. And how will we retrive all of the secrets which have same value??

managedkaos avatar
managedkaos

Yep, API key i was thinking IAM Secret Key+Secret Value. but then opted for the low hanging fruit of AWS CLI with secrets manager

DevOpsGuy avatar
DevOpsGuy

YES I am only talking about AWS SECRET MANAGER

Brij S avatar
Brij S

Hi all , has anyone been able to scale down managed nodes for EKS to 0 or 1 based on time? ie; Id like to scale the ASG down to 1 node if possible in the evenings. Is this possible? We use the cluster-autoscaler for scaling up but all my searches come up empty on if its possible to use the autoscaler to scale down.

Tim Birkett avatar
Tim Birkett

You probably want to implement the kube-downscaler to scale workloads to 0 on a schedule. That will then allow the cluster-autoscaler to scale down instances.

Steve Wade (swade1987) avatar
Steve Wade (swade1987)

We use spot instances in an ASG with a min of 0 and use cluster auto scaler to scale up and down when needed. Works like a dream

Brij S avatar
Brij S

how do you use the cluster autoscaler to scale down?

Steve Wade (swade1987) avatar
Steve Wade (swade1987)
GitHub - hjacobs/kube-downscaler: Scale down Kubernetes deployments after work hours attachment image

Scale down Kubernetes deployments after work hours - GitHub - hjacobs/kube-downscaler: Scale down Kubernetes deployments after work hours

Brij S avatar
Brij S

ah yes that in combination with the cluster-autoscaler ? nice

Steve Wade (swade1987) avatar
Steve Wade (swade1987)

Indeed

Brij S avatar
Brij S

nice I’l take a look at it

Steve Wade (swade1987) avatar
Steve Wade (swade1987)

Keep me posted

1

2021-07-18

Michael Warkentin avatar
Michael Warkentin

If anyone uses Amplify Console, I just wrote up a short piece on how to do fast rollbacks using a multi-branch approach: https://link.medium.com/YMubiWOq0hb

Fast AWS Amplify Console rollbacks with blue/green deploys

I’m a big fan of AWS Amplify Console for hosting static applications on AWS without having to manage your own pipelines, S3 buckets…

4

2021-07-16

curious deviant avatar
curious deviant

Hello, I am new to EKS Fargate and I am trying to setup a fargate cluster using the AWS TF registry module. Upon creation I observed that the coredns pods stay in pending state looking for a node to run on. Do fargate only clusters need worker nodes to run coredns ( and other system pods) ?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

@curious deviant that is how Fargate works - it will not launch any nodes until you provision something to the cluster (e.g. some k8s deployment)

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-eks-fargate-profile attachment image

Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-eks-fargate-profile

Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

which provisions a k8s deployment (after which EKS will add a node)

curious deviant avatar
curious deviant

Thank you for your response. I am going to bother you with another :slightly_smiling_face:. How about the coredns deployment? I did furthermore digging, looks like this is a known issue with fargate wherein the default coredns deployment that AWS does to the cluster, make it look for an ec2. The resolution seems to be to patch the deployment to have it run on fargate. In your experience, is it suggested to use a worker group alongside fargate profiles ?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

We don’t use fargate in production for exactly the same reasons: it’s has a lot of limitations and issues

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

But yes, you can use a managed node group alongside fargate profiles

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

CloudPosse has modules for managed and unmanaged node groups

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-eks-fargate-profile attachment image

Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
cloudposse/terraform-aws-eks-fargate-profile

Terraform module to provision an EKS Fargate Profile - cloudposse/terraform-aws-eks-fargate-profile

curious deviant avatar
curious deviant

Awesome !! Thank you so much for sharing. This was very helpful

Shreyank Sharma avatar
Shreyank Sharma

Hi, We have 2 AWS account, for some reason resource inside Account B has to access a resource which is inside Account A, is it possible to do that?? other than Access key and secrets.

is it possible by using IAM assume role?

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)

use cross-account IAM roles

Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
Working with Cross Account Roles in AWS attachment image

What to do when you need to provide a cross account access to the objects in your AWS account.

2
Andriy Knysh (Cloud Posse) avatar
Andriy Knysh (Cloud Posse)
IAM tutorial: Delegate access across AWS accounts using IAM roles - AWS Identity and Access Management

Learn the steps for delegating API access in your AWS account to an AWS Identity and Access Management (IAM) user in another account. (First of four).

1
Shreyank Sharma avatar
Shreyank Sharma

thank you

2021-07-15

OliverS avatar
OliverS

Has anyone had first-hand experience with crossplane in AWS EKS, seems awesome on paper, just wondering in practice:

• documentation: seems ok, but when the rubber hits the metal, is it adequate?

• community: active, responsive? (maintainers, users)

• robustness: should I consider it experimental or prod-level? not just for the AWS resources it manages, but for the controllers themselves (eg is crossplane easy to upgrade? what if upgrade fails partially, is it easy to rollback? are error messages adequate to troubleshoot issues with custom resources?)

• AWS resource coverage: looks minimal, eg there’s RDS and S3 but no SQS, SNS, documentDB, etc and for RDS there is no paramgroup so some things still definitely need to be provisioned outside of cluster

4
Zach avatar

There was just a discussion about this over in hangops, and the people trying it out kind of threw up their hands in frustration

Zach avatar


it absolutely hammers the k8s api and the aws api
it’s too complicated for something that should be simple

Zach avatar

and someone else mentioned that that it was nuking their cluster etcd

2021-07-14

Andy avatar
Andy
08:57:47 AM

Hi all, has anyone upgraded their AWS PostgreSQL 9.6 dbs yet? We have a master with 4 replicas that we’re looking to upgrade. We’d also like to switch to using encrypted volumes at the same time. The approach we’d use is:

  1. Take site offline
  2. Take snapshot of master
  3. Create encrypted snapshot from previous snapshot
  4. Create a new master RDS instance from the encrypted snapshot
  5. Create 4 replicas for the new master
  6. Migrate the master (RDS will then migrate the replicas in turn)
  7. Bring site back online Does that approach sound sensible? (and fastest)
Alex Jurkiewicz avatar
Alex Jurkiewicz

sounds very straight forward. Obviously the downtime will be quite large, but for extremely rare work like this I think that’s often a good tradeoff

Alex Jurkiewicz avatar
Alex Jurkiewicz

this approach also has the benefit you can completely test it beforehand. Have you done so? How long did it take?

Andy avatar

Someone on the team will be running the test today

1
Max Lobur (Cloud Posse) avatar
Max Lobur (Cloud Posse)

We did smth similar, but we were able to switch the site into readonly mode, with an appropriate maintenance message on it. Then you don’t go offline 100%, just readonly until you test the new instance of a site on the new DB, and switch the traffic.

Darren Cunningham avatar
Darren Cunningham

your plan is the arguably the fastest in regards to effort/orchestration

fastest in regards to least amount of time where your application is in either a read-only or offline state – would be to deploy the new RDS cluster and deploy a new application stack pointed to said new cluster in parallel, shutdown (either read-only on actual shutdown) “old”, do a data migration to new stack and cutover DNS

managedkaos avatar
managedkaos

For our lower environments, we created new DB instances and did a dump+restore from the old DB.

For the staging and production environments, we used Database Migration Service to keep the new DBs in sync with the old DBs (kind of a pain, honestly) in real time. No down time since we just deployed the app with the new DB settings and it cut over seamlessly.

However, I agree if you can take the time to go offline softly and do the cut that is a good way to do it.

Steve Wade (swade1987) avatar
Steve Wade (swade1987)

is there a way to only update the tags on an SSM parameter via the CLI ?

Zach avatar

aws ssm add-tags-to-resource

Zach avatar
--resource-type Parameter
--resource-id <value>
--tags <value>
Steve Wade (swade1987) avatar
Steve Wade (swade1987)

Thanks man!

1
loren avatar
loren
Corey Writes Open-Source Code for Lambda and Tailscale - Last Week in AWS attachment image

I’m afraid I come to you this morning with terrible news: I’ve been writing code again.

1
Alex Jurkiewicz avatar
Alex Jurkiewicz

i checked the repo, it’s shell code

Corey Writes Open-Source Code for Lambda and Tailscale - Last Week in AWS attachment image

I’m afraid I come to you this morning with terrible news: I’ve been writing code again.

2021-07-13

Brij S avatar
Brij S

Hi all, I’ve got two EKS related questions:

  1. has anyone managed to enable ASG metrics for managed node groups?
  2. Has anyone been able to use the cluster-autoscaler to scale down to 1/0 nodes at a given time? (ie; at night)

2021-07-12

Nishant avatar
Nishant

Hi - I’m using Cognito for authentication flow. For a demo account (only) I want to have a passwordless login or atleast have no complex password. Has anyone done anything like this? Any pointers greatly appreciated. Thank you.

Richard Pearce avatar
Richard Pearce

Free AWS training and 50% off the Exam for AWS Certified Solutions Architect - Associate https://pages.awscloud.com/GLOBAL_TRAINCERT_takethechallenge.html

Amazon Web Services (AWS) - Cloud Computing Services

Amazon Web Services offers reliable, scalable, and inexpensive cloud computing services. Free to join, pay only for what you use.

2021-07-08

Antarr Byrd avatar
Antarr Byrd

I need to create some kind of automation, maybe a runbook, whenever a step function fails. Any ideas on how to handle this?

Alex Jurkiewicz avatar
Alex Jurkiewicz

you can add a step to catch errors and run another step function, right?

2021-07-07

loren avatar
loren

this is quite nice. makes the api easier for interacting with security group rules. could lead to a number of improvements for the terraform resource/data source implementations also… https://aws.amazon.com/blogs/aws/easily-manage-security-group-rules-with-the-new-security-group-rule-id

Easily Manage Security Group Rules with the New Security Group Rule ID | Amazon Web Services attachment image

At AWS, we tirelessly innovate to allow you to focus on your business, not its underlying IT infrastructure. Sometimes we launch a new service or a major capability. Sometimes we focus on details that make your professional life easier. Today, I’m happy to announce one of these small details that makes a difference: VPC security […]

2
Alex Jurkiewicz avatar
Alex Jurkiewicz

Nice. I sort of fear it’s eight years too late. Everyone’s tooling deals with security groups not having IDs. Why bother changing

Easily Manage Security Group Rules with the New Security Group Rule ID | Amazon Web Services attachment image

At AWS, we tirelessly innovate to allow you to focus on your business, not its underlying IT infrastructure. Sometimes we launch a new service or a major capability. Sometimes we focus on details that make your professional life easier. Today, I’m happy to announce one of these small details that makes a difference: VPC security […]

loren avatar
loren

yeah, and it’s going to suck if the terraform aws provider switches to this new attribute and support for it is not implemented evenly across all partitions (govcloud, iso, etc)

2021-07-05

msharma24 avatar
msharma24

Do we have a community solution to rotate the aws org wide IAM Keys ? so far I have found this reference https://awsfeed.com/whats-new/apn/automating-rotation-of-iam-user-access-and-secret-keys-with-aws-secrets-manager

Automating Rotation of IAM User Access and Secret Keys with AWS Secrets Manager - AWS Feed attachment image

By Biswajeet Rakshit, AWS Solution Architect at TCS By Sigit Priyanggoro, Sr. Partner Solutions Architect at AWS By Will Horn, Manager – Partner Solutions

2021-07-03

2021-07-02

Shreyank Sharma avatar
Shreyank Sharma

Hi, we are migrating from Redis in ec2 to Elasticache,  and we have a lot of applications accessing that using that Redis with a password. (i.e lambda and inside Kubernetes etc….(code written in java, c#, pythons)) now if I have to Elasticache with the password I have to enable, Encryption in transit -> Redis AUTH default user. which means all connections happen with TLS (am thinking we have to make lot of changes to code just to connect to redis)

is it possible to add a basic password without encryption in transit feature.. Thanks

2021-07-01

caretak3r avatar
caretak3r

Anyone here working on AWS Quicksight? To my knowledge only the API is available, but no terraform modules. The customer I am working with wants to be able to build quicksight dashboards in various environments with CI/CD pipelines (jenkins). Wondering if anyone has done something like this/worked on anything like this? Any help is very much appreciated.

loren avatar
loren

you could probably create cloudformation templates to manage the resources, and if you want put a terraform wrapper around the cfn templates… https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-quicksight-analysis.html

AWS::QuickSight::Analysis - AWS CloudFormation

Creates an analysis in Amazon QuickSight.

caretak3r avatar
caretak3r

@loren thanks for that. I also found out that troposphere supports quicksight, https://github.com/cloudtools/troposphere/blob/master/troposphere/quicksight.py

cloudtools/troposphere attachment image

troposphere - Python library to create AWS CloudFormation descriptions - cloudtools/troposphere

caretak3r avatar
caretak3r

Might end up going that route

loren avatar
loren

same deal, troposphere is cloudformation

loren avatar
loren

if you like that, you could also do cdk

caretak3r avatar
caretak3r

Haha, you’re quick! I was just looking at that too

caretak3r avatar
caretak3r

Figured if I had to package things into containers for re-use, using the cdk might be worth my time. Never used it before.

loren avatar
loren

personally i like terraform a lot, so i’d use my first suggestion

loren avatar
loren

but totally just personal preference

Shreyank Sharma avatar
Shreyank Sharma

Hi all, is it possible to get the ip address of my elasticache nodes.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

yes, but they may vary

Cody Halovich avatar
Cody Halovich

@ you could ping the DNS endpoints. Those will not likely be static IP’s though.

Shreyank Sharma avatar
Shreyank Sharma

@Erik Osterman (Cloud Posse) @ Thank you, where i can get that info? just a nslookup to the DNS endpoint

Shreyank Sharma avatar
Shreyank Sharma

it was under EC2->Network Interfaces. thanks

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

aha, i was thikning you wanted a terraform way

    keyboard_arrow_up