#aws (2021-10)

aws Discussion related to Amazon Web Services (AWS)

aws Discussion related to Amazon Web Services (AWS)

Archive: https://archive.sweetops.com/aws/

2021-10-01

rostom avatar

anyone deployed AWS Inspector for EC2 instances in a private subnet?

mikesew avatar
mikesew

RDS Event Subscriptions: I’m trying to send RDS event subscriptions to an SNS topic, then have a lambda send those as a webhook to MS Teams. but following the AWS blog post about SNS to teams webhooks, the message I see in my teams channel is:

 {"Event Source":"db-snapshot","Event Time":"2021-10-01 02:09:04.371","Identifier Link":"<https://console.aws.amazon.com/rds/home?region=us-west-2#snapshot:id=rds:dev-mpa-spa-db-01-2021-10-01-02-09>","Source ID":"rds:dev-mpa-spa-db-01-2021-10-01-02-09","Source ARN":"arn:aws:rds:us-west-2:730458288754:snapshot:rds:dev-mpa-spa-db-01-2021-10-01-02-09","Event ID":"<http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_Events.html#RDS-EVENT-0090>","Event Message":"Creating automated snapshot"}

.. I want to customize this to something more readable. The python code in the lambda looks like below, and it’s basically blindly sending out the contents of the Message object. Anybody customized their events notifications before?

def lambda_handler(event, context): 
    url = "<https://outlook.office.com/webhook/xxxxxxx>"    
    msg = {
        "text": event['Records'][0]['Sns']['Message']
    }
    encoded_msg = json.dumps(msg).encode('utf-8')
    resp = http.request('POST',url, body=encoded_msg)
    print({
        "message": event['Records'][0]['Sns']['Message'], 
        "status_code": resp.status, 
        "response": resp.data
    })

2021-10-04

O K avatar

How can I test connectivity for created kafka service? It seems apps cannot use it Here is the config I used to create AWS MSK

module "kafka" {
  source = "../../external_modules/cloudposse/terraform-aws-msk-apache-kafka-cluster"
  # version = "0.6.3"
  namespace = "testnamesapce"
  stage     = "dev"
  name      = "msk"
  vpc_id    = module.vpc.vpc_id
  security_groups        = ["sg-XXXXXXXXXXXX", "sg-XXXXXXXXXXXX"]         
  subnet_ids             = ["subnet-XXXXXXXXXXXX", "subnet-XXXXXXXXXXXX"]
  kafka_version          = "2.8.0"
  number_of_broker_nodes = 2 # this has to be a multiple of the # of subnet_ids
  broker_instance_type   = "kafka.t3.small"
  broker_volume_size     = "100"
}

When I use netcat from EKS pods, I can reach Zookeeper nodes Error log from apps

2021-10-04 10:10:13 WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker z-2.dev.mjd92j.c17.kafka.us-east-1.amazonaws.com:2182 (id: -1 rack: null) disconnected
2021-10-04 10:10:13 WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker z-3.dev.mjd92j.c17.kafka.us-east-1.amazonaws.com:2182 (id: -2 rack: null) disconnected
2021-10-04 10:10:14 WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker z-1.dev.mjd92j.c17.kafka.us-east-1.amazonaws.com:2182 (id: -3 rack: null) disconnected
O K avatar
 ./kafka-topics.sh --create --bootstrap-server z-2.msk.xxxxxxx.c17.kafka.us-east-1.amazonaws.com:2181 --create --topic test-topic --partitions 3 --replication-factor 3 --if-not-exists

Error while executing topic command : Timed out waiting for a node assignment. Call: createTopics
[2021-10-04 11:55:27,380] ERROR org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: createTopics
 (kafka.admin.TopicCommand$)
Darren Cunningham avatar
Darren Cunningham

looks like either your security groups or route tables aren’t allowing traffic

O K avatar
nc -vz z-2.msk.xxxxxxx.c17.kafka.us-east-1.amazonaws.com 2181
Connection to z-2.msk.xxxxxxx.c17.kafka.us-east-1.amazonaws.com 2181 port [tcp/*] succeeded!
O K avatar

network looks fine

Darren Cunningham avatar
Darren Cunningham

Darren Cunningham avatar
Darren Cunningham

yeah it does

Darren Cunningham avatar
Darren Cunningham

well…silly question…are you running the netcat from outside in or from an instance within the VPC?

O K avatar

I’m running netcat from the same pod where app is running

1
Darren Cunningham avatar
Darren Cunningham

any chance the IAM policy associated doesn’t have the appropriate perms?

Darren Cunningham avatar
Darren Cunningham

honestly though I’m just throwing out random guesses in hopes that I help you stumble upon something

1
O K avatar

yep, checking everything, just wondering what might be the root cause as in overall it looks fine

Darren Cunningham avatar
Darren Cunningham

typically for me when I’ve seen unexpected timeouts it’s either networking (usually security groups) or IAM…usually with the latter though it’s not perms but rather an expired session

1
O K avatar

the problem was that by default TLS in transit was used which didn’t allowed to troubleshoot and etc… changed TLS -> TLS_PLAINTEXT and issue solved (we don’t use certificates so far)

1
Darren Cunningham avatar
Darren Cunningham

thanks for sharing the fix!

mohdmuzammilsk avatar
mohdmuzammilsk

Hi all, hope you are doing well.. currently I am using sticky session for caching the user session..for a wordpress site.. but now I am facing some performance issues with that and I am planning to move to elasticcache redis how to achieve this ?? Any reference would be really helpful.. or any basic idea

Eric Steen avatar
Eric Steen

Does anyone know if there is a cloudposse terraform package for cicd on aws that uses code commit? we have a requirement for code commit, but all the packages seem to rely on github. Thanks for any help.

msharma24 avatar
msharma24

Hey Eric,

This is my personal solution - https://github.com/msharma24/multi-env-aws-terraform and I think it can easily work in any CICD tool.

GitHub - msharma24/multi-env-aws-terraform: Multi environment AWS Terraform demoattachment image

Multi environment AWS Terraform demo. Contribute to msharma24/multi-env-aws-terraform development by creating an account on GitHub.

1
loren avatar
GitHub - plus3it/terraform-aws-codecommit-flow-ci: Implement an event-based CI workflow on a CodeCommit repositoryattachment image

Implement an event-based CI workflow on a CodeCommit repository - GitHub - plus3it/terraform-aws-codecommit-flow-ci: Implement an event-based CI workflow on a CodeCommit repository

2021-10-06

O K avatar

Is it possible to set these prarameters for AWS MSK(Kafka)

KAFKA_ADVERTISED_LISTENERS=<PLAINTEXT://kafka-server:9092>,PLAINTEXT_<HOST://localhost:29092>
O K avatar
Leader Not Available Kafka in Console Producer

I am trying to use Kafka. All configurations are done properly but when I try to produce message from console I keep getting the following error WARN Error while fetching metadata with correlation…

O K avatar

solved by adding auto.create.topics.enable=true

1
Darren Cunningham avatar
Darren Cunningham

thank you for sharing the solution. I see a few people mentioned auto creating but it doesn’t look like anybody explicitly used that – might want to share it with the community on that SO post.

1
Eric Steen avatar
Eric Steen

Good morning. I am not able to get AWS codepipeline to work with the “cloudposse/ecs-codepipeline/aws” module. I get an error:

Error: POST <https://api.github.com/repos/><name><app>/hooks: 404 Not Found []
│
│   with module.ecs_push_pipeline.module.github_webhooks.github_repository_webhook.default[0],
│   on .terraform/modules/ecs_push_pipeline.github_webhooks/main.tf line 7, in resource "github_repository_webhook" "default":
│    7: resource "github_repository_webhook" "default" {

Here is the main.tf for the pipeline:

module "ecs_push_pipeline" {
  source = "cloudposse/ecs-codepipeline/aws"
  version = "0.28.1"
  name                        = var.name
  namespace                   = var.namespace
  stage                       = "stage"
  image_repo_name             = var.imgRepoName
  region                      = var.aws_region
  github_oauth_token    = "<secure_token_from_github_org_oauth_creation>"
  github_webhooks_token = "<github_repo_webhook_secret>"
  repo_owner                  = var.owner
  repo_name                   = var.repo
  branch                      = "master"
  service_name                = "test-app-service"
  ecs_cluster_name            = "${var.name}-ecs-cluster"
  privileged_mode             = true
  cache_bucket_suffix_enabled = false # important: see <https://github.com/cloudposse/terraform-aws-codebuild/issues/91>
}

we have to use the deprecated but still supported aws oauth access instead of codestar connect is not an option as we are multi-region and it is not supported in ap or eu. Any help greatly appreciated.

Nick Kocharhook avatar
Nick Kocharhook

I’m seeing this same error with the same module setup. Were you able to resolve it?

Nick Kocharhook avatar
Nick Kocharhook

For anyone who runs across this error in the future, it’s a permissions problem. See here for more.

2021-10-09

Naveen Reddy avatar
Naveen Reddy

Hai Everyone. I have a problem with cloudformation. The current status is update_rollback_failed for ECS Service. I want to update parameter which takes docker image to latest. how can I do that as I’m unable to create change set and update the template as well. Any help would be great for me

2021-10-11

SlackBot avatar
SlackBot
03:11:17 PM

This message was deleted.

beaur97 avatar
beaur97

Hey everyone, so I have a simple question that I’m probably searching poorly for an answer for, but can’t find. So even just posting a link to an article about it is enough for me .

How do you request an internal ELB from say a node application?

For reference what I’m trying to do: My company has always worked in a monolith architecture, which is becoming a huge pain for us, so we want to split off some stuff into micro services. I’m trying to start with the most basic setup possible that allows us to easily add on to/move to better architecture in the future. We run a grails (Groovy/Java based back end) on elastic beanstalk instances. I want to launch the first service in a few EC2 instances (1 per environment) to start.

What I’m trying to figure out right now is the routing to keep the request from having to go out to the internet and back in to the ELB we are currently using. I know I can use a new ELB that’s internal to route requests dynamically based on URI so that we don’t have to hard code IPs/change per environment. Is there a specific internal IP/url the load balancer is always launched to? Or how can I consistently request it from a grails/node application. Am I overthinking it?

Alex Jurkiewicz avatar
Alex Jurkiewicz

Unless you are going to be sending terabytes of data, don’t worry about “routing to keep the request from having to go out to the internet”

beaur97 avatar
beaur97

@Alex Jurkiewicz it’s more so that this is an auth service that has zero need to even be exposed to the public

Alex Jurkiewicz avatar
Alex Jurkiewicz

Each ALB (you said ELB but I assume you aren’t using that old school thing) has a hostname, this hostname is fixed for the life of the LB.

You can CNAME a prettier name to that. For example [my-service.company.com](http://my-service.company.com) CNAMEs to [alb-1234567890abcdef.amazonaws.com](http://alb-1234567890abcdef.amazonaws.com).

Then, add a security group to the load balancer which states “only allow incoming requests on tcp/443 from security group app-servers”

Alex Jurkiewicz avatar
Alex Jurkiewicz

The security group will provide protection against public exposure. And you can also deploy the ALB into a private subnet, so it has no routable public address people can connect to.

beaur97 avatar
beaur97

Ah, that’s a great explanation. Thank you!

1
managedkaos avatar
managedkaos

just adding that when you create your ALB, configure it as internal: https://i.stack.imgur.com/uenlO.png

attachment image
managedkaos avatar
managedkaos
Is it possible to communicate to an ALB through an internal endpoint?

Setup We have an ECS cluster with 2 services (called portal-ECS-service and graph-ECS-service). Each have an ALB (portal-ALB and graph-ALB respectively). The setup is this: End user <-> porta…

2021-10-12

Brian Ojeda avatar
Brian Ojeda

Anyone noticing issues logging into AWS using the built-in SSO? Just started?

Brian Ojeda avatar
Brian Ojeda

Returning 504.

Antarr Byrd avatar
Antarr Byrd

Its down

Antarr Byrd avatar
Antarr Byrd
AWS live status. Problems and outages for Amazon Web Servicesattachment image

Real-time AWS (Amazon Web Services) status. Is AWS down or suffering an outages? Here you see what is going on.

Cody Halovich avatar
Cody Halovich

AWS Dashboard seems to be offline for the last ~30 minutes.

Cody Halovich avatar
Cody Halovich

Seems to be limited to us-east, I am able to login specifically to canada region

Alex Jurkiewicz avatar
Alex Jurkiewicz

This is why you deploy SSO in any other region

RB avatar

is it us-east-1 or all of us-east!?

2021-10-13

Steve Wade (swade1987) avatar
Steve Wade (swade1987)


*New customers can access two Availability Zones in US West (Northern California).

Steve Wade (swade1987) avatar
Steve Wade (swade1987)

is it a support ticket to get this unlocked

bradym avatar

Nope.

$ aws ec2 describe-availability-zones       
{
    "AvailabilityZones": [
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "us-west-1",
            "ZoneName": "us-west-1b",
            "ZoneId": "usw1-az3",
            "GroupName": "us-west-1",
            "NetworkBorderGroup": "us-west-1",
            "ZoneType": "availability-zone"
        },
        {
            "State": "available",
            "OptInStatus": "opt-in-not-required",
            "Messages": [],
            "RegionName": "us-west-1",
            "ZoneName": "us-west-1c",
            "ZoneId": "usw1-az1",
            "GroupName": "us-west-1",
            "NetworkBorderGroup": "us-west-1",
            "ZoneType": "availability-zone"
        }
    ]
}
1
Steve Wade (swade1987) avatar
Steve Wade (swade1987)

thanks @bradym

EvanG avatar

Does anyone have experience implementing centrailized security logging in AWS? I’ve created an Organization level cloud trail, but I need to clean up the resources in the sub accounts.

2021-10-14

Steve Wade (swade1987) avatar
Steve Wade (swade1987)
Stef avatar

Private Hosted Zone? If so, then yes.

Steve Wade (swade1987) avatar
Steve Wade (swade1987)

same but its a public zone, which confuses me

2021-10-16

Adnan avatar

Hi everyone, I seen big swings in the creation times of RDS MYSQL being created from a snapshot. Sometimes it’s 30min, other times more than 2h in which case terraform times out. Any of you had similar experience?

Darren Cunningham avatar
Darren Cunningham

yup - depends on the size of the snapshot, the size of the instance you’re restoring and the queue/availability in that AZ…which TMK is a black hole

1
Adnan avatar

thanks. it’s most likely the black hole because everything else stays the same

2021-10-17

x80486 avatar

Hello everyone! I’m looking at this module to create a website and host it in AWS S3. I see that it stores all the traffic (and what not) logs also in another S3 bucket. I wonder how do people usually go about analyzing (mostly reviewing) them :thinking_face:. Is it possible to store/send them to CloudWatch also/instead? So far, I can only download them to see what’s going on, but I’m sure that’s far from the trend

Constantine Kurianoff avatar
Constantine Kurianoff
Analyzing Amazon S3 server access logs using Amazon ES | Amazon Web Servicesattachment image

September 8, 2021: Amazon Elasticsearch Service has been renamed to Amazon OpenSearch Service. See details. When you use Amazon Simple Storage Service (Amazon S3) to store corporate data and host websites, you need additional logging to monitor access to your data and the performance of your application. An effective logging solution enhances security and improves […]

Constantine Kurianoff avatar
Constantine Kurianoff

You don’t have to actually use ElasticSearch and Kibana, but it is just so you can get an idea of how people analyze S3 access logs

x80486 avatar

Thanks @Constantine Kurianoff! I’ll check that out! I’m used to see all the logs in CloudWatch, so I was wondering that this could be the same, but probably the trend is to just use Elasticsearch to ingest them into that engine for further use.

Constantine Kurianoff avatar
Constantine Kurianoff

Yes, if you look for rich data visualization, you go with either ElasticSearch, or Grafana, or anything of that kind. I don’t have much experience with CloudWatch, but I think you can build dashboards and some good visualizations there as well

1

2021-10-18

Sarah Donehower avatar
Sarah Donehower

Hi Everyone- question about RDS/postgres and schema permissions. We have an RDS instance that was set up with a bastion host that forwards traffic from port 8887 to 5432 on the RDS instance. I’ve been accessing the instance with SSH tunneling and have successfully created and populated tables. I wanted to POC a data viz program and changed the instance to publicly accessible but only open to traffic on port 5432 for a couple of IP addresses. I can successfully connect to the instance now, but all the schemas I created while using SSH tunneling disappear. I’ve checked out permissions for the user but nothing stands out.

Is there a setting somewhere in RDS/postgres that connects port to schema privileges?

Alex Jurkiewicz avatar
Alex Jurkiewicz

did you use the same db user?

Alex Jurkiewicz avatar
Alex Jurkiewicz

are you looking in the same database? Different programs can offer a different default

Sarah Donehower avatar
Sarah Donehower

I connected to the same database as the same user. The only thing I changed was the port.

2021-10-19

Almondovar avatar
Almondovar

Hi all, we got EKS that holds our web app When we try to upload a evan a small, 250 KB file it is throwing 413 error response that is for file size limit exceed. The php.ini file seems to be ok. • post_max_size = 12M • upload_max_filesize = 10M • memory_limit = 128M Any ideas what else we should check?

bradym avatar

Check client_max_body_size in your nginx config.

1

2021-10-25

Victor Grenu avatar
Victor Grenu

Folks,

Issue #42 of my low-volume (once a week) newsletter “AWS Security Digest” is out.

What you will find:

  • Highlight of the week
  • Change since last week on AWS Managed IAM Policies
  • Curated Cloud Security Newsletters
  • AWS API changes
  • IAM Permissions changes
  • Most upvoted posts on r/AWS
  • Top shared links on Twitter (by cloudsec folks)
  • Most engaged Tweets from the community

Adopt a slow-tech approach by reading only essential, digest summary of what is going on in the AWS Security landscape.

With already 300+ subscribers with famous folks from @netflix and @amazon, you can’t go wrong :)

https://app.mailbrew.com/zoph/aws-security-digest-HrkhwqNrwBBk

AWS Security Digestattachment image

AWS Security Digest Weekly Newsletter. Curated by Victor GRENU.

2021-10-27

Tomek avatar

:wave: Is there a way to get the number of bytes written to S3 per day for a given AWS acccount (bonus if you can wildcard the bucket name, e.g. foo-dev-*)?

Alex Jurkiewicz avatar
Alex Jurkiewicz

Per bucket, use access logs

Alex Jurkiewicz avatar
Alex Jurkiewicz

Overall, you can probably read the billing report data

2021-10-28

Vlad Ionescu (he/him) avatar
Vlad Ionescu (he/him)

FYI, new feature for containers on AWS launches in less than 3 hours on Twitch: https://twitter.com/iamvlaaaaaaad/status/1453754371880230918

AWS pre-announcing a new feature is… interesting.

Uuu, a new feature for containers on AWS launches in 3-ish hours

I love the idea of doing this over Twitch! In addition to the awesome presenters, we get live interaction, Q&A, long-form demos, and thoughts from the actual devs! https://twitter.com/realadamjkeller/status/1453446212955242497

We have a special #ContainersFromTheCouch feature launch episode tomorrow @ 12pm PST. You won’t want to miss it! Links below, and get in line because from what I hear the crowds are going to be crazy!

https://www.youtube.com/watch?v=CiINJxFeNVg https://www.twitch.tv/aws

2021-10-29

pjaudiomv avatar
pjaudiomv

I made an aws profile switcher in GO based on awsp if anyone wants to check it out https://github.com/pjaudiomv/awsd

GitHub - pjaudiomv/awsd: AWS Profile Switcher in Goattachment image

AWS Profile Switcher in Go. Contribute to pjaudiomv/awsd development by creating an account on GitHub.

4
Cristian (cloudutil.io) avatar
Cristian (cloudutil.io)

Looks great, I’ll check it out. I actually submitted this sort of functionality to the AWS plugin for OhMyZSH, there’s the ‘asp’ alias if you have the plugin enabled, just type ‘asp’ and the Tab key to cycle through the available profiles

GitHub - pjaudiomv/awsd: AWS Profile Switcher in Goattachment image

AWS Profile Switcher in Go. Contribute to pjaudiomv/awsd development by creating an account on GitHub.

1
pjaudiomv avatar
pjaudiomv

Oh no way, I’ll have to check that out. I didn’t know there was a aws plug-in. Thanks

managedkaos avatar
managedkaos

Question: does this approach work with profiles updated by SAML logins?

I have several profiles but they also have the AWS_SESSION_EXPIRATION and AWS_SESSION_TOKEN values set after logging in using an SSO app.

pjaudiomv avatar
pjaudiomv

Alls it does is set or unset the AWS_PROFILE var

managedkaos avatar
managedkaos

ok. got it.

Andrea Cavagna avatar
Andrea Cavagna

@managedkaos I think Leapp is better for solving issues with SAML authentication: https://github.com/Noovolari/leapp

GitHub - Noovolari/leapp: Leapp is the DevTool to access your cloudattachment image

Leapp is the DevTool to access your cloud. Contribute to Noovolari/leapp development by creating an account on GitHub.

pjaudiomv avatar
pjaudiomv

Yea leapp is still a little green it has great promise though. But if your using saml for auth def best solution I’ve found right now

1
managedkaos avatar
managedkaos

ok! in the meantime, i am using python virtual envs to switch AWS accounts. it works good since i can set other things in the environment that are project/account specific.

2021-10-30

    keyboard_arrow_up