#aws (2021-11)
Discussion related to Amazon Web Services (AWS)
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
2021-11-01
2021-11-02
Hi everyone, I was wondering if I can categorize my logs in cloud watch based on a filter. I.e having somewhere to look at all logs that has a tag “failed-error”. If there isn’t a way to do this with cloud watch kindly suggest any third-party solution you know. Thanks
Hey, you can write queries in cloudwatch insights to categorize your logs https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html
2021-11-03
Great technical blog by Jonathan Rau, our CISO, about securing your AWS EC2 Instances with Microsoft Defender
https://blog.lightspin.io/microsoft-defender-for-endpoint-on-aws
Hi Everyone! I’m using a bastion host on AWS and am trying to add a new user. The user generates a new RSA key pair. I add the public key to the authorized_keys file on the host. But they still get Permission denied (publickey,gssapi-keyex,gssapi-with-mic). I’ve tried myself and it still fails. Its not a file/directory permissions issue because I can currently connect with the key/pair I generated a couple of months ago. How can I troubleshoot this further?
on the local system, have they set the permissions on the key to 400 or 600 so the private key is only readable by the user?
also, (if you haven’t already) try conencting with ssh -vvv to get lots of debug out put. the answer might be in there as well.
the permissions are correct.
my new findings: I can ssh into the bastion instance with the new credentials. But when I use the port forwarding command to create the ssh tunnel… that generates the permission denied error
Any thoughts on why ssh works, but ssh with port forwarding fails depending on they key used?
can you share your port forwarding command?
Typically, i use it like this:
ssh -o ExitOnForwardFailure=yes -f -N -L 5432:database-hostname.us-west-2.rds.amazonaws.com:5432 jump-host.example.com
ssh -N -i id_rsa_aws_bastion -L 8887:test-floodlight-db.ccoajwmnofma.us-east-1.rds.amazonaws.com:5432 [email protected] -v
yep, just eyeballing it that looks good to me
Ok. Well, when I originally added the new public keys to the host, I was connecting as root through the console’s EC2 Instant connect (the in-browser terminal for an ec2 instance). Then I ssh’d into the terminal as ec2-user and updated the authorized_keys file… and it worked.
Does that make sense? Can the perms of the user updating the authorized_keys file change the results? Or is the console connection a bad choice?
either way, soooooo annoying.
2021-11-04
Hello,
Sadly I’m working in an environment where most of our resources are not controlled with IaC; or else this would be relatively simple. I’m needing to keep a subset of tags (~5) from an EC2 instance in sync with its associated volumes and snapshots. An easy example of what I would be looking for is if a billing tag changed in 6 months, I would need to replicate out to its volumes and snapshots.
We already have solution with some custom coded automation, but wanted to hear if there are any projects or AWS tools that would provide this type of functionality.
might do more than you need, but there’s this one, https://github.com/GorillaStack/auto-tag
Oh, nice!
cloud custodian has a policy for this too
it’s open source and free so you’d have to come up with your own custodian policy
Does anyone have experience adding splunk universal forwarder to elastic beanstalk? Running into an “issue” that I’m stumped on and not sure where to ask anymore
2021-11-05
Any ideas how to turn off aws cloudwatch alarms at specific hours?
Have a CloudWatch Rule that runs a Lambda that toggles alarms off/on as desired
something like math expresion? Or cloudwatch Event Rules?
this is what we use:
ubuntu@Tony04:~$ crontab -l
#Ansible: mongo db backup
0 3 * * * aws cloudwatch disable-alarm-actions --alarm-names Tony04-CPU; ~/backup.daily.sh
#Ansible: re-enable CPU alarm few minutes after backup
0 4 * * * aws cloudwatch enable-alarm-actions --alarm-names Tony04-CPU
I was referring to CloudWatch Rules as you can use them like cron jobs that trigger jobs (Lambda, etc)
@ikar thank you will go this way, but will trigger those using lambda functions. Filtering through tags
2021-11-07
2021-11-08
If I will use math expression on aws cloudwatch alarms, does it mean that I will increase number of alarm querying per minute by doing that?
Has anyone ever had an issue where you can access an object in a public bucket, but when an ecs container tries to access that same object its access denied?
You may want to make sure there is a VPC endpoint so that the ecs container doesnt try to traverse the internet for it.
Is there a reason that it shouldn’t have access to a public object in a bucket on the same account? When I ssh into the fargate container I can curl objects on buckets that aren’t on the same account
From my Twitter, a thing that may be of interest to y’all//twitter.com/iamvlaaaaaaad/status/1457676356033171465)
Can we see some evidence on this claim that running kubernetes costs $1M annual please
My experience That’s it!
based on what though? the resources? training? the labor? surely you didn’t just pluck this number out of the air
You have to train people, you have to migrate to k8s. After that you have to maintain and operate k8s. All that comes with missed opportunity costs.
I added all those costs
I don’t have an Excel spreadsheet for it, it’s not that precise
awesome GraphQL for AWS https://github.com/cloudgraphdev/cli
Sorry if this is a naive question but does this try to address the pitfalls of using AWS’s advertised GraphQL solution of AppSync? And if so what are they?
no, it’s more of an inventory query solution for cloud resources, based on GraphQL syntax
CloudGraph lets any cloud professional answer questions like, “What KMS keys do I have in us-west-2?”, “How much am I paying for my environment?”, and, “What resources in my production environment aren’t tagged correctly?” in the time it takes to put on the pants you should already be wearing for your next zoom meeting. Ask any question about your cloud environments, and get back answers instantly in a single place with a single standardized API, for all of your cloud providers.
i’m not affiliated, it just looks really interesting
Oh I see it’s just using GraphQL to access thanks
Example: you can create frontend - backend webapp to query your AWS inventories using GraphQL ( new technology) VS RESt API
Imagine how this web app could be used:
• cost estimations
• managment for Landing zone
• Security Checks
General PSA when working with Fargate (via AWS Support):
Below is the summary of our conversation:
You had contacted us since you wanted to know whether STOPSIGNAL or SIGTERM sent to the running container is sent when you stop a task. While on chat, I informed you that unfortunately this feature is not available through ECS yet. The STOPSIGNAL is not supported by ECS and there is no way to specify any other custom signal for StopTask.
We are aware of this limitation and there is currently a feature request to have this available with ECS. However, there is no ETA provided as this requires extensive testing and several approvals before making any changes into production environment. You can keep an eye on the GitHub issue on the containers-roadmap repository [1] for future updates regarding this [2] - https://github.com/aws/containers-roadmap/issues/359
Later, you asked on how common is it for AWS to not have spare capacity and not replace a terminated spot task. (SERVICE_TASK_PLACEMENT_FAILURE event). I informed you that this is a known issue and there is a Feature Request open mentioning the use of “on demand” instances when spot is not available. https://github.com/aws/containers-roadmap/issues/773
2021-11-09
Hi everyone! is there someone here who partecipate to the AWS reInvent?
It could be a great idea to have the opportunity to meet each other!
Personally I’m happy to partecipate, and also, I will present with @Eric Villa Leapp at the Open-Source lounge on Monday at 5 PM
Hope to see you all
@Erik Osterman (Cloud Posse) will you partecipate?
Yep! I will be there
I’ll DM you my mobile
Awesome!
Hi all, I’m using ECS’s Docker Compose integration and everything is working well EXCEPT the output from the docker compose command is so verbose and noisy that even CircleCI wants me to write it to a file and inspect it after because it goes over the 100MB limit that there log output supports.
I’ve also noted that docker compose --loglevel doesn’t work? Has anyone else experienced this?
I guess my general question isn’t entirely AWS related but I do wonder how other infra/ops people quiet logs in this sort of situation? Are y’all just piping to grep or awk/sed?
It specifically the part where it’s creating resources in ECS, so the LogGroup, ECS Service, CloudMap etc. takes some time and each update writes another 20-30 lines to the screen - I don’t want to lose the ability to track progress of my CI pipe, but I also can’t find any flags to shut up ECS/Docker Compose
2021-11-10
Hi all, this is an EKS related question, can someone tell me how to trigger an instance refresh? is it EKS, AMI version or something on the ASG has to change to trigger it? we are looking for the minimum impact so i guess eks update is out of the question, thanks!
kubectl drain $node, then delete instance in question and let autoscaling group spin up a fresh worker node
or, if you need to refresh whole autoscaling group, use https://docs.aws.amazon.com/cli/latest/reference/autoscaling/start-instance-refresh.html
Thank you very much Igor!
2021-11-11
Hi all, which one would you use between Cloudcraft vs. Lucidchart to have automatically updated charts as we edit the infra via console or terraform? thanks!
2021-11-12
2021-11-15
A bit of a networking question, but would be glad if someone could give their take on it. It seems that AWS recently has a way to use NLB with ALB so that you can take advantage of things like SSL termination on the ALB while still using the NLB to do non HTTP traffic (https://aws.amazon.com/blogs/networking-and-content-delivery/application-load-balancer-type-target-group-for-network-load-balancer/). If I didn’t have any non HTTP needs, is there a purpose to use an NLB? The only reason we used an NLB is that our company wanted to use API Gateway and wanted to call an ECS Service behind an ALB, but API Gateway only allows NLBs to use AWS Private Link to create a connection. Otherwise, the ALB has to be public if we want to use API Gateway to reroute it but it ruins the point, since the ALB is still exposed to the public.
I guess I’m confused why AWS doesn’t provide way to use a private ALB through API Gateway directly.
Anyone know if there’s a way to set up bucket replication on the s3 bucket created by the AWS-cloudfront-s3-cdn module? A bit confused over how I can ensure the failover bucket stays up to date.
I’m evaluating migrating my CI pipelines away from CodePipeline and CodeBuild. Any recommendations? Are there hosted CI that is cheaper or comparable to CodePipeline?
depending on where your code is hosted, I would suggest GitHub Actions, GitLab CI, or Bitbucket Pipelines, These services keep your CI close to your code and the configuration for this can be versions right along with the application code. Not sure on exact pricing, but the convenience alone is worth quite a bit in my book.
2021-11-16
Hi there, Does anyone know how to set AWS Managed Apache Kafka data retention to forever?
Hi everyone,
Hoping to get some advice from some of you. I’m currently migrating our infrastructure off of Heroku into AWS. Our dev shop has one monolith and a handful of micro-services that are build with Django. The idea here is to move from a monolith to micro-services. Each application also has a celery worker and beat. I’m planning on migrating over to AWS using Fargate and initially I will be supporting the celery worker and beat with a spot instance. Has anyone had experience doing a similar migration? The two pieces that i’m trying to iron out is how we will be implementing continuous deployment and how service to service communication will work for applications that will only interact within the VPC. From what I’m reading. Codedeploy is used to help with the blue/green deployments via traffic shifting from one image to the other and AppMesh is a service mesh that will enable me to do do service to service communication. Does Codedeploy work with AppMesh?
Quick answer to your question. Yes AppMesh works with AWS.
Sorry, I don’t think my question was clear. Does Codedeploy work with AppMesh? Both are AWS services.
https://github.com/aws/aws-app-mesh-roadmap/issues/72#issuecomment-702894797
This issue made me wonder if it was possible to use both together
It’s not supported out of the box but there are ways around it.
@Yusuf Thank you! Seems like using CodeDeploy along with AppMesh is a bit redundant if I was planning on using it for Blue/Green deployments
hello all, We are facing a really strange issue. We have k8s cluster and have spot fleet for nodes. we have a java app that can start on intel all time and cannot start few times on amd epyc… Any idea ?
instance types are m5 vs m5a
HI, any logs you have to share ?
will try to collect.
need a little time to reproduce.
2021-11-17
aurora serverless? I read it somewhere the volume cluster behind this service is way slower than gp2… is that right ? If yes, then can we somehow move to gp2 or just w/o serverless?
Anyone using Cloud Custodian? As we’re moving to more accounts, we’re finding it desirable to put all the policy lambdas in one account. However, we then run into an EventBridge rule limit which we’re requesting be raised. It also seems that we could change the way the policies are packaged to not require so many lambdas. Seeing if anyone has run into this, or suggests an alternative to Cloud Custodian such as AWS Config.
Do you have AWS Control Tower enabled? are you using AWS Organizations? These tools allow you to manage many accounts under an “organizational unit” or under the entire “org”. I recommend it as you can create SCPs
@julie Yes we are using Control Tower and Orgs. Service Control Policies look like they could handle some of our rules - especially tagging. Probably not everything but worth looking into thanks!
dont quote me, but I think CT and Orgs use AWS config rules under the hood. I love using Config rules so that might be something to look into as well.
2021-11-18
Marc uses it heavily [e] oh lol I was in wrong slack. Marc isn’t in this one
cloud custodian uses aws config for inventory information. I’ve never run into the event bridge limitation tho. have you checked out the cloud custodian gitter?
kapilt and his posse are super helpful and if it’s a limitation they will know immediately
I will try the gitter, good idea. All the policies run in scheduled mode, hence the eventbridge rules, but haven’t tried Config as a data source or the Config Rule execution mode.
does anybody has good config for aurora mysql serverless? Or the default innodb and mysql conf is fine ?
Aurora does nor behave the same as another mysql
I never had to change any innodb settions or anything
and do you use serverless in prod? We are facing with slow query issue.
when we tested serverless it was slow but it all came down to the cpu size etc
and that is when they get you with the chargers
it gets pretty expensive quick
Ok thanks. Have you ever tested v2 serverless?
no
thanks
2021-11-19
oh wow built in ALB basically
can we alias a r53 record to it I wonder?
Looks like the Lambda Function URL functionality has been disabled for now.
must have been released too early
2021-11-20
2021-11-21
Interesting read on AWS Access keys - https://www.hunters.ai/blog/hunters-research-is-aws-recycling-your-access-keys
2021-11-22
2021-11-23
Hi colleagues, we are using aws iot service and we have been wondering whitch is the best place to store securely online some certificates without the need to maintain servers (like hashicorp vault etc.) Can we do it securely with some s3 buckets for example? thanks!
@Almondovar are you joining us for office hours today? #office-hours
we can discuss
hey people, I was asked if I could create regular AWS EKS control plane and make local, on premises bare-metals join that EKS – any thoughts, experience, ideas ?
(last time I did EKS it was ~1.15, I’ve just used terraform with eks module, run terraform apply - it created control plane, worker group within my vpc and that was it)
Amazon EKS Anywhere is a new deployment option that helps you create and operate Kubernetes clusters on your own on-premises infrastructure using VMware vSphere starting today, with support for bare metal deployments coming in 2022.
https://aws.amazon.com/eks/eks-anywhere/
yeah, but can I join EKS Anywhere workers to EKS cluster ? I do not want to manage control plane
well I thought thats what it did but perhaps not
yeah, I was googling and docsing week long, but havent found anything useful. in the end it makes sense, b/c how would EKS support bare-metals (and probably all other options across all the world) options and features? I guess AWS says ‘we do only our workers, get lost’
maybe they will add support joining workers populated from EKS Distro to the regular EKS, but thats not on their roadmap atm
2021-11-24
Hi all, is it possible to have gui access to an ec2 linux server, but use only aws authentication? something like using the web terminal of the SSM, but doing it with gui instead? Thanks!
linux and gui? why you need something like that, GUI on linux is just commands wrapper. Better write some bash scripts to make things easier.
As someone who uses linux as my main operating system every day, I beg to differ.
sure, you can install a desktop environment onto the ec2 instance and connect with SSH and xterm forwarding
there should be plenty of guides if you search something like “linux gui over ssh”
Apologies if i derailed the conversation - the point of this post was not to ask how to install graphical interface on an amazon linux ec2 instance, rather than, solving how people can access it the easiest but secure way
@Alex Jurkiewicz - i found this Run Your Favorite Graphical X Applications Over SSH article, you think that if we use the web terminal of systems manager, it will make it possible to achieve browser access? thank you!
perhaps you can play with pam module, because gui or cli it’s the same
In my view, it’s still early as heck for Proton, but they added Terraform support as a preview now: https://aws.amazon.com/about-aws/whats-new/2021/11/aws-proton-terraform-infrastructure/
TL;DR: Proton wants to be a pretty UI over Terraform (modules). It will help answer questions like “what’s the latest stable version of module X?” and allow non-tech users to use Terraform and Terraform modules by filling variables in a nice UI. Kind of like Service Broker, or a service platform in which you fill in details about what you want and next, next, next, finish (and then Proton commits it to your IaC repo maybe?)
^^^ I could be wrong, this is just my current understanding
So there is:
• CloudFormation (AWS)
• Terraform (Hashicorp)
• CloudFormation CDK (AWS)
• Terraform CDK (Hashicorp)
• Cloud Control (AWS) And now Proton is thrown in the mix.
It seems like AWS thinks neither CF or TF as-is are ideal, and is throwing ideas at the wall to see what sticks. So we devs are going to be stuck in a churning market for the next few years, either continuing to use CF/TF and miss out on innovation, or risk betting on a losing technology
I don’t know about anyone else, but I never adopt anything new from AWS for at least a year if I can avoid it. Their initial GA releases seem too rough around the edges to be worth touching until they’ve been proven and had some time to mature.
Eh, not really. I should do a flowchart for this too
IaC has options:
• Declarative ◦ CloudFormation ◦ Terraform ◦ Bonus declarative using k8s’ continuous reconciliation loop: ACK, Crossplane
• Imperative ◦ CDK which “compiles” to CloudFormation ◦ CDKTF which “compiles” to Terraform ◦ Pulumi which does direct changes (I think/kinda?)
• Tooling ◦ Managed applies and things around that: Atlantis, Terraform Cloud, Spacelift, Env0, Scalr ◦ Nice UIs with “Installation wizard” instead of editing HCL/YAML/Code: Proton They all solve different things. We don’t have one programing language or one IDE, so having one way to do infra as code won’t be a thing. And options are good!
I mean, cloudformation was a hackathon project that was built in 24 hours to help provision internal aws services. It wasnt originally designed for general availability
Reading through the way Proton “compiles” things, I’d liken it to something like Terragrunt + TFC
Hmmm, it relies on CodeStar for repository connections, which somehow doesn’t support CodeCommimt? Nor GitLab? But it does support BitBucket?
Can’t wait to hear what Cory Quin has to say about that
Blog post with more info on the new features and terraform support… https://aws.amazon.com/blogs/aws/new-aws-proton-supports-terraform-and-git-repositories-to-manage-templates/
Ahh, not like TFC then…
… AWS Proton is not the one managing the provision of infrastructure. Therefore it is important that in the process of provisioning the infrastructure, there is a step that notifies AWS Proton of the status of the deployment.
2021-11-25
in case folks missed https://aws.amazon.com/blogs/aws/aws-free-tier-data-transfer-expansion-100-gb-from-regions-and-1-tb-from-amazon-cloudfront-per-month/ ( after some behind the doors info been mentioned https://blog.cloudflare.com/aws-egregious-egress/. Is good to se this sort of actions from folks like AWS and co.
hell all. we are using m5 and m5a instances (intel and amd cpu). We see significant performance differences between them. is that possible ? Our java code times out on AMD cpu and we see strange issues in our app if we run on m5a types…
hi folks, i’m trying to find a sweet spot to improve the experience of developing / test lambda or ecs containers + rds (in private subnets, of course ) using least privileged access.
*Context*
Have a bunch of various solutions using lambda deployed inside VPC to interact with RDS deployed on private subnet. Equally same apply when having a container running as part of ECS talking with RDS.
*Challenges*
• developing/ debugging the lambda locally while setting breakpoints connected to RDS is very painful as you need to first overcome the network access: using a Bastion and a ssh tunnel over it is okay-ish although rough …
• because the initial RDS and the additional services were deployed using TF and the least privileges, running the code locally requires extensive effort to go over the IAM *Tried out and … failed*
• thought of moving the local dev env to Cloud9 where i could associated an existing/ working IAM locked policy as extended instance profile but … Issue #1 : by default Cloud9 gets deployed in public subnet. Issue #2: created an env using private subnet and SSM but sadly i couldn’t modify the associated instance profile from Console
Has anyone faced similar situations and if so able to share their stories ?
Have you tried localstack?
i have tried in the past life but it was a hit and miss in terms of supported services. Since then i’ve stayed away
It works pretty well for lambda at least. Can condition the endpoint for the AWS session on the env LOCALSTACK_HOSTNAME. But yeah if you start using services/APIs that aren’t yet implemented, then you end up contributing to moto a lot
Something else you might try, to simplify the networking between local dev and remote rds in private subnets, is tailscale…
that could do it in a way as i just came across https://tailscale.com/blog/github-codespaces/ time to explore. THX
Tailscale is basically magic as far as I’m concerned. One of those things that makes computers good again
I suggest using a system to upload code changes directly to an already-existing function in your development environment.
For example, we deploy dev environments using Terraform (same IaC as production). But when developers are using a dev environment, generally they only change function code. So we have a little script for them which compiles their code locally and updates an existing function. It takes ~5secs to run, which means the dev iteration loop is short.
that is an idea indeed, thanks for sharing @Alex Jurkiewicz
Hey, any good info on multi region active-active Architectures? I’d like to route users to their assigned aws region based on authentication cookie. e.g if user has eu cookie they are redirected to us-west-1 alb
you can’t do that without application logic. That is, you would need application code to read the cookie, decide the destination, and redirect to an internal hostname (or something along those lines)
The specifics of how you should implement multi-region active-active are very dependent on how your application works, and what metrics you care / don’t care about.
Take a look at AWS Global Accelerator, and RDS/DynamoDB global databases. They are often two key technologies to build a solution out of
2021-11-26
2021-11-28
2021-11-29
This’ll make a bunch of people very happy: https://aws.amazon.com/blogs/aws/announcing-pull-through-cache-repositories-for-amazon-elastic-container-registry/
@Erik Osterman (Cloud Posse) ^^
PM said on Twitter that Dockerhub support is coming: https://twitter.com/Sravan_R_/status/1465378657216593923
From this GitHub issue: all Docker official images are now mirrored to ECR Public: https://gallery.ecr.aws/docker/
Hopefully soon folks will move out of DockerHub, since the quota limit is a pain to live with (especially those with small pockets )
Anyone else at reinvent?
I am by the expo
Me too. What area?
2021-11-30
Hi guys, is there a better way to track request count with aws application load balancer? I have connected cloudwatch metrics to aws managed grafana to display daily requests on the dashboard but the data provided isn’t useful. We needed something to give us a figure of daily traffic. Any other recommendation to achieve this is appreciated too.
Thanks
anyone @ aws re:Invent? would love to meet community people
Just pulling in…