#aws (2024-06)
Discussion related to Amazon Web Services (AWS)
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
2024-06-03
Hi guys, I’m preparing an slide in a training to some mentees, I wanna present this info to explain the AWS Developer Tools in a single chart, and this summary:
• CodeStar <- Just an interface to manage several pipelines
• Cloud9 <- Just an IDE, like VSCode but in the cloud
• CodeBuild <- Similar to Github Actions
• CodePipeline <- A group of codebuilds/codedeploys
• CodeDeploy <- To deploy your code, usually to move your code from S3 to EC2
• CodeCommit <- Like GitHub My question is: does this makes sense to you, can I make it clearer? What would you change?
i feel like CodeBuild is actually pretty strong, though certainly feels different than github actions. CodeCommit is pretty awful/weak though
i have loads of use cases where i prefer codebuild over github actions, particularly anything that requires credentials or network-level access to vpcs
yeah I think CodeBuilder is a lot more powerful than GitHub Actions…it’s just that you have to build out everything. So in turn, GitHub Actions is easier to get started with and covers most scenarios for most teams and therefore is “better” in most cases.
there’s even a slick new feature that blurs the lines, where your github action runs the codebuild job directly
Ok, we can probably say that CodeBuild is harder, but not weaker
i mean, is it hard to run a shell command? that’s basically what a buildspec is doing. just a bunch of shell commands
Ok, so we can probably remove that comment, to avoid bias
i suppose if you think of github actions as a library of published “modules” that are on their own running some defined set of shell commands, then maybe so. but then of course you run into the situation where the action doesn’t support your use case. and now you have to fix the upstream, or fork it and run your own, or fall back to shell commands anyway
yeah that could be a relevant thing
just wanted to say thanks for sharing your thoughts. i never used any aws ci/cd tools, i always thought “why would i besides github actions”, but your comments made me very curious.
The secret is that most of these tools have parity in one way or another, they’re just aligned and opinionated to a sub-set of use cases. Except Jenkins, that’s fairly universally disliked.
For me the biggest advantage of GH Actions is the reusability of actions and workflows (let’s not talk about security right now) and also that someone else is managing the infrastructure .
hah, you just hit on the two biggest reasons that teams choose not to use GitHub Actions.
GHA marketplace is a significant supply chain attack risk. best mitigation is to enforce version pinning (to the hash, not just version tag), but brand-jacking and typo-squatting are possible too. People rarely actually check the code of the action they’re using. People assume nobody would put malicious code into OSS, <insert obligatory xz reference>.
Managed infrastructure is great…until you realize how much cheaper it can be at scale to run “your own”. I had one pipeline that was going to be like $4k/month on GHA (needed the largest workers they offered), set up self-hosted runners with EC2 Spot fleet and it was like $200/month. That and BYO can immensely speed up pipeline run times on occasion.
That’s why I was jokingly saying let’s not talk about security . But I didn’t necessarily mean marketplace actions. I just meant the ability to easily reuse actions and workflows. The cost is different for different orgs. In my case it’s much cheaper than your anecdote.
I get, I was just expanding for the lurkers
2024-06-04
Hello! How can I add for the msk-apache-kafka-cluster terraform module custom server configs for kafka e.g.: kafka_configuration_properties = { “auto.create.topics.enable”: true }
Error: Unsupported argument │ │ on main.tf line 91, in module “kafka”: │ 91: kafka_configuration_properties = { │ │ An argument named “kafka_configuration_properties” is not expected here.
which version of the module is used?
source = “cloudposse/msk-apache-kafka-cluster/aws” version = “2.4.0”
yeah, kafka_configuration_properties is not a variable for this module, do you use a tutorial?
I am not using any tutorial. In order to make the cluster publicly accessible, the following setting is required:
allow.everyone.if.no.acl.found = false
how to achieve this ?
this variable may be what you’re looking for, https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/blob/2.4.0/variables.tf#L204
yeah, confirmed it is
lovely, thanks alot
np
2024-06-05
Has anyone ever been confronted with: “Parameter: SpotFleetRequestConfig.IamFleetRole is invalid. ” when doing a spot-request ? The Role, Trust Policy and Policy all look fine to me. Some Redditor had the same unsolved question. It works in one region, but not in the other, it does not look like a policy issue to me.
2024-06-06
2024-06-07
Would appreciate any on this insane AWS Amplify Hosting issue: https://github.com/aws-amplify/amplify-hosting/issues/2563
cc @mike186
Thank you!
2024-06-10
Issue: Application Performance Explanation: We have deployed all our microservices on AWS EKS. Some are backend services that communicate internally (around 50 services), and our main API service, “loco,” handles logging and other functions. The main API service is accessed through the following flow: AWS API Gateway -> Nginx Ingress Controller -> Service. In the ingress, we use path-based routing, and we have added six services to the ingress, each with a corresponding resource in a single API gateway. Our Angular static application is deployed on S3 and accessed through CloudFront. The complete flow is as follows: CloudFront -> Static S3 (frontend) -> AWS API Gateway -> VPC Link -> Ingress (Nginx Ingress Controller with path-based routing) -> Services -> Container. Problem: Occasionally, the login process takes around 6-10 seconds, while at other times it only takes 1 second. The resource usage of my API services is within the limit. Below are the screenshots from Datadog traces of my API service:
• Screenshot of the API service when it took only 1 second
• Screenshot of the API service when it took 6-10 seconds Request for Help: How should I troubleshoot this issue to identify where the slowness is occurring?
@Jeremy White (Cloud Posse)
I take it you have reviewed the flame graph of the login service and done a profile of it to rule out the login service itself being the bottleneck?
I only ask because on your second image there are 4 times as many spans being indexed, so it is making me wonder whether between the two screenshots something has invalidated a cache your app relies on and it is having to rebuild that? Maybe a new pod of that service has been spun up from a scaling event and your containers don’t come with the cache prewarmed?
@Jeremy White (Cloud Posse) bumping this up
Also, as per the screenshot live-locobuzz-api-sql-server is taking almost 5x response time. Did you had a chance to check which query is expensive in the later one?
Yeah, to sort of condense what’s mentioned above, APM allows you to instrument code, effectively setting timers at different points of execution which resolve when calls return.
You’ll want to study spans captured to see which ones have the majority of time, and then further dig into (instrument) those spans
If you cannot instrument any deeper into the call (i.e. the span leaves to another service), then you’ll need to see if you can instrument that resource/dependency
if you post more about the calling functions and their dependencies, we can advise how to proceed
Hi, I’m trying to understand why the ecs cluster module is trying to recreate the policy attachments every time I add more than one module instance via a for_each. The plan shows the arn will change, but it’s a AWS managed policy, so it won’t change:
update
policy_arn :
“arniam:policy/AmazonSSMManagedInstanceCore”
change to Known after apply Forces replacement
the resource address is:
module.ecs_clusters[“xxx”].module.ecs_cluster.aws_iam_role_policy_attachment.default[“AmazonSSMManagedInstanceCore”]
@Jeremy White (Cloud Posse)
probably best to use #terraform for terraform questions
@Erik Osterman (Cloud Posse) sorry, I thought this was the terraform aws channel. Will post in terraform then
1
I’m getting an error when creating ec2-instance, trying to reference the private subnet for a dynamic_subnet I created, any ideas how to reference the private_subnet_id into ec2?
subnet = module.dynamic_subnets.private_subnet_id
That was the wrong screenshot please see below.
well, this was one short-term fix: subnet = element(module.dynamic_subnets.private_subnet_ids, 0)
I guess it randomly chooses which AZ it goes into?
2024-06-11
This policy we have to enforce mutilaz on elasticache clusters. Do we have some such policy to enforce Multi-AZ in RDS Aurora and Elasticsearch?
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Deny",
"Action": [
"elasticache:CreateCacheCluster",
"elasticache:CreateReplicationGroup"
],
"Resource": [
"arn:aws:elasticache:us-east-1:4852:replicationgroup*",
"arn:aws:elasticache:us-east-1:4852:cluster*"
],
"Condition": {
"StringNotEqualsIgnoreCase": {
"elasticache:MultiAZEnabled": true
}
}
}
]
}
We are creating AWS resources using Terraform which is using Terraform role. So we donot want to create datastores if Multi-AZ is not enabled on them. So if we have such policy for RDS Aurora and ElasticSearch then that would be great!!
RDS supports this condition in IAM policies https://docs.aws.amazon.com/service-authorization/latest/reference/list_amazonrds.html#amazonrds-rds_MultiAz
@Andriy Knysh (Cloud Posse) But this doesn’t work in RDS Aurora
i just searched for that. You can search for IAM policy keys in RDS Aurora
I couldn’t find any such parameters there. Do we have some other way to enforce such rule?
looks like there is no such policy conditions for Aurora (see https://stackoverflow.com/questions/73164178/iam-policy-to-force-enable-aurora-read-replica), although the comment was from 2 years ago
I am thinking to go for sentinel policies
or if you have idea about open policy agent. which one would be better?
where are you thinking to run the OPA agent?
in tf cloud
@Mehak do you need further assistance here?
@Gabriela Campana (Cloud Posse) yes
@Andriy Knysh (Cloud Posse)
i’m not familiar with TF cloud sentinel policies. There are many docs about that, e.g. https://developer.hashicorp.com/terraform/cloud-docs/policy-enforcement/sentinel. Maybe other people can help here
I could not find anything related to TF cloud sentinel policies in Cloud Posse projects history. @Erik Osterman (Cloud Posse) to confirm if we have any SME on TF cloud sentinel policies
2024-06-12
Can someone help me with sentinel policy to enforce multi-az on rds aurora and elasticsearch clusters. I will create policy in TF cloud?
2024-06-13
2024-06-17
I don’t think the updated cert chain will be added to this npm module before August 22. https://github.com/mysqljs/mysql/blob/master/lib/protocol/constants/ssl_profiles.js
It’ll be the kick that some need to get on the mysql2 module.
2024-06-19
2024-06-20
2024-06-21
Hi Folks, Have you ever wanted a generative AI assistant that could go through S3, Redis, RDS, Confluence, or an internet web crawler and answer questions about your product using generative AI? If you’re building this from scratch, think again. Check out Amazon Q! and How did First Orion optimize their workflow with Amazon Q? Check the link above for a detailed post describing the architecture and other aspects. Feel free to comment and share your views.
2024-06-30
Terrascan isn’t properly identifying any of the cloudposse modules for compliance. Is there a scanner that works with cloudposse modules?
The problem is complaince is based on the parameters you pass, and standards can be contradictory, for example requiring different retention periods.
We ensure our modules are sufficiently parameterized, but the end user needs to pass the parameters