#aws (2023-11)
Discussion related to Amazon Web Services (AWS)
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
2023-11-01
Hello all - I have a question. I have a bucket in the AWS console, and I configured it in Terraform. The bucket has storage class = GLACIER , and I want to update it to INTELLIGENT_TIERING can we just replace it, or do we need to add an additional resource storage class = INTELLIGENT_TIERING (is it a good practice). If replaced, will it affect the already stored data?
TIA
@smaranankari.devops
When you want to change the storage class of an S3 bucket using Terraform from GLACIER to INTELLIGENT_TIERING, it is not necessary to add an additional resource. You can modify the existing resource definition in your Terraform configuration. Terraform is designed to manage the resources it defines, so you can use the same resource block to update the storage class.
N.B: If you expect any affect of store data you can enable bucket versioning it will keep your data safe.
Anyone here have worked with Nitro Enclaves?
I’m basically trying this: https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-refapp.html
and after running this aws ec2 --region us-east-1 associate-enclave-certificate-iam-role --certificate-arn arn:aws:acm:us-east-1:123456789012:certificate/d4c3b2a1-e5d0-4d51-95d9-1927fEXAMPLE --role-arn arn:aws:iam::123456789012:role/acm-role
this encryption KMS key in the output "EncryptionKmsKeyId": "a1b2c3d4-354d-4e51-9190-b12ebEXAMPLE"
is not accessible
even if we add kms:* we get 403 on accessing the KMS key
which is created by the association of the CERT and role ( not by us)
"userAgent": "aws-nitro_enclaves-sdk-c/v0.4.0-1-g550f731",
"errorCode": "AccessDenied",
"errorMessage": "User: arn:aws:sts::12121212121:assumed-role/nvidia-lab-ec2role-proxy-server/i-0eee1113333444 is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access",
Does the user have the permission ec2:AssociateEnclaveCertificateIamRole, ec2:GetAssociatedEnclaveCertificateIamRoles, and ec2:DisassociateEnclaveCertificateIamRole ?
yeah should have, how about step 4? Associate the role with the ACM certificate
aws ec2 --region region associate-enclave-certificate-iam-role --certificate-arn certificate_ARN --role-arn role_ARN
and step 5 and 6 were also done I assume
and nginx or apache has access to the cert?
ic step 4/5 were done by others, it may be the cause
when you start the Enclave it errors out on the Decrypt
this command aws ec2 --region region associate-enclave-certificate-iam-role --certificate-arn certificate_ARN --role-arn role_ARN is the one that creates the kms key for you
2023-11-06
Hello All - In EKS, can a cluster role binding bind a k8s role to a IAM role/user or it has to be a k8s group/user ?
@Jeremy G (Cloud Posse)
You have to map the IAM role to a Kubernetes user or group first. Then you can use ClusterRoleBindings to Bind those users and groups to Kubernetes roles. See https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
2023-11-07
Does anyone have experience with AWS CDK (preferably in python)? I have a small issue that I have been stuck on for an hour now. I am letting a user pass in a CFN parameter (when the template is synthesized) and one of those parameters is a list of subnet ids.
In eks_stack.py:
subnet_ids_param = CfnParameter(
self,
"SubnetIds",
type="List<AWS::EC2::Subnet::Id>",
description="The list of IDs of the private isolated subnets",
allowed_pattern="^(subnet-[0-9a-f]{8,},)*subnet-[0-9a-f]{8,}$",
constraint_description="must be a valid list of subnet IDs.",
)
subnet_ids = subnet_ids_param.value_as_list
vpc_subnets = ec2.SubnetSelection(subnets=subnet_ids) <--- where i am having an error
Error:
TypeError: type of argument vpc_subnets must be one of (Sequence[Union[aws_cdk.aws_ec2.SubnetSelection, Dict[str, Any]]], NoneType); got aws_cdk.aws_ec2.SubnetSelection instea
pretty sure you need something like:
vpc_subnets = ec2.SubnetSelection(subnets=[ec2.Subnet.from_subnet_id(self, f"Subnet{i}", subnet_id) for i, subnet_id in enumerate(subnet_ids)])
I did try that, to pass a sequence of ec2.SubnetSelection objects, let me see…
Here’s the types from my execution:
------------------------------------------------------------
subnet_ids: ['#{Token[TOKEN.837]}']
type for subnet_ids: <class 'list'>
vpc_subnets: SubnetSelection(subnets=['#{Token[TOKEN.837]}'])
type for vpc_subnets: <class 'aws_cdk.aws_ec2.SubnetSelection'>
------------------------------------------------------------
Here’s the types from your modified edit:
------------------------------------------------------------
subnet_ids: ['#{Token[TOKEN.836]}']
type for subnet_ids: <class 'list'>
vpc_subnets: SubnetSelection(subnets=[<jsii._reference_map.InterfaceDynamicProxy object at 0x7fd6db178e10>])
type for vpc_subnets: <class 'aws_cdk.aws_ec2.SubnetSelection'>
------------------------------------------------------------
same error:
File "/home/cdk/.local/share/virtualenvs/app-4PlAip0Q/lib/python3.11/site-packages/typeguard/__init__.py", line 558, in check_union
raise TypeError('type of {} must be one of ({}); got {} instead'.
TypeError: type of argument vpc_subnets must be one of (Sequence[Union[aws_cdk.aws_ec2.SubnetSelection, Dict[str, Any]]], NoneType); got aws_cdk.aws_ec2.SubnetSelection instead
same object types, but with subnets being a different type This is being passed to the FargateCluster construct
2023-11-08
2023-11-09
Hello, does anybody know whether there’s anything I can run on a machine on-prem to check whether it’s accessing S3 via Direct Connect and not the public internet?
if you are doing it over http, access logs. if api, then cloudtrail (it should have the calling ip). If you are performing data events, e.g. s3 getobject, then turn on data event logging in cloudtrail for that bucket
Thanks! I’m looking at the data event logs: what in there would indicate that the request went or didn’t go via Direct Connect?
if it captures the ip, the ip should be different
also, if there is a marked difference in transfer speeds for transfers over the public internet vs direct connect, that might give you another way of checking the routing
i suppose you could also ask your network guy to check if the s3 addresses have been advertised to the on-prem network, which is what should happen when you set up your public VIFs for direct connect
if it captures the ip, the ip should be different The captured IP is indeed different from the IP of the on-prem machine - but does that prove that the request was routed via Direct Connect? How do I know it’s not for a firewall or some network device between the machine and the public internet? This IP address points to something that is not on-prem though.
The speeds are different, but the premises have a much better internet connection than I have! :sweat_smile: , so I don’t think it’s a definitive test.
In the interest of time, I’m trying to find a way that doesn’t require involving the network people (I was hoping there was something like traceroute that would do the trick).
cos if it goes over the internet, it will have the public ip of your public router. If it goes over DX, the ip should be something that is different from the ip of your public router. Just like egress out of a VPC to an external client - the external client sees the ip of your nat gateway
I think that traceroute from the host should also show you different path than normal traffic going via the internet (at least few first hops)
Thanks both. I tried traceroute -n -T -p 443 [s3.amazonaws.com](http://s3.amazonaws.com) and the first few hops were the same as accessing a public site, but a) I’m not sure that’s the right command to run b) I don’t recognise the hops anyways. So I may have to accept defeat and leave this one to the networking team… I guess I’ll know the answer in a couple of months
you could launch an ec2 instance and do a traceroute to that. the exit point from your intranet should be different to exit point for the public internet. At least then you will know what the routing path should look like when going over direct connect
You could be going via your http proxy server too which a traceroute wouldn’t show that. ESP if your using a browser to access s3
@Fizz If I used an EC2 instance, wouldn’t that go via the private VIF, rather than the public VIF to S3? Just wanting to make sure I don’t compare apples to oranges! @Gary Mclean I was using the aws cli to access s3, but I couldn’t see anything useful in the traceroute output so maybe that’s case
at this point, you are trying to get an idea of what the routing looks like when it leaves your company’s intranet, to compare with the s3 routing. E.g get the routing for google.com, and then get the routing for an ec2 instance, - both should have different exit points from your intranet. Then see which more closely resembles the routing for when do a traceroute to s3
An EC2 can use a private endpoint (VPC IP Address) or s3 gateway endpoint which creates a prefix list in your route tables to route PUBLIC ips via the S3 gateway
2023-11-10
hello all, we would like to start to use ecr, but my manager has a question. is the private ecr data out transfer is free until it remains in same region? and not limited?
based on this I think it will be free: https://aws.amazon.com/blogs/containers/understanding-data-transfer-costs-for-aws-container-services/ and based on calculator
yeah, but we don’t have pro support, that’t why I tried to ask here, but what I found on aws docs, it should be free.
2023-11-11
2023-11-13
I am facing an issue with my Laravel application deployment on AWS ECS. The deployment process involves Jenkins, AWS ECR, and ECS. The new task is created, but there’s an “Access Denied” error connecting to the RDS database. Due to this issue after some time service is automatically deleted
I have provided my deployment files for reference. JenkinsFile:
pipeline {
agent any
environment {
AWS_ACCOUNT_ID="794664785634"
AWS_DEFAULT_REGION="us-east-1"
IMAGE_REPO_NAME="product-mangement"
IMAGE_TAG="${BUILD_NUMBER}"
REPOSITORY_URI = "794664785634.dkr.ecr.us-east-1.amazonaws.com/product-mangement"
ECS_CLUSTER = "product-mangement"
ECS_SERVICE = "product-mangement"
}
stages {
stage('Checkout Latest Source') {
steps {
git branch: 'master',
url: '<https://github.com/jhon-123/product-mangement>',
credentialsId: 'jenkins_pta'
}
}
stage('Logging into AWS ECR') {
steps {
script {
sh """aws ecr get-login-password --region ${AWS_DEFAULT_REGION} | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"""
}
}
}
// Building Docker images
stage('Building image') {
steps{
script {
dockerImage = docker.build "${IMAGE_REPO_NAME}:${IMAGE_TAG}"
}
}
}
// Uploading Docker images into AWS ECR
stage('Pushing to ECR') {
steps{
script {
sh """docker tag ${IMAGE_REPO_NAME}:${IMAGE_TAG} ${REPOSITORY_URI}:$IMAGE_TAG"""
sh """docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${IMAGE_REPO_NAME}:${IMAGE_TAG}"""
}
}
}
stage('Deploy to ECS') {
steps {
sh "aws ecs update-service --cluster ${ECS_CLUSTER} --service ${ECS_SERVICE} --force-new-deployment"
}
}
}
}
Dockerfile:
# Use the official PHP image as a base
FROM php:8.1-fpm
ENV COMPOSER_ALLOW_SUPERUSER 1
# Arguments defined in docker-compose.yml
ARG user
ARG uid
USER root
# Install system dependencies
RUN apt-get update && apt-get install -y \
git \
curl \
libpng-dev \
libonig-dev \
libxml2-dev \
zip \
unzip
# Clear cache
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
# Install PHP extensions
RUN docker-php-ext-install pdo_mysql mbstring exif pcntl bcmath gd
# Get latest Composer
COPY --from=composer:latest /usr/bin/composer /usr/bin/composer
# Create system user to run Composer and Artisan Commands
# RUN useradd -G www-data,root -u $uid -d /home/$user $user
# RUN mkdir -p /home/$user/.composer && \
# chown -R $user:$user /home/$user
# Set the working directory
WORKDIR /var/www
# Copy the project files into the container
COPY . /var/www
# Copy .env.example to .env
COPY .env.prod .env
# Install Composer dependencies
RUN composer install
# Cache configuration
RUN php artisan config:clear
RUN php artisan config:cache
# Generate Laravel application key
RUN php artisan key:generate
# Copy the start script into the container
COPY script.sh /var/www/script.sh
# Make the script executable
RUN chmod +x /var/www/script.sh
# Expose port 8000
EXPOSE 8000
# show message
RUN echo "ehllo"
# Run the start script as the CMD
CMD ["/var/www/script.sh"]
script.sh:
#!/bin/sh
# Run Laravel migrations
php artisan migrate
# Seed Database
php artisan db:seed
echo "seeded successfully"
# Start the Laravel application
php artisan serve --host=0.0.0.0 --port=8000
Problem: The new task is created, but there is an “Access Denied” error connecting to the RDS database. The .env.prod file contains the correct RDS connection details.
.env.prod:
APP_NAME=Laravel
APP_ENV=prod
APP_KEY=base64:LyxaydSCa8HIgUdaLLQCPehtSK2siVr0o+bT6jcXWmM=
APP_DEBUG=false
APP_URL=<http://localhost>
LOG_CHANNEL=stack
LOG_DEPRECATIONS_CHANNEL=null
LOG_LEVEL=debug
DB_CONNECTION=mysql
DB_HOST=product-management.c7ebhtqyydqk.us-east-1.rds.amazonaws.com
DB_PORT=3306
DB_DATABASE=product-management
#DB_USERNAME=laravel
#DB_PASSWORD=secret
BROADCAST_DRIVER=log
CACHE_DRIVER=file
FILESYSTEM_DRIVER=local
QUEUE_CONNECTION=sync
SESSION_DRIVER=file
SESSION_LIFETIME=120
MEMCACHED_HOST=127.0.0.1
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=6379
MAIL_MAILER=smtp
MAIL_HOST=mailhog
MAIL_PORT=1025
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ENCRYPTION=null
MAIL_FROM_ADDRESS=null
MAIL_FROM_NAME="${APP_NAME}"
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_DEFAULT_REGION=us-east-1
AWS_BUCKET=
AWS_USE_PATH_STYLE_ENDPOINT=false
PUSHER_APP_ID=
PUSHER_APP_KEY=
PUSHER_APP_SECRET=
PUSHER_APP_CLUSTER=mt1
MIX_PUSHER_APP_KEY="${PUSHER_APP_KEY}"
MIX_PUSHER_APP_CLUSTER="${PUSHER_APP_CLUSTER}"
What could be causing the “Access Denied” error in the deployment process, and how can I resolve it? Any insights or suggestions for troubleshooting would be greatly appreciated.
Could trail events logs all API calls to your account, and may include a reason why you’re getting access denied. That’s where I’d start.
could be the IAM role associated with the task definition or the IAM role associated with the task or the Security groups related to the task any number of reasons it just a process of elimination to work out where it is
Hello Team - I am creating a private EKS cluster using Terraform. I am getting a error stating “Error: waiting for EKS Add-On (demcluster: vpc-cni) create: timeout while waiting for state to become ‘ACTIVE’ (last state: ‘CREATING’, timeout: 20m). Has anyone seen the error before? I get the same error for coredns aswell. It is in a VPC with no NAT gateway so I have vpc endpoints for the services as per the documentation (EC2, S3, EKS, ASG,ECR etc)
@Andriy Knysh (Cloud Posse) @Dan Miller (Cloud Posse) @Jeremy G (Cloud Posse)
first thought, try to increase the timeout, sometimes 20 minutes is not enough to create the cluster and all the addons
I think you can get this error from not having the proper IAM permissions attached to the Kubernetes service account or from not having the service account attached to the add-on. This could possibly also come from running out of private IPs in your VPC address space. I suggest increasing the timeout to 30m and then looking at the plugin progress/logs via the AWS Web Console or Kubernetes tools.
Thanks for the response everyone. I figured out the issue. There was a custom policy on S3 endpoint which wasn’t permitting the required starport registry. Upon adding it I was able to deploy the addons successfully.
2023-11-14
2023-11-15
Is there a way to share a transit gateway in another region or what would be the best approach? Maybe just use peering instead of adding another gateway?
of course theres a way to do that
2023-11-16
2023-11-18
2023-11-22
Using ecr private repo I have a vpc (same region) with transit gateway attachment attached to another vpc (nat vpc). Is the traffic free or will be charged? if charged, then how can I pull images for free ?
Service endpoints for ECR in your VPC will make it free, you’ll still be charged for network interface but not for traffic
thanks
so I will pay for vpc endpoint but not for dat
I was following this guide https://docs.aws.amazon.com/singlesignon/latest/userguide/gs-gwp.html and we got it to work, but somehow, we see no groups being synced, only users. Has anyone had a similar issue?
@Jeremy G (Cloud Posse) @Andriy Knysh (Cloud Posse) are experts in that field
the answer is :
SCIM automatic synchronization from Google Workspace only supports provisioning users; groups aren't automatically provisioned. You can't create groups for your Google Workspace users using the AWS Management Console. After provisioning users, you can create groups using a CLI or API operation
so you still need ssosync
@Dan Miller (Cloud Posse) @Jeremy White (Cloud Posse) Please confirm that we still need ssosync to sync Groups with Google
We have another long thread on the topic, and we don’t need ssosync anymore. Apparently there’s a very specific step on the Google side to enable automatic provisioning, but @Jeremy White (Cloud Posse) please verify
Sorry I lost track of this. Basically groups can either be sync’d with ssosync or manually made. Once they are present, you should see that Workspace auto-provisioning takes over assigning/unassigning users. That’s what I heard back from customers. To be clear here, I still have no direct access to any Google Workspaces, so I’ve never verified it directly. Everything has been over long calls.
Customer feedback has been that, since the groups have to be IaC in the first place when giving them permission sets, it really didn’t matter if they got made via terraform.
Your best bet is to use the identity store resource for groups and see how that fairs.
unfortunately, Cloud Posse does not have any modules nor components at this time which manage the aws_identitystore_group resource. But our terraform-aws-sso module will utilize them after you’ve made them, should you prefer using that to save yourself time making permission sets.
we’re adding this to the component as well
2023-11-23
Hello Team - I am deploying a private EKS cluster in a private VPC using terraform. I added subnets the tags of “Key – [kubernetes.io/role/internal-elb](http://kubernetes.io/role/internal-elb) ; Value – 1” & “Key – [kubernetes.io/cluster/my-cluster](http://kubernetes.io/cluster/my-cluster); Value – shared” as per the documentation. it gets deployed in my personal account successfully. But in another account (dev) account, I get a context deadline exceeded error. When i describe the service nginx-ingress-controller I see a event stating that “Error syncing load balancer: failed to ensure load balancer: could not find any suitable subnets for creating the ELB”. When we manually add a service annotation of “service.beta.kubernetes.io/aws-load-balancer-internal=true” and “terraform apply” it succeeds without any error. Is the annotation (aws-load-balancer-controller-internal) required even for nginx controller in EKS? Did anyone try to add the annotation using terraform? I am unable to find the right syntax to set on the helm_release resource for nginx-ingress-controller helm chart.
The subnet settings and LB annotation are separate settings
would be easier if the subnets are also managed by TF
Thanks Hao. What could be the fix if my subnets are managed by TF?
that would be much easier, can add tags in TF when the subnets are created
Oh my subnets are already tagged. But I still get the error.
may need to check the logs of ALB controller
or the internal ALB tag/annotation may not be correct?
That’s right @Hao Wang . When I add annotation to the svc service.beta.kubernetes.io/aws-load-balancer-internal=true it works. My question is how can I do that while deploying the helm chart of nginx-ingress-controller directly. Or add the annotation as part of the terraform execution.
oh ic, helm chart normally has annotations in values yaml file
Yeh. I was having hardtime to find the right syntax. We found the right syntax. I will post it for others to use in a bit.
I have set it up before, should be a quick fix
2023-11-24
2023-11-25
2023-11-27
one screenshot “leaks” upcoming feature that will replace aws-auth configmap
Who is at re:Invent?
2023-11-28
2023-11-29
Hello Team - I am using the fully-private-cluster terraform blueprint to create a private EKS cluster with 2 managed node groups. I am trying to restrict the ECR repositories EKS cluster can pull images from by modifying the AmazonEC2ContainerRegistryReadOnly policy(custom policy) to contain specific repositories instead of all. This setup works for the first node group. But for the second node group it fails saying policy with the same name exists. How can I make it use the existing IAM policy if it exists? I tried to use aws_iam_policy data source but now it fails on node group 1 execution itself as the IAM policy doesn’t exist with that name yet. Any guidance on troubleshooting it will be of great help.
It seems that your terraform code creates IAM policy in the same “module” that creates node group. I see two options to solve the issue
- Create IAM policy somewhere external to “module” that creates node group and then pass arn of the role as inputs to the module. Or you can use data instead of passing it as input
- Add “node-group” name as suffix to IAM policy. That way will create 2 identical IAM polices - one per node group. Technically it would work the same, terraform will manage consistency between that IAM polices on each run. It does not increase costs on AWS.
got it. Thank you Igor. I created policy externally and passing it now.