#aws (2022-02)

Good advice? No… Switch to Linux…

@emem you can upgrade it outside of the maintenance window by modifying the rds cluster/instance, set it to the desired version, and click apply immediately

Thanks

fargate has a limit of 16gb of ram

so i using fargate in eks

not sure if EKS fargate have the same

do i need to increase this memory

https://docs.aws.amazon.com/eks/latest/userguide/fargate-pod-configuration.html

Since Amazon EKS Fargate runs only one pod per node, the scenario of evicting pods in case of fewer resources doesn’t occur. All Amazon EKS Fargate pods run with guaranteed priority, so the requested CPU and memory must be equal to the limit for all of the containers. For more information, see Configure Quality of Service for Pods in the Kubernetes documentation.

oh ok thanks alot.

so in my case i have two containers in the pod one for my app and one for datadog and they node cannot manage them both. Not sure if there is a way to request more cpu on fargate

what if you set both containers to a lower setting to the maximum Node setting?

you always need to account for OS mem usage, EKS api controller running, other networking virtualization memory usa etc and what is left is what you can use

there is always and overhead on any type of virtualization

so you can’t just overcommit

true.

what i did as a stop gap was to remove the config for cpu and memory from the both containers configuration so it can provision the amount it needs for itself, until i can fix the issue

That’s an odd one. I am not sure what OCB stands for and googling for it didn’t haven’t lucky either. Others have asked about it too, but no answer. I would ask your AWS account manager.

Is that the name of your distribution ?

I found a document (written mostly in Chinese) that seems to indicate that this is Out of Cycle Billing. I noticed here that they refer to “out-of-cycle” charges which are any subscription charges that aren’t upfront reservation charges or support charges.

Thanks, I’m still waiting for a reply from the AWS Account Manager. Inside the bill it’s described as “Missed CloudFront Term Commit Charge - April 2020 - March 2021”. I’ll let you know when I have more info

i use leapp and in browser i login to my identity account using saml login via gsuite idp

then i switch roles using aws extend iam roles switcher firefox addon

FTR, I did end up switching to Codefresh. It’s got several nice features, including the “wait for approval” step I wanted. If anyone else wants to use it, I recommend starting here:

https://codefresh.io/docs/docs/yaml-examples/examples/amazon-ecs/#create-a-cicd-pipeline-for-ecsfargate

NB: the “ecs-deploy” step from the marketplace doesn’t let you provide an image tag! So use the step provided in the link above instead.

You can begin by coding your pipeline yaml in their inline editor, and then switch to a file checked into your Git repo.

One thing to be aware of: Codefresh does some behind-the-scenes ECR integration for you. If you change your access key credentials, this can be a problem. Read up on it in their docs.

I am now using Codefresh along with ecs-web-app for deployment. codepipeline_enabled is set to false. Codefresh works well, and my task definition gets updated and service re-deployed when I push to GitHub.

However, now when I make a change to Terraform and then plan and apply, the task definition gets changed. The new TD assumes that the image will be tagged with its sha, which is different from the tag I’m using on Codefresh (the short commit rev).

I haven’t set ignore_changes_task_definition, but that’s true by default. Shouldn’t this prevent the task definition from changing? If not, how do I achieve that? I am happy for the service to be managed by ecs-web-app, but I need Codefresh to be in charge of ECR and the task definition.

I suppose an alternative approach would be to have Codefresh tag the images with the sha256 as well. Feels pretty hacky, though.

I did a search and discovered another thread from a few months back on this same issue. The advice was succinct. https://sweetops.slack.com/archives/CB6GHNLG0/p1631503151175000?thread_ts=1631287236.173300&cid=CB6GHNLG0

My answer will be not EKS nor Cloud Posse-specific: the multi-region KMS key works in a way that

• you create a “master” key (aws_kms_key) in us-east-1 and set “multi_region” parameter to “true”.

• you then create a KMS key replica (aws_kms_key_replica) in another region and point it to master_key_arn (or id, please check terraform docs). Master key’s key material will be copied from the master key to the replica, and you could use master in the first region and the replica in the second region. The advantage is that data encrypted by one key may be decrypted by another because they are using the same key material. If that is what you need, then master/replica is your solution. There’s no other way of my knowing to use the same key across regions. But I may be wrong.

Thanks Contantine. All your background info is totallly correct from my additonal research. In our case it turned out that MRK don’t make sense at this point becuase we have a very predictable “active/passive” design with respect to Regions. MRK it seems would have much greater appeal if we had live applications in both regions with data and client apps present in both Regions. In our case we replicate data to the non-active region , but the client apps only ever failover when we are reacting to AWS outages or other DR/BC activities.

Thank you for your response, i am afraid i cant do that easily - i am actually using driftctl to scan terraform states that reside in these s3 buckets

uhhhh

you may have an application issue in that case

you’re allowed 3500 PUT and 5000 GET per second, per prefix

might be a KMS error instead? That’s still like 1000 per second

can you configure driftctl to skip iterating the bucket? with that many objects there’s a 99.9% chance that you will have 99.9% drift

@matt

btw, are you using https://github.com/cloudposse/terraform-aws-api-gateway?

I’d recommend to run instances via Autoscale Groups, so available capacity fishing will happen automatically.

Also additional handling is requited, maybe you just hit account limit and need to raise via support.

+1 for ASG. I also recommend you enable more than a single instance type.

That makes sense, thank you both. These are single-use CI runners, but I guess I can try launching an autoscaling group with 1 max instance just for the better support

Is this transition one of those? https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-transition-general-considerations.html#supported-lifecycle-transitions

Sort of. We have some files in legacy Glacier (now known as Flex) and we want to move to the new Glacier tier called Instant Retrieval. They were/are being sent to Glacier via those policies now, but in this case we are looking to move existing files between glacier tiers.

Our AWS contacts have explained that it isn’t as easy as a simple transition. But I figured with the new Glacier tiers, we were probably not the only ones who want to move out of flex.

Aha interesting! Good one for #office-hours

I remember looking into this a while back and at the time there was not way to move between tiers

what I was supposed to do was to pull EVERYTHING from glacier and start over but pushing it the right tier

well if you have a SCP that does not allow public IPs and you deploy only private subnets without the 0.0.0.0 route it should work no?

And what is that SCP, please tell?!?

https://stackoverflow.com/questions/66392815/is-there-aws-scp-to-block-users-from-creating-elastic-ip-ec2

https://docs.aws.amazon.com/organizations/latest/userguide/orgs_manage_policies_scps_examples_vpc.html#example_vpc_2

that will apply to new resources

Is the first answer in that stackoverflow a joke? What is an aws config SCP?

but if is a new account it will work

yes I think that is a new product that person invented

it is an alpha

lol

The second answer gets closer. I’ll try it tomorrow. Previously I had only seen that condition on ec2:RunInstances, but that’s not nearly enough

I’m working on an environment that it is exactly like that

no internet at all

vpc endpoints for everything

Yeah, same. Just not greenfield. And when you miss something like this, and think it’s ok cuz there is no IGW anyway, going back through hundreds of accounts and reviewing network captures to understand what may or may not have transpired is really not fun

I was really hoping the ingress nacl option on the subnets would work, but that broke all sorts of things in really fun and exciting and ticket-creating ways

some people use https://tugboatlogic.com/integrations/aws-integration/

Played with this some more, and still cannot find a good IAM policy (or SCP)… the second stackoverflow answer suggested a statement on the actions:

ec2:AllocateAddress
ec2:AssociateAddress
ec2:CreateNetworkInterface

using the condition key:

ec2:AssociatePublicIpAddress

but! none of those actions support that condition key. these are the actions that support the condition key:

CreateFleet
CreateNetworkInsightsPath
CreateTags (???)
DeleteNetworkInterfacePermission
ModifyFleet
ReplaceRoute
RequestSpotInstances
RunInstances
RunScheduledInstances

but you can’t use it on:

StartInstances
CreateLaunchTemplate
CreateLaunchTemplateVersion

and this is just ec2. there are plenty of other services you can deploy into a vpc and that could get a public IP…

that sucks

this is due to the 4.x provider

add a new file locally to pin it

https://sweetops.slack.com/archives/CB6GHNLG0/p1644519534463299?thread_ts=1644519506.553189&cid=CB6GHNLG0

Best to use atmos

Has the application version changed or anything else about the environment?

It sounds like the app is crashing periodically and since the app is crashing the health check may be failing and then the load balancer returns a 503 since you have no healthy nodes.

Thanks for the reply @venkata.mutyala, if I run ps -aux both httpd and tomcat processes shows its START as 2019 only

Any recent changes to the environment? New code? New settings?

No changes to environment.

I assume you went through all the log files that beanstalk provided in the console?

Have you considered replacing a node and seeing if that resolves the issue? Not saying it prevents it from happening again but given the immutable nature of beanstalk I would be curious to know if it helps.

see your HD usage df -h

see your Free Memory free -m

see your CPU usage: htop or top

Every blog post I see recommends using ec2-instance-connect to put SSH keys onto the instance, then opening an SSH tunnel to the instance to access RDS

I would ideally like to just use Session Manager to establish the tunnel if that’s possible

Yes, we do. You use SSM but it still is an SSH document being executed, temporarily, to upload your public SSH key into the remote host. Then establish the connection with SSM

Hrmm… the whole point IMO with session manager is not to use SSH keys, but IAM.

I think this pretty much describes it https://www.element7.io/2021/01/aws-ssm-session-manager-port-forwarding-to-rds-without-ssh/

There are cli’s that make it even easier

like https://github.com/claranet/sshm

Search slack for “sshm” and you’ll find related conversations

Have you used remote states?

yes, I used remote state in vpc2 to create the peering.. do I need to use something similar in vpc1?

So the way I have done it in the past is create the VPCs first and then run the peering/route table creation. Basically have VPCs live in one or more of their own states and then do all the peering/route tables independent of any VPC creation… Does this make sense… VPC1/VPC2 first, then vpc peering/routes.

I am using the cloudposse vpc_peering module, which already creates the routes in both VPCs (both VPCs they are in the same AWS account).. Should I create the vpc_peering on VPC2 and then create the routes manually in VPC1?

(should I ask this in the terraform-aws-modules channel?)

#office-hours

Cool. I’ll check this out!

Thanks awesome! let me know

like the fresh new look

Thank you! we tried to make it more a Desktop App like, with right click and double click actions

cool can’t wait to try it

I’ve got it setup for both IAM User and SSO for 10 accounts

the changelog for 0.9.0 looks promising https://www.leapp.cloud/releases

has been making working with multiple AWS accounts so much easier

Thant’s great

hmm it now takes 2 clicks to login

also is dark mode on the roadmap for the UI ? it’s very white on my dark screen

but with a double click it start straight away agah

I’m a big fan of dark mode, I think theme will be added soon in a weekend of contributing

that weekend would be a month for me of contributing

i did create a dark mode ticket though for those weekend contributors

Great idea

most likely because enable rotation isnt true by default

see this comment by bridgecrew

https://github.com/cloudposse/terraform-aws-kms-key/pull/33#discussion_r798013920

hmm well it is enabled by default already https://github.com/cloudposse/terraform-aws-kms-key#input_enable_key_rotation

cc: @Yonatan Koren i can never load bridgecrew properly. it shows up empty for me. can you see the error?

@RB no it’s blank for me as well. I’ve had some issues with the BC UI before.

I will admit, I haven’t used eksctl nor have i followed the link to see proposed solutions. So take that as you will :sweat_smile:

with that said, when i see None in something like this:

<https://sts.None.amazonaws.com/>": 

I start thinking that the AWS_DEFAULT_REGION environment variable is not set.

Most AWS services are tied to an regional endpoint and I would assume the CLIs build the URL based on the AWS_DEFAULT_REGION env var.

I would reccommend checking your configuration to make sure you are setting AWS_DEFAULT_REGION correctly for where your services are located.

eksctl:create:
  stage: deploy
  script:
    - aws configure set default.region us-east-1; aws configure set aws_access_key_id $AWS_ACCESS_KEY_ID ; aws configure set aws_secret_access_key $AWS_SECRET_ACCESS_KEY
    - if ! eksctl get cluster -n $CLUSTER_NAME ;then eksctl create cluster -f config.yaml ; else eksctl update cluster -f  config.yaml ; fi
  variables:
    CLUSTER_NAME: npd
    AWS_DEFAULT_REGION: us-east-1

I have tried the default region - no luck

i think they’re visible in CloudTrail events?

otherwise, i’ve also snagged the saml assertion using browser dev tools and decoded it to see what is sent…

(if you’re using saml!)

I am, and I do see my expected assertions in the SAML that JumpCloud’s sending. I also see the principalId in CloudTrail along with a list of sessionContext:{attributes:{...}} , but I don’t see the properties I was expecting.

Which may very well be as good as I’m going to get.

I second saml tracer (firefox) for viewing the attributes

i assume your trust policy on the iam role allows sts:TagSession? if not, i don’t think it would work at all… at least, not if the tags were set properly on the JumpCloud side…

That’s correct, the AWS SSO-managed role has sts:TagSession . Looking up the docs on sts:TagSession eventually got me to the hint I needed, which is to filter CloudTrail events to Event Name: AssumeRoleWithSaml which does have the list of principalTags written out.

Thanks for your help (and the tip on saml tracer)!

woot!