#aws (2020-01)
Discussion related to Amazon Web Services (AWS)
Discussion related to Amazon Web Services (AWS)
Archive: https://archive.sweetops.com/aws/
2020-01-02
Does anyone know how to make an internal ALB with private hosted zone use DNS cert validation ?
also, for clarification, you want a public certificate for a private zone?
(because I think private certificates address this - https://docs.aws.amazon.com/acm/latest/userguide/gs-acm-request-private.html)
cheaper to buy a domain with route53 and use public certs with a public zone than ACM Private CA
Definitely cheaper
we are using route53 private hosted zone, create a cname and attach internal alb endpoint to it
can do the same with a public zone instead. very easy. to answer the question, i think DNS cert validation will require a public zone, since the service needs to be able to perform a DNS lookup to validate the records ¯_(ツ)_/¯
that’s what i thought, was checking if it was possible to do public DNS with my current setup
the service does not have network access to your private zone to perform the DNS validation, i don’t think. not sure how that would work, would be some black magic involved…
does the public DNS need to resolve to something ?
ACM will provide you with the DNS records that you need to enter. you are proving that you own the zone associated with the ACM cert request. so the zone (or zones, can be plural) needs to match all the cert names you provide to ACM
if you can’t get there, then the ACM PCA feature that @Erik Osterman (Cloud Posse) linked ought to work. it’s just expensive to run an ACM PCA ($400/month for the PCA, plus a charge per issued cert)
@loren thanks. I will check it out
When using nginx-ingress (installed via helmfile provider), is there a good way to get the load balancer CNAME into terraform for use with populating dns (e.g. [a92da7dd42d9c11ea9d19028a775bee0-865849920.us-east-2.elb.amazonaws.com](http://a92da7dd42d9c11ea9d19028a775bee0-865849920.us-east-2.elb.amazonaws.com))
Do not use terraform for that. Instead use the external-dns controller.
This is where the power of Kubernetes controllers really shines
E.g. use the cert-manager controller to automatically generate certificates
oh yea good point, I was getting confused (I do use external-dns and cert-manager). What I actually was driving at was gcp lets you assign a static ip to a load-balancer which can be passed in as a variable to helmfile and remain constant if for some reason the cluster had to be recreated (useful if passing this ip to a third party domain you don’t control). On AWS, I don’t think there’s an ability to control the CNAME so it will change between cluster creations, is that right?
On AWS to get a static IP and attach it to load balancer, you can use https://aws.amazon.com/global-accelerator/
thanks @Andriy Knysh (Cloud Posse) I’ll give that a shot.
hmm, so global accelerator gives multiple static ip addresses, but looks like nginx-ingress is only set up for one load-balancer ip address…. though I guess just choosing one might work with some loss of HA.
https://github.com/helm/charts/blob/master/stable/nginx-ingress/values.yaml#L243
Does anyone here use RDS IAM authentication feature ?
I am trying to figure out if we would still have to generate token and use it in place of password when RDS IAM authentication is enabled
If my app is running on ec2, can i just attach iam policy and can get rid of using password as a whole ?
The documentation also says that the “The authentication token has a limited lifetime of 15 mins”, does that mean once the token is generated we have to use it with in 15 mins to make a connection
yes you app will have to use the mariadb driver to deal with the token expiry
so it will need to request a token every 15 min
once a connection is made to the database, why does the app have to make connection every 15 mins using a new token ?
because the token expires
is like the password was changed
the doc detail how to do it with some code examples
and once you are using mariadb driver you can just delete all the other user not using the driver
is a specific mysql driver that get installed in the mysql aurora server
and you create the user in a specific way
2020-01-03
I have an ACM question, are the following certs the same? (I’m trying to understand what ‘additional names’ are for) cert a:
domain name: test.domain.com
additional name: *.test.domain.com
cert b:
domain name: *.test.domain.com
additional name: -
cert b will not cover [test.domain.com](http://test.domain.com)
just all subdomains
so for example, cert b will not cover test.domain.com but aaa.test.domain.com ?
Yes
so its more ideal to go with cert a?
essentially cert a will cover [test.domain.com](http://test.domain.com) and all subdomains?
yes, create it exactly as cert a
since you use star for subdomains (all of them) and if you use DNS validation, ACM will generate two exactly similar records to put into DNS zone
and since the records are the same, you can add just one of them and all will work
but, if you request a cert for test.domain.com and just for www.test.domain.com, then ACM will generate two different records, and you will need to put both of them into DNS
2020-01-07
has anyone used Iam Roles for Service-Accounts with EKS and a cluster of just Managed Node Groups with the cluster-autoscaler (v1.14.7)?
The role has the required permissions and it is getting injected into autoscaler pod.
Aren’t the details (role Arn) added as annotations to the service account (metadata)? Away from computer (on leave)..
yes, I added it there. That gets picked up and then AWS have a mutating admission controller which injects the role and tokenfile to the pod.
we tested IAM roles for service accounts for EKS using managed Node Group https://github.com/cloudposse/terraform-aws-eks-node-group. Worked great. Maybe this will help:
Here is the Role for external-dns:
locals {
eks_cluster_identity_oidc_issuer = replace(data.aws_ssm_parameter.eks_cluster_identity_oidc_issuer_url.value, "https://", "")
}
module "label" {
source = "git::<https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.16.0>"
namespace = var.namespace
name = var.name
stage = var.stage
delimiter = var.delimiter
attributes = compact(concat(var.attributes, list("external-dns")))
tags = var.tags
}
resource "aws_iam_role" "default" {
name = module.label.id
description = "Role that can be assumed by external-dns"
assume_role_policy = data.aws_iam_policy_document.assume_role.json
lifecycle {
create_before_destroy = true
}
}
data "aws_iam_policy_document" "assume_role" {
statement {
actions = [
"sts:AssumeRoleWithWebIdentity"
]
effect = "Allow"
principals {
type = "Federated"
identifiers = [format("arn:aws:iam::%s:oidc-provider/%s", var.aws_account_id, local.eks_cluster_identity_oidc_issuer)]
}
condition {
test = "StringEquals"
values = [format("system:serviceaccount:%s:%s", var.kubernetes_service_account_namespace, var.kubernetes_service_account_name)]
variable = format("%s:sub", local.eks_cluster_identity_oidc_issuer)
}
}
}
resource "aws_iam_role_policy_attachment" "default" {
role = aws_iam_role.default.name
policy_arn = aws_iam_policy.default.arn
lifecycle {
create_before_destroy = true
}
}
resource "aws_iam_policy" "default" {
name = module.label.id
description = "Grant permissions for external-dns"
policy = data.aws_iam_policy_document.default.json
}
and here is how external-dns service account was annotated with that Role: https://github.com/cloudposse/helmfiles/pull/207/files#diff-40daec15ea9ebdc2aed4f62abba406c3R63
after all of that was deployed and EKS started a few instances from Node Group, external-dns was able to assume the role and add records to Route53 for other services
thanks, I’ll give that a try to see what I have different
ugh.. needed to include fullnameOverride: "cluster-autoscaler" in helmfile - it autogenerates a different sa name, works now.
2020-01-08
Anyone know how I can skip text when using parse on Cloudwatch Insights? I have parse message '* src="*" dst="*" msg="*" note="*" user="*" devID="*" cat="*"*' as _, src, dst, msg, note, user, devID, cat, other but I’d like to discard everything before src=
apparently there’s | display field1, field2 that filters out what fields insights diplays
(and it works on ephemeral fields created by parse )
2020-01-09
I don’t see the Opt-In option for new ARN format in ECS
Has anyone come across that before?
I am getting “The new ARN and resource ID format must be enabled to add tags to the service. Opt in to the new format and try again.” from TF, but not sure how to resolve
Just hit this same thing this morning. Solved with: aws ecs put-account-setting-default --name serviceLongArnFormat --value enabled
Thanks
I’m getting below error. Is this the right forum to ask for help?
terraform init
Initializing modules...
Downloading cloudposse/ecs-container-definition/aws 0.21.0 for ecs-container-definition...
Error: Failed to download module
Could not download module "ecs-container-definition" (ecs.tf:106) source code
from
"<https://api.github.com/repos/cloudposse/terraform-aws-ecs-container-definition/tarball/0.21.0//*?archive=tar.gz>":
Error opening a gzip reader for
we have no control over those tarball URLs
that’s the correct one provided by github
perhaps share your terraform code?
Just started to get the same oddly
source = "terraform-aws-modules/autoscaling/aws"
version = "~> 3.0"
odd
in my case updating terraform from .16 to .19 seemed to correct it so maybe a blip or maybe something broke. Hard to tell
On a serious note, I wonder how GitHub feels about all the terraformers downloading tarballs of modules. Can you imagine the thousands and thousands of tarballs being requested per second!
Microsoft can afford the bandwidth
@Bernhard Lenz better to use #terraform
Tx
2020-01-10
Anyone have experience with how much overhead RDS postgres multi-az adds? I was doing some testing on tiny instances and the added replication seemed to add a decent amount (20-30% CPU) but that was a tiny t3.small instance
don’t use tiny instances.
We don’t, but I haven’t tried every instance type to see if that impact was only because it was so small or whether replication actually adds a decent bit of overhead
Smaller instance types of anything aren’t going to give you an accurate impact assessment. How about you try using the instance types you are currently using in production.
The point of the question was to see if someone’s already done this
I don’t have an answer, but it seems a common complaint, explained in more detail: https://stackoverflow.com/questions/47162231/rds-multi-az-bottlenecking-write-performance/50441734#50441734
I suppose the amount of overhead would depend on many many factors: amount of data written, size of data written, randomness, io characteristics of the instance. The t3 class will probably be worst to use as a reference instance since they are burstable.
multi-AZ will always have (some) performance impact since Aurora does replication synchronously
Say I upload to files to an S3 bucket for a website multiple times per day: index.html and some_file.some_cache_busting_string.js, and that the new version of index.html that is uploaded will always reference the most recent js file.
index.html is a much smaller file than the js file, so when I upload these files the html file will typically complete its upload first.
If someone visits my site during the time after the HTML is uploaded, but before the js finishes uploading, the page will fail to load.
I’m guessing I’m not the first person to experience this, so how do you all handle this?
Do you have versioning enabled ?
AFAIK, it would serve old version of js file if replacing js file is not completed
sounds like a good puzzle to solve :slightly_smiling_face: could be done by uploading files to S3 sequentially or using this in index.html (in onerror, try to load the old file) https://javascript.info/onload-onerror
Hey team, am running a Hugo static site with the cloudposse CDN module with great results. Site is converted from a lamp stack minus the deprecated CMS. Looking to replace the phpmailer email form and came across this module. https://github.com/cloudposse/terraform-aws-ses-lambda-forwarder Wonder if I could stitch this and AWS API Gateway together to handle the form POST per a few blogs? Preferably in terraform but haven’t used these services before. Will need to find a way to avoid spam usage.
I would recommend a js embed to replace the forms
We use HubSpot
And outgrow
Thanks Erik, been looking at formtree but looks like plenty about. Indeed seems a lot simpler.
2020-01-11
2020-01-12
Hi guys! I faced issue with ALB’ hard limit that is 1000 Targets per ALB. I tried to get targets that are belong to ALB using aws cli, but I did not found such option there. Did someone face this issue or make work around for it?
2020-01-13
Hi guys, I have a question about EKS: You pay 0,20 USD per hour for each Amazon EKS cluster that you create (from https://aws.amazon.com/eks/pricing/) Is it apply for any type of eks confirmation (ec2, fargate ) ? Do you know why is it so expensive ? it is almost 144$/month. is it price for k8s service containers ?
That’s the price for ‘just’ the control plane. Not cheap. Otoh, setting up a HA control plane on EC2 will likely set you back for something similar or higher.
factor in the number hours that you’ll spend setting it up and keeping it running it’s inexpensive IMO
Maybe try EKS on Fargate, looks like you’re only charged for EKS as long as your pod is running
But yeah, that actually seems cheap to me too
does this (HA control plane on EC2) mean that I have to remove eks and deploy my own k8s into EC2s?
that being said, I don’t use EKS…my team is only running two containers (currently) so we’re using Fargate/ECS
That would be the alternative, yes (e.g. using Kops). But e.g. 3 m5.large instances will be as expensive. And you’ll not be able to use EKS managed nodes or EKS Fargate.
I’m using ECS as well, but I was starting to look into k8s, and I was unpleasantly surprised about price
@TBeijen thank you.
2020-01-14
Hi guys Have anyone experience with EMR auto scaling related to Presto app? We have errors in Presto when downscaling EMR, it looks like EMR does not support graceful shutdown for Presto because it is not managed by YARN. Is there any workaround? Are we doing something wrong?
Does anyone know where I can find an example of using terraform to add a trigger to a lambda function? I found aws_lambda_event_source_mapping but it seems to only support event streaming from Kinesis, DynamoDB, and SQS. I want to add a trigger for changes in an S3 object.
Thanks Joe!!
No worries. How are you finding deploying lambda functions with Terraform?
Love deploying lambda with terraform. It’s phenomenal. Especially when using the claranet/terraform-aws-lambda module
I shall look into that, thanks @loren. Have been looking for alternatives to the Serverless framework.
If you use api gateway extensively, then terraform is rather more involved to get working than serverless. But for basic lambda and integrating with other aws services, terraform is way better. And you can do api gateway also, you just need to really learn some of the complexities where serverless makes a lot of choices for you to simplify the basic interface
That’s been my takeaway from using both
Thank you for that. Current project involves kinesis stream consumers so it could work well.
2020-01-15
Has anyone tried using the reference architecture with a .ai domain? You cannot register .ai domains in AWS. I am wondering if this would work if, after provisioning, the SOA record in the apex domain’s hosted zone were deleted, and name servers at the registrar pointed to the name servers in that same zone?
@Joe Hosteny haven’t tried… but don’t see why not? ref arch does not register TLD. it just creates the zones.
so if you can create the zones, then it’s fine.
after provisioning, the SOA record in the apex domain’s hosted zone were deleted, and name servers at the registrar pointed to the name servers in that same zone?
please share context (also lets use #geodesic)
office hours starting in 10m: https://cloudposse.com/office-hours
2020-01-17
Hey peeps
I am trying to add a config rule to Landing Zone so it can be created across accounts
There’s very few (if any) tutorials on LZ - all I see are videos and articles on why LZ is good and what it does and its advantages, but barely any meaningful examples
so my CodePipieline pipelines are passing after adding the custom config rule to some template
I have also added the lambda that should get invoked (both source code and definition in the template) as well as the permissions required to execute the lambda
After the pipeline executes successfully (which takes forever to complete), the resources are still not created and at this point, I am not sure what it is that I need to understand between cloudformation and LZ
It’s hard to troubleshoot when everything is passing
Any pointers to some resource which can help me understand LZ like a 5 year old?
@Mike Crowe any resources you found helpful?
@Erik Osterman (Cloud Posse) not really, just some assistance from a friend who explained the manifest file. Still learning the ropes - info seems scanty on it
Nice little extension from a coworker: https://addons.mozilla.org/en-US/firefox/addon/amazon-web-search/
Lets you create a shortcut to open up the AWS console service list / search bar - and most importantly lets you hit esc to close it
@Mike Crowe has joined the channel
2020-01-18
@Mike Crowe sound familiar?
Ha ha btdt
AWS NAT gateways are stupidly expensive for what they do
And not documented in control tower
2020-01-19
i have to think that they’re just spinning up multiple ec2 instances to do NAT or something like that - for the price that you pay it would have to be something supper inefficient
Was thinking about this the other day… at least modifying my terraform so that the “spoke” VPCs just get a solo NAT gateway instead of 3x… but then I guess you start paying cross-AZ for 2 thirds of egress traffic anyway so I dunno
2020-01-20
can also use transit gateway to route egress traffic through one vpc, and just pay for one set of nat gateways
EC2 instances doing NAT i think
2020-01-21
2020-01-22
hey folks, a client of mine is using the AWS SSO to manage all access creds. I’m having some trouble finding a nice solution for using this with terraform.
Currently I am logging into the aws sso page, choosing the aws account, clicking “Command line or programmatic access” and copy pasting the provided env vars into my shell.
I have to do this every hour
Both aws-vault and terraform have tickets open about AWS SSO, but has anyone come up with a decent workaround?
AWS recommends that we give access/secret keys only to physical users. Machines or systems should instead assume IAM roles (e.g. an EC2 instance should never use access keys passed as environment variables).
However, how does this best practice work in real life when the system that communicates with AWS is a third party one (e.g. GitHub actions). Can I make a GitHub Actions CI pipeline communicate with AWS APIs without creating an IAM user for it? After all, GitHub is a machine, not a human. It should use IAM roles, right?
AWS means to do that in AWS eco-system. If you have a custom github runner in aws you can use IAM role
So if I use a real third party solution like Jenkins, GitHub actions, CircleCI - I have no other option but to resort to IAM users and secret keys per provider?
Yes
Yup
did you try aws-cli v2, and its new integration with aws sso? login with that to get the creds for the profile, then run terraform? https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sso.html
I ended up using aws cli v2 with a little wrapper https://github.com/linaro-its/aws2-wrap
Replaces aws-vault in my normal setup
Depending on your integration, one thing you can also consider is having some privileged task inside your infra deliver STS credentials to the third party service.
That means your infra code has to have creds to the service, but that may be more palatable
may be able to use the credential_process feature of the aws shared config file to retrieve temporary creds, if the integration is running in something that gives you that kind of access, https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html
what is the best way to implement API authentication for REST API built in nodejs (not AWS Gateway) ?
Most people use passport.js
yeah i noticed that
Little background - our apps are deployed in AWS but the APIs are not deployed to AWS gateway
so all our apps need to talk to our APIs
Passport is the right choice for api authentication in node
currently we have these APIs sitting in private layer
what do you mean by private layer?
private subnet
how does your web app JS access them?
so the app is deployed in public subnet and it has access to the APIs sitting in private subnet
using security groups
I think you’re referring to server-to-server API calls?
yes
Passport is still fine. The passport-localapikey strategy is a simple option.
thanks for your suggestion
i would like to also understand how complex would it be to use AWS API gateway
With the new HTTP API feature (in beta though) it’s not too bad but your API will still need to generate JWT’s
If you use REST API you could require an API key on certain methods or use a custom authoriser or require a certain request header. There are lots of options. It just depends on your needs.
yeah i was looking at HTTP API
any ideas and thoughts are appreciated
2020-01-23
https://awsapichanges.info/ - not sure if anyone else posted it earlier
am i nuts or is cloudwatch synthetics really really expensive
i’m calculating $73.44/month for a single per 1 minute check
Looks like it’s just really expensive. Checkout Example 10 on https://aws.amazon.com/cloudwatch/pricing/
That seems really uncompetitive. There are so many options in this space
for some of our micro services we’d be looking at spending 4x for checks than we do for running the service!
In general, cloudwatch is expensive.
Even metrics cost something like $0.30/metric
when you combine that with k8s, with has like 10,000 metrics, it’s easy to spend more on the monitoring than you would for a small cluster.
Yeah our Prometheus metrics in cloud watch would be like be 100k month.
@ $0.0017 per canary run
Just found an SES rule that was sending to an s3 bucket that used to exist, but no longer does. Is there any way to retrieve the emails it missed in the meantime?
depending on your aws support level you may try to ask aws uspport and maybe they will be able to help. I wouldn’t bet they are but there’s always a chance
2020-01-24
Has anybody run to issues with Internal DNS not resolving correctly for ECS Service Discovery? I can see the correct IP in Route53 with TTL of 60, but it’s not being picked up
You application doesn’t pick it up?
What is your zones SOA TTL?
My application doesn’t, but I also tried pinging it elsewhere, and it’s also giving me the wrong ip
SOA TTL is 900
So I would cut SOA to 30 just in general
Because that acts like a negative cache
So if the record is queried before it exists and the response is not found, that negative response will be cached for for SOA TTL
But what you are describing sounds like it could be something else.
AWS Cloud Map service is magically supposed to manage these mappings
Since you are getting an IP back but the wrong one, it doesn’t have to do with the SOA
First time I ran into an issue with it
I don’t have experience with that service so someone else here might know more like @maarten
Bah, just noticed Route53 operational issue alert in the AWS console
Been going on for over 2 hours too. SMH. Please tell me this almost never happens.
2020-01-27
#aws anyone have experience dealing with aws state machines? specifically, a missing state machine after a cloudformation (landing-zone) update, and my avm (vending machine) plays fail.
2020-01-28
hello, anyone using private aws gateway api with custom host header? Thanks
no but have you thought of using an ALB with lambda hooks instead ?
@Michel I think we may be doing that right now. We’re doing cross VPC and cross account access with a R53 private zone to a private ALB to a private API gateway. The ALB is required for cross account access otherwise the API Gateway could be shared cross VPC (same account) with just a VPC Endpoint.
2020-01-29
Since VPCs are limited in my environment, I naively thought I could just kops cluster create a new cluster inside my existing VPC. (with this snippet).
Then it blew up because of subnet conflicts and I could see where this was going.
Most of the TF vpc modules I see expect to create a new one….
Should I:
• terraform import the VPC resource,
• manually replace vpc.vpc_id with something in var.vpc_id
• stop overthinking this and use some kops flag I don’t yet know about
?
Usual approach I follow with KOPS is to create VPC and other aws reosurces using terraform and then use kops to create the actual cluster (just like you are doing). I however split the subnets cidr’s accordingly when creating the VPC from tf. In your case, I guess if you are planning to use the vpc, terraform import might help. Not sure if this helps you though
@grv Thanks for the validation, I’m more confident now. So I guess could maybe run TF against the existing VPC and have it carve out the subnets, security-groups, etc, then run kops to stich them together into a cluster.
That would make sense, but I have a feeling you might run into some kind of trouble while trying to play around existing cloud resources using tf. Again, saying maybe
Hi, has anyone deployed EKS on me-south-1 and successfully interacted with the cluster via kubectl getting credentials using aws sts assume-role ?
2020-01-30
Yes, same as other regions…
Looking for simple condition to set in EC2 instance template with ALB, that will hold the 2nd (HA instance being deployed) so I can get the first up with some manual config running then remove condition to lauch the instanceB
2020-01-31
Hello, I’ve used the AWS landing zone pattern for a while but just wondering how others use the shared account for things like Vault? I.e put Vault in here or separate account? Also usage pattern - 1 Vault for dev/prod or 2 separate installations…Just wondering what others have done…cheers
I’ve always gone with a Vault cluster per environment
Deployed into the same account unless it is a break-glass type scenario
That’s an option, I was thinking we will have multiple prod accounts so a shared prod Vault cluster would be another way
Hey people
I have a question for AWS gurus
How do I use the PutOrganizationConfigRule API in Landing Zone to create an organization config rule?
https://docs.aws.amazon.com/config/latest/APIReference/API_PutOrganizationConfigRule.html
Going to try this channel as well. Does anyone know if ECS provides an SNS topic to subscribe to events like updating service, tasks starting/stopping, autoscaling events?
I don’t believe so
Does https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwet.html get you what you need?
I guess, if that’s the only option. Thanks
Looks like it only gives you “ECS Task State Change”, “ECS Container Instance State Change” triggers
Unless they’ve adding something since I last looked. You subscribe a lambda to the event. The lambda needs to parse to figure out what the change was, then run your logic for that change type.
At that point whether you send to SNS or do anything else is up to you
Thanks @joshmyers @Steven
Not great. It needs finer events
How are people dealing with upgrading minor versions of Kubernetes in EKS in large clusters?