You are looking for ECS Connect

are you using host, bridge , or awsvpc networking mode?

awsvpc

Then I believe this is going to be the recommended solution: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-discovery.html

and then you would use DNS to reference the other service

As far as I understand ECS Connect is more modern way compared to CloudMap. Also with CloudMap you could get situations when CloudMap can return empty list of IP addresses during ECS service deployment. As far as I understand ECS Connect addresses that

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-connect.html
Amazon Amazon ECS Service Connect creates AWS Cloud Map services in your account

it uses CloudMap for private DNS, yes. But it also would deploy envoy proxies that AFAIU would deal with DNS resolution/traffic routing and do retirees if necessary

“good” there’s my comment

awesome

I just wish AWS ipv6 support was at a place where ipv4 wasn’t necessary. Definitely gonna be shutting down things in my own personal account that were handy, but won’t justify the cost anymore (I could run them for almost nothing in a public subnet)

See also: https://github.com/DuckbillGroup/aws-ipv6-gaps/pull/6

@Matt Calhoun

@Matt Calhoun @Dan Miller (Cloud Posse)

@Max Lobur (Cloud Posse) @Igor Rodionov

Hi @Renesh reddy I’m not sure what the question is. Are asking how to trigger the first repo pipeline from the second repo pipeline?

Resolved the issue. Thanks .

@Andriy Knysh (Cloud Posse)

we have SES module https://github.com/cloudposse/terraform-aws-ses, and example how to use it https://github.com/cloudposse/terraform-aws-ses/tree/main/examples/complete, which gets deployed to AWS on every PR. I don’t know about the issue described above, but you can review the module

I resolved my issue, the problem was that even you delete your configuration set if it is set by default in global settings it will return to you an error because it tries to find it.

My rule of thumb: If the bucket and the role are in the same account, only the role policy is necessary. The bucket will by default allow the role. If different accounts, then both must allow access.

makes sense, thanks!

I think it also depends on carnality. If you have several roles that needs to access the bucket, it might be easier to attach the policy to the bucket and have some rules to include them all. On the other hand if it is only one role and several buckets, it might be easier to configure the policy on the role and find a way to include in the rules all the buckets.

smart thinking! I wish there was a wiki that had tips like this

us all together we are the wiki…

You can use cw as a data source. https://grafana.com/docs/grafana/latest/datasources/aws-cloudwatch/ Extra: Anywhere cloudwatch is involved, consider if you need near real time metrics/logs. I find the service exceedingly latent all too often. Maybe it’s just me.

@Alex Atkinson Thank you.. You mean the latency is higher side by cloudwatch?

My concern is actually on sending the Prometheus metrics to cloudwatch.

Yes, anything that goes into Cloudwatch has a potential to either be visible immediately, or after much time. To my experience. They can’t even identify their own outages in a timely manner. I rely on Twitter for that.

thanks for the info. Good to know that. I’m not concerned on the latency for few mins as the dashboards are used internally.

@Matt Calhoun

and the error you receive is…?

I did not set sns to get error, When I try the batch Operations then the error is:

Error occurred when preparing manifest: Access denied when accessing arn:aws:s3:::sourcebucket. s3:PutInventoryConfiguration required for the role.

in replication status I see only it is failed

good ticket to open with AWS. It sounds entirely related to their tech

ok thanks

AFAIK Lambda functions cannot have public IPs so they cannot route to the internet without a NAT (gateway or instance). Put them in a private subnet, ensure the private subnet’s default route is a NAT in a public subnet, that the NAT has a public IP, and that the VPC has an IGW.

@Max Lobur (Cloud Posse) @Igor Rodionov

Good question, I think @Jeremy G (Cloud Posse) might suggest if he’s following these. I just scanned though our variable history and did not find any use of 100.64, however there might be older cases I’m not aware. Also might worth to fill a support case to AWS with this

The 100.64/10 block is set aside for ISP-level NAT, in particular NAT64. IIRC it was, at one point, used by Kubernetes and/or kops for the cluster’s internal address space. For EKS, they do not use 100.64/10.

At Cloud Posse, we always deploy VPCs into 10/8 and have not had a problem with that. I am not specifically aware of problems using 100.64/10 but I would avoid it unless you are doing carrier-grade NAT.

i figure they’re trying to implement suggestions for mitigating device code phishing attacks…
One suggestion is to display the code during the authorization flow and ask the user to verify that the same code is currently being displayed on the device they are setting up https://ramimac.me/aws-device-auth

@Andrea Cavagna hearing any reports like these

I haven’t seen any official communications about this from AWS

We noticed that this week, we will look at it and I think In Leapp we are going to adapt by showing the AWS sso code

@Noel Jackson

Is that the interaction you wil expect from a workflow like that?

That should work just fine. Thanks @Andrea Cavagna

Just released the fix for that, how it is working?

@Noel Jackson @Jeremy White (Cloud Posse)

@Jeremy White (Cloud Posse)

The CLI depends on several things. You might try using AWS Systems Manager. The AWS-InstallPackages can take an MSI url, so you could use

Here’s the relevant documentation: https://docs.aws.amazon.com/systems-manager/latest/userguide/walkthrough-powershell.html#walkthrough-powershell-install-application

That example is using powershell locally. You can also run these in AWS Systems Manager via console. Then you could select which ec2’s to execute on.

The run-command console is here: https://us-east-1.console.aws.amazon.com/systems-manager/run-command?region=us-east-1 (change the region according to your fleet)

If you’re interested in automating that via IaC, then terraform allows you to create an SSM association: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_association

You would just associate your windows EC2’s with the install document and add parameters to download the msi.

So, this gets to the next question, sync’ing to s3. You should be able to create a document to run aws s3 and a schedule, and AWS SSM will not only run the commands for you, it will report back failures so you can better observe when calls aren’t successful.

To do so using terraform, you can create a document with the aws_ssm_document

The document would just need to contain relevant commands. Since it can be tricky to write a full script in the SSM Document spec, make sure to consider using a separate script file if you find it too cumbersome.

Hi @Jeremy White (Cloud Posse) thanks for your response, we do have this setup in our infra, but this scenario, application team should see what happening in their task scheduler. I tried to install new cli version, but it doesn’t work, we are still facing the issue ( sometime job running and sometimes not ) fyi, we have upgraded the instances to higher

which document do you follow? seems it is a permission issue that the metrics values are not delivered to AWS

metrics from the aws services namespaces are coming through but not from the custom metrics namespaces.

i used this https://grafana.com/docs/grafana/latest/datasources/aws-cloudwatch/ and https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Agent-PrometheusEC2.html

did you set up metric_namespace?

yes.

this is the namespace.

and there are many metrics inside

may find some errors in the shipper’s pod

metrics are synced but values are not

also given the namespace in the grafana datasource configuration

ok. actually not using the k8s. its just ec2 instance

metrics are synced but values are not - how do we check this? is there any doc for this please

are you seeing any metric values in the namespace?

yes…

i sent you a screenshot above, that is from cloudwatch

i can see the metrics thre

so the only issue is the metrics go into a wrong namespace?

hmm, no. in grafana dashboard - there are no datapoints nor there is any metrics displayed

Let us take a look at the logs of pods then

Im not using the kubernetes.. the agent is configured in the ec2 instance

and the agent sends the metrics to the cloudwatch logs. so i see there are metrics available in cloudwatch logs. and then from cloudwatch logs it is sent to the cloudwatch metrics - it is also successful as i see the metrics in cloudwatch metrics as well.

Now the configuration is between the grafana and cloudwatch metrics - the problem lies here,

Oh sorry I mean to find logs in the system

We are missing logs of the agent

ok let me check them

Or try attaching admin permission to ec2

I think its something related to the aws permissions. i see in the raw response console

{"Message":"error getting accounts for current user or role: access denied. please check your IAM policy: User: arn:aws:sts::298738319810:assumed-role/mobilus-grafana-instance/1695309629321218232 is not authorized to perform: oam:ListSinks on resource: arn:aws:oam:eu-west-2:298738319810:/ListSinks","Error":"access denied. please check your IAM policy: User: arn:aws:sts::298738319810:assumed-role/mobilus-grafana-instance/1695309629321218232 is not authorized to perform: oam:ListSinks on resource: arn:aws:oam:eu-west-2:298738319810:/ListSinks","StatusCode":403}

right, it is

if admin role is attached to the instance, you should be seeing them

then you can figure out what policy is required later

attached the permission, still no luck yet my grafana is in the same organisation account, not sure why the error has come.

now the role mobilus-grafana-instance is attached

let us attach admin policy to this role

ok

Hey thanks a lot. I figured out the mistake i was doing.

Its really a silly mistake - in the datasource namespace there was a space at the start which i missed out.

i saw the query and figured it out

awesome

you are welcome

HI @Hao Wang I got stuck into new issue. now the metrics is set but the server keeps getting crashed every 5 mins. I am sending around 50+ metrics and using t2.medium and i tried with t3.medium as well.

should be memory issue, need a more powerful instance

yes. The same metrics and configuration works really great with the prometheus standalone server but when it comes to pushing to cloudwatch there is performance issues.

it makes sense in a lot of ways

this is pretty great and something that’s built up over time. Just thinking about my process of learning and applying to brownfield environments, building in security where we can, but also trying to have flexibility for teams. Definitely not easy

I BASH’d out the configs for the 76 common mime types detailed from Mozilla, which is where I ran into the limit.

You can use Edge Lambdas

… obviously.

I just don’t have much experience with CF. All my CDN experience is with limelite, Akamai, and Fastly.

tyvm

Most libraries that upload to S3 will set the content type correctly. I think it’s better for you to require this on the source object

Hmmm……….. I’m uploading from a GH action. I bet there’s a GH action to do s3 uploads and detect content-type

5 deep and can’t see any mention of content-type…. Maybe not.

If you’re using the AWS cli, this behaviour is built in

I wasn’t seeing that.

I would really recommend fixing the type on upload. You don’t want to be repeatedly recomputing something every download at the edge

I was using aws s3 cp... though. I ended up making a simple map of extensions to mime types and for-eaching through every file.

It’s slow though. And it’s a small project. I could multi-thread the cmds though. Just chop the file list into 4 and have 4 bg tasks going.

so grody

https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html
–content-type (string) Specify an explicit content type for this operation. This value overrides any guessed mime types.

Cli will guess for you

Use xargs or parallel?

…. You know what…. This all started bc of an incorrect mime type error where the file was XML instead of css, so that would have been CF responding with a 404 error.

Thanks for that, @Alex Jurkiewicz. The mention of the cli guessing automatically made me remember why this came up.

couple of questions for you to think about (i don’t have answers):

• there is data already in dynamo db. how this initial state will be extracted/processed?

• there was bug in lambda code, it was failing for more than 24 hours, dynamodb stream doesn’t contain full log anymore. what to do?

@z0rc3r

Could I ask what you mean by “dynamodb stream doesn’t contain full log anymore”

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html#Streams.DataRetention

@z0rc3r I don’t believe this will be an issue as the Lambda function will hourly on a daily basis so the data will always be up to date. I’m unsure if DynamoDB streams is a bit of an overkill for what my end goal is.

What’s your use case? Would one of the options presented here be acceptable?

@Christopher Wade To extract data from DynamoDB Table in to a CSV file then push it to an S3 Bucket so external users are able to grab the CSV file

I don’t believe this will be an issue as the Lambda function will hourly on a daily basis so the data will always be up to date. Imagine there was some changes in data structure, corrupted data in dynamodb, or aws deprecated lambda runtime. Lambda execution for some reason failed for more than 24 hours straight.

There will be some changes to the data structure if you mean by new records inputted. Unsure what you mean by “Corrupted Data in DynamoDb”. Can you elaborate further about what you have mentioned about Lambda execution failed for more than 24 hours straight? @z0rc3r

You expect lambda to always succeed and consume dynamodb stream. What happens when lambda execution fails for some reason? What happens when dynamodb stream is truncated after 24 hours of lambda failures?

@z0rc3r It’s something I’ve not thought about but I do believe these do need to be consider. Ideally I think it would be better to run the Lambda on daily basis rather than hourly. This would remove of using DynamoDB Streams.

Probably you don’t understand what dynamodb stream is. It’s sequence of changes to data, not data itself. If you don’t have complete history of changes (truncation after 24 hours) you cannot construct exact data state between current and last successful run.

It’s something that I’m not used too. As you’ve mentioned it can’t construct exact data state between current and last successful run. I do believe it’s a bit of an overkill for what we’re needing.

If you have ever done a similar piece of architecture work that would help I would be grateful to hear @z0rc3r

I didn’t

https://github.com/cloudposse/terraform-aws-eks-node-group/blob/main/examples/complete/main.tf

the complete example which gets deployed to AWS on every PR

uses VPC, subnets, EKS cluster and EKS Node Group modules

does it use them to create new ones?

or does it hook to an existing VPC/Subnet configuration?

and use that?

you def can use the subnets module or not use it. The subnets are inputs to the EKS modules

https://github.com/cloudposse/terraform-aws-eks-node-group/blob/main/examples/complete/main.tf#L107

https://github.com/cloudposse/terraform-aws-eks-node-group/blob/main/examples/complete/main.tf#L124

provide your own subnets there

There’s only reference there to public_subnet_id

how do I pass a private_subnet_id to spin a node_group in?

the EKS modules def don’t create or lookup any subnets since its not their responsibility

if you are using the subnets module, the private subnets are in the outputs https://github.com/cloudposse/terraform-aws-dynamic-subnets/blob/main/outputs.tf#L23

if you create your own subnets, you have their IDs already (from the terraform-aws-subnet resource)

This is true

I’m currently passing in a list of 6 subnets, 3 of which are private, 3 of which are public to the module:

variable "aws_eks_subnets" {
  type        = list(string)
  description = "Subnets for worker nodes"
  default = [
    "subnet-0961f52276f66803a",
    "subnet-0628e61bc2cbf07ab",
    "subnet-05a72053829efed5c"
  ]
}

imagine there are 6 subnets there

there’s no distinction on that list between a public subnet vs a private one

oh sec I think I know what I’m doing wrong

ah I see, or I guess (if I’m using the aws vpc module) I could also use the output from module.subnets.private_subnet_ids

I’m going to just try passing the list of private subnet id’s, instead of the entire list

have anyone used this network setup?

does cloudposse @Erik Osterman (Cloud Posse) used this?

I wonder how performant, pricy this could be

Which training is this ?

AFT and control tower in Udemy

Seems a bit old school with a single point of failure and centralised control, so I guess if you were super paranoid like a bank it could work.

Other more distributed approaches like maybe VPC Lattice could be explored.

GCP introduced some features that make this approach easier to implement including multi-cluster ingress, cross-region and cross-project load balancer references, distributed in-line firewall and hierarchical org-level firewal.

https://www.udemy.com/course/aws-secure-landing-zone-with-control-tower-and-aft/

I set something like this up in a PCI environment to be able to audit all in/egress traffic in 1 place, meaning all public endpoints is then in the “public” account, including loadbalancers etc. not with AFT though

all cross account traffic is through transit gateway

single place to monitor wafs etc… but we had to use global accellerator for some of the loadbalancers, because of this design

@Eamon Keane no single point of failure here, this are not 1 device per component this are autoscalable devices to speeds of up to 45 Gbps

and multi-az etc

is a very simplified diagram

probably single point in the sense that a admin with acess to the public account can ruin the day

well anyone with access to an BGP router can ruin the day, that is a people problem

its a balance yes - if you have autonomous groups or divisions, you could argue that each account running their own in/egress is more “redundant”

in the training they mention that the Ingres part of this approach in AWS is not quite flexible , so a the end you need IGWs fro ingress for your LBs

egress goes through the network account

we used ingress on public account only, then loadbalancers would use target groups in destination accounts - but this was the reason we used global accellerator if I remember correctly - also its different for elb and layer4 one

IP targets?

I remember doing something like that with VPC peering, is a bit limiting

yes… not good, I know

we do it with transit gateway - not optimal, but it works

yes, I think with TGs is better now

Yes, we’ve implemented this before.

Most recently we helped @johncblandii

I understand the value from a compliance perspective, but I don’t like it from an IaC perspective.

what is the problem from IaC? Isolation of components per team?

I share this article a lot. It covers all the ways to manage centralized ingress in AWS… https://aws.amazon.com/blogs/networking-and-content-delivery/design-your-firewall-deployment-for-internet-ingress-traffic-flows/

there are interesting options around centralized firewall management but distributed ingress points in there also

to be honest I’m not looking at implementing this but mostly to understand if this is a common pattern, if is used and if it have any gotchas

i’ve implemented the ELB sandwich with TG a couple times (from the article i shared, last diagram on the left). the biggest gotcha is just the complexity, and understanding impacts of changes. second biggest is lower agility due to division of responsibility, as any ingress requires coordination between the app team and the network/firewall team

yes, that will be difficult and I guess you will have to get static EIPs to make sure to never change the ingress ips if your products are whitelisted fromt he client side etc, which ads external coordination

This set up adds like $0.02 per request on ingress so take your current costs and x2 then

egress and Ingres x2?

On which metric/resource @johncblandii?

ingress which is why we only did egress

and only for non-prod

prod has too much traffic so 2x’ing our prod traffic bill is a non-starter

“Ingress” is not a billing metric though, so just trying to understand which item of the bill you are referring to

https://aws.amazon.com/transit-gateway/pricing/

egress, not ingress

Transit Gateway data processing charge across peering attachments: 1 GB was sent from an EC2 instance #1 in a VPC attached to Transit Gateway #1 (N. Virginia region) over a peering attachment to Transit Gateway #2 (Oregon region), where it will reach EC2 instance #2 within a VPC. The total traffic related charges will result in a charge of $0.04. This charge comprises $0.02 for Transit Gateway Data Processing on Transit Gateway #1 along with $0.02 for outbound inter-Region data transfer charges. Here Transit Gateway #2 will not incur data processing charges, as they do not apply for data sent from a peering attachment to a Transit Gateway. As inbound inter-Region data transfer charges are free, no further charges apply to the Transit Gateway #2 (Oregon region) side.

but…Account A to B adds $0.02 per request so it is ingress, technically (at the start of the chain), but the charge is on the egress to the other account

so just keep that in mind

Oh yeah, TG traffic pricing is absurd

yeah, so we abandoned centralized ingress entirely

It is $0.02 per-GB, though, not per-request….

correct

so we did the math and that 2x on our GB’s was a no go

That might be a good reason to use one of the distributed ingress designs, from the article I posted (presuming you also have managed firewall requirements, anyway)

mmm that is good information