#aws (2022-03)

Its been a while since i used transfer but i would say check the filesystem or bucket that you are using on the backend. in either case, make sure either the location is created and/or the user has permission to write. if i can recall the steps i will update.

same, been a while - I recall needing to set scope down policy .. details here: https://docs.aws.amazon.com/transfer/latest/userguide/users-policies.html

The problem was that Policy attached to the user. Needed to remove it and attach a role with the correct permissions

try to look at centralised logging and create a cloudwatch alarm when backup fails message is seen in logs

https://aws.amazon.com/premiumsupport/knowledge-center/rds-mysql-communication-packet-error/

There is the issue tracker to report any bugs or file feature requests.

Thanks @Baronne Mouton for creating https://github.com/cloudposse/terraform-aws-backup/issues/35

no problem

here’s how I ended up resolving.

aws ec2 describe-subnets --filters "Name=tag:Name,Values=*private*" \
   | jq -r '.Subnets[] | (.Tags[]//[]|select(.Key=="Name")|.Value) as $name | "\(.SubnetId) \($name)" '
subnet-0ceb34435b2c1d11 vpc-private1-us-east-1a
subnet-0b233052g4876c07 vpc-private2-us-east-1b

Sorry for the delayed reply but you can do all of the filtering in the CLI call so you don’t need to pipe over to jq:

 aws ec2 describe-subnets --query='Subnets[].{SubnetId:SubnetId, Name:Tags[?Key==`Name`].Value[] | [0]}' --output=text

So with that query clause, I believe that it did not support wildcards. UPDATE: I tried combining your query clause with my above filter clause and you’re right, I got the same result without having to resort to parsing with JQ.

aws ec2 describe-subnets  \
--filters "Name=tag:Name,Values=*private*" \
--query='Subnets[].{SubnetId:SubnetId, Name:Tags[?Key==`Name`].Value[] | [0]}' --output=text

the terminology in leapp sure takes some getting used to, when coming from directly managing awscli profiles

Yeah — Agreed.

though, i can’t really tell whether it’s awscli that is inconsistent with aws terminology, or if it is leapp

Just you wait for the concurrent multi console extension coming in a few weeks @Matt Gowie

Thanks for sharing. Just tried it out. Nice look and feel. Thinking it would be cool if there was a way to “import” existing profiles. Getting a backup of the existing credential file was nice though.

Question, is there a way to trigger leap from the CLI? My workflow is to create an AWS profile and then tie it to a python virtualenvironment. when i start the environment, it hooks into the AWS credentials and sets the environment for the profile associated with the environment.

With leapp, it looks like I would have to open the app and start a session before activating my python environment. If i could pull leapp into my current worfklow (without having to go to a GUI), it would be awesome.

Also, for folks that switch contexts often, is there a way to activate multiple profiles at the same time? I could, of course, go into the UI and start multiple sessions, but would be cool if there was a way to link sessions so they are activated at the same time; for example, i need to sessions in dev and prod accounts at the same time.

CLI is coming in a week or two and we’re evaluating the importing process to make onboarding smoother

Bravo

yeah it was kind of shocking to not see my existing profiles

Yeah I know better make it clear to people that Leapp is going to clean the profiles

@Nicolò Marchesi first thing I tried when I got the update was starting session #1 and then going to start session #2 to see if the multi console sessions stuff that you shared was included in this update. It didn’t work of course, but I am excited to test out that upcoming functionality!

@managedkaos those were some good feedback items — I’d like to see that stuff too! Nicolo + team have been super responsive and active in GH, so I’m sure adding an issue there or checking out the issues they have open would be a good call.

indeed. i will copy-pasta this over to GH!

Thanks guys! also you can take a look at https://roadmap.leapp.cloud for a high-level overview

For reference, the Multi Console Browser Extension is now available.https://sweetops.slack.com/archives/CCT1E7JJY/p1646935545276399?thread_ts=1646926352.409299&cid=CCT1E7JJY

1. Virtual Service can apparently be directed to Virtual Node directly, but when I last implemented it, I used a Virtual Router with a single Virtual Node target.
2. I haven’t used backends before, but I believe you only need to define a backend if you’re going to be using the service mesh to provide discovery and routing to that backend. That’s why backends are all Virtual Services.
3. I suspect no, but I’m curious to hear if you get this to work.

Awesome, thanks! I would use the SweetOps terraform module, but it has a lot of moving parts. So I figured struggling through it myself will probably be beneficial

(also doesn’t help that the module is outdated by a few years and uses deprecated features, like template_file in a data block)

My problem seems identical to this: https://docs.aws.amazon.com/app-mesh/latest/userguide/troubleshooting-setup.html#ts-setup-lb-mesh-endpoint-health-check

I deployed a Bastion and can hit my core app, but not the virtual gateway backed by the envoy proxy

It immediately refuses connections on port 80 (which is correctly mapped), and hangs on curl to 9901 (envoy admin interface port)

The virtual gw is a LB from what I remember, make sure your bastion host is in the same VPC and allow incoming traffic on the sg of the LB. Check for logs on envoy in cloudwatch or VPC flow log on the LB and work back from there

Yep, VGW is what the LB is trying to connect to. Bastion is on same VPC, and can reach the main app ECS but can’t reach VGW.

Logs seem ok, I can see it responding to the ECS health checks and it looks good

[2022-03-13 05:48:44.554][14][debug][conn_handler] [source/server/active_tcp_listener.cc:140] [C321] new connection from 127.0.0.1:39462
...
':authority', '127.0.0.1:9901'
':path', '/stats?format=json'
...
[2022-03-13 05:48:44.554][14][debug][admin] [source/server/admin/admin_filter.cc:66] [C321][S2351538453251859789] request complete: path: /stats?format=json

So something with external connectivity? The ports seem bound correctly:

80:80/tcp, 9901:9901/tcp

Same thing on the sidecar (minus the port 80, since that’s for the app itself)

When you try to connect from bastion host to the VGW, the connection source should be from a private IP not local 127.0.0.1

You’re going to love this: it was a typo in my APPMESH_RESOURCE_ARN. For any future readers who stumble onto this… virtualGateway vs virtualNode will cause frustration

if you haven’t already, please for the sanity of the next person report that to the document maintainer.

another reason I hate AppMesh

Wait where is the typo? I don’t see anything

there are no circular dependencies

it uses the same module to create a TGW https://github.com/cloudposse/terraform-aws-transit-gateway/blob/master/examples/multi-account/main.tf#L14

and to create VPC attachments https://github.com/cloudposse/terraform-aws-transit-gateway/blob/master/examples/multi-account/main.tf#L85

thanks for your quick reply! gonna try it out!

it doesnt work anymore after version 0.4.1

@Frank hmmm that may be a bug. are you giving it an existing transit gateway id?

https://github.com/cloudposse/terraform-aws-transit-gateway/blob/ef27722d92a9871e47ff9570900ef780f2481a01/main.tf#L10

usually when we see this issue triggered by a string variable type, we change it to a single item list of string to get around this error

@RB only for module.transit_gateway_vpc_attachments_and_subnet_routes_shared like the multiaccount example

the multi-account example is not automatically deployed to AWS by terratest so it could be out of date

dd you try this example https://github.com/cloudposse/terraform-aws-transit-gateway/tree/master/examples/complete - it’s deployed to AWS on each open PR so it should work w/o issues

@Frank

in general, in situations like

The "count" value depends on resource attributes that cannot be determined until apply

we can split the TF code into 2 components and apply them separately in two steps

(this is not an easy error to solve, it depends on many-many factors, it depends on all other code and dependencies b/w all your code. In some instances the same code might work, but just adding some other resources/deps could break it)

thanks @Andriy Knysh (Cloud Posse)! I’ll try to split up the components and see if that works out, havent tried the complete example yet. its a tricky error to solve indeed

I have tried by creating the transit gateway first, that results in the same errors.

https://gist.github.com/fspijkerman/5e6d8c46a94ad0f1033b576362dedf23

I’ve solved it by making the transit gateway first.

module "transit_gateway" {
  source                     = "cloudposse/transit-gateway/aws"
  version                    = "0.6.1"
  ram_resource_share_enabled = true
  route_keys_enabled = true

  create_transit_gateway                                         = true
  create_transit_gateway_route_table                             = false
  create_transit_gateway_vpc_attachment                          = false
  create_transit_gateway_route_table_association_and_propagation = false

  context = module.this.context
  providers = {
    aws = aws.shared
  }
}

transit_gateway => routes and association => vpc attachment

had to define a depends_on to module.transit_gateway in order to get rid of the count error

I don’t think so. cluster mode redis is fundamentally different from non-cluster mode redis

you can check with support, this is the exact sort of question they are fantastic at answering

Thanks @Alex Jurkiewicz

I have seen example of full account access

@Yonatan Koren

Approved

The former supports one lambda function invoked by separate event rules for each configuration. The latter requires a separate lambda function for each configuration…

I have a lambda that for some reason can only connect to https

if you change the config of the endpoint to http still tries to connect to https

endpoints within the vpc

I’ve never done it before but depending on how you did the self-managed install you might need to install some additional drivers so that your cluster is aware it’s running on AWS.

Beyond that, I imagine with the right IAM roles it will just work…

Per Lambda VPC updates in 2019 (see: https://go.aws/3JyDitA ), Lambda service still creates ENI in customer VPC when Lambda function has in-VPC configuration, but the IPs of the Hyperplane ENIs managed by AWS are ephemeral. In this architecture invocation of AWS Lambda functions (regardless of the style of invocation) is still a public API call. That means that Aurora PostgreSQL DB instances living in private subnets would still need route to the public internet to invoke Lambda function via public API. This route for DB private subnets does exist, however, it is routed via Transit Gateway in another AWS account. This means that eventually traffic from TGW hosting VPC would exit AWS root account boundaries and Lambda invocation would still happen via the Internet (public Lambda service API endpoints). Additionally, this includes multiple chargebacks for traffic routing between our accounts and eventually for the public API requests (despite being public Lambda service API endpoints). Therefore it still seems that required solution (as advocated by AWS docs linked above) is still to introduce VPC Endpoints for Lambda service inside VPC hosting Aurora PostgreSQL DB instances.

Ended up doing something like this … but still need to verify everything will work….

# Below module deploys VPC Endpoint (of type Interface) for invoking AWS Lambda service via private network (uses AWS PrivateLink) from within
# the private subnets in which Aurora PostgreSQL DB instances are deployed (in DB VPC). Despite this being set of networking resources,
# we opted-out from placing this module in Terraform profiles/vpc for VPC, because having VPC Endpoints for Lambda service is not requirement in each of our VPCs.
# It is so far only a requirement for a VPC hosting Aurora PostgreSQL DBs.
module "lambda_service_vpc_endpoints" {
  source  = "cloudposse/vpc/aws//modules/vpc-endpoints"
  version = "0.28.1"

  context = module.this.context

  interface_vpc_endpoints = tomap({
    "lambda" = {
      # Required
      # NOTE: Regarding 'name' attribute below, we have to feed only AWS service name as it relies on data source 'aws_vpc_endpoint_service' to lookup regional service endpoint.
      # See this for reference:
      #  1) <https://github.com/cloudposse/terraform-aws-vpc/blob/0.28.1/modules/vpc-endpoints/main.tf#L11-L15>
      #  2) <https://github.com/cloudposse/terraform-aws-vpc/blob/0.28.1/modules/vpc-endpoints/main.tf#L49>
      # name = "com.amazonaws.${var.region}.lambda"
      name = "lambda"
      # NOTE: Regarding 'security_group_ids' attribute below - the default SG for VPC allows all inbound and outbound traffic
      #       which is desirable. For now, we don't want to control access for individual and specific Lambda functions (via ARNs)
      #       to Lambda service VPC endpoint interfaces via Security Groups.
      security_group_ids  = [var.vpc_resources.default_security_group_id]
      private_dns_enabled = true

      # Optional
      # NOTE: Regarding 'policy' attribute below - since the default is to allow full access to the endpoint service (Lambda),
      #       we will rely only on RDS IAM Role for control.
      # DOCS:
      #       1) <https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html#vpc-endpoint-policies>
      #       2) <https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc-endpoints.html#vpc-endpoint-policy>
      policy     = null
      subnet_ids = var.vpc_resources.private_subnet_ids
    }
  })
  vpc_id = var.vpc_resources.vpc_id
}

Bit late here, but in short - by having either NAT gateway or VPC endpoints you are basically routing your request from resource in private subnet to AWS Lambda endpoint.

Which is either reachable using public route (NAT Gateway) - or using private route - traffic never leaves the VPC using VPC endpoint.

I would look first at https://aws.amazon.com/blogs/aws/introducing-karpenter-an-open-source-high-performance-kubernetes-cluster-autoscaler/

ssm + https://github.com/segmentio/chamber is great, chamber exec dev1/applicationname — python main.py would pull all those env vars within that session, without leaving anything local on the instance

Is this package necessary? Seems like ECS is able to pull in SSM vars if permissions are setup properly

maybe so, there can be a lot of situations it’s useful

Chamber is great if you want portability and want to be able to move your workloads to K8s one day or something along those lines but don’t want to configure them differently.
I was thinking about storing it into SSM and have the value pulled in programmatically with boto3 Why use boto3 if you can ref the environment variable or secret in the task def (which you mentioned above)? That is the way to do it properly.

Why use boto3 if you can ref the environment variable or secret in the task def (which you mentioned above)? That is the way to do it properly. - not planning on using boto anymore

Do any of you guys have experience doing this within a Private VPC? Apparently when you use the latest Farget launch type, 1.4.0, AWS has changed the networking model on this version and its necessary for me to either make the instance public (not possible with our configuration) or use a service like AWS PrivateLink

What are the queries doing?

various read selects for Ops/ Dev folks to lookup data should they need to dig into the prob

this might be worth a ticket @azec please create one in the repo with all of the necessary module inputs and we’ll investigate further

Where can I open a ticket @RB? I have never done this, mostly opened some PRs directly ….

I’d open the ticket in the modules repo

https://github.com/cloudposse/terraform-aws-rds-cluster/issues

Ah gotcha!

As a secondary question is there a nice tool for calculating multiple subnet sizes given a starting VPC CIDR?

As a secondary question is there a nice tool for calculating multiple subnet sizes given a starting VPC CIDR? https://tidalmigrations.com/subnet-builder/ might be able to help you here

Thanks @Lucky just what I was after

Why a /19? Start with like a whole /16 and you won’t have nearly so much worry about running out of space

@Alex Jurkiewicz - it’s a company policy thing to use /19

thanks @Wil, one comment

Ah, I did that 3 times!

Pushed the changes, thank you.

One note, that enables this feature which is a change in functionality. I should change it to default to off so there’s an option to enable it.

yes please set it to false by default

Some workflows stuff came in from the fork, would you like me to leave that?

the GH workflows are auto-updated

GH actions are having issues now, so we’ll wait for the tests to start and finish, then will merge

ack

It works, at least for the scenario where KMS key is used to encrypt the AWS Backup Vault in centralized account. I could strip to OU instead of using * for principal but I would need to test that.

 statement {
    sid = "Allow attachment of persistent resources"
    actions = [
      "kms:CreateGrant",
      "kms:ListGrants",
      "kms:RevokeGrant"
    ]
    resources = ["*"]

    principals {
      type        = "AWS"
      identifiers = ["*"]
    }

    condition {
      test     = "StringLike"
      variable = "kms:ViaService"
      values = [
        "rds.${local.region}.amazonaws.com",
        "backup.${local.region}.amazonaws.com"
      ]
    }

    condition {
      test     = "Bool"
      variable = "kms:GrantIsForAWSResource"
      values   = [true]
    }

That effectively allows usage of KMS key in centralized backup account to other accounts.

hey @Thomas Hoefkens! This is some very deep stuff and unfortunately I haven’t worked with Cognito (yet). You might have better luck with posting your question in https://repost.aws/, specifically under following topics:

1. Serverless
2. Security Identity & Compliance
3. https://repost.aws/tags/TAkhAE7QaGSoKZwd6utGhGDA/amazon-cognito

Maybe also check out awsiq.slack.com or awsdevelopers.slack.com

If you have your own auto scaling logic working, you probably want to disable the ASG’s equivalent logic (Launch and Terminate). https://docs.aws.amazon.com/autoscaling/ec2/userguide/as-suspend-resume-processes.html

Yeah we ended up suspending the AZRebalance operation, looking into why this is happening in the first place though and not our other clusters

The most common reason is that AWS don’t have capacity of the instance types you request, and one AZ is floating close to zero capacity. As you scale up there’s no free capacity in one AZ, due to other customer’s pre-existing workloads. Later one, those workloads are stopped, and the rebalance occurs.

Simplest solution is to add more instance types to the ASG, for example, both m6g and m5 and m4

Can you not use a lifecycle policy?

https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html

no because mfa delete protection has been set on that bucket. that is my isue, that’s why etcd manager cannot delete old backups

fyi… just created a cronjob that uses aws cli and with that I can delete the files W/o iisue

issue

you can’t change the region of a cluster without destroying it

a region is like another Datacenter

Is it not possible to only delete the cluster but only the node in that region or by adding to another region.

I do not think so

Regional products are created on that region

the global cloud is a myth

Yea, I would be interested to know too

Right? Alternative is just snapshots and scripting to achieve this but a managed solution would be appealing

ya, would love that. it would be really nice for our customers. inevitably the obfuscation would be a bit custom, but would like to have a turnkey pattern we could implement.

That said, anyone who has tried using DMS with terraform will know it’s a real PIA. Can’t modify running jobs and no way to natively stop jobs in terraform.