#aws (2021-08)

sounds like a case for aws support. Does the Lambda VPC configuration need to access a MSK endpoint in your VPC? Does the security group allow this?

Is that what you want? https://aws.amazon.com/fr/blogs/aws/new-vpc-traffic-mirroring/

Are you using CodeDeploy?

Yes @Mohammed Yahya

you need to make sure traffic shifting happened on all listeners http and https - the question is why you need both,

For the SSL. i believe if i didn’t add the HTTPS listener to the ALB the subdomain won’t be available on the API once pointed to the ALB. Correct me if am wrong please

you need one listener HTTPS, and create redirect rule on ALB from HTTP to HTTPS, the api, or any subdomain should work - you can create rules on the ALB for subdomains

you can publish the api two ways, using subdomain or using sub paths

Oh, Now I get. I should

1. add API subdomain to with HTTP host Header and redirect to HTTPS
2. On HTTPS add subdomain again and forward to the target group. Is that correct?

Yes.

Thanks so much for your time @Mohammed Yahya

I would say least privilege would be specifying exactly what can be done and by whom.

In your first policy, any of those resources can perform any lambda or event action. To me that seems dangerous as it could allow a lambda function to create/delete other lambda functions at will.

The second policy is what I would consider a good least-privilege policy. A user/role with that policy attached can upload to a single bucket. There may even be cases where you’d want to restrict that further and only give access to upload to a single path inside the single bucket, but that’s just a further enhancement that may or may not be useful.

So with the first one, do you find any value on the lambda name prefixes? Or is wildcarding actions just a bad idea?

There are exceptions to every rule, but I’d avoid wildcarding actions unless you have a really good reason.

One problem with a wildcard is it can actually grant more permissions over time if/when AWS adds more features to an API. And suddenly your policy allows more permissions than you intended.

It takes more effort to get the list of actions right, but IMO it’s worth it.

Good points. Thank you!

i’d agree that least privilege involves restricting both actions and resources

One thing I don’t like in the initial policy is how it combines two service namespaces into a single statement. Eg allowing lambda and event namespaced actions, and relying on the fact their ARNs are different. I prefer to explicitly separate them like:

    {
        "Sid": "",
        "Effect": "Allow",
        "Action": [
            "lambda:*",
        ],
        "Resource": [            "arn:aws:lambda:region:123456789012:layer:*",
"arn:aws:lambda:region:123456789012:function/my-pattern-*"
        ]
    },
    {
        "Sid": "",
        "Effect": "Allow",
        "Action": [
            "events:*"
        ],
        "Resource": [  "arn:aws:events:region:123456789012:rule/my-pattern-*"
        ]
    }

in our org, we’re given a standard set of roles by the platform team’s automation whenever a new account gets spun up:

• org.app-or-program-name.nonprod.administrator

• org.app-or-program-name.nonprod.operator

• org.app-or-program-name.nonprod.readonly

• org.app-or-program-name.prod.administrator

• org.app-or-program-name.prod.operator

• org.app-or-program-name.prod.readonly the readonly accounts have access to AWS Support, so they can at least open support tickets.

i dont like it . none of it is mapped to AD. It’s still nice to use AWS SSO but still… bleh

you will need all that

you can find any module we have like for ecs eks and look at the /example/complete folder to get an idea of what is needed to get it running

Could the issue you’re experiencing be the same as this one?

https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/pull/24

@Ashwini Thillai can you try that feature branch and see if it solves your issue?

module "kafka" {
  source = "git::ssh://[email protected]/krainet/terraform-aws-msk-apache-kafka-cluster.git?ref=krainet:bugfix/broker-dns-iam"

  # ...
}

@RB I have tried to run this, the module still throws me the Invalid index error .

Also discovered that the module does not calculate the bootstrap_brokers_combined_list properly . I am running totally 4 brokers on my cluster, but it consolidates that to 3 brokers

@Ashwini Thillai could you create a ticket in the module repo with the exact terraform code youre using ?

Think there is already an open issue, related to what am facing - https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/issues/17

@Ashwini Thillai try this

https://github.com/cloudposse/terraform-aws-msk-apache-kafka-cluster/issues/17#issuecomment-899904249

Thank you , I’ll give a try today and update you.

@RB Amazing ( pr #29 ) fixed my issue. However, just discovered that the broker endpoints is outputting only 3 brokers even though the cluster runs 4 brokers . I think fixing bootstrap_brokers in the outputs file should resolve the issue

I also read this from the tf page - so am kinda doubtful if it is the module or if this is how AWS works ,

• bootstrap_brokers - Comma separated list of one or more hostname:port pairs of kafka brokers suitable to bootstrap connectivity to the kafka cluster. Contains a value if encryption_info.0.encryption_in_transit.0.client_broker is set to PLAINTEXT or TLS_PLAINTEXT. The resource sorts values alphabetically. AWS may not always return all endpoints so this value is not guaranteed to be stable across applies.

Yes, i included that info in the pr. It’s not a stable output unfortunately

Ah, sorry ! just noticed it . When can we expect the PR to be merged ?

As soon as it gets approved. I’ll merge

@Ashwini Thillai try using 0.6.2 of the module and it should work

Amazing, thanks for creating the release . Appreciate it

I reached out to AWS support to understand why only 3 endpoints are being outputted . Here’s the response from them . Incase it helps someone,

In order to get more insight, I reached out to our MSK service team and they informed me that this is an expected behavior that both UI and getBootstrapBrokers API gives maximum 3 brokers as the output as you will only need a maximum of 3 broker node details to successfully connect or programmatically connect to the MSK cluster at any given moment. The broker node details presented may randomly change as part of the connection string  when retrieving the broker node connection string, alternatively you can still connect to any other available broker nodes on your cluster or make use of, as a connection string (Thus, you will only ever need a maximum of 3 broker nodes to ensure high availability when connecting, producing or consuming from the cluster).

Having said that, only 3 brokers will be listing in bootstrap list irrespective of the number of brokers configured in the cluster and this is by design. However, all brokers will be part of the cluster.

Very interesting. Thanks for reaching out to them and sharing

@Ashwini Thillai our slack removes messages over time as this is a free slack so if you want to share this, it might be better to share it on the same github issue so it persists

sure, i can do that I think there is also a workaround for getting the list of all the nodes . I’ll add that to the github page too !

https://docs.aws.amazon.com/pinpoint-sms-voice/latest/APIReference/welcome.html

Yep. That’s the APIs, but I wasn’t able to find any api specific for requesting new SMS numbers.

hello Amit, did you ever find a solution for this? I’m trying to achieve something similar.

Will try, what’s the issue you’re having?

aws-nuke is your friend for this. I use it regularly and it’s awesome.

One cool aspect of it is that it will loop through the destruction process, to solve the dependency issue you’re mentioning.

oh nice! i’ve never seen this tool before. thank you!

aws-nuke is good… hopefully you know what you’re doing gl

clear the cache

of your browser

will do!

did clearing the cache actually fix that?!

I actually does in many of this weird issues

once I had a problem where I could not see any ec2 instances, tried in another browser a d it worked so I cleaned the cache and everything was fine

I didn’t get back to it! Will check now

if no issues are reported by Amazon then I will have to think the problem is between the chair and the keyboard

:)

ease of administration? None of those

Try Tailscale or ZeroTier or …

We’ve used the AWS VPN Client in one case. We didn’t already have AD so getting that in place was more pain than I’d do again. I like OpenVPN. If I didn’t have a k8s cluster already I’d just go with the AMI.

Tailscale & ZeroTier, very nice!! I’ll check out both

ZeroTier pricing is fantastic

Have you used it before @Alex Jurkiewicz ?

Teleport can also be used for this as an alternate to VPNs: https://github.com/gravitational/teleport

We use zerotier currently yes Ryan. We evaluated tailscale, teleport and strongdm earlier this year. All looked great, but the project we wanted them for never got off the ground so we didn’t pull the trigger

thx everyone, appreciate the tips!!

I would go with Client VPN, you can automate everything with Terraform, if you choose certificate authentications - you can even automate the creation of TLS certs, and you can easily integrate it with SAML like Google or Azure AD, and just yesterday there is a blog post about integration with AWS SSO, so if you are looking for simple and easy way this is it.

https://aws.amazon.com/blogs/security/authenticate-aws-client-vpn-users-with-aws-single-sign-on/

1

You will be up with vpn within 15 minutes

I believe tailscale doesn’t meet compliance requirements for companies that need it. i don’t know which benchmarks in particular, just I know it was ruled out by one of our customers for that reason.

https://tailscale.com/blog/how-tailscale-works/

I’m blown away with STUN and their DERP implementations

It’s very impressive. I’m going to take this for a spin. Worst case can fall back to another solution

Client VPN w/AWS’s SSO is pretty painless, assuming you’re already using AWS SSO

bonus: you get redundancy out of the box (set up endpoints in whatever AZ’s you like), unlike tailscale or similar which is going to be a whole other thing to manage

Cloudflare also has a product in this area, “Cloudflare Teams”, which can ensure things like anti-virus are active on clients

it does require you use Cloudflare for DNS to get the most out of it

@Matt Gowie is also a big fan of Tailscale

Big of tailscale, but I ran into the compliance problem with a customer. Though I know they’re going through a SOC2 audit now or soon, so I’m sure that’ll change things.

StrongDM is a good one — I like them and their support.

I’d recommend against Client VPN only because who wants to deal with VPN config files. They’re archaic and troublesome.

Though I’d definitely go AWS Client VPN over running my own VPN host.

Zscaler is the other one that a few of our customers use

Oh yeah. How are you guys liking Zscaler compared to SDM Erik?

we’re not involved as end users, just helping deploy the agents

Got it.

@Ben Arent

cidr_blocks = concat(
        module.global.le_egress_cidr,
        module.global.shared_services_cidr,
        module.global.ibm_egress_tdq
      )

this looks like you are passing strings to concat, which expects 2 or more lists

the cidr_blocks input is itself just a list so you should be able to just do

cidr_blocks = [
        module.global.le_egress_cidr,
        module.global.shared_services_cidr,
        module.global.ibm_egress_tdq
]

Thanks @Zach, i tried this but still seeing an error

Error: Invalid value for module argument

  on security_groups.tf line 5, in module "sg":
   5:   rules = [
   6:     {
   7:       type        = "ingress"
   8:       from_port   = 80
   9:       to_port     = 80
  10:       protocol    = "tcp"
  11:       cidr_blocks = [module.global.le_egress_cidr, module.global.shared_services_cidr, module.global.ibm_egress_tdq]
  12:       self        = null
  13:       description = "Allow HTTP access from egress"
  14:     },
  15:     {
  16:       type        = "ingress"
  17:       from_port   = 80
  18:       to_port     = 80
  19:       protocol    = "tcp"
  20:       cidr_blocks = []
  21:       self        = true
  22:       description = "Allow HTTP from inside the security group"
  23:     },
  24:     {
  25:       type        = "egress"
  26:       from_port   = 0
  27:       to_port     = 65535
  28:       protocol    = "all"
  29:       cidr_blocks = ["0.0.0.0/0"]
  30:       self        = null
  31:       description = "Allow egress to anywhere"
  32:     }
  33:   ]

The given value is not suitable for child module variable "rules" defined at
.terraform/modules/sg/variables.tf:33,1-17: all list elements must have the
same type.

I think you might be running into issues where you have self = null on 2 of the rules but self = true on the third

Yes please. PRs are always welcome

I must have missed the other lambda-* files. good catch

doh. i missed these

• https://github.com/cloudposse/terraform-aws-datadog-lambda-forwarder/blob/df3934704ba7849a22f938e658dac62de7913e60/lambda-log.tf#L71

• https://github.com/cloudposse/terraform-aws-datadog-lambda-forwarder/blob/df3934704ba7849a22f938e658dac62de7913e60/lambda-vpc-logs.tf#L79

• https://github.com/cloudposse/terraform-aws-datadog-lambda-forwarder/blob/df3934704ba7849a22f938e658dac62de7913e60/lambda-rds.tf#L78

@Brandon Metcalf ping me when you open a pr and i’ll review

https://github.com/cloudposse/terraform-aws-datadog-lambda-forwarder/pull/9. Thanks!

@Brandon Metcalf 0.1.3 released. thanks for the pr

it should also support china region, aws_partition data source cloud be used

@Mohammed Yahya yes it uses the data.aws_partition :)

@RB I’m thinking we should have a github action with a conftest policy that checks for arn:aws: and errors if any found.

probably would catch 99% of these

sounds like a good idea to me

agreed. seems like that crops up a lot. not many people use govcloud.

I’m working on a China region project and I’m crying everyday from these ARNs

i think the conftest would only prevent new additions of the arn:aws string, no?

we love the pull requests to fix these issues. please submit them when you see it and ill personally try to approve and merge them!

Do a wild sed to replace it if you want