#terraform-aws-modules (2020-04)

One repo per module, yes definitely! On the strict aws resource split, I think this should never be a strict rule and should be evaluated on a per-case basis. If you are extremely strict with the one-resource per module then in many cases just just create a simple “abstraction” around a resource which doesn’t need abstraction because it’s super simple to begin with. I think it’s better to define in the team how great modules look like by having certain community modules as example. Whenever there is a debate on style or structure, a team member should be able to argue that it makes sense to do so because x and y. Cheers.

Thank you so much for your response .. That’s definitely the way to go.

IMO, repo per module always.

In my case we’re doing modules not for each resource but for each “infrastructure component”. For example we have a module for codepipeline which include codebuild, codecommit and all the IAM around it.

Looks like it is failing because of an unrelated README change?

(I’m the author)

This does seem a bit of an odd failure. @Maxim Mironenko (Cloud Posse) if there’s anything I can do, let me know

@bazbremner having some issues with GitHub actions. They recently did some changes related to tokens. I am on it

Maybe the module terraform-aws-vpc module fits your need.

i had used that module but i needed more fine-grained control over the subnets created. Essentially needed 4 subnets per AZ with a greater IP range in the private ones. I ended up rolling it with the existing TF resources.

@cabrinha there is a minor change request for your PR. also, after it will be addressed I will run rebuild README.md routine, so please, make sure your repo allow write access for our bot

Thanks for looking at this so quickly @Maxim Mironenko (Cloud Posse) – I’ve updated the PR with your suggestion.

How do I allow write access for the bot?

no need, we are fine, bot works well

How can I duplicate the README and FMT commands you guys run on your PRs in my own org?

it is as easy as running:

make init
make readme/deps
make readme

and for FMT:

make terraform/install TERRAFORM_VERSION=0.12.19
terraform fmt -recursive

if you don’t want to do so on your host machine, you can use docker image

for example this one: https://github.com/cloudposse/geodesic

Would I be able to copy this file into my own repos and use it the same way?

https://github.com/cloudposse/terraform-aws-emr-cluster/blob/master/.github/workflows/slash-command-dispatch.yml

@Andriy Knysh (Cloud Posse) is the master of how that works. @Maxim Mironenko (Cloud Posse) though is getting caught up

It’d also be great if we could get read-only access to these firebase workflows that are running against these modules too. Not a necessity but, interesting.

<https://github.com/cloudposse/terraform-aws-emr-cluster/actions> and <https://github.com/cloudposse/actions/actions>

@cabrinha you need a few steps to be able to use GitHub actions like we do:

1. Copy slash-command-dispatch.yml to your repo

1. Add the repo access token as secret

1. The action above calls these workflows https://github.com/cloudposse/actions/tree/master/.github/workflows

which you need to have as well

we separated the dispatched from the executor since we use one executor for all our repos (we just add the dispatcher to them) - so it’s easy to update it in one place

but you can use the dispatcher and the executor from just one repo

for cross-account access, you need to add permissions on both sides

on the one side, add an S3 bucket policy with permissions for the other account’s entities (users, groups or roles) to access the bucket

yeah this i have added

on emr side on ec2 roles ?

on the other side, add permissions to users/groups/roles to access the bucket

(I don’t know about your architecture so can’t advise on where to add those roles, emr or ec2)

hmm,

it also depends on how you use it, just EC2 or Kubernetes

this is pure emr on aws

but the description above applies to any case

yes

@navdeep EMR is a complicated topic. If you show me the code where you think you should do it, I would be able to help you

take a look at https://github.com/cloudposse/terraform-aws-emr-cluster/blob/master/main.tf#L294

give me about 30 mins, I’ll find some code for EMR

@navdeep on the bucket side, you add this:

@navdeep actually, what problem are you trying to solve? Why your bucket is in different account from the EMR cluster?

what we did for a client, we created EMR cluster and S3 bucket in one account (let’s call it data. Then created Firehoses in other accounts (e.g. prod, staging). Then we added a bucket policy to allow access from those Forehoses (cross-account). Then allowed the Firehoses to write to the bucket (cross-account)

so company has multiple accounts and thats because of different business verticals, i tried to put above policy before too,thanks for mentioning ,

The applications deployed in the other accounts (prod, staging, dev) have permissions to write data to the corresponding Firehoses (in the same account). Then, the Firehoses send data to the bucket in the data account. The EMR cluster in the data account (specifically, Hive and Presto) can access the S3 bucket in that account

Note that you can’t have a Firehose in the data account and push data from apps in other accounts - ASW SDKs don’t have the possibility to push to Firehose in another account

that’s why we created Firehoses in all other accounts and allowed them to write to the datalake bucket in data account

hmm correct this seems to be a good design, this is more of a legacy we are carrying

But to do what you mentioned (EMR in one account, the bucket in another), I think you need to add resource "aws_iam_role_policy_attachment" (with permissions to accesss the bucket cross-account) to these roles:

https://github.com/cloudposse/terraform-aws-emr-cluster/blob/master/main.tf#L280

https://github.com/cloudposse/terraform-aws-emr-cluster/blob/master/main.tf#L315

(not sure to both or just one of those, did not test it cross-account)

hmm ok i will check and if it can be configured i will push a PR,

thanks

you can add additional variables to add additional policies to those two roles

(would be a good addition to the module)

hey yup, though what worked us is adding read policy to give access to ec2 role we are creating in this module , shall i push a PR to add this in documentation ? if you need to read data from different account give following policy to ec2 role getting created

and hey thanks again !!

thanks @navdeep

PRs always welcome

Thanks @Andriy Knysh (Cloud Posse)!! Hope you’re good!