#terraform (2019-09)
Discussions related to Terraform or Terraform Modules
Archive: https://archive.sweetops.com/terraform/
2019-09-02
Hi, I have been using the resource aws_ami_from_instance to create AMI. The problem with this approach is that I cannot delete instance after creating the AMI. The instance is useless after this. So basically my workflow is as follows:
- Create an Instance
- Run some script inside the instance
- Create an AMI from the instance
- Terminate the instance I have been recommended to use packer for this but problem with packer is that it is doesn’t have a good integration with Terraform (Also I’m passing a lot of variables in the scripts in step 2) Any suggestions please?
@jaykm FYI
@sahil I haven’t used packer but I am not sure why would you want to control packer with terraform.
@Nikola Velkovski So the script I’m talking about in step 2 takes a lot of variables computed while running terraform. In case of aws_ami_from_instance, I can easily pass those variables inside the bash script, but same is not true for packer + terraform
hmmm you might want to drop terraform for that and maybe stick to aws cli since blocking/maintaing a terraform state for baking amis doesn’t sound quite right.
what kind of data you are computing with terraform? I am guessing ids/arns of resources ?
@Nikola Velkovski Yes, ids and arns.
that is easily doable with aws cli
you are most probably using it for baking the ami, the dependency of terraform just makes it more complex.
I guess I’ll have to use aws cli instead. Thanks for your help.
You are welcome, usually I do not use terraform for things that are contstantly changing, deploys etc.
It gets cumbersome pretty quickly.
How come you aren’t using Packer to bake AMIs?
—deleted because i’ve realized im stupid and can’t read —
@sahil you can use terraform to setup codebuild with packer to setup your ami building pipeline.
After that you can use the aws_ami datasource to get the last built ami.
@sahil what about building a base AMI and passing in the Terraform computed variables as part of the user_data script?
@davidvasandani Actually that might work. Thanks!
@sahil No problem! Building a base AMI in Packer that can be used in both a staging and prod environment with different vars loaded at boot via Terraform make testing much easier! Keep us updated with your progress, if you run into any issues, or with a success!
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Sep 11, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
2019-09-03
I’m not sure if this is a TF12 problem or not, but I made another module just recently and this seemed to work, however - providers and their aliases are not found by the module anymore:
Error: Provider configuration not present
To work with
module.cicd-web.aws_iam_policy_attachment.cloudformation_policy_attachment its
original provider configuration at module.cicd-web.provider.aws.nonprod is
required, but it has been removed. This occurs when a provider configuration
is removed while objects created by that provider still exist in the state.
Re-add the provider configuration to destroy
module.cicd-web.aws_iam_policy_attachment.cloudformation_policy_attachment,
after which you can remove the provider configuration again.
I found this link: https://github.com/hashicorp/terraform/issues/21472 that states that providers need to be explicitly passed down to the module, which I tried but still doesnt work
2019-09-04
Cross posting this from r/terraform: https://old.reddit.com/r/Terraform/comments/czjnvq/analysis_paralysis_bootstrapping_a_new_terraform/ Anyone here have good examples for bootstrapping a clean parameterized Terraform deployment on AWS?
@Matt is this your thread? If so #geodesic is a great tool that many of us will talk to you about
It avoids Workspaces & wrappers like Terragrunt
Rationale for “great tool” https://github.com/osulli/geodesic-getting-started/blob/master/docs/why-geodesic.md
yes, that’s my thread @oscar
I will take a look at Geodesic
@Matt tonight join https://zoom.us/meeting/register/dd2072a53834b30a7c24e00bf0acd2b8
It is Sweetops weekly hour session
Great chance to get a demo and ask Qs.
#office-hours starting in 15 minutes
#office-hours starting now! ask questions, get answers. free for everyone. https://zoom.us/j/508587304
not sure I can make it @oscar, not this week
but this is one of my major grips about Terraform which I generally like a lot
otherwise
2019-09-05
Sorry, for your modules, they compatible with Terraform version 0.12+?
not all of them are converted to 0.12 yet (we are working on it)
those that were converted have hcl2 label https://github.com/cloudposse?utf8=%E2%9C%93&q=hcl2&type=&language=
May I can help you to convert it into hcl2?
you can thanks. We also adding Codefresh instead of Travis, and adding tests which are deployed to AWS using Codefresh pipelines (this complicates the task for you)
https://github.com/terraform-providers/terraform-provider-aws/issues/9995
Add your thumbs up if that would be useful to you.
being quite busy the past few weeks (going back to fixing terraform-lsp on the weekend) was working on a nice research and production project
the project being openstack on nomad
Now I’m just curious was CloudKitty is
Billing and Chargeback service of OpenStack
2019-09-06
Hey guys, I am looking for a way to have my AWS autoscaling group perform a shutdown script before scaling down. The only way I can find to do this is using lifecycle hooks > Cloudwatch Events > lambda > SSM . But this seems quite a chain to string together. Any suggestions?
the life cycle hook is probably the way to go, but you could try https://opensource.com/life/16/11/running-commands-shutdown-linux as well
i’m not sure what your use case is and how important it is to get the shutdown script to run appropriately OR if you OS is event linux.
unless the OS goes bad, the scale in in an ASG will try to let the instance shutdown gracefully - this would let units in /usr/lib/systemd/system-shutdown/ run. i’m not sure what the timeouts would be before a forceful termination by the ASG.
K99runmycommandatshutdown from the link above works really well in both ASG’s and SpotFleet instances.
Thanks!
I will give it a crack and see if this is for for purpose as it has a lot less moving parts. Thanks for the assistance @Jonathan Le @davidvasandani.
imho that’s the only way
it is normally used for ecs/ecs2 connection draining on scale in(down)
but in your case it might be even more complex since the script has to report success
Thanks @Nikola Velkovski
you are welcome
was hoping someone with regex expertise can help me out here, Im trying the following:
${replace(var.project, "/\\s$/", "")}
where var.project is a string that will end in the letter s. I’m trying to strip the s at the end but im not having any luck. When I run this the s remains. Any ideas?
You need to use $1 as described here - https://nedinthecloud.com/2018/08/27/terraform-fotd-replace/
@Brij S Your regex is replacing \s, not s
Try ${replace(var.project, "/s$/", "")}
The other approach, to @antonbabenko point, is to match the whole string: ${replace(var.project, "/^(.*)s$/", "$1")}
or just use substr("str", 0, length("str") - 1) without messing up with regex (https://blog.codinghorror.com/regular-expressions-now-you-have-two-problems/)
2019-09-09
Hi, How can I reference resources created with for_each?
Below the example what I try to accomplish:
locals {
users = ["user1", "user2"]
}
resource "aws_iam_user" "this" {
for_each = toset(local.users)
name = "${each.value}"
}
resource "aws_iam_access_key" "this" {
for_each = toset(local.users)
user = # Reference above created users
}
Hi @Michał Czeraszkiewicz
user = aws_iam_user.this[each.key].name
@maarten thx 
Hello
is terragrunt considered a “best practice” tool to be using?
I don’t know if I consider it a “best practice” tool… most of the must have features of terragrunt (i.e. state locking during apply) have made it to terraform, my answer would be different if this was asked a year or so ago
just asking because i previously worked in a place that we had one terraform repo per environment(dev,prod,staging)
and we had to do alot of repeatable work in each env
It could be used for a very specific style of writing TF to keep things DRY, though you can do this now with workspaces, as well… I’m a fan of this workflow: https://github.com/cochransj/tf_dynamic_environment_regions
Note: a lot of that assumes you are using TF 0.12+
thanks
another question
imagine that i have an RDS instance in prod env, but i do not have it in dev env
i would be able to accomplish this with terragrunt or workspaces?
sure
conditional statement on count which i believe is possible in 0.12, psuedo code: IF $terraform.workspace == “prod” THEN count = 1 ELSE count = 0 on the RDS resource
When we talking about multple accounts, teams, environments…for me, terragrunt has been totally necessary to keep my terraform code organized and without so much boilerplate code
thx
one last question
i used to work with 1 big tfstate per environment
is a best approach to be using multiple tfstates per resourcegroup?
so we can manage VPCs individually, EC2s individually, etc….
we like to split tfstate on multiple dimensions… such as team and stage and app and stateful/stateless…
though, “individually” is a bit relative… for many actions, you can use -target to restrict the scope of an action… splitting tfstate helps reduce the blast radius of accidents better IMO though
nice
also see the recent thread/posts by @Erik Osterman (Cloud Posse) in #geodesic for another approach… https://sweetops.slack.com/archives/CB84E9V54/p1567187759027700
each team has autonomy to apply without interfering in other team infrastructure
basically, i just started in a new company, nothing in IAC yet
and i’m researching good strategies/architectures to start our environments
and i’m thinking about splitting into multiple tfstates, because we have a project to make a disaster recovery plan. We should be able to disaster recover only some portions of our infrastructure, into another region
we use terragrunt because it is visual (hierarchy/tfstate by directory structure) and easy to comprehend. tf workspaces are less visible in that sense and IMO harder to “know” where you are working. geodesic is solving similar problems in another way entirely
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Sep 18, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
does anyone know how to get the function name of a created lambda?
resource "aws_lambda_function" "s3_metadata" , can this resource be accessed via aws_lambda_fuction.s3_metadata.id ?
the docs dont make it apparent..
aws_lambda_fuction.s3_metadata.function_name
since tomorrow is HashiConf
is that a prediction or a question
I am giving my prediction that maybe they will officially announce packer 2.0 with HCL2?
how can i pass db_subnet_group_name to aws_rds_cluster resource using data object ?
currently i am trying to use
db_subnet_group_name = "${element(data.my_state.networking.database_subnets,1)}"
2019-09-10
does anyone know how to read subnet name from state file ?
@rohit doesn’t look like there is a data source for this yet: https://github.com/terraform-providers/terraform-provider-aws/pull/9525
@sarkis thanks. I will try a different alternative then
terraform plan getting a cost estimation feature on TF Cloud interesting…
I didn’t find any references to ECS Service Discovery in CP modules. Is it because everyone is running an alternative solution?
For someone getting started with containers, and not having more than 3-4 services at the most, should I even bother with orchestration and/or sophisticated methods of service discovery?
Or will ALB/ECS combo get the job done?
for that we usually deploy https://istio.io/docs/concepts/what-is-istio/ in the k8s cluster
don’t have anything in TF
That’s what AWS AppMesh does, right? I wonder if that’s an overkill for my use case though.
yes AppMesh should do similar things. we did not use it yet
for 3 static services might be an overkill but at the same time, you get an experience and be able to use it with tens of services
Has anyone played with the new Terraform SaaS offering?
Looks like TF cloud has hit GA
2019-09-11
Not yet but took a read. Keen to hear someone’s experience & comparison to local Geodesic workflow / CI tools using Geodesic workflow / Atlantis
definitely +1 on this. This is the workflow that we’ve just committed to, so keen on hearing peoples experiences!
I have a hard time getting the cloudposse modules to work with the recent terraform version (v0.12.8). I feel like I’m missing something, any ideas?
make sure all modules you are using are converted to TF 0.12
For example, this one is now cloudposse/terraform-aws-alb
Don’t know about all modules in terraform-aws-modules/....... (they are CloudPosse’s)
Yeah, so it is indeed an issue with the module implementation itself?
not implementation
the modules that are still in TF 0.11 syntax will not work in TF 0.12 (with a few small exceptions)
Try:
on .terraform/modules/alb_magento2/main.tf line 33, in resource "aws_security_group_rule" "http_ingress":
33: cidr_blocks = [var.http_ingress_cidr_blocks]
Change: removal of "${ and }"
If that doesn’t work, try:
cidr_blocks = var.http_ingress_cidr_blocks
.. since it is already a list
#office-hours starting now! ask questions, get answers. free for everyone. https://zoom.us/j/508587304
2019-09-12
How are folks doing multi region as far as Terraform goes…?
provider per region, pass the provider explicitly to each module/resource
these guys have the best reference i’ve seen for it, https://github.com/nozaq/terraform-aws-secure-baseline/blob/master/providers.tf
Workspaces ?
I’m more interested in things like what you do with the state file
is it the age-old question of one giant state, or many smaller states? i think either way it would be controlled by the backend config…
you can have a backend config with a credential that keeps it all in one region if it is one state, should work fine, even if the resources are in multiple regions
or a backend config per state where you apply some rationale/logic to where you want that state stored…
I don’t think this is so simple. You can’t have state for multi regions all in a bucket in one of the regions
the region goes down, which maybe the reason you have gone multi region in the first place, now you can’t get to your TF state
why not?
that’s a different issue, not a technical limitation of tf
I wasn’t talking specifically about restrictions by TF, I’m wondering how people are doing it in a sane way
re a conversation I’ve just had with @Nikola Velkovski
would cross-region bucket replication be sufficient?
set that up on your backend, then repoint your backend config in tf if you need to use another region
Yeah that could get you out of a bit of a hole, but I don’t want to have to repoint backends etc
what is your backend? can you do consul or something in a cross-region way?
S3
This is what I found regarding remote state and workspaces
sorry here it is for s3
hmm no mention of changing the bucket with workspaces
IIRC you can’t use interpolation in the backend block
with s3, to avoid manually re-jiggering your backend, you would need to be managing the s3 endpoint rather explicitly, doing some kind of health check on the real endpoints and re-pointing things as necessary
and you may still hit problems when running tf, since you’d have to also be quite careful about targeting resources to avoid running against the downed region
I haven’t yet seen a setup that actually addresses these problems. setting up multiple providers in the same state feels like half a solution, and one that will likely bite you when you need to reach for it
It isn’t easy
yeah, if this is that big a concern, you may be best off confining a state to a single region as much as possible, and setting up your app accordingly (deploy independently to multiple regions)
still may need some coordination layer perhaps that your app states depend on, but now your cross-region blast radius is confined to just that resource
which goes into a state bucket for each said region
replicate between each other maybe
using one bucket, different paths of state per regions you can do that manually means having below tree:
providers/aws
├── eu-east-1
│ ├── dev
│ ├── pre
│ ├── pro
│ └── qa
└── eu-west-1
├── dev
├── pre
├── pro
└── qa
or you can use terraform workspaces
terraform workspaces can’t be interpolated into backend config AFAICR
I think one bucket isn’t ideal…
for me one bucket seems ideal, and you can only play with paths inside it.
and if eu-west-1 goes down? you can’t provision in eu-west-1 OR other region ?
s3 is a global service
buckets are regional, have def seen S3 in a region go down before (not often but has happened and one of the drivers for going multi region for me)
Ah my bad S3 bucket name is unique globally, confused totally agree with you on that spin up a bucket for each region is ideal
@joshmyers would Aurora Serverless Postgres as a TF backend solve this problem?
I believe that if a region went down the DNS would just failover to the new promoted master in a new region.
or leveraging Minio distributed across multiple regions (or even cloud providers!) https://dickingwithdocker.com/2019/02/terraform-s3-remote-state-with-minio-and-docker/
Thanks @davidvasandani, will have a look!
Let us know what you end up going with? I know at some point I’ll need to address a more robust TF backend.
continuing the thread in order to go multi region/environment we can do something like this
locals {
environment = element(split("_", terraform.workspace), 1)
region = element(split("_", terraform.workspace), 0)
}
output "region" {
value = local.region
}
output "environment" {
value = local.environment
}
and the the workspace should be set like
eu-west-1_staging
it’s a bit hacky but does the trick
backends don’t allow interpolation, so you are gonna need some kind of wrapper to get different buckets per region without inputting vars etc
yes it also doesn’t tackle the state problem
but it sounds like you don’t want to put your state in an s3 bucket
maybe other backends might work for you ?
No I think S3 is fine, but it needs to be regional specific and therefore named buckets, so need some way to easily toggle the backend bucket too
The Aurora Postgres idea is interesting, but a few things. Requires much more setup, automating that is possible, but a pain. Requires credentials. Doesn’t solve one of the problems we spoke about. State would be all good in the case of a regional failure as DNS should flip over to the other region and should be all good, but if you have multi region provider in a single run anyway, you are gonna have a hard time if one of those regions is down
Half of your apply or so is gonna fail, potentially leaving you in an interesting state
Really good point.
Have you thought of ways to simulate this?
Nope, and my guess is that when AWS breaks in such a way, all bets are off anyway, but moving onto a client where this is a major concern and wanted to know others feelings
@Erik Osterman (Cloud Posse) any thoughts?
I believe the plan is to usually decouple the infrastructure and application so that the application self heals until the provider resolves the outage (ie don’t try to terraform while S3 is offline )
but looking to hear Erik’s thoughts on this.
terragrunt is a wrapper that lets you use some interpolation in the backend config, it resolves it and constructs the init command for you 
heh, I know, that is about all I want out of it at this point! lol
+1 for #terragrunt
@Mike Whiting can you share how you are invoking the module?
@Mike Whiting has joined the channel
@Nikola Velkovski yep that’s the one
oh that woiuld be me
what a coincidence
are you using terraform 0.12 by any chance ?
yeah
unfortunately it has not been ported yet
but I can dedicate some time and do it
that would be awesome
cool
just to clarify…
this will enable ec2 instances to log to cloudwatch events
what do you mean by cloudwatch events
Cloudwatch events are cron like jobs
ah ok
it will add additional metrics to cloudwatch
I just want to see logs from docker
ah that’s not it.
in order to see the logs
from docker you’‘ll need to have:
- the dockerized app to write to stdout
- iam role for the ec2 machines to write to cloudwatch logs
- a log group in cloudwatch logs
I think that should do it
sounds good…
however let me explain how I got here
oh and this
I’m creating a aws_ecs_task_definition and I suspect the service is failing to start because the docker image resides on a gitlab image registry and I imagine it’s not possible to use a docker image somewhere where the authentication isn’t through AWS
but I was hoping to see evidence of that through some kind of logging
if it’s ECS/EC2 then you can ssh into the machine and check the agent logs
otherwise you’ll need to set up logging
from experience the most usual problem is that the instances do not have internet
you can try with a simple docker image
e.g. nginx
and see if it works
if I use a vanilla docker image e.e. jenkins:lts which is available publically then everything works
*e.g
so it’s not an internet issue
you should see how to authenticate through docker with gitlab
makes sense.. I suppose actually I just need to perform the authentication through the user_data field of aws_launch_configuration
pretty much
thanks.. that’s given me some direction
@Erik Osterman (Cloud Posse) what’s the workflow in this case, should Mike create an issue ?
Ya if he needs it now, the best bet is to fork and run terraform upgrade
cool thanks
hey guys i am trying to create a ec2 (after taint’ing the existing ec2), attaching the ebs volume(using aws_volume_attachment), and using an user-data script in my tf to mount the volume (which is /home), and also trying to import some data from /home to the newly created instance, problem is many times the /home is not mounted, and the /var/log/cloud-init-output.log shows No such file or directory to the files i am trying to import, any thoughts on this ?
^ hope that question is not confusing
problem is many times the /home is not mounted
is it never mounted, or just sometimes fails?
well most of the times it is never mounted(9/10 times) , i manually ssh into the instance and do a sudo mount -a and it mounts, i tried adding sudo mount -a to the user-data script itself – doesn’t help
might be some race conditions
e.g. something (EBS) is not ready yet, but the code tries to mount it
try to add a delay for testing
or maybe there are some settings to wait
i tried adding sleep 60 in the user-data script which didn’t work, OR should i add something to the terraform itself for the wait thingee ?
no, tf does not wait for ramdom things, just for resources to be created (and not for all in all cases)
oh got it, will try some combinations of wait in my user-data script itself
still trying ways mentioned from ^ stackoverflows, came up with
if sudo blkid | grep /dev/xvdb > /dev/null; then
sudo mount -a
else
sleep 10
fi
any elegant approaches to make that in a loop ?
update: right now adding a direct sleep 10 (without any loop) seems to have solved the problem
hey all, all of a sudden i’m getting this error when running terraform apply
Error: error validating provider credentials: error calling sts:GetCallerIdentity: NoCredentialProviders: no valid providers in chain. Deprecated.
I have no idea why this is happening. The only thing I did was add some credentials to my .aws/credentials file.
My providers look like this
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "storesnonprod"
alias = "nonprod"
}
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "storesprod"
alias = "prod"
}
does anyone know what might be causing this?
@Brij S is it possible you mucked up your .aws/credentials toml format so it’s not being parsed correctly by the TF provider?
@sarkis, it seems when I add some old profiles back to the credentials file it works. But when I remove them i get the error
do your profiles depend on each other? i think there was something like source_profile i can’t remember the exact param
no
two seperate profiles as you see in the snippet above
can you share your ~/.aws/credentials file in DM and redact sensitive data
sure
Anyone using a tool like drifter or terraform-monitor-lambda to detect state drift?
Any success or best practices for identifying and correcting Terraform changes over time?
While these can be useful in a small environment, they are supporting a problematic process and are not going to scale well
If using micro services, there will be a state file per microservice. Lets say you a small environment with only 10 service and you have 3 environments. That’s 30 state files plus a few more for environment infrastructure and a few for account infrastructure. This can grow fast
Then there is the bad process that it is supporting. People making production changes from their local systems with possibly no testing or any audit tracking. A much better process would be to commit the change to git repo and have that trigger the terraform run. This gets rid of the drift issue due to uncommitted changed. It also allows you to add testing and ensure it is run, as well as having an audit trail
master brach reflects what is deployed to prod. With all the history from the PRs. There should be no drift since what’s is currently in master is deployed to prod with terraform/helm/helmfile
what we usually do to make and deploy a change to apps in k8s and serverless: create a new branch, make changes, open a PR, automatically (CI/CD) deploy the PR to unlimited staging so people could test it, approve the PR, merge the PR to master, cut a release which gets automatically deployed to staging or prod depending on release tag
for infrastructure (using terraform): create new branch, open a PR, make changes, run terraform plan automatically, review the plan, approve the PR, run terraform apply, if everything is OK, merge the PR to master
we use atlantis for that
2019-09-13
@Brij S you can verify if the profiles are properly set with aws cli
e.g. aws s3 ls --profile storesnonprod
because terraform uses that
or by
AWS_PROFILE=profilename aws s3 ls
I need something like “random_string” resource, but with a custom command. So, execute a command only if the resource isn’t in the state yet (or was tainted), and use commands output as a value to put in the state. Any idea what kind of magic to use to achieve such result?
if you provide the param in var, use it
if not, use random string to generate it
Thank you. Unfortunately it’s not the thing I look for. I need something like:
resource "somemagicresource" "pass" {
command = "openssl rand -base64 12"
}
Unfortunatelly provisioners doesn’t store any kind of result in a state.
I’ll try to go with https://github.com/matti/terraform-shell-resource , but thanks for all the links provided
Have you guys seen this before?
Error: Error creating S3 bucket: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
status code: 409, request id: aaaa, host id: aaaa//bbbb+cxxx=
It doesn’t exist on ANY of our accounts
its a very, very, specific and niche bucket name
the changes someone else owns it is extremely slim
I’m sure I saw this about 4-5 months ago, but it actually was created on one of our accounts
we saw that happens when you create a resource (e.g. bucket) not using the TF remote state. Then the state was lost on local machine. Then TF tries to create it again, but it does exist in AWS
check that you use remote state and not losing it
Yeh that adds up with what potentially happened
That local state file is long gone
How can I recover the S3 bucket? :S
you need to find it in AWS
and either destroy it manually in the console, or import it
It isn’t there
I’ve searched hte account (it has no buckets) - new account
make sure you have permissions to see it (maybe it was created under diff permissions)
Yeh I’ve checked sadly with Admin permissions in console
It genuinely isn’t there, even got one of the IT guys to look
I’ve opened a ticket with AWS but slow
S3 is global, so you need to check all your accounts, even those you don’t know about
Wait so, it could be on a different aws account to that on which I ran terraform?!
it could be. I don’t remember if AWS shows the error Error creating S3 bucket: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it in this case
AWS console just shows name already in use
when attempting to replicate but in the console
aghhh
so yes, you created it before and lost the state (if you are saying that the chance is very slim that some other people used the same name)
But surely even if I lost the state file
the s3 bucket would be on the aws account
btw the context is the backend module
Weird it has happened again on another account
it is in my local state file
a resource
but it isn’t on the console
what is going on
you have to follow exact steps when provisioning the TF state backend because you don’t have the remote backend yet to store the state in
Yeh no I know
it was an accident
I’m familiar with it
you have to provision it first with local state, then update TF code to use remote S3 backend, then import the state
Probably the 18th account I’ve used your module on.. just something weird happened this time
Ya, I do this:
run:
# Apply Backend module using local State file
direnv allow
bash toggle-s3.sh local
terraform init && terraform apply
# Switch to S3 State storage
bash toggle-s3.sh s3
terraform init && terraform apply
and my toggle-s3.sh script basically comments out hte backend
ok
It’s worked plenty time
not sure what happened this time though
i guess the bucket with that name exists for any reason (you created it, other people created it on diff account, or other people from diff orgs created it). Try to use a diff name
No I think something weird is happening
on a second account I’m getting this
Error: Error in function call
on .terraform/modules/terraform_state_backend/main.tf line 193, in data "template_file" "terraform_backend_config":
193: coalescelist(
194:
195:
196:
|----------------
| aws_dynamodb_table.with_server_side_encryption is empty tuple
| aws_dynamodb_table.without_server_side_encryption is empty tuple
Call to function "coalescelist" failed: no non-null arguments.
I’ve followed the same pattern and commands as many accounts previously
all version locked etc
No clue why it isn’t having it today
annnnnd its working again
what the ?
what theee
that magic bucket thats there but not there?
can see it on aws cli
but not console
whaat
same permissions (iam role)
@oscar I think you mixed up TF versions
if you use 0.11, use 0.11 state backend
same for 0.12
Its aws
look at htis
✗ . (none) state_storage ⨠ aws s3 rm <s3://xxx-terraform-state>
-> Run 'init-terraform' to use this project
⧉ xxx
✗ . (none) state_storage ⨠ aws s3 ls
2019-09-13 14:20:17 xxx-terraform-state
even after removing it is stil there hahaaha jeez
or, another posibility, aws provider was not pinned, got updated, and the new one has issues
we had a few 0.11 modules basted after the aws provider was updated
I think you nailed it akynsh#
andriy*
managed torecover the state file by cli#
Error: Failed to load state: Terraform 0.12.6 does not support state version 4, please update.
that was released just this morning
I wonder if I had the .terraform/modules directory already there
so the conclusion is, always pin everything, TF modules, TF version, providers, etc.
Damn I have it pinned to major
aws = “~> 2.24”
so I found out why my module was using different credentials. It was because i had a [main.tf](http://main.tf) in my module with the following content:
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "ambassadorsnonprod"
alias = "nonprod"
}
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "ambassadorsprod"
alias = "prod"
}
However, when I remove this [main.tf](http://main.tf) file from the module, and run tf plan with configuration that references this module I get the following error:
To work with
module.cicd-web.aws_iam_policy_attachment.cloudformation_policy_attachment its
original provider configuration at module.cicd-web.provider.aws.nonprod is
required, but it has been removed. This occurs when a provider configuration
is removed while objects created by that provider still exist in the state.
Re-add the provider configuration to destroy
module.cicd-web.aws_iam_policy_attachment.cloudformation_policy_attachment,
after which you can remove the provider configuration again.
I have a main.tf that is setup so I’m not sure why im getting this error
ominously similar to my situ
I am looking at using the SweetOps s3_bucket module but I am not sure how to enable server access logging using the module. Does the module support enabling server access logging?
depending on the s3 module you want to use
this one supports it https://github.com/cloudposse/terraform-aws-s3-website/blob/master/main.tf#L48
I’m having some issues with Terraform and terraform-aws-rds-cluster module, I’m creating a Global cluster ( I forked the cloudposse module and added one line) but this is not just related to to global aurora clusters but the problem is that the cluster finish creating but terraform for some reason keeps pooling for status until it times out after 1 hour , this is what I see :
module.datamart_secondary_cluster.aws_rds_cluster.default[0]: Creation complete after 9m10s [id=example-stage-pepe1secondary]
module.datamart_secondary_cluster.aws_rds_cluster_instance.default[1]: Creating...
module.datamart_secondary_cluster.aws_rds_cluster_instance.default[0]: Creating...
module.datamart_secondary_cluster.aws_rds_cluster_instance.default[0]: Still creating... [10s elapsed]
module.datamart_secondary_cluster.aws_rds_cluster_instance.default[1]: Still creating... [10s elapsed]
module.datamart_secondary_cluster.aws_rds_cluster_instance.default[1]: Still creating... [20s elapsed]
module.datamart_secondary_cluster.aws_rds_cluster_instance.default[0]: Still creating... [20s elapsed]
that will continue for 1 hour…..
and the console will show it as available
you see that eveytime you provision or just saw one time?
have you seen this before ?
no
if it only once, I’d say your session had expired
it is pretty consistant
the workaround was to create the secondary cluster with 0 instances
then change it two instances
pretty much every time
I mean I have not been able to successfully complete the creation the cluster
has anyone used multiple providers for a module?
Ive done this multiple times with success, but now I’m facing an issue where all resources are created in one account(provider) and not the other and i’m not sure why
@Brij S you’ll need to post some code or errors for us to help you diagnose.
I had an issue like that yesterday, the name of the resource needs to be different and you need to pass the provider alias to every resource
in my /terraform/modules/cicd folder Ive got a [main.tf](http://main.tf) file with the following:
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
alias = "nonprod"
}
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
alias = "prod"
}
in my /terraform/cicd/stores folder Ive got a [main.tf](http://main.tf) with the following:
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "storesnonprod"
alias = "nonprod"
}
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "storesprod"
alias = "prod"
}
and ive got a /terraform/cicd/stores/web.tf file Ive got
module "cicd-web" {
source = "../../modules/cicd-web"
providers = {
aws.nonprod = "aws.nonprod"
aws.prod = "aws.prod"
}
........
in all of my resources ive got either a provider = "aws.nonprod" or provider = "aws.prod" but they all get created in aws.nonprod
@davidvasandani ^
However, I realized that if I put profiles in /terraform/modules/cicd/main/tf then it works! However, that defeats my purpose of the module since id want to use different profiles for different accounts
there is no difference between these providers
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
alias = "nonprod"
}
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
alias = "prod"
}
they are the same
thats a good point.. didnt notice that
they need to have some diff, e.g. region
but the region is the same as well
if i remove that main.tf from the module I get an error saying it needs it
they have to be diff otherwise why do you need them
provider "aws" {
region = "us-west-2"
shared_credentials_file = "/Users/tf_user/.aws/creds"
profile = "customprofile"
}
diff region, or diff profile, or diff shared_credentials_file
right, I can add profile but if that lives in the module I cant reuse it
for another set of accounts
which tells the provider to use diff credentials from diff profile to access diff account
you need to add that
if I leave profile in the [main.tf](http://main.tf) in the module, then I cant reuse the module
because another account will have a different profile
whatever you are saying you can’t reuse, does not make any diff for terraform
so in my module, ``/terraform/modules/somemodule` i have a main.tf which includes a profile which is used for account A
you create a set of providers (they should differ by region or profile)
differ by profile, yes
then for each module, you send a set of required providers
and in each resource use the provider aliases
there is no other way of doing it
wait, in the module, the main.tf if I put a profile in
how does the module become reusable if the profile is hardcoded for a certain account
the module is reusable because you send it a list of providers (which can contain only one)
and the module uses that provider
w/o knowing the provider details
yes I understand that
so that means, I remove [main.tf](http://main.tf) from my module?
(which causes errors)
not sure I understand that
ok let me explain
you fix the error in main.tf
not remove it
in /terraform/modules/somemodule/main.tf I have:
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
alias = "nonprod"
}
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
alias = "prod"
}
In /terraform/folder/main.tf I have:
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "storesnonprod"
alias = "nonprod"
}
provider "aws" {
version = "~> 2.25.0"
region = var.aws_region
profile = "storesprod"
alias = "prod"
}
In /terraform/folder/web.tf I have:
module "cicd-web" {
source = "../../modules/somemodule"
providers = {
aws.nonprod = "aws.nonprod"
aws.prod = "aws.prod"
}
that is how im using the providers
can you have multiple providers in a module ?
you can. kinda need to when you want to implement a cross-account workflow, for things like vpc peering, resource shares, etc…
I think you can but should you do it ?
if I remove /terraform/somemodule/main.tf I get this error:
Error: Provider configuration not present
To work with
module.somemodule.aws_iam_policy_attachment.codepipeline_policy_attachment its
original provider configuration at module.somemodule.provider.aws.nonprod is
required, but it has been removed. This occurs when a provider configuration
is removed while objects created by that provider still exist in the state.
Re-add the provider configuration to destroy
module.somemodule.aws_iam_policy_attachment.codepipeline_policy_attachment,
after which you can remove the provider configuration again.
in my case I instantiate the module twice one with one provider and one with the other
look at this example
they don’t create one resource within two providers
in my module I have multiple resources that have either provider = "aws.nonprod" or provider = "aws.prod"
mmm, maybe moving to a module that can do do any provider and the pass one provider to the module
ok, we are mixing up at least 4-5 diff concepts here
sorry
- @Brij S if you created resources using a provider, you can’t just remove it. Delete the resources, then remove the providers from
[main.tf](http://main.tf), then re-apply again
- @Brij S your providers must be different (that’s after you do #1). Otherwise TF uses just the first one since they are the same (that’s why eveything gots created in just one account)
- @jose.amengual you create a module, but don’t hardcode any provider in it. You can send the provider(s) to it IF nessessary
- But in (almost) all cases, it’s not necessary. The only use-case you need to send provider(s) to a module is when your module is designed in such a way so it creates resources in diff regions or in diff accounts (bad idea)
- Creating such a module that creates resources in diff region is OK (in this case you can send it a list of providers that differ by region)
- Creating such a module that creates resources in diff accounts is bad idea
@Andriy Knysh (Cloud Posse) could I show you the problem Im having? I dont have any resources created but i’m still getting the error
sounds like you have resources created
To work with
module.somemodule.aws_iam_policy_attachment.codepipeline_policy_attachment its
original provider configuration at module.somemodule.provider.aws.nonprod is
required, but it has been removed. This occurs when a provider configuration
is removed while objects created by that provider still exist in the state.
Re-add the provider configuration to destroy
i just ran terraform destroy, no resoruces found
could we zoom possibly?
regarding #6 above: instead of thinking of creating modules that uses providers for diff accounts, it’s better to create yourself an environment which will allow you to login into diff accounts (by using diff profiles in ~./aws, and eveb better by assuming roles)
2019-09-15
Are you using some of our terraform-modules in your projects? Maybe you could leave us a testimonial! It means a lot to us to hear from people like you.
Hey guys! I’m looking for some advice on how to approach an issue. I’m trying to figure out a way to use Terraform to provision a Windows Server 2016 instance that will run this cloud prep tool once it’s provisioned. I want to do something with Packer down the line but right now I’m just trying to make an easy way to spin up cloud gaming rigs on AWS for myself.
Prep tool: https://github.com/jamesstringerparsec/Parsec-Cloud-Preparation-Tool
Is what you’re after. There are plenty of examples out there on how to pass user-data to an ec2 instance in terraform.
@James D. Bohrman this link didn’t work for me.
2019-09-16
Hi @James D. Bohrman this might help “Deploying a Windows 2016 server AMI on AWS with Packer and Terraform. Part 1” by Bruce Dominguez https://link.medium.com/8hIu8JaK1Z
Does anyone have a good suggestion on creating a snapshot from a Rds database (that’s encrypted) and restoring it to a Dev/testing Env and doing some data scrubbing?
Suggestions yes, any of them any good? Not so sure
Have seen this done in several ways, none of which were particularly nice
@Bruce https://github.com/hellofresh/klepto looked interesting in this space last time I checked
(probably not a discussion for this particular channel)
Thanks @joshmyers I will check it out.
is anyone able to advise on aws_ecs_task_definition. If I specify multiple containers in the task definition file then neither of the containers come up.
but if I have just one it works
@Mike Whiting you are really going to need to post your instantiation of the Terraform resource or whatever. What you expected. What the actual error message is etc
did you mean to @ me?
these are the resources:
resource "aws_ecs_task_definition" "jenkins_simple_service" {
// volume {
// name = "docker-socket"
// host_path = "/var/run/docker.sock"
// }
volume {
name = "jenkins-data"
host_path = "/home/ec2-user/data"
}
family = "jenkins-simple-service"
container_definitions = file("task-definitions/jenkins-gig.json")
}
resource "aws_ecs_service" "jenkins_simple_service" {
name = "jenkins-gig"
cluster = data.terraform_remote_state.ecs.outputs.staging_id
task_definition = aws_ecs_task_definition.jenkins_simple_service.arn
desired_count = 1
iam_role = data.terraform_remote_state.ecs.outputs.service_role_id
load_balancer {
elb_name = data.terraform_remote_state.ecs.outputs.simple_service_elb_id
container_name = "jenkins-gig"
container_port = 8080
}
}
[
{
"name": "jenkins-gig",
"image": "my-image",
"cpu": 0,
"memory": 512,
"essential": true,
"portMappings": [
{
"containerPort": 8080,
"hostPort": 8000
}
],
"environment" : [
{
"name" : "VIRTUAL_HOST",
"value" : "<host>"
},
{
"name": "VIRTUAL_PORT",
"value": "8080"
}
],
"mountPoints": [
{
"sourceVolume": "jenkins-data",
"containerPath": "/var/jenkins_home",
"readOnly": false
}
]
},
{
"name": "nginx-proxy",
"image": "jwilder/nginx-proxy",
"cpu": 0,
"memory": 512,
"essential": true,
"portMappings": [
{
"containerPort": 80,
"hostPort": 80
}
]
}
]
if I remove the nginx-proxy container from the definition then ecs-agent successfully pulls and launches the jenkins container but with it included nothing happens
nb: ‘my-image’ is from a private registry and nginx-proxy is public
Do you have any error events being logged?
Are there creds for the private repo?
I’m just observing the ecs-agent logs currently (within the instance)
I followed this guide for the private registry stuff https://docs.aws.amazon.com/AmazonECS/latest/developerguide/private-auth-container-instances.html
as I say, the container from the private image launches fine when I don’t specifiy the proxy container in the definition file.. i.e. one container object
You hadn’t specific which one you can bring up on it’s own, or that one is in a private registry at that point
ECS agent logs should give you an idea
I can bring up the jenkins container (private image) on it’s own
when the nginx-proxy definition is present ecs-agent just sits idle
does that make sense?
yes
what do you think I should try?
starting to wonder if terraform is really for me if I can’t get help
(from anywhere)
Terraform is just making API calls for you
yep
The tags for this module are so confusing: https://github.com/cloudposse/terraform-aws-rds/releases
I’ve been using 0.11 by mistake as I took the ‘latest’ tag
but that’s actually just a hotfix
the latest .12 tag ios 0.10
true I could have read the list and lesson learned, but had me stumped for a while as to why it wasn’t working!
@oscar don’t pin to master/latest, always pin to a release. In the module, TF 0.12 started with tag 0.10.0, but when we needed to add some features to TF 0.11 code, we created the tag 0.9.1 which is the latest tag, but not for TF 0.12 code
Yes that’s what I mean
a 0.11 tag is at the top of the tags list
bamboozled me, logically I would have thought only 0.12 tags would be at the top of the ‘releases’ tab
that’s how GitHub works
so I had it pinned to a 0.11 until I realised what was going on
i don’t even see a 0.11 tag in there. there is a 0.11 branch…
exactly
0.9.1 is a TF 0.11 tag
oh you mean the 0.9.1 tag only supports tf 0.11
not that there is a 0.11 tag
Aye
confusing
bamboozles
we did not find a better way to support both code bases and tag them
Haha its fine, I was just pointing out it is a bamboozle
so we started a TF .12 code with some tag and continue incresing it for 0.12
It makes sense
what you are doing makes sense to me, releasing patch fixes on the 0.9 minor stream
for 0.11, usually increase the last tag for 0.11 branch
The lesson learned was ‘don’t just grab the top most tag’
would be cool if tf/go-getter supported more logic in the ref than an exact committish… a semver comparator (like ~>0.9) would be awesome
tf/go-getter waht does this do?
terraform uses go-getter under the covers to retrieve modules specified by source
I see, yeh that would be smart
checks the versions.tf file and cehcks for compatibility
@Andriy Knysh (Cloud Posse) I think I was doing the PR as you commented! https://github.com/cloudposse/terraform-aws-rds/pull/38
thanks @oscar, looks good (you even updated the README
running automatic tests now, if ok will merge
where abouts are your tests?
I couldn’t see them
I noted Codefresh wasn’t in the PR either
testing this example https://github.com/cloudposse/terraform-aws-rds/tree/master/examples/complete
Oh I see. When I navigated the test/ directory it looked like an example
but I realise now that examples_complete_test.go is related ot the examples/ dir
and that examples/ isn’t just documentation. Nice
Yah that’s some nice gitops
I was expecting a trigger
but that’s cooler
it is a trigger, but we have to trigger it (for security when dealing with PRs from forks)
Oh that makes sense actually
otherwise you could DDOS it
Yeh
merged and released 0.11.0 (now you have that tag ) thanks
woop, thanks
Debate/Conversation:
“We should enable deletion_protection for production RDS”
https://www.terraform.io/docs/providers/aws/r/db_instance.html#deletion_protection
For: anyone in console / terraform cannot accidentally delete (assuming IAM permissions are not super granular & TF is being operated manually)
Against: presumably this would mean the resource cannot be updated? I’m not too familiar with RDS so unsure on how many settings actually cause a re-create
better to enable it, but usually when you want to delete an RDS instance aws takes a snapshot of it as back up.
guys do you know when we will have count enabled for module
Not seen an ETA yet, just that it is reserved alongside for_each
?
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Sep 25, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
I’ve had a brainwave that perhaps I need to add another dedicated aws_ecs_service resource for the nginx-proxy - see my example code above. is this a possibility?
Is there a MKS/Kafka module anywhere?
Has anyone solved a solution for dynamically determining which subnets are free in a given VPC to then use for deploying some infrastructure into? Or know of some examples?
what do you mean by are free?
available ip address space
that’s not easy
yup; plus we have multiple cidr blocks (secondaries) being added to the VPC so in some cases the the secondary blocks are barely usable because subnets created off of them don’t garnish many ip address space (e.g. \28)
so yeah - in those cases basically need a way to filter away “unusable” subnets
the closet thing i’ve found is running a local cmd and finding a way to stuff it into a data template to somehow use downstream - kind of like the solution here: https://medium.com/faun/invoking-the-aws-cli-with-terraform-4ae5fd9de277
but all very ugly
you can use https://www.terraform.io/docs/providers/aws/d/subnet_ids.html to get all subnets for a VPC
does TF support inline code for lambda functions like cloudformation?
2019-09-17
Inside terraform(.tf) i can use assign dynamic stuff using variables like - key_name = "${var.box_key_name}" for different environments, how can i do the same inside the user-data scripts attached to tf, i am tyring to have unique values for sudo hostnamectl set-hostname jb-*environtmenthere* in the user-data script
hi gents, has any one of you succesfully created s3 bucket module with dynamic cors configuration?
not sure what you mean by ‘dynamic configuration’, but take a look here https://github.com/cloudposse/terraform-root-modules/blob/master/aws/docs/main.tf#L79
by dynamic configuration, I thought about utilizing terraform’s ‘dynamic’ feature
the same approach you linked I use right now but it forces to have any kind of CORS configuration applied to the bucket, even when you do not need CORS at all
with dynamic configuration I thought I will be able to create s3 buckets with or without cors configuration
that’s easy to implement
I ended up with something like this:
dynamic "cors_rule" {
for_each = var.cors_rules
content {
allowed_headers = [lookup(cors_rule.value, "allowed_headers", "")]
allowed_methods = [lookup(cors_rule.value, "allowed_methods")]
allowed_origins = [lookup(cors_rule.value, "allowed_origins")]
expose_headers = [lookup(cors_rule.value, "expose_headers", "")]
max_age_seconds = lookup(cors_rule.value, "max_age_seconds", null)
}
}
when variable cors_rules is a list of maps like this:
cors_rules = [{
allowed_origins = "*"
allowed_methods = "GET"
}]
however, this approach is still not perfect, because values not mentioned in the cors_rules variable will be applied anyway with default values
am I missing something ?
i don’t think it’s possible to do it, unless you want to use many permutations of dynamic blocks with for_each with different conditions
I see
thanks for answering
that’s how we deploy https://docs.cloudposse.com/
Here’s a little tool I’ve been working on that the gamers here might like. I used a lot of Cloud Posse modules also
for discoverability, have you considered renaming it to terraform-aws-parsec-instance? this is the format hashicorp suggests for the registry
I haven’t but I will probably do that!
@Andriy Knysh (Cloud Posse) finally so apparently A) Terraform state loading is private to itself in the UI and Command code, so I will need to talk to either paul or terraform team about it, B) and good news finally find out that loading in terraform is implicit cascading
terraform will declare main.tf to be empty
and skip reading
is useful, since I need it to do resource & data types gathering for error checking
2019-09-18
Anyone seen the issue where you curl from an EKS worker node to the cluster and get SSL issues?
Using CP worker / cluster / asg modules.
curl: (60) SSL certificate problem: unable to get local issuer certificate
… this is curling the API endpoint as per EKS
@Addison Higham I’m using your branches from here https://sweetops.slack.com/archives/CB6GHNLG0/p1566415698381800
Error: Invalid count argument
on .terraform/modules/eks_workers.autoscale_group/ec2-autoscale-group/main.tf line 120, in data "null_data_source" "tags_as_list_of_maps":
120: count = var.enabled ? length(keys(var.tags)) : 0
The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.
but getting the following error. Could you provide any guidance on what you think that might be?
yeah, that was an oopsie, a fix got merged… but maybe it didn’t make it onto the branch I was trying to upstream
lemme find it
Thanks. If possible could you push it to your fork’s master? :slightly_smiling_face: I did try your inst-* branch but that didn’t seem to quite fix it
oh that is a different issue @oscar, what are you passing to tags? as the error message says, it can’t have anything dynamic being passed in
tags is actually empty
I’m passing var.tags which is an empty {} in my terraform proejct that calls your eks_worker module
so am I correct in using your worker & cluster branches @master branch?
because I’m aware you also have the ASG one updated, but do the master branches of worker and cluster point to that?
oh yeah, so that is why we use the inst-version, which does this: https://github.com/instructure/terraform-aws-eks-cluster/pulls?utf8=%E2%9C%93&q=is%3Apr
to be safe, whenever I change refs, I also just delete .terraform directory and re-init
it is sorta weird, we didn’t want to open a PR to our updated module, but they do need to merge them in order for these to work
Ya I understand the need for the branch. I’ll give another go later on.
So worker inst Cluster master
And that should fix my previous issue with count?
I think so? at least that is what we have and don’t have any issues
public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304
@Addison Higham - darn still got the same issue
module "eks_cluster" {
source = "git::<https://github.com/instructure/terraform-aws-eks-cluster.git?ref=master>"
...
}
module "eks_workers" {
source = "git::<https://github.com/instructure/terraform-aws-eks-workers.git?ref=inst-version>"
...
same error?
Yeh
Error: Invalid count argument
on .terraform/modules/eks_workers.autoscale_group/main.tf line 120, in data "null_data_source" "tags_as_list_of_maps":
120: count = var.enabled ? length(keys(var.tags)) : 0
The "count" value depends on resource attributes that cannot be determined
until apply, so Terraform cannot predict how many instances will be created.
To work around this, use the -target argument to first apply only the
resources that the count depends on.
module "eks_workers" {
source = "git::<https://github.com/instructure/terraform-aws-eks-workers.git?ref=inst-version>"
namespace = var.namespace
stage = var.stage
name = var.name
tags = var.tags
...
}
var.tags is empty (defaulting to {})
is your cluster_name dynamic? see https://github.com/instructure/terraform-aws-eks-workers/blob/master/main.tf#L2, the workers module computes some tags, so your cluster_name needs to be known at plan time
Omg
that must be it
that is why in the example you see them use the label module to compute the name of the cluster in multiple distinct places
Interestingly though… https://github.com/cloudposse/terraform-root-modules/blob/master/aws/eks/eks.tf#L77
# mine
cluster_name = "${module.eks_cluster.eks_cluster_id}"
Will hardcode to a string for now
Super thanks. Cluster and workers up now
But back to workers not connecting to cluster.
@oscar did you apply this https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/kubectl.tf ?
Ah - no thank you. I saw this before but didn’t honestly understand it! Should this be run at cluster creation or can be applied afterwards?
so at the time we did it, in some cases there were some race conditions, that’s why we did not enable it by default
after the cluster applied, we set the var to true and applied that
Many thanks
but now you can test it with the var enabled from start
we did that almost a year ago so a lot prob has changed
and we will convert the EKS modules to 0.12 and add auto-tests this/next week (finally )
Would love to get a hold of those updated modules
Andriy you are my hero
My workers are now connected
TF weirdly got an unauthorized response when applying the command:
kubectl apply -f config-map-aws-auth-xxx-development-eks-cluster.yaml --kubeconfig kubeconfig-xxx-development-eks-cluster.yaml
but my kubectl already had the context activated
so I just ran the apply configmap without the –kubeconfig
@Andriy Knysh (Cloud Posse) XD XD XD XD XD, so I found the biggest issue that is causing vs code users for using the terraform lsp plugin, I forgot to omit the hover provider from the first release that I was trying out(so is very error prone), since I only use vim, so there is no hover that get activated
so now is alot more stable for any GUI based Editor that is going to use terraform-lsp
nice @Julio Tain Sueiras
@justingrote btw, didn’t realize you were on sweetops. We discussed your comment today #office-hours today https://github.com/hashicorp/terraform/issues/15966#issuecomment-520102463 (@sarkis had originally directed my attention to it)
@justingrote has joined the channel
i am facing issues with pre-commit when using in my terraform project
repos:
- repo: <git://github.com/antonbabenko/pre-commit-terraform>
rev: v1.15.0
hooks:
- id: terraform_fmt
- id: terraform_docs_replace
i receive the following error
pkg_resources.DistributionNotFound: The 'pre-commit-terraform' distribution was not found and is required by the application
any ideas on what could be the problem ?
@antonbabenko
@rohit Not sure if it will fix anything, but you can try changing the git:// to https://. Here’s mine for reference:
- repo: <https://github.com/antonbabenko/pre-commit-terraform>
rev: v1.19.0
hooks:
- id: terraform_fmt
- id: terraform_docs
i think the problem is with terraform_docs_replace
and maybe it has terraform version 0.11.13
i want to replace the README file automatically as part of commit
do you know if the same can be achieved using terraform_docs ?
its possible. I contributed terraform_docs_replace several months ago, it probably hasn’t been touched since then.
i think terraform_docs_replace is only supported in terraform v12
terraform_docs just makes changes to an existing README between the comment needles
<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
stuff gets changed here
<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
terraform_docs_replace was made quite a while ago, before 12 came out
when i update variables and their description in [variables.tf](http://variables.tf), my README.md files does not gets updated using terraform_docs
good read
if ive got a module such as:
module "vpc_staging" {
source = "./vpc_staging"
}
can I access a variable/output created in that module in another module like so?
module "security-group" {
source = "terraform-aws-modules/security-group/aws"
version = "1.25.0"
name = "sg"
description = "Security group for n/w with needed ports open within VPC"
vpc_id = "${module.vpc_staging.vpc_id}"
}
Would I use the variable name, or output id? What do I reference basically?
The second module can use the outputs of the first module. So, the vpc_staging module would need an output called vpc_id for that example you gave.
right! I thought so, just wanted to confirm
thanks
Hey guys, I do have a problem with the examples on the eks_cluster, more specifically on the subnets module It has an unssoported argument there:
An argument named “region” is not expected here.
unsupported
module subnets on main.tf: this line -> region = “${var.region}” Terraform complains about it not being an expected argument
the example is not actually correct since the EKS modules are TF 0.11, but the subnet module are pinned to master which is already 0.12
we are working on converting EKS modules to 0.12
for now, pin the subnet module to a TF 0,11 release
module "subnets" {
source = "git::<https://github.com/cloudposse/terraform-aws-dynamic-subnets.git?ref=tags/0.12.0>"
pin to 0.4.1 which is TF 0.11
Thank you mate
2019-09-19
How come only the creator of the EKS cluster can connect using the CP moduels?
By default, only the creator of the cluster has access to it using IAM. The aws-auth ConfigMap in the kube-system namespace controls it. You can add an IAM role mapped to a K8s group that will give anyone who is able to assume that role the ability the log in. Looks like CloudPosse’s implementation of the terraform-aws-eks-workers module doesn’t make this configurable yet.
Looks like the template for the ConfigMap is here: https://github.com/cloudposse/terraform-aws-eks-workers/blob/master/config_map_aws_auth.tpl
The EKS cluster example shows it being applied here: https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/kubectl.tf
Here’s an example of what it would look like with an IAM role bound to a K8s group that would give anyone that is able to assume the role my-eks-cluster-admin the ability to log into the cluster with cluster-admin privileges:
apiVersion: v1
kind: ConfigMap
metadata:
name: aws-auth
namespace: kube-system
data:
mapRoles: |
- rolearn: arn:aws:iam::REDACTED:role/REDACTED
username: system:node:{{EC2PrivateDNSName}}
groups:
- system:bootstrappers
- system:nodes
- rolearn: arn:aws:iam::REDACTED:role/my-eks-cluster-admin
username: my-eks-cluster-admin
groups:
- system:masters
mapUsers: |
mapAccounts: |
Then, you would change the command being run in your kubeconfig to use the role by using the -r flag in the aws-iam-authenticator token command.
Be advised that this will defeat some auditability because Kubernetes will see everyone as the user my-eks-cluster-admin. You can do a very similar thing with the mapUsers section in order to map each user you want to give access to with a username in Kubernetes.
The syntax for mapUsers is
mapUsers: |
- userarn: <theUser'sArn>
username: <TheUsernameYouWantK8sToSee>
groups:
- <TheK8sGroupsYouWantTheUserToBeIn>
Thank you we found the answer to this earlier on! Really apporeciate your detail!
We’re planning to fork it when 0.12 of the module goes live to support this customizability
thanks guys, we will add additional roles and users mapping (working on 0.12 of the modules now)
Ah that’s cool. My new firm is really keen to use CP’s own version of 0.12 (not the fork/PR branch). We have our own customizability reqs so once 0.12 is done and pushed we can start extending
https://github.com/hashicorp/terraform/issues/22649 anyone experiencing this out of nowhere? (All devs using the state file are on 0.12.6)
they have been busy adding new features
usually that happened when using 0.12 then trying to read the state with 0.11
but now looks like any version bump causes that
But everyone (2 people - we’re next to eachother) using that project are using the same geodesic shell and have the same version 0.12.6… yet the statefile in S3 says 0.12.7 O.O
neither of us have 0.12.7 which is super weird
geodesic has 0.12.6 as well?
yep!
or rather
we are both in geodesic
and terraform version is 0.12.6 on both our PCs
No one else feasibly ran this
inside geodesic, terraform version is 0.12.6 as well?
Yes
on our locals: 0.12.0
on our geodesics: 0.12.6
Whilst we’d like to know why, we’re happy to use 0.12.9 etc
.. but we’re using cloudposses terraform_0.12
@Andriy Knysh (Cloud Posse) I see that 0.12.7 is in your packages https://github.com/cloudposse/packages/blob/master/vendor/terraform-0.12/VERSION
however apk add –update –no-cache terraform_0.12 does not work as expected
Ok updated my geodesic FROM to 0.122.4 and that cleared the cache
now on 0.12.7
i thought you needed apk update && apk add --update terraform_0.12@cloudposse is the @cloudposse not required?
doh that must be it
merci beaucoup
granted, using the newest geodesic is also nice~ features and bugfixes, oh my
I was only coming from 0.119 - not that far behind!
I also usually customize that in my own dockerfile that wraps geodesic:
RUN apk add terraform_0.12@cloudposse terraform@cloudposse==0.12.7-r0
Is what’s in ours, but we only have one or two 0.12 projects, everything is mostly on 0.11 still
stupid question I’m using
locals {
availability_zones = slice(data.aws_availability_zones.available.names, 0, 2)
}
but sometimes my resources end up in the same AZ
better to just hardcode them ?
@jose.amengual what do you mean by sometimes? When in diff regions?
the code above is ok and should work
same region
no need to hardcode anything
I’m using the terraform terraform-aws-rds-cluster module
which I’m going to send a PR to support global clusters
you have to make sure you create the subnets in diff AZs https://github.com/cloudposse/terraform-aws-rds-cluster/blob/master/examples/complete/main.tf#L40
that example worked many times
I know is weird because if I recreate the cluster then it will work
I wonder now….maybe I just have a problem in one region
we use TF to create the accounts so in every region we subnets for every AZ
I was wondering if for some reason we made a mistake or something
but I’m using a data lookup to find them base on tags
yea check the data lookup if it returns the correct result
exactly what I’m doing
I’m gettin 3 subnet ids in us-east-1 and 4 in us-west-2
so the data lookups are good
hmm… maybe we need to specify AZs now https://www.terraform.io/docs/providers/aws/r/rds_cluster.html#availability_zones
cluster_size = 2 and I pass 4 subnets, then it should be ok
Has anyone else run into the issue where you can’t pass variables via the command line when using the remote backend since last week when they released terraform cloud?
Error: Run variables are currently not supported
The "remote" backend does not support setting run variables at this time.
Currently the only to way to pass variables to the remote backend is by
creating a '*.auto.tfvars' variables file. This file will automatically be
loaded by the "remote" backend when the workspace is configured to use
Terraform v0.10.0 or later.
Additionally you can also set variables on the workspace in the web UI:
<https://app.terraform.io/app/Boulevard/sched-dev-feature-branch-environments/variables>
Global cluster support PR @Andriy Knysh (Cloud Posse) https://github.com/cloudposse/terraform-aws-rds-cluster/pull/56
thanks @jose.amengual
commented
fixed
did you run
make init
make readme/deps
make readme
looks like README was not updated
and docs/terraform.md was deleted
weird
mmm
❰jamengual❙~/github/terraform-aws-rds-cluster(git:globalclusters)❱✔≻ make readme 5.2s Thu 19 Sep 18:51:17 2019
curl --retry 3 --retry-delay 5 --fail -sSL -o /Users/jamengual/github/terraform-aws-rds-cluster/build-harness/vendor/terraform-docs <https://github.com/segmentio/terraform-docs/releases/download/v0.4.5/terraform-docs-v0.4.5-darwin-amd64> && chmod +x /Users/jamengual/github/terraform-aws-rds-cluster/build-harness/vendor/terraform-docs
2019/09/19 18:51:24 At 3:16: Unknown token: 3:16 IDENT var.namespace
make: *** [docs/terraform.md] Error 1
hmmm
looks like something is broken (will have to look)
Looks like an old build harness
The ident error tells me that it’s using an old version of terraform
Terraform-docs does not support it natively, so we have a wrapper around terraform docs
ohhhh
one sec
I have two binaries
Also might get fixed if you blow away build harness and rerun make init. Just a hunch.
(On my phone so cant provide more detail)
done
Unknown token: 3:16 IDENT happened to me when TF versions mismatched
thanks guys
tested on AWS and merged
I am currently working on a new fix for the terraform-docs.awk wrapper here: https://github.com/antonbabenko/pre-commit-terraform/issues/65
If there are any other issues coming up, let me know
2019-09-20
Azure
Hi everyone, I’m about to move my big terraform configuration into separate modules, but I have a question about best practice regarding resource-groups.
If I will create resource-group resource in every of my modules, it will be fine, because it will be created once, but when for some reason I will remove entire module or I will try to redeploy it, wouldn’t Terraform want to delete my resource group (and all other resources-modules)? Should I rather use data resource to make reference to resource group created in another module or what are your ideas?
Thanks
hey guys… not sure whats going on but it looks like the 0.9.0 - terraform-aws-cloudfront-s3-cdn module is creating ARN IDs like
“arn
iam:user/CloudFront Origin Access Identity XXXXXXXXXXXXXXXX”
for S3 policies to allow Cloudfront access
ah that’s a new issue
I just encountered it today
oh thank god.
AWS changed how the API behaves
in the background
if you need a quick fix
I literally thought i was going crazyh
hahah it happened to me as well
i do ,. please
sec
the glorious fix is
aaaaah
can i just downgrade my provider version?
lets take a peak
principals {
type = "AWS"
identifiers = [replace("${aws_cloudfront_origin_access_identity.this.iam_arn}", " ", "_")]
}
Thank you!
for now you should be able to patch it until @Andriy Knysh (Cloud Posse) or @Erik Osterman (Cloud Posse) wake up and officialy fix it
haha Erik is a long time friend of mine, i can hold something over him i think to get it fixed
although, I was the one who was usually embarrassing themselves…
I think using the replacements only works for current state files, if you’re doing new policies you have to use type CanonicalUser and identifier s3_canonical_user_id
aaaah
nope that’s not going to work
It just applied for me.
even though CanonicalUser and identifier s3_canonical_user_id will pass tf apply
try it again
aws is changing it in the background
you’ll get a change on every apply
really
ugh
at least that’s what happened to me
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
damn
It wont take the replace suggestion, keeps telling me bad
Error: Error putting S3 policy: MalformedPolicy: Invalid principal in policy
gonna try something
it was too early, i was using dashes lol….
underscores work
thanks for the help Nikola!
gonna lurk here now….
you are welcome
2019-09-22
guys, any update on this https://sweetops.slack.com/archives/CB6GHNLG0/p1566415698381800 ? Really looking forward to use these modules with TF 0.12
yes, we are working on that now, will be done in the next 2-3 days
thanks for the update @Andriy Knysh (Cloud Posse)
2019-09-23
If I create NATs in one module, is there a way to get a list of NAT GW and pass it to a new sg with TF?
Your module can output the list of NAT GWs and you can do whatever you desire with that list
is that only if I am creating that sg within the same module?
Nope.
So there are levels. Think of them as boxes. Terrafrom resources have attributes( variables you set say, ami_name for an EC2 instance) and outputs( say instance_name). You can take that output and play around with it in the same module. Or you can get that output and push it out of your module — your module now outputs that value too.
Thank you! Is there another way to do it with using just data? like data aws_nat_gatway and then scrape for a list with tags
there are examples in terraform-root-modules of reading the output of other modules using their remote state.. https://github.com/cloudposse/terraform-root-modules/blob/9301b150c89a5543bdd2785ecdacf000ee6c5561/aws/iam/audit.tf#L15
@pericdaniel I believe this post will answer your questions https://blog.gruntwork.io/how-to-manage-terraform-state-28f5697e68fa#7077
Thank you!
@Andriy Knysh (Cloud Posse) https://github.com/cloudposse/terraform-aws-rds/pull/41
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Oct 02, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
Hi guys,
Any of you has experience with maintenance of SaaS environments? What I mean is some dev, test, prod environments separate for every Customer?
In my case, those environments are very similar, at least the core part, which includes, vnet, web apps in Azure, VM, storage… All those components are currently written as modules, but what I’m thinking about is to create one more module on top of it, called e.g. myplatform-core. The reason why I want to do that is instead of copying and pasting puzzles of modules between environments, I could simply create env just by creating/importing my myplatform-core module and passing some vars like name, location, some scaling properties.
Any thoughts about it, is it good or bad idea in your opinion?
I appreciate your input.
the idea is good. That’s how we create terraform environments (prod, staging, dev, etc.). We have a catalog of terraform modules (just code without settings/configs). Then for each env, we have a separate GitHub repo where we import the modules we need (using semantic versioning so we know exactly which version we are using in which env) and provide all the required config/settings for that environment, e.g. AWS region, stage (prod, staging, etc.), and security keys (from ENV vars or AWS SSM)
As I understand, you’re actually not creating a Terraform Module of your core/base infra, but instead you have catalogs/repos per environment with versioned “module puzzles”?
for example, we have a catalog of TF modules - reusable code which we can use in any env (prod, staging, dev, testing) https://github.com/cloudposse/terraform-root-modules/tree/master/aws
the code does not have any identity, it could be deployed anywhere after providing the required config/settings
then for example, in testing env, we create projects for the modules we need (e.g. eks), https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/eks
and load the module code from the catalog https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/eks/.envrc (uisng semantic versioning)
but all the config/settings are provided from a few places:
- Dockerfile (in which we have settings common for all modules in the project) https://github.com/cloudposse/testing.cloudposse.co/blob/master/Dockerfile
- Secrets are from ENV vars (which get populated from diff sources, e.g. AWS SSM, Secrets Manager, Vault, etc.) when the CI/CD deployment pipeline runs, or on dev machine by executing some commands)
I see, thank you very much I started with different approach, I keep all my environments in one Terraform Repository with projects and I include modules from external git repositories (each module in separate git repository)
that’s what we do too
https://github.com/cloudposse/terraform-root-modules is a (super)-catalog of top level modules which are aggregations of low-level modules
each projects in there connects low-level modules together into a reusable top-level module https://github.com/cloudposse/terraform-root-modules/blob/master/aws/eks/eks.tf#L31
ah, right
those aggregations are opionated since you can have many diff ways to connect low-level modules to create a top-level module you need
Interesting approach. I was reading quite a lot recently, best practices with Terraform,TF Up & Running etc. and in most cases people don’t recommend using nested modules, but it looks really reasonable in your case.
they are not nested (in that sense)
those are module instantiation and connecting them together into a bigger module
that’s why we have modules in TF - to reuse them
in other modules
Actually that was my understanding of nested word, sorry. English is not my first language
By nested modules, they mean modules of modules. Cloudposse stuff does use modules of modules e.g. module A may use module B, and module B may use module C
It works fine, but can be interesting to debug several layers down
If you want composable modules, there isn’t much of a way around that
Any samples/examples for implementing Cloudwatch events>create new rule>Ec2 Instance State-Change Notification > Target > SNS > email, currently going through official docs
@Hemanth you cannot create email subscription to an SNS topic with terraform, because they require a confirmation
Hey All, has anyone had issues creating azure resources with an s3 backend?
@Andriy Knysh (Cloud Posse) have you ever used an s3 backend with other providers for resources? I’m getting an issue where my declared resources are being pick up in the state file
did not use azure, but you can give more details about the issue, maybe somebody here will have some ideas
Otherwise, you just want to create the following resources: aws_cloudwatch_metric_alarm, aws_sns_topic, and aws_sns_topic_subscription
maybe you could find some ideas from https://github.com/cloudposse?utf8=%E2%9C%93&q=alarm&type=&language=
@Hemanth ^
@Andriy Knysh (Cloud Posse) the https://github.com/cloudposse/terraform-aws-ec2-cloudwatch-sns-alarms is empty. but thanks those samples are helpful
that one was not implemented
2019-09-24
re: https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/pull/51 @Andriy Knysh (Cloud Posse) if you make this change every terraform plan will produce a change
you just need to replace ” “, “_” on the old value
You guys using terraform cloud at all yet?
What are the overall benefits?
Visualization into runs via the web UI. You can see whats been applied recently and how that run went.
You can lock down certain users, you can also plan/apply automatically based on changes to git.
Interesting. I’ll have to check it out. Used to getting the auto features baked into my CI workflow, so if tf-cloud can potentially simplify that, it could be a win.
Does the visualization piece look at anything outside the tf-state?
Using #atlantis for now, as it is more flexible
Though terraform cloud does look appealing
Can you use terraform_remote_state data source as an input attribute for subnet in the cloudposse aws ec2 module?
I am using the terraform approved aws vpc module to create my VPC, and have correctly setup all my outputs, one specific being a public_subnet ID and I am trying to reference said subnet ID as a terraform_remote_state data source as the subnet attribute but am not sure of the proper syntax
@leonawood take a look here https://sweetops.slack.com/archives/CB6GHNLG0/p1569238378151700
thank you!
I have a terraform module in which we use to setup new AWS accounts with certain resources. So this module is generic enough to use on ‘dev’ aws account, ‘qa’ account and ‘prod’ account for say. However, I need to only create some resources based on the environment. How can I achieve this with a module? I saw this online: https://github.com/hashicorp/terraform/issues/2831
is this still the best way?
was about to try that out but read that if the count is set to 0, it would destroy the resource ?
for all resources in the module, you could use count = var.environment == "prod" ? 1 : 0 or count = var.environment == "qa" ? 1 : 0 etc.
or any combination of the conditions
so adding count = var.environment == "prod" ? 1 : 0 would ensure the resource is only created in prod?
it will ensure that if var.environment == "prod" then the resource will be created. If you run it in prod, it will be in prod.
at the same time, you could make a mistake and set var.environment == "prod" and run it in dev, then it will be created as well in dev
@Brij S you need some kind of container (or separate repo) where you set all configs for let’s say prod (e.g. region and AWS prod account ID) and where you set var.environment == "prod"
when you run it, it will be used only in the prod account and since var.environment == "prod", the resource will be created
so a better strategy would be not to create a super-giant module with many conditions to create resources or not
divide the big module into small modules
then use some tools to combine only the required modules into each account repo
the tool could be terragrant or what we do using geodesic and remote module loading https://www.reddit.com/r/Terraform/comments/afznb2/terraform_without_wrappers_is_awesome/
anyone here split up state files? we use tf workspaces and it works quite nicely. I am interested if theres a way to combine all the outputs into one file tho for reference?
so I can just send to our sys admin and it contain all the relevant details
@Andriy Knysh (Cloud Posse) i will look into terragrunt, as for now Id like to use the above suggestion with TF11, but having some issue with syntax:
${var.aws_env} == "prod" ? "1" : "0" doesnt work - what am i missing?
"${var.aws_env == "prod" ? 1 : 0}"
what about the closing }
need it too
cool let me try that
PR for more alb ingress config options: https://github.com/cloudposse/terraform-aws-alb-ingress/pull/22
I did not check the provider versions so unsure if it’ll break consumers or not
added a simple example too
thanks @johncblandii
no prob
is there a way to create an IAM user, generate access keys and plug them into paramstore with terraform?
2019-09-25
Components for secure UI hosting in S3
• S3 — for storing the static site
• CloudFront — for serving the static site over SSL
• AWS Certificate Manager — for generating the SSL certificates Route53 — for routing the domain name to the correct location Did anyone come across any modules for this in terraform ?
thankq @Andriy Knysh (Cloud Posse)
Hi all,
I’ve been using terraform (0.10 & 0.11) for close to three years now and as terraform 0.12 gets more support/becomes more of the industry standard, my team is looking to adopt it in a way where we can rearchitect our terraform structure, and reduce the general number of pain points across the team.
Currently we are a multi-region AWS shop that has single terraform repos for every service we deploy, with modules at the root of the repo, and directories representing each of our environments (qa-us-east-1, qa-eu-west-1). We run terraform from within those environment specific directories and push remote state to S3 to maintain completely separate state.
We’re thinking about how we can merge all of this into a single repo where:
- There are modules that can be reused across all of our different services (they’d either live at the root of the base terraform repo or in a separate terraform modules repo that we can reference from within our base repo)
- We duplicate as little code as possible (probably obvious but still worth mentioning)
- We continue to keep all state separate on a per environment basis
- Follow terraform best practices to make sure that upgrade paths continue to be easy/straightforward
We also want to keep in mind that we are shifting to a multi account AWS organization where our terraform will be deploying into different AWS accounts as well.
The team so far has demoed both Terragrunt and Terraform Workspaces. We are also considering not using workspaces or Terragrunt but still migrating to the single repo structure. There have been mixed opinions about all options considered. I’d love to get feedback from the community if anyone has opinions based on current or previous experiences with either.
Please note that we are currently not using Terraform Enterprise but that has been an option that could be considered as well
Regarding the multiple AWS account, we have a similar setup where, depending on the env directory you’re in, we hop into the correct AWS Account. Would that work for you are are you planning on deploying the same environment within multiple accounts?
it would be different environments within multiple accounts. The rough plan is to have each of our teams have a production & development/test account. So one thought was that the specific account would be another extracted layer of directories, either a level above or below the env directory
@kj22594 take a look here, similar conversation https://sweetops.slack.com/archives/CB6GHNLG0/p1569261528160800
thanks. I’ll take a look
in short, we use the following:
- Terraform modules to provision resources on AWS https://github.com/cloudposse?utf8=%E2%9C%93&q=terraform&type=&language=
- A catalog of top-level modules where we assemble the low-level modules together and connect them. They are completely identity-less and could be deployed in any AWS account in any region https://github.com/cloudposse/terraform-root-modules/tree/master/aws
- A container (
geodesichttps://github.com/cloudposse/geodesic( with all the tools required to provision cloud infrastructure
- Then, for a specific AWS account and specific region, we create a repo and Docker container, e.g. https://github.com/cloudposse/testing.cloudposse.co
it provides:
1) all the tools to provision infrastructure
2) Settings and configs for a specific environment (account, region, stage/env, et.) NOTE that secrets are read from ENV vars or SSM using chamber
3) The required TF code for each module that needs to be provisioned in that account/region gets loaded dynamically https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/eks/.envrc
4) to login to AWS, an AIM role gets assumed in the container (we use aws-vault)
so once inside that particular container (testing.cloudposse.co), you have all the tools, all required TF code, and all the settings/configs (that specify where and how the modules get provisioned)
so the code (logic) is separated from data (configs) and the tools (geodesic), but get combined in a container for a particular environment
Wow, thanks. That makes a ton of sense and seems to be a very sound way of approaching this problem. I do really like the idea of having root level modules repo where you can interconnect different modules for use cases that happen numerous times but also having the modules split out so that they can be reused separately too
yes
also, while terragrunt helps you to organize your code and settings, this approach gives you much more -code/settings/tools in one container related to a particular environment
(terragrunt still can be used to organize the code if needed) https://www.reddit.com/r/Terraform/comments/afznb2/terraform_without_wrappers_is_awesome/
the nice part about all of that is that the same container could be used from three different places: developer machine, CI/CD pipelines (those that understand containers like Codefresh or GitHub Actions), and even from GitHub itself using atlantis (which is running inside geodesic container) - that’s how we do deployment and testing of our modules on real AWS infrastructure
That is really cool. Atlantis is something that I’ve had conversations with a friend about but we’ve never actually implemented it or even tested it
I really appreciate this, this is all great knowledge and insight
public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304
32019-09-26
Hey does anyone have a terraform party slackmoji?
I will trade you one terraform-unicorn-dab slackmoji.
lol
I’d love a terraform-parrot
hahaha
I was hoping for something like my kubernetes party:
I stole that form kubernetes.slack.com
Nice!
I just made it!
Probably not my best work, but not bad for a first gif
¯_(ツ)_/¯
Thanks
Niiice!
Where do I get that unicorn XD
I really need it in my workspace
1) download the icons above
2) go here https://$[team.slack.com/customize/emoji](http://team.slack.com/customize/emoji) where $team is your slack team
XD
2019-09-27
@here I am trying to upgrade from v.11.14 to v.12 and after going through the upgrade steps and fixing some code changes … now I am seeing following issue
Error: Missing resource instance key
on .terraform/modules/public_subnets.public_label/outputs.tf line 29, in output "tags":
29: "Stage", "${null_resource.default.triggers.stage}"
Because null_resource.default has "count" set, its attributes must be accessed
on specific instances.
For example, to correlate with indices of a referring resource, use:
null_resource.default[count.index]
did anyone faced similar issue and was able to fix it
try "${join("", null_resource.default..*.triggers.stage}"
Hi
I would apreciate some help with the terraform-aws-elasticsearch module
when trying to use it from the complete example
i get in a plan the following
tha’s one example but i get that for all the variables
you have to provide values for all variables
it seems as if it were not reading the set variables
yeah, but in the variables.tf file?
or to use the .tfvar files, use :
terraform plan -var-file="fixtures.us-west-1.tfvars"
terraform apply -var-file="fixtures.us-west-1.tfvars"
hoooo i see
but don’t use out values
change them
so the modules as module "elasticsearch" { blah blah should be empty of values if i use a tfvars file rigth ?
you can provide values for the vars from many diff places https://www.terraform.io/docs/configuration/variables.html
how do you provide credentials to private terraform github repository module ?
like this in your providers.tf
then just set in variables.tf files the values
thanks
and how do i provide the path to github module if it is not at the root level
for example, source = "[email protected]:hashicorp/example.git"
but my main.tf is under modules directory
@AgustínGonzalezNicolini how would i access it ?
[email protected]:hashicorp/example.git//myfolder?ref=tags/x.y.z
thanks
Thanks @Andriy Knysh (Cloud Posse)!!!
2019-09-29
Hey guys, I am looking for the best way to roll back a change to an ASG to the previous known working ami as part of CICD pipeline with Terraform. Thinking of using a script to tag the previous AMI and using that to identify last known config. Has anyone else solved this problem?
I’ve been asked to provision 3 EKS clusters: Dev, Staging, and Prod. What is the way that you guys do this? Currently, I’m thinking of
- Having 3 branches in my git repo called “dev”, “staging”, and “prod”
- Having 3
.tfvarsfiles calleddev.tfvars,staging.tfvars,prod.tfvars - If I commit to
dev, My CICD runsterraform applyusing a workspace calleddev, usingdev.tfvars
Hi @roth.andy, personally I am a fan of workspaces. We used to have to have this setup but without the fixed branches, CI/CD automaticaly deployed a branch to staging and for prod it was a interactive apply ( if tests passed )
2019-09-30
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Oct 09, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
Guys, i’ve been using https://github.com/cloudposse/terraform-aws-vpc-peering to peer two vpcs and it works awesome. On current project I need to peer N numbers of VPC’s all with each other. As number of VPCs grows it become pretty hard to manage everything even with terraform. Is there any way to dynamically create peering mesh? CIDRs are carefully chosen so there will be no overlapping and I can fetch all vpcs with single data source. This shot from AWS describes my setup perfectly
I’d suggest using transit gateway
you can now use share vpcs : https://docs.aws.amazon.com/vpc/latest/userguide/vpc-sharing.html
@Milos Backonja I would look into Transit Gateway. This allows you to have a hub and spoke type of network and manage the routing tables centrally.
second this
Awesome, thanks a lot, This simplifies my setup enormously. I will need to check/estimate costs.
you can do a bunch of other cool things with transit gateways, like centralize nat gateways, or hook in a central direct connect
Hi Guys, does anyone here using Terraform Enterprise ?
Hub n’ Spoke with VPC Transit Gateway
does anyone know how to add private subnets to the default vpc using terraform?
Don’t use the default pvc, it is bad practice…
is there a module that creates a vpc with private subnet?
yes, just go to clousposse github and search for vpc and subnets
we use their modules and they work great
thanks @jose.amengual
@Brij S take a look at this example https://github.com/cloudposse/terraform-aws-emr-cluster/blob/master/examples/complete/main.tf
specifically lines 5-24, right?
yes
does aws_alb_listener resource multiple certificate_arn ?
For those interested in the EKS modules, we’ve converted them to TF 0.12