#terraform (2019-12)
Discussions related to Terraform or Terraform Modules
Archive: https://archive.sweetops.com/terraform/
2019-12-01
TIL that terraform support list explode notation( the three dot)
2019-12-02
hey,
I currently have to split up a terraform project into multipile ones since the providers needs data from a resource.
Does anyone have an idea on how to share the state / resources? For example I have azure Database and would like to use the terraform postgres provider. I would use the data source but the name I generate for the database has a random string attached to it.
Typically tags can be used to more verbosely describe a resource (assuming that exists in azure similar to aws). So if you can load based on e.g. app=myapp, stage=production, resulting in the resource.
you can use the remote state https://www.terraform.io/docs/providers/terraform/d/remote_state.html
yes, remote state. Or you can save all params to SSM when you create a resources and then read them from SSM in other resources or in applications. E.g. https://github.com/cloudposse/terraform-root-modules/blob/master/aws/grafana-backing-services/aurora-mysql.tf#L176
Thanks a lot ! I went with remote state!
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Dec 11, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
Hi Guys, do we really have to tag VPC, to get a EKS cluster created? does the subnet level tagging will do as well?
tags = {
"kubernetes.io/cluster/${local.cluster_name}" = "shared"
}
asking this because we dont have permissions to add tags at VPC level but we can at Subnet level, Couldn’t find much info hence asking this.
My understanding from docs is that you need to tag VPC if multiple eks clusters will be deployed in the VPC. This is the same for subnets. Subnets have additional tag requirement telling AWS that they can be used for pods and services etc.
So you might be ok if only one cluster.
@kskewes Thanks, its failing for us even for one cluster, we see events in CloudTrail saying “VPC tagging permissions denied”
i think when i was playing around with stuff it was working without the vpc tags but i have no idea what piece depends on it and i’d be very hesitant to move forward without it
2019-12-03
I agree completely with @Chris Fowles. While spinning my EKS cluster with TF, I forgot to tag my VPC appropriately. It seemed to work fine for the tests I carried out (deploy an API with an external ELB). However I fixed it once I caught it since it is indeed recommended by AWS
Hi all! Quick question about TF - is it possible to check in AWS if private IP address in subnet is taken or not?
I’m writing a TF script to deploy Exasol into AWS - it ties to specific IPs where first is for management node and following are for DB nodes. I’m able to get CIDR of VPC subnet and using cirdhost assign address to network interface - the only thing missing is precaution to check whether this address isn’t already taken in the subnet.
It’s not an option - because the address should be sequential - just for sake of mind of data engineers
consider - it’s a special requirement
but - looks like I’ve solved it by myself. You can’t create a network interface if there already exists network interface with address specified
Terraform support for the new AWS EKS Fargate feature.
Ah looks like a provider dev put one in just after me: https://github.com/terraform-providers/terraform-provider-aws/issues/11110
Hello all. Forgive me, but I’m new to terraform as well as CloudPasse. I do have an issue that may or may not be a bug. I tried to use terraform-aws-cloudfront-s3-cdn, but am geting the error: Error downloading modules: Error loading modules: module cdn: Error parsing .terraform\modules\8b3cb57814301845ecb9970841a29803[main.tf](http://main.tf): At 3 Unknown token: 3:16 IDENT var.namespace Looking at the code I see: module “origin_label” { source = “git://github.com/cloudposse/terraform-terraform-label.git?ref=tags/0.4.0>” namespace = var.namespace stage = var.stage name = var.name delimiter = var.delimiter attributes = compact(concat(var.attributes, [“origin”])) tags = var.tags } Shouldn’t that be namespace = ${var.namespace} ??
@Wayne Johnson you’re using TF 0.12 version of the module, but with terraform binary 0.11
Either pin a module to a release for 0.11, or use terraform binary 0.12
Thanks. I’ll look into upgrading.
Since terraform-docs doesn’t support TF 0.12 yet, how are you guys maintaining your module docs?
Ya… same here. This is based on what we started with build-harness
The sad truth is the amount of time that has been invested on this hack, we could have had a bonafide solution
I can imagine. I’ve been reading some threads on this.
i’m surprised no one has forked terraform-docs to make it work
sad that hashicorp wouldn’t accept a PR https://github.com/hashicorp/terraform-config-inspect/pull/17
2019-12-04
2019-12-05
Hey! Does anyone have a good example of creating dynamic blocks or content for the aws_security_group_rule resource? I need to create an ingress resource that has quite a few rules.
have you thought of using the largest community module for security groups ?
I have in the past but in this instance I am importing an existing security group. I wasn’t sure if this fits the use case.
what about creating a new security group and applying next to the old one
and then drop the old one
Hello there, do you know any aws application load balancer terraform module?
Terratest for the example: https://github.com/cloudposse/terraform-aws-ecs-web-app/blob/master/test/src/examples_complete_test.go
@Hugo Lesta ^
THanks @Andriy Knysh (Cloud Posse)
On your terraform modules why do you choose to reference resources that have a count for enable/disable using a join() instead of [0]?
join("", aws_ecr_repository.default.*.name)
aws_ecr_repository.default[0].name
resource "aws_ecr_repository" "default" {
count = var.enabled ? 1 : 0
name = var.use_fullname ? module.label.id : module.label.name
image_scanning_configuration {
scan_on_push = var.scan_images_on_push
}
tags = module.label.tags
}
resource "aws_ecr_lifecycle_policy" "default" {
count = var.enabled ? 1 : 0
repository = join("", aws_ecr_repository.default.*.name)
More context
join always works regardless of whether the resource list is empty or not
and it worked in TF 0.11 (.0. or [0] did not)
if enabled=false, this will not work aws_ecr_repository.default[0].name
b/c the list is empty
that makes sense. thanks for explaining!
in TF 0.12, the ternary operator could be used: var name = var.enabled ? aws_ecr_repository.default[0].name : “”
which did not work in TF 0.11 b/c it always evaluated both sides
and got errors on getting the name for an item from the empty list
join works in all cases in 0.11 and 0.12
good to know! thanks again
does anyone know how to transfer state from TFE to s3?
@Brij S you would need to add the configuration in terraform for that https://www.terraform.io/docs/backends/types/s3.html
just make sure the bucket is not public and it has versioning enabled
2019-12-06
Hello, I am new to terraform. Terraform plan don’t catch all issue and often they are discovered during the terraform apply . How do you validate the terraform apply before pushing in production ? is that the purpose of workspace ?
Hi Pierre, that is usualy because of the api
e.g. the most common thing is when you’ve policies, plan doesn’t have a way to validate a policy in aws
yes, I know but then should I deploy like this ? create new workspace -> deploy to test -> validate -> delete all resource in the test workspace -> deploy to prod workspace
hmm that’s a good question
I think you’ll find this tool useful
ha nice I found a kitchen-terraform module , I’ll check this
@Nikola Velkovski if I change the backend to s3, wont it try to destroy the current resources and rebuild them?
nope
it will seamlessy migrate to the new backend
so let me get this straight, I simply switch the backend to s3, run tf init/plan/apply etc and it will not try to destroy the already created resources and say no changes necessary?
yup
you’ll see on STDOUT, terraform will ask you if you would like to migrate to the new backend and so on.
beer is on me if that’s not the case
Does anyone know if there is a way to group resource variables in the .tfvars file by the uniquely named resource? An example would be where I have 2 AWS RDS resources that share the same variable names, but I want the variable inputs to be unique for each resource. I know that this can be done with a module: terraform.tfvars file module_name = { engine = “postgres” engine_version = “11.5” } However, I would like to do this without having to create a module.
same way, just use objects in the .tfvars and in the variable definition…
engine1 = {
engine = "postgres"
engine_version = "11.5"
}
engine2 = {
engine = "mysql"
engine_version = "11."
}
then lookup the values from that object in your tf config…
resource "thing" "engine1" {
engine = var.engine1["engine"]
engine_version = var.engine1["engine_version"]
}
resource "thing" "engine2" {
engine = var.engine2["engine"]
engine_version = var.engine2["engine_version"]
}
Thanks, using a map variable for all the resource value assignments with lookup is the right path. In my case I will have to create a variable for each type per resource, i.e. map(string), map(number), and map(bool) and group the inputs by the type per resource.
Has anyone used modules to import existing resources?
As in a module that is just data resources and outputs? I use this for commonly imported groups of things, like VPC parameters, KOPS parameters, etc. Easy to just put a single module block in terraform rather than copy paste all the stuff I need. It’s a bit more DRY that way.
Thanks Alex
2019-12-07
2019-12-08
Can the RDS module be used for creating a cross region replica?
it’s for creating cross-region replicas. PR https://github.com/cloudposse/terraform-aws-rds-replica/pull/4/files added possibility to create replicas in the same region as the master
thanks @Andriy Knysh (Cloud Posse)!
2019-12-09
is there some naming convention recommended when naming resources ? ( like in some programming language ) should my resource be named az_vnet_internal or azVnetInternal ? what about the object name ? how would you rewrite the resource below ?
resource "azurerm_virtual_network" "internal" {
name = "az_vnet_internal"
...
}
Usually, the value that you specify for the name argument needs to follow the guidelines of the relevant API. Terraform providers aren’t necessarily aware of such conventions, so the best place to look in the respective API documentation.
For example, here’s what Azure has to say about storage accounts: <https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-manager-storage-account-name-errors>
ok I see, I was expecting something more like in python naming convention: “Use the function naming rules: lowercase with words separated by underscores as necessary to improve readability.” <https://www.python.org/dev/peps/pep-0008/#method-names-and-instance-variables>
yeah, I figured as much. Unfortunately, there isn’t consistency across providers and usually not even between resources under a terraform-provider-* category, so there’s no terraform convention here.
Do you mean the resource named as viewed by the cloud provider or the name of the resource in Terraform, e.g. resource <name>?
teh resource name as viewed by the could provider
by example I have found out the following:
azurerm_storage_account object name
can only consist of lowercase letters and numbers, and must be between 3 and 24 characters long
azurerm_storage_container :
only lowercase alphanumeric characters and hyphens allowed in "name"
Right, exactly, that’s the example I shared in the docs link above to Azure
So I think I will name all object in camelCase
or not can only consist of lowercase letters
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Dec 18, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
Thanks to @Andriy Knysh (Cloud Posse) we now have support for AWS-managed EKS node pools: https://github.com/cloudposse/terraform-aws-eks-node-group
2
Just used this to deploy a node group, killed off the old “workers” module, and it was relatively smooth. I had to deal w/ the taints of the old workers, etc, but it is all running pretty smoothly.
The only issue I have now is getting the kubectl/aws calls to work on Terraform Cloud. I’ll work on that tomorrow, though.
Good stuff!
I believe @Andriy Knysh (Cloud Posse) has this working on TF cloud
and it downloads those utils
yeah, i’m going to dig in tomorrow. i was at a rancher mini-conf while deploying today
#livingdangerously
lol
@Andriy Knysh (Cloud Posse) @Erik Osterman (Cloud Posse) since you’re not returning the kubeconfig anymore, what do you think is the best practice for me running kubectl/helm outside of the modules?
Store it in SSM I think
it isn’t returned anymore, right? if not, we can’t access it to store it
let me look. maybe i missed it
yeah, it isn’t an output on the cluster anymore
Think @Andriy Knysh (Cloud Posse) had a solution for this
Will show how in an hour
we constructed kubeconfig before (in 0.11 version), but it was not a good idea for a few reasons:
shows up in state
- kubeconfig is always present on the cluster, just need to read it
- We contructed it for
aws-authenticator, but the newest EKS version usesaws eks get-token- it’s always be not the latest setup
so, if you use CLI, do it like this https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf#L139
or, if you need to construct kubeconfig, do it like this https://github.com/cloudposse/terraform-aws-eks-cluster/blob/0.11/master/kubeconfig.tpl
all those vars for the template are in the outputs
(we did the same for kops in geodesic - read kubeconfig from the cluster, not constructed it)
ah, fair enough
do you place it in the default kubeconfig location or just write the file and --kubeconfig to use it
or KUBECONFIG env var
either way
in a container, you can place it in the default kubeconfig location since it’s only on that container
this is terraform cloud so it’ll be gone after that run anyway
if you want to use helm, then use KUBECONFIG
on TF Cloud, I used --kubeconfig param
what i found odd was it said kubectl was not found when i tried to use it even when the “install” kubectl param was passed in
i tried to use just my own install (apt-get install kubectl)
it’s complicated on TF Cloud
for a few reasons:
- You can’t execute under
sudo, so the only place you can install external binaries is under theterraformuser folder
- But if you install binaries there, you have to put the location into the
PATHvariable in order to use the binaries
you can sudo
our make:
init-kubectl:
@sudo apt-get install -y apt-transport-https && \
curl -s <https://packages.cloud.google.com/apt/doc/apt-key.gpg> | sudo apt-key add - && \
echo "deb <https://apt.kubernetes.io/> kubernetes-xenial main" | sudo tee -a /etc/apt/sources.list.d/kubernetes.list && \
sudo apt-get update && \
sudo apt-get install -y kubectl
- But it does not work either since once the shell that downloaded the binaries exits, the env vars are gone and
PATHis not longer what you expect
on TF Cloud? No
on Terraform Enterprise (on-prem), yes
I added var
external_packages_install_path: "/home/terraform/.terraform/bin"
to TF Cloud workspace
and then to call a binary, just do /home/terraform/.terraform/bin/helmfile for example
or use `
binary = "${var.external_packages_install_path}/helmfile"
in other TF code
we’re not on-prem. this is TFC
maybe they just ignore our sudo
or we were grandfathered in
maybe just ignore
but if you try to use locations other then the terraform user folder, you get error
gotcha
so let me try out of that folder and see
fwiw
yeah, @Erik Osterman (Cloud Posse), but we have working code in TFC using sudo in the Makefile and in a local-exec:
resource "null_resource" "make_install" {
provisioner "local-exec" {
command = "sudo apt-get update && sudo apt-get install make"
}
provisioner "local-exec" {
command = "make init-ci"
}
triggers = {
run = uuid()
}
}
I can’t speak to why it works against their docs…but it does
from the run:
null_resource.make_install (local-exec): Executing: ["/bin/sh" "-c" "sudo apt-get update && sudo apt-get install make"]
In what path does it get installed?
i assume the default dir for apt-get resources. i will check, though
just noticed this in the makefile:
init-aws-auth: ## Initialize aws-iam-authenticator
@curl -o aws-iam-authenticator <https://amazon-eks.s3-us-west-2.amazonaws.com/1.12.7/2019-03-27/bin/linux/amd64/aws-iam-authenticator> && \
chmod +x ./aws-iam-authenticator && \
sudo mv ./aws-iam-authenticator /usr/local/bin
running now to check
null_resource.make_install (local-exec): /usr/bin/kubectl
this does not work, though:
external_packages_install_path = "/usr/local/bin"
i’m assuming it is a sudo issue
install_aws_cli=true
if [[ "$install_aws_cli" = true ]] ; then
echo 'Installing AWS CLI...'
mkdir -p /usr/local/bin
cd /usr/local/bin
curl -LO <https://s3.amazonaws.com/aws-cli/awscli-bundle.zip>
unzip ./awscli-bundle.zip
./awscli-bundle/install -i /usr/local/bin
export PATH=$PATH:/usr/local/bin:/usr/local/bin/bin
echo 'Installed AWS CLI'
which aws
aws --version
fi
not sure what’s working and what’s not. I spent just a few days on it, but there were a lot of issues using paths other than the terraform user folder (permissions etc.)
Error: Error running command ' set -e
install_aws_cli=true
if [[ "$install_aws_cli" = true ]] ; then
echo 'Installing AWS CLI...'
mkdir -p .terraform/modules/eks_cluster/.terraform/bin
cd .terraform/modules/eks_cluster/.terraform/bin
curl -LO <https://s3.amazonaws.com/aws-cli/awscli-bundle.zip>
unzip ./awscli-bundle.zip
./awscli-bundle/install -i .terraform/modules/eks_cluster/.terraform/bin
export PATH=$PATH:.terraform/modules/eks_cluster/.terraform/bin:.terraform/modules/eks_cluster/.terraform/bin/bin
echo 'Installed AWS CLI'
which aws
aws --version
fi
install_kubectl=true
if [[ "$install_kubectl" = true ]] ; then
echo 'Installing kubectl...'
mkdir -p .terraform/modules/eks_cluster/.terraform/bin
cd .terraform/modules/eks_cluster/.terraform/bin
curl -LO <https://storage.googleapis.com/kubernetes-release/release/$(curl> -s <https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl>
chmod +x ./kubectl
export PATH=$PATH:.terraform/modules/eks_cluster/.terraform/bin
echo 'Installed kubectl'
which kubectl
fi
aws_cli_assume_role_arn=
aws_cli_assume_role_session_name=
if [[ -n "$aws_cli_assume_role_arn" && -n "$aws_cli_assume_role_session_name" ]] ; then
echo 'Assuming role ...'
mkdir -p .terraform/modules/eks_cluster/.terraform/bin
cd .terraform/modules/eks_cluster/.terraform/bin
curl -L <https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64> -o jq
chmod +x ./jq
source <(aws --output json sts assume-role --role-arn "$aws_cli_assume_role_arn" --role-session-name "$aws_cli_assume_role_session_name" | jq -r '.Credentials | @sh "export AWS_SESSION_TOKEN=\(.SessionToken)\nexport AWS_ACCESS_KEY_ID=\(.AccessKeyId)\nexport AWS_SECRET_ACCESS_KEY=\(.SecretAccessKey) "')
echo 'Assumed role '
fi
echo 'Applying Auth ConfigMap with kubectl...'
aws eks update-kubeconfig --name=xolv-prod-tools-cluster --region=us-west-2 --kubeconfig=~/.kube/config
kubectl version --kubeconfig ~/.kube/config
kubectl apply -f .terraform/modules/eks_cluster/configmap-auth.yaml --kubeconfig ~/.kube/config
echo 'Applied Auth ConfigMap with kubectl'
': exit status 1. Output: Installing AWS CLI...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 12.3M 100 12.3M 0 0 72.0M 0 --:--:-- --:--:-- --:--:-- 72.0M
Archive: ./awscli-bundle.zip
inflating: awscli-bundle/install
inflating: awscli-bundle/packages/colorama-0.3.9.tar.gz
inflating: awscli-bundle/packages/ordereddict-1.1.tar.gz
inflating: awscli-bundle/packages/argparse-1.2.1.tar.gz
inflating: awscli-bundle/packages/pyasn1-0.4.8.tar.gz
inflating: awscli-bundle/packages/PyYAML-3.13.tar.gz
inflating: awscli-bundle/packages/docutils-0.15.2.tar.gz
inflating: awscli-bundle/packages/PyYAML-5.1.2.tar.gz
inflating: awscli-bundle/packages/urllib3-1.25.7.tar.gz
inflating: awscli-bundle/packages/python-dateutil-2.8.0.tar.gz
inflating: awscli-bundle/packages/awscli-1.16.302.tar.gz
inflating: awscli-bundle/packages/python-dateutil-2.6.1.tar.gz
inflating: awscli-bundle/packages/colorama-0.4.1.tar.gz
inflating: awscli-bundle/packages/jmespath-0.9.4.tar.gz
inflating: awscli-bundle/packages/urllib3-1.22.tar.gz
inflating: awscli-bundle/packages/futures-3.3.0.tar.gz
inflating: awscli-bundle/packages/s3transfer-0.2.1.tar.gz
inflating: awscli-bundle/packages/rsa-3.4.2.tar.gz
inflating: awscli-bundle/packages/botocore-1.13.38.tar.gz
inflating: awscli-bundle/packages/six-1.13.0.tar.gz
inflating: awscli-bundle/packages/simplejson-3.3.0.tar.gz
inflating: awscli-bundle/packages/virtualenv-15.1.0.tar.gz
inflating: awscli-bundle/packages/setup/setuptools_scm-1.15.7.tar.gz
Running cmd: /usr/bin/python virtualenv.py --no-download --python /usr/bin/python .terraform/modules/eks_cluster/.terraform/bin
Running cmd: .terraform/modules/eks_cluster/.terraform/bin/bin/pip install --no-cache-dir --no-index --find-links file:///terraform/.terraform/modules/eks_cluster/.terraform/bin/awscli-bundle/packages/setup setuptools_scm-1.15.7.tar.gz
Traceback (most recent call last):
File "./awscli-bundle/install", line 162, in <module>
main()
File "./awscli-bundle/install", line 151, in main
pip_install_packages(opts.install_dir)
File "./awscli-bundle/install", line 114, in pip_install_packages
pip_script, setup_requires_dir, package
File "./awscli-bundle/install", line 49, in run
p.returncode, cmd, stdout + stderr))
__main__.BadRCError: Bad rc (127) for cmd '.terraform/modules/eks_cluster/.terraform/bin/bin/pip install --no-cache-dir --no-index --find-links file:///terraform/.terraform/modules/eks_cluster/.terraform/bin/awscli-bundle/packages/setup setuptools_scm-1.15.7.tar.gz': /bin/sh: 1: .terraform/modules/eks_cluster/.terraform/bin/bin/pip: not found
aws cli won’t install w/ the default path (from the module)
module "eks_cluster" {
source = "git::<https://github.com/cloudposse/terraform-aws-eks-cluster.git?ref=0.13.0>"
namespace = var.namespace
stage = var.stage
name = var.name
attributes = var.attributes
tags = module.xoot.default_tags
apply_config_map_aws_auth = true
install_aws_cli = true
install_kubectl = true
kubeconfig_path = local.kubeconfig_filename
allowed_cidr_blocks = var.allowed_cidr_blocks_cluster
allowed_security_groups = var.allowed_security_groups_cluster
kubernetes_version = var.kubernetes_version
region = var.region
subnet_ids = module.subnets.public_subnet_ids
workers_role_arns = [module.eks_node_group_default.eks_node_group_role_arn]
workers_security_group_ids = []
vpc_id = module.vpc.vpc_id
...
}
looks like some bad paths in the pip install command
https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf#L108 already has /bin in the path and it appends it again
you’re already cd’d into the local install path then pass -i of the same folder you’re in: https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf#L107
yeah…at a bit of a loss. I can’t use the internal tools due to the above and I can’t use my own installed tools for {not sure why, but insert reason here}.
I confirmed the tools are installed and work:
null_resource.make_install (local-exec): /usr/bin/kubectl
null_resource.make_install (local-exec): /usr/bin/aws
null_resource.make_install (local-exec): aws-cli/1.14.44 Python/3.6.8 Linux/4.15.0-1044-aws botocore/1.8.48
My other make commands work with those same tools, but the module’s use of these tools does not.
i haven’t confirmed, due to these issues, but it seems a bit cart before horse on the --kubeconfig param pointing to kubeconfig_path which the file won’t exist yet because the cluster isn’t back so I can’t write it.
gotta jet, but i’ll dig in more tomorrow
The module works on TF cloud. I’ll send you the variables I used later today
this is the code that works on TF Cloud:
provider "aws" {
region = var.region
assume_role {
role_arn = var.aws_assume_role_arn
}
}
module "label" {
source = "git::<https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.16.0>"
namespace = var.namespace
name = var.name
stage = var.stage
delimiter = var.delimiter
attributes = compact(concat(var.attributes, list("cluster")))
tags = var.tags
}
locals {
# The usage of the specific kubernetes.io/cluster/* resource tags below are required
# for EKS and Kubernetes to discover and manage networking resources
# <https://www.terraform.io/docs/providers/aws/guides/eks-getting-started.html#base-vpc-networking>
tags = merge(module.label.tags, map("kubernetes.io/cluster/${module.label.id}", "shared"))
eks_worker_ami_name_filter = "amazon-eks-node-${var.kubernetes_version}*"
}
module "vpc" {
source = "git::<https://github.com/cloudposse/terraform-aws-vpc.git?ref=tags/0.8.1>"
namespace = var.namespace
stage = var.stage
name = var.name
attributes = var.attributes
cidr_block = var.vpc_cidr_block
tags = local.tags
}
module "subnets" {
source = "git::<https://github.com/cloudposse/terraform-aws-dynamic-subnets.git?ref=tags/0.18.1>"
availability_zones = var.availability_zones
namespace = var.namespace
stage = var.stage
name = var.name
attributes = var.attributes
vpc_id = module.vpc.vpc_id
igw_id = module.vpc.igw_id
cidr_block = module.vpc.vpc_cidr_block
nat_gateway_enabled = var.nat_gateway_enabled
nat_instance_enabled = var.nat_instance_enabled
tags = local.tags
}
module "eks_workers" {
source = "git::<https://github.com/cloudposse/terraform-aws-eks-workers.git?ref=tags/0.11.0>"
namespace = var.namespace
stage = var.stage
name = var.name
attributes = var.attributes
tags = var.tags
instance_type = var.instance_type
eks_worker_ami_name_filter = local.eks_worker_ami_name_filter
vpc_id = module.vpc.vpc_id
subnet_ids = module.subnets.public_subnet_ids
associate_public_ip_address = var.associate_public_ip_address
health_check_type = var.health_check_type
min_size = var.min_size
max_size = var.max_size
wait_for_capacity_timeout = var.wait_for_capacity_timeout
cluster_name = module.label.id
cluster_endpoint = module.eks_cluster.eks_cluster_endpoint
cluster_certificate_authority_data = module.eks_cluster.eks_cluster_certificate_authority_data
cluster_security_group_id = module.eks_cluster.security_group_id
# Auto-scaling policies and CloudWatch metric alarms
autoscaling_policies_enabled = var.autoscaling_policies_enabled
cpu_utilization_high_threshold_percent = var.cpu_utilization_high_threshold_percent
cpu_utilization_low_threshold_percent = var.cpu_utilization_low_threshold_percent
}
module "eks_cluster" {
source = "git::<https://github.com/cloudposse/terraform-aws-eks-cluster.git?ref=tags/0.13.0>"
namespace = var.namespace
stage = var.stage
name = var.name
attributes = var.attributes
tags = var.tags
region = var.region
vpc_id = module.vpc.vpc_id
subnet_ids = module.subnets.public_subnet_ids
kubernetes_version = var.kubernetes_version
kubeconfig_path = var.kubeconfig_path
local_exec_interpreter = var.local_exec_interpreter
configmap_auth_template_file = var.configmap_auth_template_file
configmap_auth_file = var.configmap_auth_file
oidc_provider_enabled = var.oidc_provider_enabled
install_aws_cli = var.install_aws_cli
install_kubectl = var.install_kubectl
kubectl_version = var.kubectl_version
jq_version = var.jq_version
external_packages_install_path = var.external_packages_install_path
aws_eks_update_kubeconfig_additional_arguments = var.aws_eks_update_kubeconfig_additional_arguments
aws_cli_assume_role_arn = var.aws_cli_assume_role_arn != "" ? var.aws_cli_assume_role_arn : var.aws_assume_role_arn
aws_cli_assume_role_session_name = var.aws_cli_assume_role_session_name != "" ? var.aws_cli_assume_role_session_name : module.label.id
workers_role_arns = [module.eks_workers.workers_role_arn]
workers_security_group_ids = [module.eks_workers.security_group_id]
}
this is the variables and ENV variables for the EKS workspace (we have a TF generator that parses the config YAML and provisions TF Cloud workspaces):
environments:
- name: "testing"
env_vars:
AWS_ACCOUNT_ID: "xxxxxxxxx"
vars:
stage: "testing"
region: "us-east-2"
aws_assume_role_arn: "arn:aws:iam::xxxxxxxx:role/OrganizationAccountAccessRole"
workspaces:
- name: "eks"
enabled: true
repo_name: "cloudposse/terraform-cloud-reference-architectures"
repo_branch: "master"
repo_working_directory: "blueprints/eks"
env_vars:
CONFIRM_DESTROY: 1
hcl_vars:
availability_zones: ["us-east-2a", "us-east-2b"]
vars:
name: "eks"
enabled: true
vpc_cidr_block: "172.16.0.0/16"
instance_type: "t2.small"
kubernetes_version: "1.14"
associate_public_ip_address: true
max_size: 3
min_size: 2
autoscaling_policies_enabled: true
nat_gateway_enabled: false
nat_instance_enabled: false
configmap_auth_file: "/home/terraform/.terraform/configmap-auth.yaml"
kubeconfig_path: "/home/terraform/.kube/config"
external_packages_install_path: "/home/terraform/.terraform/bin"
@johncblandii ^
https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf#L108 already has /bin in the path and it appends it again
the first bin is what we just selected (see variables above, external_packages_install_path), it could be anything (your choice). The second bin is what the AWS CLI creates automatically and puts its scripts in there. So it just happens to be bin/bin
I’m reading on my phone, but that looks like what I’m using except the path. I’ll try that path tomorrow
Any thoughts as to why your commands don’t work with my installed packages?
- the installed packages are not in PATH
- or no permissions if they are downloaded into diff location (not inside
/home/terraform/user folder)
They must be because I do no PATH tweaking.
I’ll find out for sure
ok…running it real quick then will be back later to do more
@Andriy Knysh (Cloud Posse) what are your thoughts on the cart before horse nature of needing a kubeconfig in the module before one is written because we need the results from the module?
Why before? We provision the cluster and it has kubeconfig. Then we read it from the cluster to execute other commands
Did I miss where you were writing the file somewhere inside of the module? I didn’t see that so it looks like it’s reading from a file that doesn’t exist yet.
So it looks like this works, but I haven’t verified whether i can use the tools for my other commands (got an error for aws eks not being available).
apply_config_map_aws_auth = true
configmap_auth_file = "/home/terraform/.terraform/configmap-auth.yaml"
external_packages_install_path = "/home/terraform/.terraform/bin"
install_aws_cli = true
install_kubectl = true
kubeconfig_path = local.kubeconfig_filename
aws eks update-kubeconfig reads kubeconfig from the cluster and saves it to the file system at --kubeconfig=${var.kubeconfig_path}
but I haven’t verified whether i can use the tools for my other commands (got an error for aws eks not being available).
once the shell that executes commands in https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/auth.tf exits, you lose all env vars exported there (so PATH will be reverted to default)
in other shells, you have to call the binaries by prefixing them with /home/terraform/.terraform/bin, e.g. /home/terraform/.terraform/bin/bin/aws
aws eks update-kubeconfig reads kubeconfig from the cluster and saves it to the file system at –kubeconfig=${var.kubeconfig_path}
AHHHHHHHHHH…ok. That was what I was completely missing, man. I knew a write had to happen somewhere.
anyone have any prior art for a terraform provider that downloads binary dependencies?
I don’t know of anything that downloads the cli from a remote source and installs it (which is what I assume is the intended behavior you’re describing), but <https://github.com/terra-farm/terraform-provider-virtualbox> leverages <https://github.com/terra-farm/go-virtualbox> which just checks if VBoxManage is installed locally on your machine.
go-getter is similar, in that it requires git to be present and in the path
thanks @Arthur Burkart and @loren!
@Andriy Knysh (Cloud Posse)
I think @Andriy Knysh (Cloud Posse) also mentioned using go-getter
e.g. imagine a terraform provider that wrapped a cli tool
that provider would depend on the cli tool in order to function
2019-12-10
@Andriy Knysh (Cloud Posse) has just finished implementing support for EKS Fargate node pools!
Terratest provisions a Node Group with two worker nodes AND a Fargate Profile for k8s default namespace. Then it creates a stub k8s deployment (using k8s go-client) in the default namespace in order for Fargate to add a Fargate node to the cluster to provision the deployment. Then it waits for all three nodes to join the cluster, then deletes the k8s deployment, and finally destroys all AWS resources with terraform destroy
Creating Kubernetes deployment ‘demo-deployment’ in the ‘default’ namespace...
Created Kubernetes deployment ‘demo-deployment’
Waiting for worker nodes to join the EKS cluster...
Node ip-172-16-97-1.us-east-2.compute.internal has joined the EKS cluster at 2019-12-10 03:49:29 +0000 UTC
Node ip-172-16-137-51.us-east-2.compute.internal has joined the EKS cluster at 2019-12-10 03:49:37 +0000 UTC
Node fargate-ip-172-16-53-87.us-east-2.compute.internal has joined the EKS cluster at 2019-12-10 03:51:06 +0000 UTC
All nodes have joined the EKS cluster
Listing deployments in namespace ‘default’:
* Deployment ‘demo-deployment’ has 1 replica(s)
Deleting deployment ‘demo-deployment’ ...
Deleted deployment ‘demo-deployment’
(heh, slack has broken markdown)
yea
This is slick testing!
We’ve been using terratest also, nice work here!
These slides are really good, https://www.infoq.com/presentations/automated-testing-terraform-docker-packer
coming for this module to test next.
Do you guys see any problems of adding
lifecycle {
create_before_destroy = true
}
to : https://github.com/cloudposse/terraform-aws-rds-cluster/blob/master/main.tf#L90
any objections if I send a PR?
PRs always welcome
what’s the purpose of having that?
well I gave it more tough and I do not think is needed
the idea behind was to always create new nodes first before deleting
in case one was terminates by hand but created with TF
for example
or in a cluster with size 4 the writer is being destroy by TF when is set to 2
2019-12-11
Hey guys, Please advise on what is the structure of this value so that I could provide additional S3 bucket policy? https://github.com/cloudposse/terraform-aws-cloudfront-s3-cdn/blob/a05cffd71b6b5ea8d9a881e20c8b41d038dc9167/variables.tf#L66 Whatever I provide it blames on this or that character somewhere. Can’t find example of what it actually expects.
Hey @maarten. Thanks. Applied in this form and all works like a charm. I was thinking that it might expect TF style like described here: https://www.terraform.io/docs/providers/aws/d/iam_policy_document.html And not official one for AWS what you sent.
Nevertheless it’s fixed already. Thank a lot!
2019-12-12
Hi, I have a module that isn’t playing well in 0.12 specifically because it previously referenced private_ip[0] to name an aws_route53_record. I previously used replace() to change the . to -
Hey @NVMeÐÐi 
I answered your question in Gitter where you asked it earlier.
Oh wow, I didn’t see that alert pop up
THANK YOU AGAIN!
I wouldn’t say I’m to advanced with Terraform, still learning every day.
0.11 to 0.12 upgrades are kicking my butt.
Yeah, it’s just an oddity. You don’t usually expect the types to change underneath your feet.
If you haven’t read the upgrade guide, I strongly recommend it.
• https://www.terraform.io/upgrade-guides/0-12.html#referring-to-list-variables
• https://github.com/hashicorp/terraform/blob/master/CHANGELOG.md#0120-may-22-2019
It seems to cover all the gotchas
My question is this… Is there anyway to change a tuple to a string in 0.12? (Interpolation doesn’t seem to do it anymore).
Example below, I get the tuple error for the name, but records is fine.
resource "aws_route53_record" "this" {
name = "ip-${replace(module.this_mybox.private_ip[0], ".", "-")}"
type = "A"
zone_id = "${var.dns_zone_id}"
records = "${module.this_mybox.private_ip[0]}"
ttl = "600"
}
Oof, got it resolved. Artburkart got me.
2019-12-13
It finally happened… https://github.com/terraform-aws-modules/terraform-aws-eks/issues/635
It was only a matter of time…
I think the approach we took with our EKS modules will continue to scale better.
Hey folks, any guidance on this issue I’m running into with the elasticache-memcached module? https://github.com/cloudposse/terraform-aws-elasticache-memcached/issues/8
unfortunately that module was not converted to TF 0.12 yet. We have automated tests for all modules that are converted to 0.12. The tests provision the module on real AWS account. We have not tested that module for a while, will have to look into that (and convert to 0.12)
Any Terratest users? Tried to find a better channel (maybe #terragrunt would have been better here) - but I’m wondering, how can I manage which Terraform version I would be using for doing the runs…? Tried to read the module’s source code also but did not find anything obvious… only TerraformBin variable but I’d guess that does not help crazy much…
I haven’t used Terratest but given these lines of code TerraformBinary and the documentation in their basic terraform example, https://github.com/gruntwork-io/terratest/tree/master/examples/terraform-basic-example “Install Terraform and make sure it’s on your PATH.” that Terratest runs the terraform binary that is in your path.
This is the function that runs the Terraform command https://github.com/gruntwork-io/terratest/blob/9c546f282a74359d7d17697a504268b190bb5e35/modules/terraform/cmd.go#L48-L63
To manage the terraform versions, I would use a tool such as tfenv.
@Joe Presley ok, thanks. I currently plugging our Terratest-orchestrated Terraform tests into GitHub Actions, and I was surprised when I was able to do a Terratest-run with just installing the module-dependencies and running “go test” - no need to install TF in PATH… Hence I starting doubting on what’s happening
@Joe Presley I was also using tfenv before locally, but I found tfswitch more… ux-friendly maybe
Does the GitHub actions use HashiCorp’s terraform GitHub action? https://www.terraform.io/docs/github-actions/getting-started.html I’ll have to checkout tfswitch.
@Joe Presley I could use those yes… And, I’d need to find a small balance here, as I’d need to pull dependent TF modules from TF Enterprise’s Module Registry - meaning, I anyway need to configure ~/.terraformrc somehow…
I might build a fork of the official Terraform Actions which just includes an additional command, “test” (which executes a “make test”… Might be the easiest at the end…
Sent my question to the Terratest team: https://github.com/gruntwork-io/terratest/issues/419 - let’s see…
2019-12-16
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Dec 25, 2019 11:30AM.
Register for Webinar
#office-hours (our channel)
chatting in general re nodegroups vs separate ASGs per zone for stateful apps/services in eks kube. the convo being more appropriate here. . .
since using TF to manage. I was refactoring to support PVCs and dealing with some very active operators which are running tons of node-shells that I did’nt know use ‘nodeSelector’ when templatizing the helm chart and applying the manifests.
so also curious how folks are managing stateful apps/services like operators, kafka, etc that set ‘nodeSelector’, it seems like I need to find a dynamic solution to keeping the ‘nodeSelector’ s updated as scaling events occur, etc.
2019-12-17
btw, I don’t know why @maarten didn’t share this yet, but this looks really rad: https://github.com/Flaconi/terraform-aws-bastion-ssm-iam
Thanks @Erik Osterman (Cloud Posse), maybe we can have an #I-created-this-take-a-look-announce or something
Yea, that’s a good suggestion!
I just created #community-projects
@antonbabenko I saw your post wishing for an “AWS Bastion” service. Have you seen this interesting bastion developed by @maarten?
Hi guys! I saw that one (and 10+ other similar projects). The post was about “AWS should take the undifferentiated load from users and make a proper service for us to just use without asking us to run and manage it”. In one of my current project I am going to use Teleport, because I can’t use “ssm session manager” and “ec2-instance-connect” by AWS.
Teleport is rad, but definitely a lot of work.
Does anyone know good teleport module which I can get as basic (skyscrapers have one)
We deploy teleport on kops/kubernetes
but have a related module for backing services
I need teleport in first place to setup RKE cluster on EC2
Not sure I follow…
Do you mean you want to deploy the core teleport architecture on dedicated EC2 VMs?
(if so, it’s not a bad idea)
Brb :)
Bastion server with zero outside ports open
Hi so I’m using this : https://github.com/cloudposse/terraform-aws-rds-cluster/blob/master/main.tf#L163 option to create cluster CNAMES to the cluster endpoint byt my app are not able to resolve the CNAME records, so I’m guessing maybe this is a problem wit the way that aurora cluster endpoint work on route53? If the default cluster endpoints are CNAMEs then my CNAMES will point to a CNAME than then it will point to A record that is the rds instance endpoint so maybe java has an issue with following multiple CNAMES or somesthing?
I never had this problem before but I never pointed CNAMEs to cluster endpoint, I was wondering to maybe switch to custom endpoints
What happens when you dig it?
Does it resolve correctly?
we use CNAMEs with Aurora clusters all the time with no issues
works with any SQL client. Also works with Metabase, which is Java-based
in your app, try to use the cluster endpoint first to check if it’s working
it could be any other issue (e.g. SSL etc.)
in our app the endpoint works but the CNAME does
from the instance that runs the app I can resolve no ptoblem
; ANSWER SECTION:
team-staging-data-reader-us-east-2.staging-ds.sonatype.com. 300 IN CNAME team-staging-us-east-2-data.cluster-ro-xxxxxxxx.us-east-2.rds.amazonaws.com.
team-staging-us-east-2-data.cluster-ro-xxxxxxxx.us-east-2.rds.amazonaws.com. 1 IN CNAME team-staging-us-east-2-data-2.xxxxxxxx.us-east-2.rds.amazonaws.com.
team-staging-us-east-2-data-2.xxxxxxxx.us-east-2.rds.amazonaws.com. 5 IN A 10.10.10.5
but is a CNAME to a CNAME to an A record
so I wonder if dropwizard does not like that somehow
Ya maybe trying to use SSL? That won’t work with the CNAME
we are not using SSL yet
@channel, need help with below module for ALB. This module is a combined one which created ALB, target group, listerner, Listener rule, attaching certificate. However, as you may see, we are passing a list of elements to this module which is causing this module to break.
It breaks if I remove any element from the list and but this, all the element post the one removed gets deleted and recreated.
Please suggest if anyone has faced such issues and how have they solved. This is happening to any resources is in list. Could be because terraform refresh doesn’t try to re-map the index of elements if anything is missing or deleted or added. Appreciated your help in advance
@jha.bikal see if this issue discusses what you’re seeing, and the linked issue also that describes how to use for_each in terraform 0.12 to address it… https://github.com/hashicorp/terraform/issues/14275#issuecomment-361408631
Thanks @loren. Yes, they have touched the exact pain point I’m facing now.
Any idea if we can use different terraform version for different workspace(in the same server)
2019-12-18
hei guys, I’ve started recently with terraform-aws-rds(https://github.com/cloudposse/terraform-aws-rds.git?ref=tags/0.9.3)
) and terraform-aws-vpc (https://github.com/cloudposse/terraform-aws-vpc.git?ref=tags/0.7.0)
yet it seems they are completely incompatible.
As soon as I run terraform plan, it gives me errors like the ones below, which require me to go an tweak the generated .terraform modules files, in order for them to work together.
Error: Unsupported block type
on .terraform/modules/rds_instance/main.tf line 71, in resource "random_string" "default":
71: keepers {
Blocks of type "keepers" are not expected here. Did you mean to define
argument "keepers"? If so, use the equals sign to assign it a value.
Error: Incorrect attribute value type
on .terraform/modules/rds_instance/main.tf line 104, in resource "aws_db_subnet_group" "default":
104: subnet_ids = ["${var.subnet_ids}"]
Inappropriate value for attribute "subnet_ids": element 0: string required.
Error: Incorrect attribute value type
on .terraform/modules/rds_instance/main.tf line 134, in resource "aws_security_group_rule" "ingress_cidr_blocks":
134: cidr_blocks = ["${var.allowed_cidr_blocks}"]
Inappropriate value for attribute "cidr_blocks": element 0: string required.
Any idea if that is norm ? is there some best practices to avoid having such incompatibilities ?
I don’t think its ok for me to tweak the generated .terraform modules, just to make those work
looks like you are mixing up TF 0.11 and 0.12. cidr_blocks = ["${var.allowed_cidr_blocks}"] is 0.11 syntax
take a look at the working example here https://github.com/cloudposse/terraform-aws-rds/blob/master/examples/complete/main.tf
Terratest that deploys the example on AWS https://github.com/cloudposse/terraform-aws-rds/blob/master/test/src/examples_complete_test.go
for TF 0.12, use the latest versions of the modules (that’s what the tests usually do)
for TF 0.11, find the prior releases for 0.11 and use all modules for 0.11
https://github.com/cloudposse/terraform-aws-rds?ref=tags/0.9.3 is 0.11 version
I’m using terraform v.0.12.17 locally now. Is there a way to check for each module release if its on Terraform 12 or 11 ?
there is no automatic way to check it
look into examples/complete of each module
they should use the latest TF 0.12 version (or almost latest)
also, look into release, e.g. https://github.com/cloudposse/terraform-aws-rds/releases/tag/0.10.0
it will say “Convert to TF 0.12”. All tags after that are 0.12
sometimes we make new releases for 0.11, but the tag will be smaller than the first 0.12 release, e.g. https://github.com/cloudposse/terraform-aws-rds/releases/tag/0.9.3 is a bug fix release for 0.11 version
alright, thanks a lot, will check now 
I’m curious what CloudPosse uses as a rule of thumb for when to have a module call another module vs defining the terraform resource. For instance the aws-ecs-alb-service-task module doesn’t reference the aws-iam-role module for creating its roles. https://github.com/cloudposse/terraform-aws-ecs-alb-service-task/blob/master/main.tf https://github.com/cloudposse/terraform-aws-iam-role
I think the iam role module was created much, much later.
In our case, we use the IAM role module to provision roles needed by pods in kubernetes
and use it more in a standalone context
So if the ecs-alb-service-task was written from scratch today, it would likely reference the iam-role module?
So in < 0.12, using modules like this could very often lead to “count of … could not be computed” https://docs.cloudposse.com/troubleshooting/terraform-value-of-count-cannot-be-computed/
post 0.12, we encounter that much less often, so perhaps.
the main reason we have the iam role module is to enforce consistent names
however, if we’re just provisioning some roles in another module, we probably wouldn’t use this module because we’re already using the null label module for that everywhere else
i don’t think this is the “rule of thumb” you were looking for though
another example: https://github.com/cloudposse/terraform-aws-ssm-parameter-store
We wanted to use this module all over the place, but ended up not being able to because of the count-of problems. Thus, we just ended up using the raw resources everywhere.
Thanks Erik. I appreciate it. So I guess you could say the rule of thumb is to reference other modules unless it creates more problems than benefits.
That’s a good paraphrasing.
Is there a way to release the lock on terraform file in dynamo db through terraform CLI
have you tried terraform force-unlock ?
Just tried it, thanks - it worked
Just need the lock id
But if you give it the wrong one, it will tell you the existing one
Hello all, In the last 24 hours, all of our terraform-null-label modules started failing, with the following error, anyone have any ideas?
Error: Failed to download module
Could not download module "s3_bizrewards_dev_label" (s3_bizrewards.tf:31)
source code from
"<https://api.github.com/repos/cloudposse/terraform-null-label/tarball/0.16.0//*?archive=tar.gz>":
Error opening a gzip reader for
/var/folders/1d/gpvdrwrd0y1_d0jv64w76j645nyvdq/T/getter001152442/archive: EOF.
@Matt Law how do you load the module? In terraform (source=…), or something else? That link you posted is 404, so if it was working before, then something changed in GitHub
Hi, Ive been calling it like this:
module "sqs_bizrewards_dev_label" {
source = "cloudposse/label/null"
version = "0.16.0"
but changed to this, which works, but all of our labels are like the above.
source = "git::<https://github.com/cloudposse/terraform-null-label.git?ref=tags/0.16.0>"
hmm, never seen that error before. Looks like something is going on between the terraform registry and GitHub API
` source = “cloudposse/label/null”`
maybe there was something broken with the terraform registry?
(we still use the git-style URLs everywhere)
…anyways, we have no control or influence over how the registry URL work. they basically proxy/redirect to github.
We experience similar issue
$ terraform --version
Terraform v0.12.18
some more info on this: Hashicorp said they are investigating a “random issue” pertaining to this. Also when I used 0.12.18 it worked ok, Versions 0.12.15->0.12.17 seem to have the problems Thanks for all the replies.
thanks for the follow up!
Hi everyone,
Could you please check the following issue & let me know what’s wrong here.
Error: Missing resource instance key
on terraform-aws-codebuild/main.tf line 144, in data “aws_iam_policy_document” “permissions_cache_bucket”: 144: “${aws_s3_bucket.cache_bucket.arn}”,
Because aws_s3_bucket.cache_bucket has “count” set, its attributes must be accessed on specific instances.
For example, to correlate with indices of a referring resource, use: aws_s3_bucket.cache_bucket[count.index]
Error: Missing resource instance key
on terraform-aws-codebuild/main.tf line 144, in data “aws_iam_policy_document” “permissions_cache_bucket”: 144: “${aws_s3_bucket.cache_bucket.arn}”,
Because aws_s3_bucket.cache_bucket has “count” set, its attributes must be accessed on specific instances.
For example, to correlate with indices of a referring resource, use: aws_s3_bucket.cache_bucket[count.index]
Error: Missing resource instance key
on terraform-aws-codebuild/main.tf line 145, in data “aws_iam_policy_document” “permissions_cache_bucket”: 145: “${aws_s3_bucket.cache_bucket.arn}/*”,
Because aws_s3_bucket.cache_bucket has “count” set, its attributes must be accessed on specific instances.
For example, to correlate with indices of a referring resource, use: aws_s3_bucket.cache_bucket[count.index]
Error: Missing resource instance key
Hi @foqal Thanks for the update! Let me take a look into this, basically I would like to achieve the following things, let me know if you can help me out on this:- My stack is: AWS CodePipeline (github integration) + CodeBuild (lambda deploy) packed in terraform. AWS Lambda + API gateway dev environment and a commit in prod branch goes to AWS Lambda + API gateway prod environment
2019-12-19
looks like you are using the old TF 0.11 version of the module which has that issue (and newer version of terraform complains about it, but did not do it before)
take a look at the new 0.12 version (where the issue is fixed) https://github.com/cloudposse/terraform-aws-codebuild
Hey there!
The iam_access_key doc (https://www.terraform.io/docs/providers/aws/r/iam_access_key.html#secret) says for the secret attribute:
secret- The secret access key. Note that this will be written to the state file. If you use this, please protect your backend state file judiciously. Alternatively, you may supply apgp_keyinstead, which will prevent the secret from being stored in plaintext, at the cost of preventing the use of the secret key in automation. The If you use this part means if you use the resource as a whole? Or just the attribute? Is there a way how to prevent storing secret in the state file? Theterraform applyknows it’s a sensitive value: ``` $ terraform apply
An execution plan has been generated and is shown below. Resource actions are indicated with the following symbols:
- create
Terraform will perform the following actions:
# module.elasticprop.aws_iam_access_key.elasticprop_access_key will be created
- resource “aws_iam_access_key” “elasticprop_access_key” {
- encrypted_secret = (known after apply)
- id = (known after apply)
- key_fingerprint = (known after apply)
- secret = (sensitive value)
- ses_smtp_password = (sensitive value)
…
```
…using terraform
0.12.18
Hello, can resources be generated from an interpolated variable ? or may be i should use templates ? by example will this work :
resource "azurerm_managed_disk" "Disk${var.ServerName}" {}
probably not
but you don’t need to do that
just name it for example default
and use count or for_each
then select attributes of all resources from the list, e.g. name = azurerm_managed_disk.default[count.index) (not real, just an example)
ok I see, I will fix that ( taking over a project and some bad naming usage were implemented in object )
2019-12-20
Rejoice!!!
Pr merged
finally!!!!
I was just wondering what is the best option for supplying secrets to Terraform. For example, when writing secrets to parameter store.
Options I can see are:
- Create an input variable and supply via cli, prompt or ENV var
- Create a gitignored tfvars file and store secrets in there for use in input vars
- Same as (2) but use git-crypt on the tfvars file and keep the encrypted version in git
- Manually add to Vault (or some other secure place that Terraform can access - SecretHub?) and retrieve value I guess the best option will partly depend on how many people are running the Terraform modules.
Interested in what people have found is most practical. Related GitHub issue: https://github.com/hashicorp/terraform/issues/516
Secrets generated programmatically as part of terraform, we just write directly into SSM. This will still have the same problems that those must be also stored in the statefile. The only way to keep the statefile remotely secure is via gitops and eliminate human plan/apply phase.
The other kinds of secrets which originate from humans are populated by humans, usually using command line tools. It has all the same problems one might expect: human forgets to populate equivalent secret to another environment. This tends to be more of a coldstart problem when bringing up a new service.
The alternative: git-crypt of secrets. For those unfamiliar, you can basically encrypt a config file using the public key. then at checkout, the build platform has the secret to decrypt.
While the automation is somewhat nice, I don’t like it. The secrets are exchanged on github and the PR review process does nothing. Someone can be changing a secret they shouldn’t be changing or fat-fingering the config file (human error). Hard to catch that.
We haven’t come up with the perfect way yet.
https://github.com/ContainerSolutions/externalsecret-operator/ looks pretty neat
using 1Password as the gui to manage your secrets
Thanks @Erik Osterman (Cloud Posse) - ran out of time to reply in 2019!
Anyway, I appreciate your insights. It kind of confirms my experience.
Great point about secrets being available in TF state.
Regarding the issue with git-crypt, I think this is an issue with external secret storage too. Vault will allow you to validate input if you use Sentinel, but without you could easily have the situation where someone enters a malformed secret. As you said, this type of thing is hard to catch.
For the current client I’m working with I’ve decided to use TF var user input because we’re not automating Terraform plan/apply at all. The inputs are being written to SSM Parameter Store. Parameter Store paths are locked down using IAM roles.
The client is using Dashlane for general password management so we’ll use this to store arbitrary strings too.
Not a great solution but at least it’s simple.
I used dashlane for a ~~~1 year (migrated from 1p). But lost total confidence in Dashlane when passwords would fail to replicate across devices. Also, during that time 1password improved by leaps and bounds, plus introduced 1p for teams. Moved back ~~~years ago and very happy.
That’s good to know. I read elsewhere that Dashlane was flakey.
I’ve tried a few including LastPass. I need to try 1password, esp since they integrate with tools like the one you posted above!
I’m not a fan of LastPass either. Doesn’t support TOTP
Yeah it has its own authenticator app for MFA
And it doesn’t support shared MFA, right?
You mean like a team can each have their own MFA device on the same account?
If so, then yes. I have had two phones attached.
No, like AWS Master Account with MFA - stored in LastPass.
Oh right. Good question. I’m about to set up Teams for another client so I’ll let you know.
2019-12-21
2019-12-22
Folks, any comments here pls ? I have created a security group in account x. But later on updated it with a different privider (account Y); with this updated a security group created in an accountY; but then should the old resource be left in accountX; that’s what I see … is that expected ?
Yes absolutely what you should see. Your state is tied to the account. So if you apply with account X and then account Y you have two states and thus two of the resource
Thanks … what’s the way “terraform show” works with multiple providers ?
2019-12-23
Has anyone been able to create a module for creating a lambda to rotate RDS credentials for Postgres in AWS Secrets manager? There is a CloudFormation template for it but I’m finding it a real pain to replicate it.
@Bruce Have you thought of using aws_cloudformation_stack in terraform. It’s maybe not the nicest solution, but it will save you time and you can still deploy it with terraform.
I was trying to avoid that but that may be the solution
:zoom: Join us for “Office Hours” every Wednesday 11:30AM (PST, GMT-7) via Zoom.
This is an opportunity to ask us questions on terraform and get to know others in the community on a more personal level. Next one is Jan 01, 2020 11:30AM.
Register for Webinar
#office-hours (our channel)
2019-12-24
Is “aws organisations” the best way to deal with creating and maintaining users across environments (Dev/Stage/Prod) ? Wondering what will be the differences of creating user base and role access to different environment in terraform versus using “aws organizations” ? Will there be any specific advantages of using “organizations” as such ?
Many thanks in advance.
It doesn’t matter for cross account role assumption specifically. The best way, regardless of using Organizations or not is to simply not have any users except for in one account. Or, if using an auth provider, not having any AWS users at all.
Thanks; and you mean “auth provider” means for ex: SAML ?
except for in one account: So do you mean like master account will have all the AWS users in a company and they assume roles when they need to access other accounts ?
Yes, SAML. And yes, users in an “ops” or “admin” account and then assume roles across… Keeping it clean.
Hi i have just used the eks nodegroup module
Great effort
Btw i spent 2 hours to find out an issue
It seems that there is a bug on aws API if you come from ipv6 ip
I have switched to ip4 and it is fixed
2019-12-26
Hi I have just started using terraform. I was reading elasticsearch aws terraform module : https://github.com/cloudposse/terraform-aws-elasticsearch
I did not find any precision about if its includes logstash in the stack or not.
I am trying to build an ELK cluster on AWS
2019-12-29
Hi Guys, is there a way to for loop in templatefale through miltiple lists/tuples? something like this for example:
%{ for name lastname age in names lastnames ages ~}
echo ${name} ${lastname} ${age}
%{ endfor ~}
Are they the same length?
yes
2019-12-30
any recommendations for terraform version manager ?
thanks
Hello Guys
Stuck in a aws_launch_template user data issue since a long time.
Trying to deploy this template:
data "template_file" "user-data"{
template = <<EOF
echo export DB_CONNECTION="${var.rds_endpoint}" >> /etc/profile
EOF
}
resource "aws_launch_template" "django-launch-template" {
image_id = "${data.aws_ami.django-ami.id}"
instance_type = "${var.instance_type}"
name = "${var.template_name}"
key_name = "${var.key_pair}"
user_data = "${base64decode(data.template_file.user-data.rendered)}"
}
Ans this is the error I get:
Error: Error in function call
on main.tf line 41, in resource "aws_launch_template" "django-launch-template":
41: user_data = "${base64decode(data.template_file.user-data.rendered)}"
|----------------
| data.template_file.user-data.rendered is " echo export DB_CONNECTION=\"Test.String.com\" >> /etc/profile\n"
Call to function "base64decode" failed: failed to decode base64 data ' echo
export DB_CONNECTION="Test.String.com" >> /etc/profile
’=
try without the base64decode()
Ahh.. My bad Used the wrong function Sorry
Works with base64encode()
how would we translate
"${var.var1}${var.var2}"
with the exact same notation
yeah that worked. Thanks
in terraform 12 ?
2019-12-31
when upgrading from terraform 0.11.x to 0.12, what is the best practice to deal with state stored in S3?
It should update without problems
If you don’t have versioning turned on (which you should), I would make sure to have a backup
i am not sure if versioning is turned on not, how do you enable versioning ?
Haven’t had any problems with the state updating automatically. Update to 0.11.14 first, then 0.12.x