#atlantis (2020-05)

interesting

I just set it up for one account per environment

but the catch is that the TF repo is for both envs, stage and prod

so I started freaking out when I realize that people can access the prod state but….it was not possible since the S3 bucket lives in another account

so in your case is a bit harder

you could deploy 3 different atlantis with 3 different web hooks and 3 different repos

and then by instance profile restrict which remote state they can use

I’m assuming you are using remote state

otherwise restricting using the githib groups could be another way but that is on the clouposse repo

is a fork of atlantis

then you can name a team that can run the apply

and only that team can do it

When I looked at the clouposse module it looked like it did not support gitlab, am I wrong in thinking that?

generally your idea of 3 Atlantis makes sense - the only drawback of this is not having a clear way to actually write an instance profile that can’t touch prd resources

the state yes, but I guess you could manually import a prd instance then destroy it?

have considered using an approach like this: https://docs.aws.amazon.com/iot/latest/developerguide/tagging-iot-iam.html

I haver seen that

you could do TAG base access

https://aws.amazon.com/blogs/security/simplify-granting-access-to-your-aws-resources-by-using-tags-on-aws-iam-users-and-roles/

ABACs are called

Attribute base access

Oh nice.. that seems like quite a good solution

are they in different VPCs ?

yes.. we’ve done what we can with the one account restriction

do you have a specific reference for the cloudposse groups approach, not clear to me from their github repo what they’ve done in the fork

one sec

by the way : https://sweetops.slack.com/archives/CDKPAK81Z/p1587861631054200?thread_ts=1587160463.053900&cid=CDKPAK81Z

https://github.com/cloudposse/atlantis/commit/edd82fe5de051e881d13e375b19364dcd1d41c4e#diff-1427b112aa72f64adcf839007e954d56R131

awesome, tanks a bunch

I’ll go through in detail

thanks a lot for the help, will let you know how I get on

no problem, I hope this help

By this do you mean a repo per environment?

yes

so that each repo have it’s own webhook

i do that already

what if you add something like :

autoplan:
    when_modified: ["*stg.tf", "../modules/**stg.tf"]

ahhhhhhh…..

I missed that

not totally sure this will work, but may do

yes, that could work too

but in my case I will still have two webhooks

right, but one of them may not actually trigger anything

correct

(hopefully)

I wonder if there is a way to do something like assume role before the different workflows

that would simplify this

you could run aws-vault in the init step

and apply step

ah yeah.. that might be enough

like the way they show to run Terragrunt and custom workflows

I think what I’m going to try later today (I’m in Singapore, so guessing not the same time zone) is one atlantis with assume roles for env

you will need to package aws-vault plus .aws config dir in the container

where do you run atlantis btw? on fargate?

Fargate

Looks like now that this guys are in hashicorp, they are not so willing to improve Atlantis

hmm thats a shame

what is in the fork that you need not existing in the main branch?

Github group whitelist

I don’t know if it works for gitlab

Basically only certain groups can do apply

Ah okay. Yeah I’m going with the fargate setup currently. but some minor blockers around getting my config to work

was planning on just using codeowners for that

Github does not have code owners I believe

https://github.com/runatlantis/atlantis/issues/653

it was abandon

Oh … sad

Wondering if there is any real alternative

(except terraform enterprise)

atlantis have like 300 forks

I guess a lot of people have think some options are lacking..

either the Atlantis devs have really dropped the ball here or people just don’t understand open source

lol

try terragrunt-atlantis-config it will generate atlantis.yaml according to your HCL dependencies

omg sorry for that it terragrunt not terraform

would this work if a module was referenced using a git source as well ?

Don’t know

probably not, you can look into the atlantis code

I’m not so versed in golang yet

what problem are you trying to solve?

you want atlantis to run diff workflows for staging and prod when diff files are modified?

and you are using one repo?

suggest you create a folder conf and add staging.tfvars and prod.tfvars in it

sorry I made a mistake

if you modify the main module, both staging and prod will run

if you modify only e.g. staging.tfvars, then only the staging atlantis would run

one aws account per environment, one atlantis server per aws account

one repo for all environments?

yes

so you need to separate prod config from staging config

they should not be in [variables.tf](http://variables.tf) for all environments

I run them like this, this is the server config

staging-us-west-2:
    plan:
      steps:
      - run: rm -rf .terraform
      - init:
          extra_args: [-backend-config=staging-backend-us-west-2.tfvars]
      - plan:
          extra_args: [-var-file=staging-us-west-2.tfvars]

only the code (logic) is common for all

the idea was that we could test config changes of the main module in staging without triggering a prod even to the prod atlantis

that is what we usually do

so one repo terraform-aws-rds for multiple environments and we have different .tfvars and backen-configs

for example

but that should work

you already disabled autoplan

when you run atlantis plan -p production-us-east-2, only prod should run

yes but by repo is for multiple environments so I haver multiple webhooks

one point from the same repo to the staginmg atlantis and another to the prod atlantis

I want the staging atlantis to ignore the productions workflows

so I thought I could use :

version: 2
projects:
- name: staging-us-east-2
  dir: .
  workflow: staging-us-east-2
  terraform_version: v0.12.24
  autoplan:
    when_modified: 
      - "*.tf"
    enabled: false
- name: production-us-east-2
  dir: .
  workflow: production-us-east-2
  terraform_version: v0.12.24
  autoplan:
    when_modified: 
      - "prod.txt"
    enabled: false

and so I create a PR, then go and run manually atlantis plan -p staging-use-east-2 then once approved then I run atlantis apply and after is tested, I can user the same PR or a new PR , modify the prod.txt file and then run atlantis plan -p production-use-east-2

but what I really wanted was that the atlantis in staging ignore any call to production workflows

to avoid this :

I want to avoid having devs to be able to run atlantis plan against prod and getting those errors in the comments

since the atlantis servers are separated is not a security concern

is just more annoying that anything

you are talking about two diff things. If you run those atlantis plan xxxxxxxx commands manually, then it should run only the workflow specified in the command (prod or staging)

for the autoplan, enable it, then make each autoplan (prod and staging) depend on diff tfvars files

in this case, if you update the module(s), both autoplans will run and show the changes

if you update the settings for e.g. staging, only the staging autoplan will run

I prefer to run autoplan manually than automatically but I will try this

do you know if the when_modified array is like an AND for every item ?

like *.tf AND prod.tfvars need to change for it to trigger ?

It’s OR

If you modify the code, then *.tf will trigger both

If you modify settings for one environment, only one should trigger

ok I see

thanks a lot

well, yesterday I realize how stupid I’m and why I never thought about having two separated atlantis-repo-config.yaml files, one for the altantis in prod, one for the atlantis in staging

that will solve this issue

that’s a good idea

each one could be configured and allowed to do only specific things

exactly

and I can still do the specific files so that it does not comment back to the repo saying that that project does not exist

etc

This is what we do, we have Atlantis look for a configuration file that matches the env name. We have a patched version of Atlantis that does this. We still get the extra status checks showing up in the PR, but I think some of that is fixed in the latest version of Atlantis

if you use a WAF rule you can avoid the calls from github to altantis base on some text in the payload post from the webhook but if you can deal with the extra status then that is cheaper

I saw a delete old messages option or something in the new version

They added the –silence-vcs-status-no-plans flag that silences VCS commit status when autoplan finds no projects to plan

Haven’t tried it yet

can you share your repo and altantis yaml ?

I mean a piece of it

ohh but you said you have a patched version, what patches do you have ?

to support different Atlantis configs was a quick hack. We set an env variable that tells Atlantis what config file to load

So in the repo we have 3 Atlantis config files, prod.yaml dev.yaml sandbox.yaml.

ahhh ok I know the feature request you are altking

that it never got merged

Yeah, that PR did it the better way but it requires lots of changes. Ours is like 5 lines of code

We were thinking of reimplementing that PR and submitting it again

https://github.com/runatlantis/atlantis/pull/310

can you share the code ?

https://github.com/mediwareinc/atlantis-1

It’s in the multi-env branch

https://github.com/mediwareinc/atlantis-1/commit/cf71a8f8debc126b2210bda9eee97a9b4eb7339d

NO Fing way……..

how can someone deny to merge that?

if you create the pull request and starts huge discussion and it takes months to merge, I will just flip the table

I think that PR you linked should have been included, not sure why they rejected it

but you solution is totally valid too

for people that can deploy multiple altlantis

We are going to update to the latest Atlantis next week and try our change with the –silence-vcs-status-no-plans flag

But ultimately would like to get something submitted back to atlantis

I’m kinda concern with the project because both guys work for hashicorp

basically getting pay to have a conflict of interest

they should hand over the project

For 1) I tested it locally using docker run --user atlantis -it 2febd567f15e bash after chowning to root:atlantis which allowed reading users but not writing / appending

atlantis has a global config file. It does not answer all the questions or allow you to config everything you asked, but might help

Yep, this has support for that, but it’s a beast of a module

thanks guys - I was using the other terraform on fargate module, will go through this one

Pretty cool. Does it only work with terraform cloud and enterprise or also for other backends?

This works with terraform open source

if you have TF enterprise you probably don’t need this?

I was under the impression they offered integrations of many build tools

someone must be working in an atlantis feature complete clone, right…..?

I heard a rumour

Isn’t the guy that started atlantis ilya working directly got hashicorp? Wouldn’t he be the one working on this as well?

both guys are working for hashicorp

thread start

do i need to define both of these in my repo config ?

- id: "/.*/"
  workflow: default
- name: secrets
  dir: secrets
  workflow: secrets

even tho my server config has this already ?

- id: "/.*/"
  workflow: default

which for of atlantis are you using ?

using runatlantis/atlantis:v0.12.0

looks like the latest version

you need to allow custom workflows if you are defining workflows in your repo config

otherwise is not going to pick them up

it does pick it up for my custom workflow, just not for my not defined modules. this is my atlantis server config

that should apply to all repos

it should but after i added a repo level config that i shared above, it seemed to only allow the custom one to work

the others i have to manually run atlantis plan -d mytfdir

mmm so is the autoplan you are expecting it to run?

yessir

I hate autoplan so I disable every time

autoplan does have a default for the files it tracks

“/.tf”*

something like that

I can type two *

freaking annoying slack

@Andriy Knysh (Cloud Posse) I was trying make dist and got a bunch of errors, I’m wondering how you guys run the build ?

I have no idea what I’m doing…. lol

make build-service

which is CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o atlantis .

and that exploded

do I have to have a specific version of go ?

i tested it with 1.11 and 1.2 before, now I have 1.13.1, all works

are you building `

runatlantis/atlantis

?

or CloudPosse repo?

runatlantis prob have made a lot of changes since the last time we forked it

cloudposse repo

make sure you copy it into <GOPATH>/src/github.com/runatlantis/atlantis

b/c all imports are using runatlantis

that’s how Go works

golang 101 lol

this looks better

at least it did more stuff

I moved to runatlantis/atlantis

and in there I run make build-service

better but not quite

my ultimate goal is to patch current atlantis with the github code you guys did and add the atlantis.yaml specific settings I need

but for now building cloudposse fork will be great

so I ended up downloading lates atlantis src

compiled

added few thing worked

and I decided to add the code from PR #30 from cloudposse

after a few fixes

compiled

I deployed

but I’m stuck

fo whatever reason it keeps saying : Error: User @jamengual does not have permissions to execute 'plan' command.

which is checkUserPermissions returning false

but I do not know why yet

any pointers @Andriy Knysh (Cloud Posse)

this is the bit of code is doing the call

/ Check if the user who commented has the permissions to execute the 'plan' or 'apply' commands
	ok, err := e.checkUserPermissions(baseRepo, user, parseResult.Command)
	if err != nil {
		e.Logger.Err("unable to comment on pull request: %s", err)
		return
	}
	if !ok {
		e.commentUserDoesNotHavePermissions(baseRepo, pullNum, user, parseResult.Command)
		e.respond(w, logging.Warn, http.StatusForbidden, "User @%s does not have permissions to execute '%s' command", user.Username, parseResult.Command.Name.String())
		return
	}

@jose.amengual did you specify the parameter gh-team-whitelist?

https://github.com/cloudposse/atlantis/pull/30/files#diff-1427b112aa72f64adcf839007e954d56R132

and then did you add your user to the whitelisted team (on GitHub)?

the code in this PR https://github.com/cloudposse/atlantis/pull/30 makes gh-team-whitelist param optional. If not specified at all, the default functionality is in place. If specified, then the code checks if the user is in a whitelisted team. If not, the error is thrown

yes I did

basically I had atlantis already running with cloudposse fork

and then I I forked latest and added your code

and deployed to the same fargate task

I have everything in Parameter store

I added debug too

did you use the correct format for gh-team-whitelist?

hds-team:plan,hds-team:apply

(looks like you did since you used CP fork before)

I’m part of that team

I used the terraform atlantis module from you guys

it is weird, I can run help and that , it does not crash or anything, plus returns my user correctly

well… atlantis made a lot of changes, and GitHub made a lot of changes. If you copy the code correctly and it does not work, then something has been changed in the GitHub API

somehow something in team_whitelist_checker.go does not work

if you run our fork now, does it work?

I can roll back, one sec

somehow something in team_whitelist_checker.go does not work

that is my theory

we made changes a few times as well b/c GitHub kept changing some API

ohhhhh so maybe I did not push those updates ?????

I did this : https://github.com/cloudposse/atlantis/pull/30/files

(don’t know )

I checked like 40 times

after #30 I do not see anything about github

in situations like this, and since you are modifying the code anyway, would be great to update the version number and display it on the screen (I think it’s displayed)

so you can know for sure which version got deployed

https://github.com/cloudposse/atlantis/blob/master/main.go#L23

mmmm

I see :

2020-05-29 18:15:42
2020/05/30 01:15:42+0000 [INFO] server.go:462 server: Atlantis started - listening on port 4141

yes so I have a custom version

atlantis version                                                     8.9m  Fri 29 May 20:34:46 2020 atlantis 0.12.0-multienv-debug1

I see the same if I docker build locally

I wonder if I could shot the version on fargate logs

you should see the version in the UI, in the bottom-right corner

I added this too https://github.com/cloudposse/atlantis/pull/32/files

which I though it was the problem

the ui…ok let me check I have not used the UI ever

mmm the ui is 503…..

the ui is on /?

ok, listener rule error , it shows my version

v0.12.0-multienv-debug0

so at least I know is the one I compiled

and this was updated too

func (g *GithubClient) GetTeamNamesForUser(repo models.Repo, user models.User) ([]string, error) {
	var teamNames []string
	opts := &github.ListOptions{}
	org := repo.Owner
	for {
		teams, resp, err := g.client.Teams.ListTeams(g.ctx, org, opts)
		if err != nil {
			return nil, err
		}
		for _, t := range teams {
			membership, _, err := g.client.Teams.GetTeamMembership(g.ctx, t.GetID(), user.Username)
			if err == nil && membership != nil {
				if *membership.State == "active" && (*membership.Role == "member" || *membership.Role == "maintainer") {
					teamNames = append(teamNames, t.GetName())
				}
			}
		}
		if resp.NextPage == 0 {
			break
		}
		opts.Page = resp.NextPage
	}
	return teamNames, nil
}

yes / https://github.com/cloudposse/atlantis/blob/master/server/server.go#L435

do you see this error Error: User @jamengual does not have permissions to execute 'plan' command as GitHub comment?

also, you sure that the user is a member of the team, and the team name is correct?

and also, is our version working for you?

yes I do see it as a github comment

yes I’m a member of that team and the team is correct

you version was working ok but my need is that I use one repo for all environment so I deploy and atlantis per AWS account and without the option to define a custom atlantis.yaml it does not work for me so I added https://github.com/runatlantis/atlantis/compare/master...mediwareinc:multienv-0.11.0

which is very very simple and fit my needs

but I really like the team whitelist option and that is why I’m here

so it’s the same team whitelist code in both repos, one is working, the other is not

correct

Unless I missed something

and the code gets executed and checks for team membership

as fas as I can tell

so 1) you missed something; 2) atlantis made other changes that affected it; 3) you are using new GitHub API that changed the logic (they did it in the past)

I’m not a go experts by any means but I was thinking to add e.Logger.Err

one of the 3 but I think 3 is unlikely because this was working with cloudposse fork and I did not change anything related to apis for github

since our fork does not use Go modules (was not in atlantis at the time), and the new atlantis uses Go modules, the GitHub APi could be completely different

(the sad thing is, w/o Go modules it was not possible to pin it to a particular version)

so you alway get the latest

but new atlantis uses Go modules so you can see what version you are using

you alway get the latest - if you don’t use Go modules, which was created to fix those issues

I see ok

as I said, w/o Go modules it was aways the latest version, and GitHub updated the API a few times and broke our code

with Go modules in the latest atlantis, they pinned it to a particular version, but.. that version could be new with some API changes from GitHub (just an idea)

so you think when I did go get [github.com/runatlantis/atlantis](http://github.com/runatlantis/atlantis)

ad then go build I got a version of the github client api that is not compatible ?

might be

we had such issue before

not saying that they change the API all the time, but they did, and broke our code

I’m running my version now using ngrok and using chamber to populate the ENV variables to my local

so I can debug faster

intersting

I added this :

// checkUserPermissions checks if the user has permissions to execute the command
func (e *EventsController) checkUserPermissions(repo models.Repo, user models.User, cmd *events.CommentCommand) (bool, error) {
	if cmd.Name == models.ApplyCommand || cmd.Name == models.PlanCommand {
		teams, err := e.VCSClient.GetTeamNamesForUser(repo, user)
		if err != nil {
			return false, err
		}
		e.Logger.Err("PEPE teams: ", teams)
		ok := e.TeamWhitelistChecker.IsCommandAllowedForAnyTeam(teams, cmd.Name.String())
		if !ok {
			return false, nil
		}
	}
	return true, nil
}

e.Logger.Err("PEPE teams: ", teams)

mmm there is some sensitive information on all this

basically

terrafom-atlantis is a team

but the other is the description of the ream not the team name

no wait….those are the names

they just have spaces

we did not test teams with spaces

that is it

I added terraform-atlantis to the team whitelist

and now is working

but where is hds-team in the list?

yes it was

and it did work with the cloudposse fork

atlantis_gh_team_whitelist = hds-team:plan,hds-team:apply,terraform-atlantis:plan

what do you mean “it was”? it’s not in the list [Ops Team Product Engineering Employees terraform-atlantis]

I mangled the output

so you are not part of that team

sorry

PEPE teams: %!(EXTRA []string=[IQ Ops HDS Team Product Engineering Employees terraform-atlantis])

HDS Team is that team

and I’m part of it

ok so…….

HDS Team:plan,hds-team:apply that works

so this must be something to do with the the github client response that is now not passing some sort of alternated name for the team ?

I guess a team with a name with space will be converted to something else ?

I’m trying to understand why it worked with the cloudposse fork before

@Andriy Knysh (Cloud Posse) do you happen to know if there is a way to shortcircuit atlantis execution after the call to ParseRepoCfg I was trying to stop atlantis if :

if projectName != "" && yaml.AtlantisYAMLFilename != "atlantis.yaml" {
		return
	}

OR by using a new env variable ATLANTIS_MULTISERVERbut I’m not familiar with the code and after many hours the plan command keeps executing

my guess that whatever I do it will have to be in the Event controller but I’m not so sure anymore

then i see this in github

�[31m
�[1m�[31mError: �[0m�[0m�[1mError loading state: state snapshot was created by Terraform v0.12.25, which is newer than current v0.12.24; upgrade to Terraform v0.12.25 or greater to work with this state�[0m

�[0m�[0m�[0m

im using atlantis 12 and i dont believe the new 13 that was just released fixes this, according to their changelog

any idea what im doing wrong ? or is this a bug ?

the init seems to work but the make diff which calls terraform refresh seems to use the default terraform instead of the terraform_version

perhaps v0.12.25 needs to be 0.12.25 according to this

this is interesting too

https://github.com/runatlantis/atlantis/issues/910

Looks like @lkysow says to use

- run: terraform$ATLANTIS_TERRAFORM_VERSION plan

so perhaps i need to do that in my Makefile to use the correct version of terraform …

terraform_version in atlantis workflow never worked for us as well

in my make diff, i changed it so my Makefile uses the $ATLANTIS_TERRAFORM_VERSION, not sure what else to do

wish atlantis could be smart enough to check the tfstate, get the correct version, and use the same version in the state

dude youre right, WTF, tested it out and that env variable wasnt even filled

so i guess im not allowed to use Makefile targets in my run atlantis command if it requires a non default terraform version

https://github.com/runatlantis/atlantis/issues/1065

yes it’s like that

you can fork it and improve

i figured it out https://github.com/runatlantis/atlantis/issues/1065#issuecomment-645404842

atlantis, when using repo configs, is using the PR branch’s config instead of the one in master. this wasn’t clear to me and was confusing.