#release-engineering (2020-01)

Remote worker’s but this might sadly not be an option for travic ci.

via their support found that the premium vm’s are still there

will try that first

you could maybe connect your repo to gitlab-ci with a “ci-only” project, and manage your own runners as powerful as you need?

Drop Travis?

5K year for 10 concurrent is really pricey. I used to run Gitlab’s CI with docker-machine launching on-demand spot instances for less than $50/month for the runners.

in :

Provision the module with Terraform. Terraform GitHub Provider will use the github_webhooks_token to create webhooks on the repo

as I understand it I need to provision the module meaning provided the necessary input variables to be able to run apply, correct ?

correct

thanks

mmm I’m getting this error :

I used something like :atlantis_repo_whitelist = ["cloudposse/terraform-aws-ecs-atlantis"]

that variable is set to list(string)

https://github.com/cloudposse/terraform-aws-ecs-atlantis/blob/master/variables.tf#L123

so this should work

I know why

the example is incorrect

https://github.com/cloudposse/terraform-aws-ecs-atlantis/blob/master/examples/with_cognito_authentication/variables.tf#L78

if you add * it doesnt like it somehow

this example https://github.com/cloudposse/terraform-aws-ecs-atlantis/tree/master/examples/complete

is always correct since it’s auto-tested on AWS with terratest

at least it ran OK the last time a PR was opened for the repo

https://github.com/cloudposse/terraform-aws-ecs-atlantis/blob/master/examples/complete/variables.tf#L131

https://github.com/cloudposse/terraform-aws-ecs-atlantis/blob/master/examples/complete/fixtures.us-east-2.tfvars#L45

they are different

different how?

the example itself defaults it to empty list

and we provide the concrete value in tfvars

the default is different :

that is from the cognito example

but in the cognito example there is no .tfvars, just defaults

if I’m not mistaken, is just the cognito-example folder what have the problem

I switched to the complete example and it works

mmmm the provisioning of the module should work for private repos too , right ?

The user you create the token for, must be an admin in the repo to be able to create webhooks

I added to a team that is admin

That’s the dropdown with Read, Write and Admin values

exactly

I just added it as a single collaborator with admin and does not work

I forgot to add the scope!!!!

admin:repo_hook

another issue

the webhooks are created in the repo

but every time the apply happens wants to create the webhooks again

so I’m guessing webhook_enabled = true should be set to false after initial bootstrap ?

I wonder how this will work for multiple repos

I think I figure out that part too no

but I wonder about this piece : https://github.com/cloudposse/terraform-aws-ecs-atlantis#caveats

so the repo where the webhook were created should have a dockerfile ?

or should I point it to : github.com/cloudposse/atlantis

ahhh I’m so stupid

that will have to be the and ECR repo image or docker registry of atlantis

mmmm that does not seems to work

if the example in https://github.com/cloudposse/terraform-aws-ecs-atlantis/tree/master/examples/complete did not help, this is how we deployed atlantis in our test account (it’s working now) https://github.com/cloudposse/testing.cloudposse.co/tree/master/conf/ecs

uses https://github.com/cloudposse/terraform-root-modules/tree/master/aws/ecs

mmm

the environment variables of the ECS task are missing

I mean , the only one that is created is ATLANTIS_ENABLED

but atlantis is expecting other ENV vars

I have no clue

I can;t use default-backend as an image_url for the ECS task since that thing is the pretty 404 page

so I changed it to cloudposse/atlantis:latest

and now atlantis does not start

but I’m confused about how codepipeline/codebuild work since I never used them before

that is why I don’t know if my repo have to have a Docker file to build atlantis and then that is build and upload to ECR and then run, or atlantis runs all the time and then it pulls the repo and run TF etc

need to read the docs more

base on this : https://www.runatlantis.io/docs/deployment.html#architecture-overview

altantis tuns all the time and recieve the webhook requests

so it needs to be up and running to do so

this is what I do not see : https://github.com/terraform-aws-modules/terraform-aws-atlantis/blob/master/main.tf#L28

which I think I’m missing somehow

do I have to have a Docker file like this in eve repo that I run atlantis to run ? https://github.com/cloudposse/atlantis/blob/master/Dockerfile

so Codepipeline can build the image and push it to ECS?

is that the idea?

as we did it, we install atlantis in Docker file for each environment we use

e.g. for our testing account https://github.com/cloudposse/testing.cloudposse.co/blob/master/Dockerfile#L69

so the CodePipeline builds the image on merging to master branch

and saves it to ECR

and then deploys it to ECS

so yes, in our case, we install atlantis in the same repo on which atlantis operates after it gets installed

but you can choose any other solution that fits your needs

in our case, atlantis is deployed from the repo, and then operates on the same repo using the config file https://github.com/cloudposse/testing.cloudposse.co/blob/master/atlantis.yaml

(we did it to minimize the number of repos to create and maintain. You can have atlantis defined in one repo, but operating on a different infra repo. But in this case, the number of repos doubles)

(the confusion here is because we merged two diff concepts together to minimize the number of repos: GitOps for atlantis itself (how to deploy it into each environment), and GitOps for the resources that atlantis controls)

if you use atlantis itself from a different repo (not on which it operates), this file becomes very important from security point of view

https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/ecs/atlantis-repo-config.yaml#L11

id: github.com/cloudposse/testing.cloudposse.co gives atlantis permissions to the specified repo only

another important part of the whole solution is https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh

since all Dockerfiles have geodesic as base, atlantis.sh gets executed

and all ENV vars get exported from SSM to the shell using chamber https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh#L45. Write them to SSM as described here https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/ecs/atlantis.auto.tfvars#L1

@jose.amengual you were asking about missing vars

(who said the whole solution would be easy )

hahaha

now the missing pieces are surfacing…

now I understand a bit more

in our case we have several repos that build one environment, where we do have a ECS cluster

and the many services inside are build from a different TF repo

so I was thinking to deploy atlantis as a service that is part of the same ECS cluster

and trigger the builds trough the webhooks

yes, so you prob will have to do some digging since our stuff was made to work with geodesic which has lots of things for atlantis

and the idea behind this is to minimize the “access” of atlantis and atlantis apply to that specific aws account/cluster

if that’s a question, then yes

and to minimize the number of repos

if you have 7 environments, you would have 14 repos (7 for atlantis) if you deployed atlantis from separate repos

ok, that confuses me a bit

that’s why we put atlantis in the Dockerfile for the the same repo on which it operates in the same account

I though atlantis will clone and run any repo that hits the public endpoint for the webhook, right ?

it can

but you need to deploy atlantis itself

preferably with GitOps

imagine that I gave atlantis running already

then I add the webhooks to the repo, give access to the atlantis bot user to the repo then atlantis should be able to run terraform apply from that repo on a github event

cloudposse/testing.cloudposse.co is the repo that deploys atlantis and the repo on which the deployed atlantis operates

for that environment

as you described yes, it will do it

so one atlantis per environment basically

no matter how many terraform repos that environment could have?

yes

but as I mentioned, you have to deploy atlantis somehow into each env

that’s when it becomes more complicated

correct I understand

and dont’ forget https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/ecs/atlantis-repo-config.yaml

and that is why you run atlantis from whiting the TF repo

to avoid having two each

haha yes, been trying to say that for 30 mins

hahahahahaha

to minimize the number of repos

and I guess another advantage of running from the sam repo is that you could somewhat customize atlantis for repo specific configs

Yes

For example, it uses the same credentials or assume role as you do when executing into that container

So you specify it only once in one place

exactly

thanks so much for the time, sorry for being too slow

np, that’s not easy solution

I was wondering yesterday, building atlantis every time could take some time

do you happen to know if there is a notable difference from having atlantis running all the time?

CodePipeline will build on merge to master

and deploy to ECS

but ECS will switch to the new version only if the deployment is a success

ok

and this approach should not be an issue if I want to setup atlantis to build from a branch ?

not just merge to master

yes

we selected master

in CodePipeline config

ahhhh right that is Codepipeline not atlantis, ok cool

so I’m back on working on this :

I’m creating my Docker file and atlantis-repo-config.yaml

I was thinking to use this Docket file : https://github.com/cloudposse/atlantis/blob/master/Dockerfile

and ad this to it : https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh#L45

• some cleaning and that should get me going

or ? or not ?

nope….the Dockerfile needs to be different

mmm not working

I wonder if that is atlantis complaining

or just codepipeline

mmm is pointing to master….I need to point it to my branch

I never used codebuild before….

now I have a buildspec.yaml

I got it running…

just having problem with the repo.yaml

thanks for sharing, this is awesome

will add that

I would argue though the price of Jenkins is the effort needed to manage it, solve for plugin compatibility issues etc.etc.

Sure, but once you get that stuff figured out there aren’t really any ongoing costs, vs a subscription

Once again I’m not saying Jenkins is the best and everyone should be using it, but it doesn’t deserve all the flak that gets shot at it sometimes.

It’s the result of years of hard work, done for nothing in return

I think the best use case for Jenkins is a small, self-sufficient devops team. Governance and multi tenant is where jenkins starts to fall over hard

cloud resource cost is not free as well.

True, although as Erik pointed out in his post you can take advantage of spot instances to make it very cheap. Jenkins ephemeral workers are particularly well suited to spot instances

Great write-up @Erik Osterman (Cloud Posse) - you’ve covered a lot of ground over one weekend :smile:.

Here are some things off the top of my head that might be worth mentioning:

plugin management • I haven’t had that many problems re plugins and upgrades but it really depends on how many plugins you use and, as you mentioned, the current maintenance. I have moved towards using command line tools in containers as opposed to installing some of the more exotic plugins. • although I haven’t tried it myself, there is also a Cloudbees “tried and tested” version of Jenkins (probably limited in plugins but more stable) - https://www.cloudbees.com/products/cloudbees-jenkins-distribution/overview. They have the option to run on kubernetes as well: https://docs.cloudbees.com/docs/cloudbees-jenkins-distribution/latest/jenkins-lts-to-cloudbees-jenkins-distribution/#_kubernetes_deployment Docker builds in kubernetes: • another alternative would be to use kaniko - https://github.com/GoogleContainerTools/kaniko • Makisu also looks promising: https://github.com/uber/makisu Containers in pod templates: • Yes, this is a pain, and whilst it’s easy enough to “fix” with JCasC and a Jenkins restart, it’s not easily done on a per build basis unless, as you mentioned, your Jenkinsfile/project contains the yaml. • To get away from the ugliness of it all (yes, it is ugly :slightly_smiling_face:) you could use the readFile or readTrusted steps, but again it’s not that simple.

• Another thing to think about is the potential resource usage when using multiple containers in a single pod as opposed to build your own image with all the necessary tools - each container would have resource required and limits set - a single container would save resources in this respect. I think @roth.andy mentioned this somewhere.

Jenkins “Configuration as Code” plugin supports nearly everything you’d need to run Jenkins itself in a “GitOps” compliant manner And if it doesn’t there is always the https://github.com/jenkinsci/configuration-as-code-groovy-plugin which allows you to run arbitrary groovy scripts for the bits you need.

Oh, and a small thing - I think the Github Organisation Plugin (as opposed to the workflow-multi-branch plugin) automatically detects projects with Jenkinsfiles. The multi-branch just detects branches, tags, etc, from within a configured repo.

Thanks @Steve Boardwell !! Will update the post tomorrow with your feedback and corrections :-) very much appreciated

Thanks @Steve Boardwell I didn’t know about Makisu! Have you used the buildah/podman from Fedora?

+1 for podman

isn’t it from Red Hat though?

Yes… well these days Fedora is basically the RedHat inclubator…

@Blaise Pabon no, it’s on the list though.

@Steve Boardwell turns out the github-branch-source plugin is what I configured for the repository scanning capabilities https://docs.cloudbees.com/docs/admin-resources/latest/plugins/github-branch-source

Ah, yes, that was it - not the “Github Organisation Plugin”. I was getting my plugins crossed thinking of the “Pipeline Multibranch Plugin”. We don’t scan our org automatically so haven’t needed to configure for a while. Thanks for the heads up.

https://major.io/2019/05/24/build-containers-in-gitlab-ci-with-buildah/