#release-engineering (2020-01)
All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.
CI/CD Discussions
Archive: https://archive.sweetops.com/release-engineering/
2020-01-10

Hey does anyone have any examples of production/dev/staging environments integrated with terraform and orchestrated by a Jenkins pipeline?
2020-01-12

any GoCD
users here?
2020-01-17

We have been using travisci pro for years, and pay a whopping 5k a year for 10 concurrent builds. The cost or amount of builds is not really the issue though. It is the power of the underlying machines. Our builds flows are optimised to the max, but we have for example a 10 minute create-react-app build (webpack) which cannot be sped up in any way, except bigger CPU’s. 2 min on my top spec mbp vs 12 minutes on travis. But they do not have plans with faster machines, so not really an option. How are you all handling these kinds of problems?

Remote worker’s but this might sadly not be an option for travic ci.

via their support found that the premium vm’s are still there

will try that first

you could maybe connect your repo to gitlab-ci with a “ci-only” project, and manage your own runners as powerful as you need?


5K year for 10 concurrent is really pricey. I used to run Gitlab’s CI with docker-machine launching on-demand spot instances for less than $50/month for the runners.
2020-01-22
2020-01-23

so I’m using : https://github.com/cloudposse/terraform-aws-ecs-atlantis and I’m going trough the setup but
Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

in :
Provision the module with Terraform. Terraform GitHub Provider will use the github_webhooks_token to create webhooks on the repo
Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

as I understand it I need to provision
the module meaning provided the necessary input variables to be able to run apply, correct ?

correct

thanks

mmm I’m getting this error :

on main.tf line 95, in module "atlantis":
95: atlantis_repo_whitelist = [var.atlantis_repo_whitelist]
The given value is not suitable for child module variable
"atlantis_repo_whitelist" defined at ../../variables.tf:123,1-35: element 0:
string required.

I used something like :atlantis_repo_whitelist = ["cloudposse/terraform-aws-ecs-atlantis"]

that variable is set to list(string)

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

so this should work

I know why

the example is incorrect

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

if you add *
it doesnt like it somehow

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

is always correct since it’s auto-tested on AWS with terratest

at least it ran OK the last time a PR was opened for the repo

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

they are different

different how?

the example itself defaults it to empty list

and we provide the concrete value in tfvars

the default is different :

variable "atlantis_repo_whitelist" {
type = list(string)
description = "Whitelist of repositories Atlantis will accept webhooks from"
default = ["github.com/example/*"]
}

that is from the cognito example

but in the cognito example there is no .tfvars, just defaults

if I’m not mistaken, is just the cognito-example folder what have the problem

I switched to the complete example and it works

mmmm the provisioning of the module should work for private repos too , right ?

Error: POST <https://api.github.com/repos/company/terraform-rds/hooks>: 404 Not Found []
on .terraform/modules/atlantis.ecs_web_app.ecs_codepipeline.github_webhooks/main.tf line 6, in resource "github_repository_webhook" "default":
6: resource "github_repository_webhook" "default" {

The user you create the token for, must be an admin in the repo to be able to create webhooks

I added to a team that is admin

That’s the dropdown with Read, Write and Admin values

exactly

I just added it as a single collaborator with admin and does not work

I forgot to add the scope!!!!

admin:repo_hook

another issue

the webhooks are created in the repo

but every time the apply happens wants to create the webhooks again

so I’m guessing webhook_enabled = true
should be set to false after initial bootstrap ?

I wonder how this will work for multiple repos

I think I figure out that part too no

but I wonder about this piece : https://github.com/cloudposse/terraform-aws-ecs-atlantis#caveats
Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

so the repo where the webhook were created should have a dockerfile ?

or should I point it to : github.com/cloudposse/atlantis

ahhh I’m so stupid

that will have to be the and ECR repo image or docker registry of atlantis

mmmm that does not seems to work

[31mError: --gh-user/--gh-token or --gitlab-user/--gitlab-token or --bitbucket-user/--bitbucket-token must be set[39m

if the example in https://github.com/cloudposse/terraform-aws-ecs-atlantis/tree/master/examples/complete did not help, this is how we deployed atlantis
in our test account (it’s working now) https://github.com/cloudposse/testing.cloudposse.co/tree/master/conf/ecs
Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis
Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

mmm

the environment variables of the ECS task are missing

I mean , the only one that is created is ATLANTIS_ENABLED

but atlantis is expecting other ENV vars

I have no clue

I can;t use default-backend as an image_url for the ECS task since that thing is the pretty 404 page

so I changed it to cloudposse/atlantis:latest

and now atlantis does not start

but I’m confused about how codepipeline/codebuild work since I never used them before

that is why I don’t know if my repo have to have a Docker file to build atlantis and then that is build and upload to ECR and then run, or atlantis runs all the time and then it pulls the repo and run TF etc

need to read the docs more

Atlantis: Terraform Pull Request Automation

altantis tuns all the time and recieve the webhook requests

so it needs to be up and running to do so

this is what I do not see : https://github.com/terraform-aws-modules/terraform-aws-atlantis/blob/master/main.tf#L28
Terraform configurations for running Atlantis on AWS Fargate. Github, Gitlab and BitBucket are supported. - terraform-aws-modules/terraform-aws-atlantis

which I think I’m missing somehow

do I have to have a Docker file like this in eve repo that I run atlantis to run ? https://github.com/cloudposse/atlantis/blob/master/Dockerfile
GitOps for Teams (experimental hard fork of atlantis) - cloudposse/atlantis

so Codepipeline can build the image and push it to ECS?

is that the idea?

as we did it, we install atlantis in Docker file for each environment we use

e.g. for our testing account https://github.com/cloudposse/testing.cloudposse.co/blob/master/Dockerfile#L69
Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

so the CodePipeline builds the image on merging to master
branch

and saves it to ECR

and then deploys it to ECS

so yes, in our case, we install atlantis in the same repo on which atlantis operates after it gets installed

but you can choose any other solution that fits your needs

in our case, atlantis is deployed from the repo, and then operates on the same repo using the config file https://github.com/cloudposse/testing.cloudposse.co/blob/master/atlantis.yaml
Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

(we did it to minimize the number of repos to create and maintain. You can have atlantis defined in one repo, but operating on a different infra repo. But in this case, the number of repos doubles)

(the confusion here is because we merged two diff concepts together to minimize the number of repos: GitOps for atlantis itself (how to deploy it into each environment), and GitOps for the resources that atlantis controls)

if you use atlantis itself from a different repo (not on which it operates), this file becomes very important from security point of view

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

id: github.com/cloudposse/testing.cloudposse.co gives atlantis permissions to the specified repo only

another important part of the whole solution is https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh
Geodesic is a cloud automation shell. It's the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

since all Dockerfiles have geodesic
as base, atlantis.sh
gets executed

and all ENV vars get exported from SSM to the shell using chamber https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh#L45. Write them to SSM as described here https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/ecs/atlantis.auto.tfvars#L1
Geodesic is a cloud automation shell. It's the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

@jose.amengual you were asking about missing vars

(who said the whole solution would be easy )

hahaha

now the missing pieces are surfacing…

now I understand a bit more

in our case we have several repos that build one environment, where we do have a ECS cluster

and the many services inside are build from a different TF repo

so I was thinking to deploy atlantis as a service that is part of the same ECS cluster

and trigger the builds trough the webhooks

yes, so you prob will have to do some digging since our stuff was made to work with geodesic which has lots of things for atlantis

and the idea behind this is to minimize the “access” of atlantis and atlantis apply to that specific aws account/cluster

if that’s a question, then yes

and to minimize the number of repos

if you have 7 environments, you would have 14 repos (7 for atlantis) if you deployed atlantis from separate repos

ok, that confuses me a bit

that’s why we put atlantis in the Dockerfile for the the same repo on which it operates in the same account

I though atlantis will clone and run any repo that hits the public endpoint for the webhook, right ?

it can

but you need to deploy atlantis itself

preferably with GitOps

imagine that I gave atlantis running already

then I add the webhooks to the repo, give access to the atlantis bot user to the repo then atlantis should be able to run terraform apply from that repo on a github event

cloudposse/testing.cloudposse.co is the repo that deploys atlantis and the repo on which the deployed atlantis operates

for that environment

as you described yes, it will do it

so one atlantis per environment basically

no matter how many terraform repos that environment could have?

yes

but as I mentioned, you have to deploy atlantis somehow into each env

that’s when it becomes more complicated

correct I understand

and dont’ forget https://github.com/cloudposse/testing.cloudposse.co/blob/master/conf/ecs/atlantis-repo-config.yaml
Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

and that is why you run atlantis from whiting the TF repo

to avoid having two each

haha yes, been trying to say that for 30 mins

hahahahahaha

to minimize the number of repos

and I guess another advantage of running from the sam repo is that you could somewhat customize atlantis for repo specific configs

Yes

For example, it uses the same credentials or assume role as you do when executing into that container

So you specify it only once in one place

exactly

thanks so much for the time, sorry for being too slow

np, that’s not easy solution

I was wondering yesterday, building atlantis every time could take some time

do you happen to know if there is a notable difference from having atlantis running all the time?

CodePipeline will build on merge to master

and deploy to ECS

but ECS will switch to the new version only if the deployment is a success

ok

and this approach should not be an issue if I want to setup atlantis to build from a branch ?

not just merge to master

yes

we selected master

in CodePipeline config

ahhhh right that is Codepipeline not atlantis, ok cool

so I’m back on working on this :

I’m creating my Docker file and atlantis-repo-config.yaml

I was thinking to use this Docket file : https://github.com/cloudposse/atlantis/blob/master/Dockerfile
GitOps for Teams (experimental hard fork of atlantis) - cloudposse/atlantis

and ad this to it : https://github.com/cloudposse/geodesic/blob/master/rootfs/etc/init.d/atlantis.sh#L45
Geodesic is a cloud automation shell. It's the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

- some cleaning and that should get me going

or ? or not ?

nope….the Dockerfile needs to be different

mmm not working

[Container] 2020/02/07 00:21:33 Waiting for agent ping
[Container] 2020/02/07 00:21:35 Waiting for DOWNLOAD_SOURCE
[Container] 2020/02/07 00:21:35 Phase is DOWNLOAD_SOURCE
[Container] 2020/02/07 00:21:35 CODEBUILD_SRC_DIR=/codebuild/output/src274700738/src
[Container] 2020/02/07 00:21:35 Phase complete: DOWNLOAD_SOURCE State: FAILED
[Container] 2020/02/07 00:21:35 Phase context status code: YAML_FILE_ERROR Message: YAML file does not exist

I wonder if that is atlantis complaining

or just codepipeline

mmm is pointing to master….I need to point it to my branch

I never used codebuild before….

now I have a buildspec.yaml

I got it running…

just having problem with the repo.yaml
2020-01-24

has anybody built a Codefresh pipeline for a Maven project that uses https://github.com/GoogleContainerTools/jib ?
Build container images for your Java applications. - GoogleContainerTools/jib
2020-01-27

I spent some time this weekend getting caught up on the state of the “Jenkins” ecosystem (not Jenkins X)

thanks for sharing, this is awesome

Here’s my braindump: https://cloudposse.com/devops/jenkins-pros-cons-2020/

I think there are many here with more experience. Let me know if I got anything wrong or something else I should add!


@roth.andy

@michal.matyjek @Steve Boardwell

That’s a nice writeup. I found it incredibly painful to develop and debug groovy scripts in Jenkins, bt perhaps it is better as of late. We have been making a push on Concourse, and you’d have to pull me kicking and screaming to go back at this point.

You’re missing the #1 pro that makes Jenkins still the most used tool in its space: price.
It certainly has its barnacles, and I’m not defending it, but it’s pretty damn good for being free.

will add that

I would argue though the price of Jenkins is the effort needed to manage it, solve for plugin compatibility issues etc.etc.

Sure, but once you get that stuff figured out there aren’t really any ongoing costs, vs a subscription

Once again I’m not saying Jenkins is the best and everyone should be using it, but it doesn’t deserve all the flak that gets shot at it sometimes.

It’s the result of years of hard work, done for nothing in return

I think the best use case for Jenkins is a small, self-sufficient devops team. Governance and multi tenant is where jenkins starts to fall over hard

cloud resource cost is not free as well.

True, although as Erik pointed out in his post you can take advantage of spot instances to make it very cheap. Jenkins ephemeral workers are particularly well suited to spot instances

Great write-up @Erik Osterman (Cloud Posse) - you’ve covered a lot of ground over one weekend :smile:.
Here are some things off the top of my head that might be worth mentioning:
plugin management
• I haven’t had that many problems re plugins and upgrades but it really depends on how many plugins you use and, as you mentioned, the current maintenance. I have moved towards using command line tools in containers as opposed to installing some of the more exotic plugins.
• although I haven’t tried it myself, there is also a Cloudbees “tried and tested” version of Jenkins (probably limited in plugins but more stable) - https://www.cloudbees.com/products/cloudbees-jenkins-distribution/overview. They have the option to run on kubernetes as well: https://docs.cloudbees.com/docs/cloudbees-jenkins-distribution/latest/jenkins-lts-to-cloudbees-jenkins-distribution/#_kubernetes_deployment
Docker builds in kubernetes:
• another alternative would be to use kaniko - https://github.com/GoogleContainerTools/kaniko
• Makisu also looks promising: https://github.com/uber/makisu
Containers in pod templates:
• Yes, this is a pain, and whilst it’s easy enough to “fix” with JCasC and a Jenkins restart, it’s not easily done on a per build basis unless, as you mentioned, your Jenkinsfile/project contains the yaml.
• To get away from the ugliness of it all (yes, it is ugly :slightly_smiling_face:) you could use the readFile
or readTrusted
steps, but again it’s not that simple.
• Another thing to think about is the potential resource usage when using multiple containers in a single pod as opposed to build your own image with all the necessary tools - each container would have resource required and limits set - a single container would save resources in this respect. I think @roth.andy mentioned this somewhere.

Jenkins “Configuration as Code” plugin supports nearly everything you’d need to run Jenkins itself in a “GitOps” compliant manner
And if it doesn’t there is always the https://github.com/jenkinsci/configuration-as-code-groovy-plugin which allows you to run arbitrary groovy scripts for the bits you need.

Oh, and a small thing - I think the Github Organisation Plugin (as opposed to the workflow-multi-branch plugin) automatically detects projects with Jenkinsfiles. The multi-branch just detects branches, tags, etc, from within a configured repo.

Thanks @Steve Boardwell !! Will update the post tomorrow with your feedback and corrections :-) very much appreciated

Thanks @Steve Boardwell I didn’t know about Makisu! Have you used the buildah/podman
from Fedora?

+1 for podman

isn’t it from Red Hat though?

Yes… well these days Fedora is basically the RedHat inclubator…

@Blaise Pabon no, it’s on the list though.

@Steve Boardwell turns out the github-branch-source
plugin is what I configured for the repository scanning capabilities https://docs.cloudbees.com/docs/admin-resources/latest/plugins/github-branch-source
The GitHub Branch Source plugin allows you to create a new project based on the repository structure from one or more GitHub users or organizations. You can either:

Ah, yes, that was it - not the “Github Organisation Plugin”. I was getting my plugins crossed thinking of the “Pipeline Multibranch Plugin”. We don’t scan our org automatically so haven’t needed to configure for a while. Thanks for the heads up.


My team at Red Hat depends heavily on GitLab CI and we build containers often to run all kinds of tests. Fortunately, GitLab offers up CI to build containers and a container registry in every repository to hold the containers we build. This is really handy because it keeps everything together in one place: your container build scripts, your container build infrastructure, and the registry that holds your containers. Better yet, you can put multiple types of containers underneath a single git repository if you need to build containers based on different Linux distributions.

While I am still supprting Jenkins and groovy scripts, I am in transition of moving a lot of stuff over on Drone CI (and yeah its free as well)