#release-engineering (2020-01)

jenkins_ci All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2020-01-30

2020-01-29

2020-01-28

2020-01-27

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I spent some time this weekend getting caught up on the state of the “Jenkins” ecosystem (not Jenkins X)

michal.matyjek avatar
michal.matyjek

thanks for sharing, this is awesome

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
:--1:3
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

I think there are many here with more experience. Let me know if I got anything wrong or something else I should add!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@roth.andy

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@michal.matyjek @Steve Boardwell

Joe Hosteny avatar
Joe Hosteny

That’s a nice writeup. I found it incredibly painful to develop and debug groovy scripts in Jenkins, bt perhaps it is better as of late. We have been making a push on Concourse, and you’d have to pull me kicking and screaming to go back at this point.

roth.andy avatar
roth.andy

You’re missing the #1 pro that makes Jenkins still the most used tool in its space: price.

It certainly has its barnacles, and I’m not defending it, but it’s pretty damn good for being free.

:100:3
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

will add that

michal.matyjek avatar
michal.matyjek

I would argue though the price of Jenkins is the effort needed to manage it, solve for plugin compatibility issues etc.etc.

roth.andy avatar
roth.andy

Sure, but once you get that stuff figured out there aren’t really any ongoing costs, vs a subscription

roth.andy avatar
roth.andy

Once again I’m not saying Jenkins is the best and everyone should be using it, but it doesn’t deserve all the flak that gets shot at it sometimes.

roth.andy avatar
roth.andy

It’s the result of years of hard work, done for nothing in return

roth.andy avatar
roth.andy

I think the best use case for Jenkins is a small, self-sufficient devops team. Governance and multi tenant is where jenkins starts to fall over hard

:--1:1
btai avatar

cloud resource cost is not free as well.

roth.andy avatar
roth.andy

True, although as Erik pointed out in his post you can take advantage of spot instances to make it very cheap. Jenkins ephemeral workers are particularly well suited to spot instances

:--1:1
Steve Boardwell avatar
Steve Boardwell

Great write-up @Erik Osterman (Cloud Posse) - you’ve covered a lot of ground over one weekend :smile:.

Here are some things off the top of my head that might be worth mentioning:

plugin management

• I haven’t had that many problems re plugins and upgrades but it really depends on how many plugins you use and, as you mentioned, the current maintenance. I have moved towards using command line tools in containers as opposed to installing some of the more exotic plugins.

• although I haven’t tried it myself, there is also a Cloudbees “tried and tested” version of Jenkins (probably limited in plugins but more stable) - https://www.cloudbees.com/products/cloudbees-jenkins-distribution/overview. They have the option to run on kubernetes as well: https://docs.cloudbees.com/docs/cloudbees-jenkins-distribution/latest/jenkins-lts-to-cloudbees-jenkins-distribution/#_kubernetes_deployment Docker builds in kubernetes:

• another alternative would be to use kaniko - https://github.com/GoogleContainerTools/kaniko

• Makisu also looks promising: https://github.com/uber/makisu Containers in pod templates:

• Yes, this is a pain, and whilst it’s easy enough to “fix” with JCasC and a Jenkins restart, it’s not easily done on a per build basis unless, as you mentioned, your Jenkinsfile/project contains the yaml.

• To get away from the ugliness of it all (yes, it is ugly :slightly_smiling_face:) you could use the readFile or readTrusted steps, but again it’s not that simple.

• Another thing to think about is the potential resource usage when using multiple containers in a single pod as opposed to build your own image with all the necessary tools - each container would have resource required and limits set - a single container would save resources in this respect. I think @roth.andy mentioned this somewhere.

Steve Boardwell avatar
Steve Boardwell


Jenkins “Configuration as Code” plugin supports nearly everything you’d need to run Jenkins itself in a “GitOps” compliant manner
And if it doesn’t there is always the https://github.com/jenkinsci/configuration-as-code-groovy-plugin which allows you to run arbitrary groovy scripts for the bits you need.

Steve Boardwell avatar
Steve Boardwell

Oh, and a small thing - I think the Github Organisation Plugin (as opposed to the workflow-multi-branch plugin) automatically detects projects with Jenkinsfiles. The multi-branch just detects branches, tags, etc, from within a configured repo.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Thanks @Steve Boardwell !! Will update the post tomorrow with your feedback and corrections :-) very much appreciated

Blaise Pabon avatar
Blaise Pabon

Thanks @Steve Boardwell I didn’t know about Makisu! Have you used the buildah/podman from Fedora?

roth.andy avatar
roth.andy

+1 for podman

roth.andy avatar
roth.andy

isn’t it from Red Hat though?

Blaise Pabon avatar
Blaise Pabon

Yes… well these days Fedora is basically the RedHat inclubator…

Steve Boardwell avatar
Steve Boardwell

@Blaise Pabon no, it’s on the list though.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Steve Boardwell turns out the github-branch-source plugin is what I configured for the repository scanning capabilities https://docs.cloudbees.com/docs/admin-resources/latest/plugins/github-branch-source

GitHub Branch Source plugin | CloudBees Docs

The GitHub Branch Source plugin allows you to create a new project based on the repository structure from one or more GitHub users or organizations. You can either:

Steve Boardwell avatar
Steve Boardwell

Ah, yes, that was it - not the “Github Organisation Plugin”. I was getting my plugins crossed thinking of the “Pipeline Multibranch Plugin”. We don’t scan our org automatically so haven’t needed to configure for a while. Thanks for the heads up.

Blaise Pabon avatar
Blaise Pabon
Build containers in GitLab CI with buildah :: major.io — attachment image

My team at Red Hat depends heavily on GitLab CI and we build containers often to run all kinds of tests. Fortunately, GitLab offers up CI to build containers and a container registry in every repository to hold the containers we build. This is really handy because it keeps everything together in one place: your container build scripts, your container build infrastructure, and the registry that holds your containers. Better yet, you can put multiple types of containers underneath a single git repository if you need to build containers based on different Linux distributions.

1
1
grv avatar

While I am still supprting Jenkins and groovy scripts, I am in transition of moving a lot of stuff over on Drone CI (and yeah its free as well)

2020-01-24

Darren Cunningham avatar
Darren Cunningham

has anybody built a Codefresh pipeline for a Maven project that uses https://github.com/GoogleContainerTools/jib ?

GoogleContainerTools/jib

Build container images for your Java applications. - GoogleContainerTools/jib

2020-01-23

PePe avatar

so I’m using : https://github.com/cloudposse/terraform-aws-ecs-atlantis and I’m going trough the setup but

cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

PePe avatar

in :

Provision the module with Terraform. Terraform GitHub Provider will use the github_webhooks_token to create webhooks on the repo
cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

PePe avatar

as I understand it I need to provision the module meaning provided the necessary input variables to be able to run apply, correct ?

Adrian avatar
Adrian

correct

PePe avatar

thanks

PePe avatar

mmm I’m getting this error :

PePe avatar
on [main.tf](http://main.tf) line 95, in module "atlantis":
  95:   atlantis_repo_whitelist    = [var.atlantis_repo_whitelist]

The given value is not suitable for child module variable
"atlantis_repo_whitelist" defined at ../../variables.tf:123,1-35: element 0:
string required.
PePe avatar

I used something like :atlantis_repo_whitelist = ["cloudposse/terraform-aws-ecs-atlantis"]

PePe avatar

that variable is set to list(string)

PePe avatar
cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

PePe avatar

so this should work

PePe avatar

I know why

PePe avatar

the example is incorrect

PePe avatar

if you add * it doesnt like it somehow

aknysh avatar
aknysh
cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

aknysh avatar
aknysh

is always correct since it’s auto-tested on AWS with terratest

aknysh avatar
aknysh

at least it ran OK the last time a PR was opened for the repo

PePe avatar
cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

PePe avatar

they are different

aknysh avatar
aknysh

different how?

aknysh avatar
aknysh

the example itself defaults it to empty list

aknysh avatar
aknysh

and we provide the concrete value in tfvars

PePe avatar

the default is different :

PePe avatar
variable "atlantis_repo_whitelist" {
  type        = list(string)
  description = "Whitelist of repositories Atlantis will accept webhooks from"
  default     = ["<http://github.com/example/*|github.com/example/*>"]
}
PePe avatar

that is from the cognito example

PePe avatar

but in the cognito example there is no .tfvars, just defaults

PePe avatar

if I’m not mistaken, is just the cognito-example folder what have the problem

PePe avatar

I switched to the complete example and it works

PePe avatar

mmmm the provisioning of the module should work for private repos too , right ?

PePe avatar
Error: POST <https://api.github.com/repos/company/terraform-rds/hooks>: 404 Not Found []

  on .terraform/modules/atlantis.ecs_web_app.ecs_codepipeline.github_webhooks/main.tf line 6, in resource "github_repository_webhook" "default":
   6: resource "github_repository_webhook" "default" {
aknysh avatar
aknysh

The user you create the token for, must be an admin in the repo to be able to create webhooks

PePe avatar

I added to a team that is admin

aknysh avatar
aknysh

That’s the dropdown with Read, Write and Admin values

PePe avatar

exactly

PePe avatar

I just added it as a single collaborator with admin and does not work

PePe avatar

I forgot to add the scope!!!!

PePe avatar

admin:repo_hook

PePe avatar

another issue

PePe avatar

the webhooks are created in the repo

PePe avatar

but every time the apply happens wants to create the webhooks again

PePe avatar

so I’m guessing webhook_enabled = true should be set to false after initial bootstrap ?

PePe avatar

I wonder how this will work for multiple repos

PePe avatar

I think I figure out that part too no

PePe avatar
cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

PePe avatar

so the repo where the webhook were created should have a dockerfile ?

PePe avatar

or should I point it to : github.com/cloudposse/atlantis

PePe avatar

ahhh I’m so stupid

PePe avatar

that will have to be the and ECR repo image or docker registry of atlantis

PePe avatar

mmmm that does not seems to work

PePe avatar
[31mError: --gh-user/--gh-token or --gitlab-user/--gitlab-token or --bitbucket-user/--bitbucket-token must be set[39m
aknysh avatar
aknysh

if the example in https://github.com/cloudposse/terraform-aws-ecs-atlantis/tree/master/examples/complete did not help, this is how we deployed atlantis in our test account (it’s working now) https://github.com/cloudposse/testing.cloudposse.co/tree/master/conf/ecs

cloudposse/terraform-aws-ecs-atlantis

Terraform module for deploying Atlantis as an ECS Task - cloudposse/terraform-aws-ecs-atlantis

cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

aknysh avatar
aknysh
cloudposse/terraform-root-modules

Example Terraform service catalog of “root module” blueprints for provisioning reference architectures - cloudposse/terraform-root-modules

PePe avatar

mmm

PePe avatar

the environment variables of the ECS task are missing

PePe avatar

I mean , the only one that is created is ATLANTIS_ENABLED

PePe avatar

but atlantis is expecting other ENV vars

PePe avatar

I have no clue

PePe avatar

I can;t use default-backend as an image_url for the ECS task since that thing is the pretty 404 page

PePe avatar

so I changed it to cloudposse/atlantis:latest

PePe avatar

and now atlantis does not start

PePe avatar

but I’m confused about how codepipeline/codebuild work since I never used them before

PePe avatar

that is why I don’t know if my repo have to have a Docker file to build atlantis and then that is build and upload to ECR and then run, or atlantis runs all the time and then it pulls the repo and run TF etc

PePe avatar

need to read the docs more

PePe avatar
Deployment | Atlantis

Atlantis: Terraform Pull Request Automation

PePe avatar

altantis tuns all the time and recieve the webhook requests

PePe avatar

so it needs to be up and running to do so

PePe avatar
terraform-aws-modules/terraform-aws-atlantis

Terraform configurations for running Atlantis on AWS Fargate. Github, Gitlab and BitBucket are supported. - terraform-aws-modules/terraform-aws-atlantis

PePe avatar

which I think I’m missing somehow

PePe avatar

do I have to have a Docker file like this in eve repo that I run atlantis to run ? https://github.com/cloudposse/atlantis/blob/master/Dockerfile

cloudposse/atlantis

GitOps for Teams (experimental hard fork of atlantis) - cloudposse/atlantis

PePe avatar

so Codepipeline can build the image and push it to ECS?

PePe avatar

is that the idea?

aknysh avatar
aknysh

as we did it, we install atlantis in Docker file for each environment we use

aknysh avatar
aknysh
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

aknysh avatar
aknysh

so the CodePipeline builds the image on merging to master branch

aknysh avatar
aknysh

and saves it to ECR

aknysh avatar
aknysh

and then deploys it to ECS

aknysh avatar
aknysh

so yes, in our case, we install atlantis in the same repo on which atlantis operates after it gets installed

aknysh avatar
aknysh

but you can choose any other solution that fits your needs

aknysh avatar
aknysh

in our case, atlantis is deployed from the repo, and then operates on the same repo using the config file https://github.com/cloudposse/testing.cloudposse.co/blob/master/atlantis.yaml

cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

aknysh avatar
aknysh

(we did it to minimize the number of repos to create and maintain. You can have atlantis defined in one repo, but operating on a different infra repo. But in this case, the number of repos doubles)

aknysh avatar
aknysh

(the confusion here is because we merged two diff concepts together to minimize the number of repos: GitOps for atlantis itself (how to deploy it into each environment), and GitOps for the resources that atlantis controls)

aknysh avatar
aknysh

if you use atlantis itself from a different repo (not on which it operates), this file becomes very important from security point of view

aknysh avatar
aknysh
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

aknysh avatar
aknysh

id: github.com/cloudposse/testing.cloudposse.co gives atlantis permissions to the specified repo only

aknysh avatar
aknysh
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

aknysh avatar
aknysh

since all Dockerfiles have geodesic as base, atlantis.sh gets executed

aknysh avatar
aknysh
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

aknysh avatar
aknysh

@PePe you were asking about missing vars

aknysh avatar
aknysh

(who said the whole solution would be easy )

PePe avatar

hahaha

PePe avatar

now the missing pieces are surfacing…

PePe avatar

now I understand a bit more

PePe avatar

in our case we have several repos that build one environment, where we do have a ECS cluster

PePe avatar

and the many services inside are build from a different TF repo

PePe avatar

so I was thinking to deploy atlantis as a service that is part of the same ECS cluster

PePe avatar

and trigger the builds trough the webhooks

aknysh avatar
aknysh

yes, so you prob will have to do some digging since our stuff was made to work with geodesic which has lots of things for atlantis

PePe avatar

and the idea behind this is to minimize the “access” of atlantis and atlantis apply to that specific aws account/cluster

aknysh avatar
aknysh

if that’s a question, then yes

aknysh avatar
aknysh

and to minimize the number of repos

aknysh avatar
aknysh

if you have 7 environments, you would have 14 repos (7 for atlantis) if you deployed atlantis from separate repos

PePe avatar

ok, that confuses me a bit

aknysh avatar
aknysh

that’s why we put atlantis in the Dockerfile for the the same repo on which it operates in the same account

PePe avatar

I though atlantis will clone and run any repo that hits the public endpoint for the webhook, right ?

aknysh avatar
aknysh

it can

aknysh avatar
aknysh

but you need to deploy atlantis itself

aknysh avatar
aknysh

preferably with GitOps

PePe avatar

imagine that I gave atlantis running already

PePe avatar

then I add the webhooks to the repo, give access to the atlantis bot user to the repo then atlantis should be able to run terraform apply from that repo on a github event

aknysh avatar
aknysh

cloudposse/testing.cloudposse.co is the repo that deploys atlantis and the repo on which the deployed atlantis operates

PePe avatar

for that environment

aknysh avatar
aknysh

as you described yes, it will do it

PePe avatar

so one atlantis per environment basically

PePe avatar

no matter how many terraform repos that environment could have?

aknysh avatar
aknysh

yes

aknysh avatar
aknysh

but as I mentioned, you have to deploy atlantis somehow into each env

aknysh avatar
aknysh

that’s when it becomes more complicated

PePe avatar

correct I understand

aknysh avatar
aknysh
cloudposse/testing.cloudposse.co

Example Terraform Reference Architecture that implements a Geodesic Module for an Automated Testing Organization in AWS - cloudposse/testing.cloudposse.co

PePe avatar

and that is why you run atlantis from whiting the TF repo

PePe avatar

to avoid having two each

aknysh avatar
aknysh

haha yes, been trying to say that for 30 mins

PePe avatar

hahahahahaha

aknysh avatar
aknysh

to minimize the number of repos

PePe avatar

and I guess another advantage of running from the sam repo is that you could somewhat customize atlantis for repo specific configs

aknysh avatar
aknysh

Yes

aknysh avatar
aknysh

For example, it uses the same credentials or assume role as you do when executing into that container

aknysh avatar
aknysh

So you specify it only once in one place

PePe avatar

exactly

PePe avatar

thanks so much for the time, sorry for being too slow

aknysh avatar
aknysh

np, that’s not easy solution

PePe avatar

I was wondering yesterday, building atlantis every time could take some time

PePe avatar

do you happen to know if there is a notable difference from having atlantis running all the time?

aknysh avatar
aknysh

CodePipeline will build on merge to master

aknysh avatar
aknysh

and deploy to ECS

aknysh avatar
aknysh

but ECS will switch to the new version only if the deployment is a success

PePe avatar

ok

PePe avatar

and this approach should not be an issue if I want to setup atlantis to build from a branch ?

PePe avatar

not just merge to master

aknysh avatar
aknysh

yes

aknysh avatar
aknysh

we selected master

aknysh avatar
aknysh

in CodePipeline config

PePe avatar

ahhhh right that is Codepipeline not atlantis, ok cool

PePe avatar

so I’m back on working on this :

PePe avatar

I’m creating my Docker file and atlantis-repo-config.yaml

PePe avatar

I was thinking to use this Docket file : https://github.com/cloudposse/atlantis/blob/master/Dockerfile

cloudposse/atlantis

GitOps for Teams (experimental hard fork of atlantis) - cloudposse/atlantis

PePe avatar
cloudposse/geodesic

Geodesic is a cloud automation shell. It&#39;s the fastest way to get up and running with a rock solid, production grade cloud platform built on top of strictly Open Source tools. ★ this repo! h…

PePe avatar
  • some cleaning and that should get me going
PePe avatar

:–1: or :-1: ? or not ?

PePe avatar

nope….the Dockerfile needs to be different

PePe avatar

mmm not working

PePe avatar
[Container] 2020/02/07 00:21:33 Waiting for agent ping
[Container] 2020/02/07 00:21:35 Waiting for DOWNLOAD_SOURCE
[Container] 2020/02/07 00:21:35 Phase is DOWNLOAD_SOURCE
[Container] 2020/02/07 00:21:35 CODEBUILD_SRC_DIR=/codebuild/output/src274700738/src
[Container] 2020/02/07 00:21:35 Phase complete: DOWNLOAD_SOURCE State: FAILED
[Container] 2020/02/07 00:21:35 Phase context status code: YAML_FILE_ERROR Message: YAML file does not exist
PePe avatar

I wonder if that is atlantis complaining

PePe avatar

or just codepipeline

PePe avatar

mmm is pointing to master….I need to point it to my branch

PePe avatar

I never used codebuild before….

PePe avatar

now I have a buildspec.yaml

PePe avatar

I got it running…

PePe avatar

just having problem with the repo.yaml

2020-01-22

2020-01-17

Roderik van der Veer avatar
Roderik van der Veer

We have been using travisci pro for years, and pay a whopping 5k a year for 10 concurrent builds. The cost or amount of builds is not really the issue though. It is the power of the underlying machines. Our builds flows are optimised to the max, but we have for example a 10 minute create-react-app build (webpack) which cannot be sped up in any way, except bigger CPU’s. 2 min on my top spec mbp vs 12 minutes on travis. But they do not have plans with faster machines, so not really an option. How are you all handling these kinds of problems?

Pierre Humberdroz avatar
Pierre Humberdroz

Remote worker’s but this might sadly not be an option for travic ci.

Roderik van der Veer avatar
Roderik van der Veer

via their support found that the premium vm’s are still there

Roderik van der Veer avatar
Roderik van der Veer

will try that first

loren avatar
loren

you could maybe connect your repo to gitlab-ci with a “ci-only” project, and manage your own runners as powerful as you need?

:100:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Drop Travis?

1
Maycon Santos avatar
Maycon Santos

5K year for 10 concurrent is really pricey. I used to run Gitlab’s CI with docker-machine launching on-demand spot instances for less than $50/month for the runners.

:100:1

2020-01-12

Mahesh avatar
Mahesh

any GoCD users here?

2020-01-10

Rob Rose avatar
Rob Rose

Hey does anyone have any examples of production/dev/staging environments integrated with terraform and orchestrated by a Jenkins pipeline?

    keyboard_arrow_up