#release-engineering (2020-03)

jenkins_ci All things CI/CD. Specific emphasis on Codefresh and CodeBuild with CodePipeline.

CI/CD Discussions

Archive: https://archive.sweetops.com/release-engineering/

2020-03-31

PePe avatar

Anyone have experience with canary deploys for apps that do not have an LB and that they join a cluster and receive traffic trough a queue instead of an LB?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Can you give some more context? Are you using Kubernetes with Prometheus? ;)

PePe avatar

ECS

PePe avatar

ECS+EC2

PePe avatar

the instances join a cluster and elect a leader and then the Queue is created

PePe avatar

then the instances pull from the queue and when there is an event they start processing

PePe avatar

so we could say the queue is like the ALB for the members of the queue

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Have you seen Harness.io?

PePe avatar

no…

PePe avatar

what do you think it could be useful ?

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Harness is all about how to automate the non-trivial continuos deployment process. It has connectors for ECS and Cloud Watch. Perhaps it could simplify this process. At the very least, watch a demo of it. It opened my eyes as to what a well-designed CD process process could look like.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Harness makes it easier to implement the kind of business logic you are describing because it’s does one thing and only one thing: manage deployments.

PePe avatar

cool, I will

PePe avatar

we were thinking of using codedeploy

PePe avatar

but then we started thinking and without duplicating the ingress part for the request and creating a new queue was not possible

Chris Fowles avatar
Chris Fowles

well if it’s a queue then you’d be polling to pull messages off it - so you could just scale your canary’s poll rate to reduce the amount of “traffic” it’s receiving yeh?

Chris Fowles avatar
Chris Fowles

from there it’s just regular canary management - monitor errors and roll back if you see something unexpected

PePe avatar

correct we can have the same queue or another queue and change the thread count

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

that’s the right algorithm, but how is it orchestrated?

PePe avatar

this is in TF+ ECS+EC2

PePe avatar

and a go app to deploy the task def and image to ECR

Chris Fowles avatar
Chris Fowles

code deploy is pretty limited from my experience in dealing with a multi stage deploy

PePe avatar

sadly to change the Thread count it need restart

PePe avatar

so we will have to redeploy to increase it

PePe avatar

which is ok

PePe avatar

I like CodeDeploy and the recent version has some good features

Chris Fowles avatar
Chris Fowles

code deploy is fine for deploying - you might need something higher level to orchestrate the canary process though

Chris Fowles avatar
Chris Fowles
AWS CodeDeploy now supports linear and canary deployments for Amazon ECS | Amazon Web Services attachment image

AWS CodeDeploy has extended blue/green deployment support for Elastic Container Service (Amazon ECS) to include canary and linear deployments for applications hosted on AWS Fargate or Amazon Compute Cloud (Amazon EC2). Blue/green deployments are a safe deployment strategy provided by AWS CodeDeploy for minimizing interruptions caused by changing application versions. This is accomplished by creating […]

Chris Fowles avatar
Chris Fowles

hadn’t seen that

PePe avatar

that is why I said that the old CodeDeploy did not have much features

PePe avatar

that is a very recent post too

PePe avatar

that is and interesting read

2020-03-27

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Adding @ bot

discourse_forum avatar
discourse_forum
10:06:22 PM

@ has joined the channel

2020-03-25

sheldonh avatar
sheldonh

I’m tasked as objective this year to improve consistency in our terraform releases. I’ve done lots of various research and testing and am currently looking the following main approaches I’ve evaluating. I think it would be useful to dialogue on this with ya’ll and save myself some repeat work. I’m going to do more detail in a thread. Please comment in there

  1. Terraform Cloud
  2. Jenkins
  3. Azure DevOps Pipelines
  4. Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in.
sheldonh avatar
sheldonh

Not a comprehensive list but this is the main summary I currently am thinking

Background: Currently we use TFS 2018 for internal stuff, but most of the jobs are being built in Jenkins, but only now are they starting to do more multistage builds (first one). I use Azure DevOps + Terraform Cloud for tasks.

** KEY REQUIREMENTS **

  1. Manage continuing number of workspaces with regular CICD
  2. Promote remote runs rather than from laptop to centralize the team.
  3. Promote CICD to QA
  4. Promote CD with manual approval of changes noted
  5. Reduce the overall complexity to get up and running for a team member. If recommended something say that required us all to learn and self-host Kubernetes just to have agents… that would block us. I deal with 80% operational focused folks that have terraform/scripting knowledge, but are not more disciplined developers. I need to make this more approachable.
sheldonh avatar
sheldonh

Main Pros

  1. Terraform Cloud: The easiest way I see to get a centralized management of all terraform plans with continual integration and deployment triggered and queued for approval.
  2. Jenkins: 2-3 team members have experience with it. Well used in industry. Can apply pipeline as code approach with work, and multistage approvals.
  3. Azure DevOps Pipelines: Our MSDN licenses cover usage for most of my team already (so free), and we have free parallel hosted agents for 2000 mins + due to enterprise licenses. Intuitive to me and supports both powershell + bash without complexity. Yaml based pipeline is better than learning groovy to me. Big fan overall.
  4. Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in: Would be tough sell. Gitlab seems possible, but cost for enterprise is very high. No msdn benefits, and heard it’s not as extensible with plugins or visual gui for pipelines. Not sure the pipelines would gain me much over AzureDevOps
  5. Github actions: As enterprise user, this is also promising. However, not sure it gains us anything really over using azure devops pipelines as it’s less featured at this time and not in centralize pipeline management location.
sheldonh avatar
sheldonh

Cons

  1. Terraform Cloud: Current dashboard/centralized management could be really cluttered like CF stacks without using Terraform to manage and keep organized. Not able to leverage more advanced pipeline steps with powershell/bash wrapping commands.
  2. Jenkins: Not a fan of more self-hosted, and to me (as newbie with it), feels far inferior as a deployment pipeline management tool compared to Azure DevOps. Requires groovy instead of just using YAML pipelines. More infra management vs using Microsoft Hosted containers and more docker.
  3. Azure DevOps Pipelines: New tech to most of team, as I’ve advocated for it but not had any takers for a while. Not sure what other major cons it has, as I’ve run into very few overall problems with it.
  4. Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in. They have their own perks of course
Nikola Velkovski avatar
Nikola Velkovski

Cons.

  1. Jenkins update.
sheldonh avatar
sheldonh

I put in more infra management vs hosted service. Big deal to me for sure

sheldonh avatar
sheldonh

Updated release tooling with key requirements. I would love insight into everyone’s thoughts based on your experience with the various solutions so I can summarize my findings later in a blog post. As a big fan of Azure DevOps I want ensure I’m not biased in recommending it and have more community insight. Appreciate it!

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Also have this post on Jenkins Pros/Cons https://cloudposse.com/devops/jenkins-pros-cons-2020/

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We don’t have first hand experience with it, but the vertical integration looks pretty nice (if that’s valued) and have heard only good things from others using it.

Zachary Loeber avatar
Zachary Loeber

having done both groovy based jenkins pipelines and an extensive amount of azure devops pipelines I see them both about the same. Big difference worth noting is that the release pipeline as code for ADO (aka. multi-stage pipelines) is still rather new and does not have parity with its web console ‘legacy’ release pipeline counterpart.

Zachary Loeber avatar
Zachary Loeber

ADO is also has some inherent issues which can be overcome with either clever hacks or by using other services in tandem

Zachary Loeber avatar
Zachary Loeber

for instance, their artifacts solution is total garbage

Zachary Loeber avatar
Zachary Loeber

IMHO

Zachary Loeber avatar
Zachary Loeber

unless you are pure maven and/or npm (and even then, you are limited to caching upstream dependencies to maven central)

Zachary Loeber avatar
Zachary Loeber

plus, Azure devops simply does not accommodate for various permission requirements well. Ultimately you will find you are forced to create a per-user PAT for a pipeline which is particularly crappy

Zachary Loeber avatar
Zachary Loeber

it will work, don’t get me wrong, but I find it to be ultimately unsupportable.

Zachary Loeber avatar
Zachary Loeber

also keep in mind that even MS employees will tell you that they push for new ADO users to use github over the baked in vsts repos included with ADO

Zachary Loeber avatar
Zachary Loeber

I get why but it feels a bit skeevy

Zachary Loeber avatar
Zachary Loeber

my 2 cents.

sheldonh avatar
sheldonh

Very nice. We use github repos. The yaml features should be at parity i think now, i think all plugins work in both. The main difference in yaml = one pipeline. There is no seperate release + build pipelines now.

sheldonh avatar
sheldonh

I would like your experience on jenkins compared to yaml pipelines then if you can add some notes on this. Assume only 2-3 people know jenkins. The rest know bash or powershell. Assuming that would you go with azure pipelines over self-managed jenkins or would you put everyone on jenkins?

Zachary Loeber avatar
Zachary Loeber

I really like using ADO yaml in a central repo of pipeline code and using it across multiple repositories to standardize my work. It helped me wrangle some 70 repos into a cohesive state and completely chuck the legacy pipeline manual garbage out the window.

Zachary Loeber avatar
Zachary Loeber

going one step further, to automate the creation and management of the pipelines themselves I was able to craft nifty solutions using Jenkins Job builder and a nominal amount of python that built several dozen pipelines from some yaml.

Zachary Loeber avatar
Zachary Loeber

it was beautiful… to do the same in ADO, well you aren’t out of luck but have to do some preview az cli work or use preview features of the devops API

Zachary Loeber avatar
Zachary Loeber

oh, in Azure stuff stays in preview like forever…

sheldonh avatar
sheldonh

Good point. I have automatic pipeline creation with terraform cloud down pretty well, and also managing github. I never even thought about looking for an azuredevops provider.

I haven’t had the need to automate pipeline creation yet, but that makes since to avoid issues for sure.

Zachary Loeber avatar
Zachary Loeber

the ADO provider is super beta, if MS paid me I’d help them finish the thing

sheldonh avatar
sheldonh

ha! Ok, i will check it out sometime too. Thanks for the great tips.

Zachary Loeber avatar
Zachary Loeber

sure, I’ve been on both sides of this particular fence, glad to be opinionated

sheldonh avatar
sheldonh

Ok as far as I’ve seen Scalr is the closest I’ve seen in comparison to Terraform cloud.

I’m think a dedicated terraform workflow system is going to be a better fit than a general release tool in the long term.

sheldonh avatar
sheldonh

Update this this … Rethinking perspective…. Adopting Azure DevOps this next week and considering impact. No one else uses pipelines right now. Trying to think of what I could do to make things accessible and convince of benefit to less Jenkins and more azdos .

sheldonh avatar
sheldonh

Here’s my cross post from another community . Thinking it might be better to deal with working through azdos templates and all rather than terraform cloud because then the other team can continue to use make files and all, others can use PowerShell, etc but it’s all still in common running location.

sheldonh avatar
sheldonh

Here’s longer content on this. Posting to main room too so it’s visible.

sheldonh avatar
sheldonh

I’ve been using terraform cloud, so I’ve been interested in using multi-stage pipelines in Azure DevOps. One of the challenges is because we’re using GitHub it’s not an auto discovered pipeline so it’s another step to set up.

So I’m curious if anyone has set up terraform template steps that provide back to a GitHub pool request a comment showing the prospective plan directly in line kind of like Atlantis. I know you can go see the plan artifact, I was thinking of what would be a great spelling point for using multi-state pipelines with approval and slack for example. One great feature is that the bill could batch request so I don’t go incrementally create through five different plans if several changed were done quickly.

I’m trying to set up my team for better success with a consistent pipeline since there’s a mix of Jenkins and terraform cloud. This next week we’re going to be flipping from TFS to Azure DevOps, it was considering if I could make things a lot easier by azdos

Pierre Humberdroz avatar
Pierre Humberdroz

I think what you are looking for is something like this:

    - terraform plan -out=$PLAN
    - echo \`\`\`diff > plan.txt
    - terraform show -no-color ${PLAN} | tee -a plan.txt
    - echo \`\`\` >> plan.txt
    - sed -i -e 's/  +/+/g' plan.txt
    - sed -i -e 's/  ~/~/g' plan.txt
    - sed -i -e 's/  -/-/g' plan.txt
    - MESSAGE=$(cat plan.txt)
    - >-
      curl -X POST -g -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}"
      --data-urlencode "body=${MESSAGE}"
      "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/discussions"

2020-03-21

2020-03-16

Zachary Loeber avatar
Zachary Loeber

I think I’ve mentioned this one before but I’m constantly using my own wrapper app for cloudposse/packages to download and install new versions of github releases based apps. I’ve added in an ‘auto’ task recently that detects pretty much all aspects of an app release, creates a packages app, then runs it. https://github.com/zloeber/ghr-installer (I’ve even set it up with an alias locally called githubapp so I can run it from anywhere on my workstation to quickly grab github releases.

zloeber/ghr-installer

Github Releases Installer. Contribute to zloeber/ghr-installer development by creating an account on GitHub.

Laurynas avatar
Laurynas

Hi do you use one ECR for Dev/test/ prod environments or separe repositories? Id like to do image promotion from dev to test to prod but I’m not sure if this is a best practice or are there any pitfalls

imiltchman avatar
imiltchman

Non-Prod and Prod services should be hosted on separate AWS accounts, and thus separate ECRs

Laurynas avatar
Laurynas

We have separe prod and non prod accounts. We use separate aws account that only has ECR. When we ready to deploy test image to prod we just change the tag in prod deployment

joshmyers avatar
joshmyers

You can accomplish this by proper releasing…

joshmyers avatar
joshmyers

I’m personally a fan of a single source of truth ECR repo and images get tagged etc as they get promoted through environments…

:--1:2
Steven avatar
Steven

We have 2 repos. 1 for master builds, that is only for dev testing, and 1 for release candidates. The release candidates repos are in a single account that all other accounts can access

imiltchman avatar
imiltchman

I did not know this was possible. You can set up cross-region/cross-account ECR access?

imiltchman avatar
imiltchman

(For ECS)

joshmyers avatar
joshmyers

Yes….

Laurynas avatar
Laurynas

@joshmyers could you tell me how you tag your images? We also need a proper tagging strategy

Steven avatar
Steven

Well technically you can do cross region. But usually you’ll want a repo in each region. Our build process for release candidates pushes the images to repos in every region where we have services

Steven avatar
Steven

Our tags reflect tags from git. This way developers have more control over the tagging scheme

Zachary Loeber avatar
Zachary Loeber

Decentralizing your container repos means you veer from the build one time only devops mantra. I’ve done this in my recent past due to there being a lack of centralized infrastructure in the environment (no one wanted to own it from a billing perspective). I survived the ordeal with clever pipeline as code until we finally settled on a single subscription for a container registry but I’ll never do that again if I can help it….

:100:1
Zachary Loeber avatar
Zachary Loeber

just my 2 cents

Chris Fowles avatar
Chris Fowles

we tag with short hash of the commit that built the container

Chris Fowles avatar
Chris Fowles

i think this is fine for most scenario for un published containers

Chris Fowles avatar
Chris Fowles

semver is often an unneeded overhead

Chris Fowles avatar
Chris Fowles

use a single repo. single artifact that is promoted. technically if you have multiple ecrs you have multiple artifacts

:--1:1
imiltchman avatar
imiltchman
npm is joining GitHub - The GitHub Blog attachment image

We’re excited to announce that npm will be joining GitHub.

2020-03-12

MattyB avatar
MattyB

Any lightweight release tools that you guys use? I’m guessing if we already use Jira then the best method would be to make sure we’re utilizing fields properly. What about change requests, e-mails, approvals, etc..? Making sure people own what they’re doing.

2020-03-10

Gabe avatar

Hey everyone, I created a GitHub application that will check Terraform Formatting (0.12) and allows you to fix any issues by clicking a button in the GitHub UI. My goal was to not have to set this up on multiple repos and avoid the manual work of having to fix formatting when something slips through the cracks. In case this might help you as well, check it out: https://github.com/apps/terraform-format-0-12

GitHub

GitHub is where people build software. More than 40 million people use GitHub to discover, fork, and contribute to over 100 million projects.

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also feel free to add to #community-projects

GitHub

GitHub is where people build software. More than 40 million people use GitHub to discover, fork, and contribute to over 100 million projects.

:--1:1
    keyboard_arrow_up