#release-engineering (2020-03)

also feel free to add to #community-projects

Non-Prod and Prod services should be hosted on separate AWS accounts, and thus separate ECRs

We have separe prod and non prod accounts. We use separate aws account that only has ECR. When we ready to deploy test image to prod we just change the tag in prod deployment

You can accomplish this by proper releasing…

I’m personally a fan of a single source of truth ECR repo and images get tagged etc as they get promoted through environments…

We have 2 repos. 1 for master builds, that is only for dev testing, and 1 for release candidates. The release candidates repos are in a single account that all other accounts can access

I did not know this was possible. You can set up cross-region/cross-account ECR access?

(For ECS)

Yes….

@joshmyers could you tell me how you tag your images? We also need a proper tagging strategy

Well technically you can do cross region. But usually you’ll want a repo in each region. Our build process for release candidates pushes the images to repos in every region where we have services

Our tags reflect tags from git. This way developers have more control over the tagging scheme

Decentralizing your container repos means you veer from the build one time only devops mantra. I’ve done this in my recent past due to there being a lack of centralized infrastructure in the environment (no one wanted to own it from a billing perspective). I survived the ordeal with clever pipeline as code until we finally settled on a single subscription for a container registry but I’ll never do that again if I can help it….

just my 2 cents

we tag with short hash of the commit that built the container

i think this is fine for most scenario for un published containers

semver is often an unneeded overhead

use a single repo. single artifact that is promoted. technically if you have multiple ecrs you have multiple artifacts

Not a comprehensive list but this is the main summary I currently am thinking

Background: Currently we use TFS 2018 for internal stuff, but most of the jobs are being built in Jenkins, but only now are they starting to do more multistage builds (first one). I use Azure DevOps + Terraform Cloud for tasks.

** KEY REQUIREMENTS **

1. Manage continuing number of workspaces with regular CICD
2. Promote remote runs rather than from laptop to centralize the team.
3. Promote CICD to QA
4. Promote CD with manual approval of changes noted
5. Reduce the overall complexity to get up and running for a team member. If recommended something say that required us all to learn and self-host Kubernetes just to have agents… that would block us. I deal with 80% operational focused folks that have terraform/scripting knowledge, but are not more disciplined developers. I need to make this more approachable.

Main Pros

1. Terraform Cloud: The easiest way I see to get a centralized management of all terraform plans with continual integration and deployment triggered and queued for approval.
2. Jenkins: 2-3 team members have experience with it. Well used in industry. Can apply pipeline as code approach with work, and multistage approvals.
3. Azure DevOps Pipelines: Our MSDN licenses cover usage for most of my team already (so free), and we have free parallel hosted agents for 2000 mins + due to enterprise licenses. Intuitive to me and supports both powershell + bash without complexity. Yaml based pipeline is better than learning groovy to me. Big fan overall.
4. Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in: Would be tough sell. Gitlab seems possible, but cost for enterprise is very high. No msdn benefits, and heard it’s not as extensible with plugins or visual gui for pipelines. Not sure the pipelines would gain me much over AzureDevOps
5. Github actions: As enterprise user, this is also promising. However, not sure it gains us anything really over using azure devops pipelines as it’s less featured at this time and not in centralize pipeline management location.

Cons

1. Terraform Cloud: Current dashboard/centralized management could be really cluttered like CF stacks without using Terraform to manage and keep organized. Not able to leverage more advanced pipeline steps with powershell/bash wrapping commands.
2. Jenkins: Not a fan of more self-hosted, and to me (as newbie with it), feels far inferior as a deployment pipeline management tool compared to Azure DevOps. Requires groovy instead of just using YAML pipelines. More infra management vs using Microsoft Hosted containers and more docker.
3. Azure DevOps Pipelines: New tech to most of team, as I’ve advocated for it but not had any takers for a while. Not sure what other major cons it has, as I’ve run into very few overall problems with it.
4. Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in. They have their own perks of course

Cons.

1. Jenkins update.

I put in more infra management vs hosted service. Big deal to me for sure

Updated release tooling with key requirements. I would love insight into everyone’s thoughts based on your experience with the various solutions so I can summarize my findings later in a blog post. As a big fan of Azure DevOps I want ensure I’m not biased in recommending it and have more community insight. Appreciate it!

Also have this post on Jenkins Pros/Cons https://cloudposse.com/devops/jenkins-pros-cons-2020/

We don’t have first hand experience with it, but the vertical integration looks pretty nice (if that’s valued) and have heard only good things from others using it.

having done both groovy based jenkins pipelines and an extensive amount of azure devops pipelines I see them both about the same. Big difference worth noting is that the release pipeline as code for ADO (aka. multi-stage pipelines) is still rather new and does not have parity with its web console ‘legacy’ release pipeline counterpart.

ADO is also has some inherent issues which can be overcome with either clever hacks or by using other services in tandem

for instance, their artifacts solution is total garbage

IMHO

unless you are pure maven and/or npm (and even then, you are limited to caching upstream dependencies to maven central)

plus, Azure devops simply does not accommodate for various permission requirements well. Ultimately you will find you are forced to create a per-user PAT for a pipeline which is particularly crappy

it will work, don’t get me wrong, but I find it to be ultimately unsupportable.

also keep in mind that even MS employees will tell you that they push for new ADO users to use github over the baked in vsts repos included with ADO

I get why but it feels a bit skeevy

my 2 cents.

Very nice. We use github repos. The yaml features should be at parity i think now, i think all plugins work in both. The main difference in yaml = one pipeline. There is no seperate release + build pipelines now.

I would like your experience on jenkins compared to yaml pipelines then if you can add some notes on this. Assume only 2-3 people know jenkins. The rest know bash or powershell. Assuming that would you go with azure pipelines over self-managed jenkins or would you put everyone on jenkins?

I really like using ADO yaml in a central repo of pipeline code and using it across multiple repositories to standardize my work. It helped me wrangle some 70 repos into a cohesive state and completely chuck the legacy pipeline manual garbage out the window.

going one step further, to automate the creation and management of the pipelines themselves I was able to craft nifty solutions using Jenkins Job builder and a nominal amount of python that built several dozen pipelines from some yaml.

it was beautiful… to do the same in ADO, well you aren’t out of luck but have to do some preview az cli work or use preview features of the devops API

oh, in Azure stuff stays in preview like forever…

Good point. I have automatic pipeline creation with terraform cloud down pretty well, and also managing github. I never even thought about looking for an azuredevops provider.

I haven’t had the need to automate pipeline creation yet, but that makes since to avoid issues for sure.

the ADO provider is super beta, if MS paid me I’d help them finish the thing

ha! Ok, i will check it out sometime too. Thanks for the great tips.

sure, I’ve been on both sides of this particular fence, glad to be opinionated

Ok as far as I’ve seen Scalr is the closest I’ve seen in comparison to Terraform cloud.

I’m think a dedicated terraform workflow system is going to be a better fit than a general release tool in the long term.

Update this this … Rethinking perspective…. Adopting Azure DevOps this next week and considering impact. No one else uses pipelines right now. Trying to think of what I could do to make things accessible and convince of benefit to less Jenkins and more azdos .

Here’s my cross post from another community . Thinking it might be better to deal with working through azdos templates and all rather than terraform cloud because then the other team can continue to use make files and all, others can use PowerShell, etc but it’s all still in common running location.

Here’s longer content on this. Posting to main room too so it’s visible.

I’ve been using terraform cloud, so I’ve been interested in using multi-stage pipelines in Azure DevOps. One of the challenges is because we’re using GitHub it’s not an auto discovered pipeline so it’s another step to set up.

So I’m curious if anyone has set up terraform template steps that provide back to a GitHub pool request a comment showing the prospective plan directly in line kind of like Atlantis. I know you can go see the plan artifact, I was thinking of what would be a great spelling point for using multi-state pipelines with approval and slack for example. One great feature is that the bill could batch request so I don’t go incrementally create through five different plans if several changed were done quickly.

I’m trying to set up my team for better success with a consistent pipeline since there’s a mix of Jenkins and terraform cloud. This next week we’re going to be flipping from TFS to Azure DevOps, it was considering if I could make things a lot easier by azdos

I think what you are looking for is something like this:

    - terraform plan -out=$PLAN
    - echo \`\`\`diff > plan.txt
    - terraform show -no-color ${PLAN} | tee -a plan.txt
    - echo \`\`\` >> plan.txt
    - sed -i -e 's/  +/+/g' plan.txt
    - sed -i -e 's/  ~/~/g' plan.txt
    - sed -i -e 's/  -/-/g' plan.txt
    - MESSAGE=$(cat plan.txt)
    - >-
      curl -X POST -g -H "PRIVATE-TOKEN: ${GITLAB_TOKEN}"
      --data-urlencode "body=${MESSAGE}"
      "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/discussions"

here is how it looks: https://gitlab.com/gitops-demo/infra/aws/-/merge_requests/6

Can you give some more context? Are you using Kubernetes with Prometheus? ;)

ECS

ECS+EC2

the instances join a cluster and elect a leader and then the Queue is created

then the instances pull from the queue and when there is an event they start processing

so we could say the queue is like the ALB for the members of the queue

Have you seen Harness.io?

no…

what do you think it could be useful ?

Harness is all about how to automate the non-trivial continuos deployment process. It has connectors for ECS and Cloud Watch. Perhaps it could simplify this process. At the very least, watch a demo of it. It opened my eyes as to what a well-designed CD process process could look like.

Harness makes it easier to implement the kind of business logic you are describing because it’s does one thing and only one thing: manage deployments.

cool, I will

we were thinking of using codedeploy

but then we started thinking and without duplicating the ingress part for the request and creating a new queue was not possible

well if it’s a queue then you’d be polling to pull messages off it - so you could just scale your canary’s poll rate to reduce the amount of “traffic” it’s receiving yeh?

from there it’s just regular canary management - monitor errors and roll back if you see something unexpected

correct we can have the same queue or another queue and change the thread count

that’s the right algorithm, but how is it orchestrated?

this is in TF+ ECS+EC2

and a go app to deploy the task def and image to ECR

code deploy is pretty limited from my experience in dealing with a multi stage deploy

sadly to change the Thread count it need restart

so we will have to redeploy to increase it

which is ok

I like CodeDeploy and the recent version has some good features

code deploy is fine for deploying - you might need something higher level to orchestrate the canary process though

although https://aws.amazon.com/blogs/containers/aws-codedeploy-now-supports-linear-and-canary-deployments-for-amazon-ecs/

hadn’t seen that

that is why I said that the old CodeDeploy did not have much features

that is a very recent post too

https://github.com/runatlantis/atlantis/issues/384

that is and interesting read