Anyone have experience with canary deploys for apps that do not have an LB and that they join a cluster and receive traffic trough a queue instead of an LB?
Can you give some more context? Are you using Kubernetes with Prometheus? ;)
the instances join a cluster and elect a leader and then the Queue is created
then the instances pull from the queue and when there is an event they start processing
so we could say the queue is like the ALB for the members of the queue
what do you think it could be useful ?
Harness is all about how to automate the non-trivial continuos deployment process. It has connectors for ECS and Cloud Watch. Perhaps it could simplify this process. At the very least, watch a demo of it. It opened my eyes as to what a well-designed CD process process could look like.
Harness makes it easier to implement the kind of business logic you are describing because it’s does one thing and only one thing: manage deployments.
cool, I will
we were thinking of using codedeploy
but then we started thinking and without duplicating the ingress part for the request and creating a new queue was not possible
well if it’s a queue then you’d be polling to pull messages off it - so you could just scale your canary’s poll rate to reduce the amount of “traffic” it’s receiving yeh?
from there it’s just regular canary management - monitor errors and roll back if you see something unexpected
correct we can have the same queue or another queue and change the thread count
that’s the right algorithm, but how is it orchestrated?
this is in TF+ ECS+EC2
and a go app to deploy the task def and image to ECR
code deploy is pretty limited from my experience in dealing with a multi stage deploy
sadly to change the Thread count it need restart
so we will have to redeploy to increase it
which is ok
I like CodeDeploy and the recent version has some good features
code deploy is fine for deploying - you might need something higher level to orchestrate the canary process though
AWS CodeDeploy has extended blue/green deployment support for Elastic Container Service (Amazon ECS) to include canary and linear deployments for applications hosted on AWS Fargate or Amazon Compute Cloud (Amazon EC2). Blue/green deployments are a safe deployment strategy provided by AWS CodeDeploy for minimizing interruptions caused by changing application versions. This is accomplished by creating […]
hadn’t seen that
that is why I said that the old CodeDeploy did not have much features
that is a very recent post too
Hi there, Watching the video below I wonder what are the key differentiators to GitHub Actions? https://blog.github.com/2018-10-17-action-demos/#hashicorp-terraform-simplified-pull-request-review-w…
that is and interesting read
Adding @ bot
@ has joined the channel
I’m tasked as objective this year to improve consistency in our terraform releases. I’ve done lots of various research and testing and am currently looking the following main approaches I’ve evaluating. I think it would be useful to dialogue on this with ya’ll and save myself some repeat work. I’m going to do more detail in a thread. Please comment in there
- Terraform Cloud
- Azure DevOps Pipelines
- Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in.
Not a comprehensive list but this is the main summary I currently am thinking
Background: Currently we use TFS 2018 for internal stuff, but most of the jobs are being built in Jenkins, but only now are they starting to do more multistage builds (first one). I use Azure DevOps + Terraform Cloud for tasks.
** KEY REQUIREMENTS **
- Manage continuing number of workspaces with regular CICD
- Promote remote runs rather than from laptop to centralize the team.
- Promote CICD to QA
- Promote CD with manual approval of changes noted
- Reduce the overall complexity to get up and running for a team member. If recommended something say that required us all to learn and self-host Kubernetes just to have agents… that would block us. I deal with 80% operational focused folks that have terraform/scripting knowledge, but are not more disciplined developers. I need to make this more approachable.
- Terraform Cloud: The easiest way I see to get a centralized management of all terraform plans with continual integration and deployment triggered and queued for approval.
- Jenkins: 2-3 team members have experience with it. Well used in industry. Can apply pipeline as code approach with work, and multistage approvals.
- Azure DevOps Pipelines: Our MSDN licenses cover usage for most of my team already (so free), and we have free parallel hosted agents for 2000 mins + due to enterprise licenses. Intuitive to me and supports both powershell + bash without complexity. Yaml based pipeline is better than learning groovy to me. Big fan overall.
- Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in: Would be tough sell. Gitlab seems possible, but cost for enterprise is very high. No msdn benefits, and heard it’s not as extensible with plugins or visual gui for pipelines. Not sure the pipelines would gain me much over AzureDevOps
- Github actions: As enterprise user, this is also promising. However, not sure it gains us anything really over using azure devops pipelines as it’s less featured at this time and not in centralize pipeline management location.
- Terraform Cloud: Current dashboard/centralized management could be really cluttered like CF stacks without using Terraform to manage and keep organized. Not able to leverage more advanced pipeline steps with powershell/bash wrapping commands.
- Jenkins: Not a fan of more self-hosted, and to me (as newbie with it), feels far inferior as a deployment pipeline management tool compared to Azure DevOps. Requires groovy instead of just using YAML pipelines. More infra management vs using Microsoft Hosted containers and more docker.
- Azure DevOps Pipelines: New tech to most of team, as I’ve advocated for it but not had any takers for a while. Not sure what other major cons it has, as I’ve run into very few overall problems with it.
- Others (harder to sell) could be Gitlab/CodeFresh or another if I got buy in. They have their own perks of course
- Jenkins update.
I put in more infra management vs hosted service. Big deal to me for sure
Updated release tooling with key requirements. I would love insight into everyone’s thoughts based on your experience with the various solutions so I can summarize my findings later in a blog post. As a big fan of Azure DevOps I want ensure I’m not biased in recommending it and have more community insight. Appreciate it!
Also have this post on Jenkins Pros/Cons https://cloudposse.com/devops/jenkins-pros-cons-2020/
We don’t have first hand experience with it, but the vertical integration looks pretty nice (if that’s valued) and have heard only good things from others using it.
having done both groovy based jenkins pipelines and an extensive amount of azure devops pipelines I see them both about the same. Big difference worth noting is that the release pipeline as code for ADO (aka. multi-stage pipelines) is still rather new and does not have parity with its web console ‘legacy’ release pipeline counterpart.
ADO is also has some inherent issues which can be overcome with either clever hacks or by using other services in tandem
for instance, their artifacts solution is total garbage
unless you are pure maven and/or npm (and even then, you are limited to caching upstream dependencies to maven central)
plus, Azure devops simply does not accommodate for various permission requirements well. Ultimately you will find you are forced to create a per-user PAT for a pipeline which is particularly crappy
it will work, don’t get me wrong, but I find it to be ultimately unsupportable.
also keep in mind that even MS employees will tell you that they push for new ADO users to use github over the baked in vsts repos included with ADO
I get why but it feels a bit skeevy
my 2 cents.
Very nice. We use github repos. The yaml features should be at parity i think now, i think all plugins work in both. The main difference in yaml = one pipeline. There is no seperate release + build pipelines now.
I would like your experience on jenkins compared to yaml pipelines then if you can add some notes on this. Assume only 2-3 people know jenkins. The rest know bash or powershell. Assuming that would you go with azure pipelines over self-managed jenkins or would you put everyone on jenkins?
I really like using ADO yaml in a central repo of pipeline code and using it across multiple repositories to standardize my work. It helped me wrangle some 70 repos into a cohesive state and completely chuck the legacy pipeline manual garbage out the window.
going one step further, to automate the creation and management of the pipelines themselves I was able to craft nifty solutions using Jenkins Job builder and a nominal amount of python that built several dozen pipelines from some yaml.
it was beautiful… to do the same in ADO, well you aren’t out of luck but have to do some preview az cli work or use preview features of the devops API
oh, in Azure stuff stays in preview like forever…
Good point. I have automatic pipeline creation with terraform cloud down pretty well, and also managing github. I never even thought about looking for an azuredevops provider.
I haven’t had the need to automate pipeline creation yet, but that makes since to avoid issues for sure.
the ADO provider is super beta, if MS paid me I’d help them finish the thing
ha! Ok, i will check it out sometime too. Thanks for the great tips.
sure, I’ve been on both sides of this particular fence, glad to be opinionated
Ok as far as I’ve seen Scalr is the closest I’ve seen in comparison to Terraform cloud.
I’m think a dedicated terraform workflow system is going to be a better fit than a general release tool in the long term.
I think I’ve mentioned this one before but I’m constantly using my own wrapper app for cloudposse/packages to download and install new versions of github releases based apps. I’ve added in an ‘auto’ task recently that detects pretty much all aspects of an app release, creates a packages app, then runs it. https://github.com/zloeber/ghr-installer (I’ve even set it up with an alias locally called githubapp so I can run it from anywhere on my workstation to quickly grab github releases.
Github Releases Installer. Contribute to zloeber/ghr-installer development by creating an account on GitHub.
Hi do you use one ECR for Dev/test/ prod environments or separe repositories? Id like to do image promotion from dev to test to prod but I’m not sure if this is a best practice or are there any pitfalls
Non-Prod and Prod services should be hosted on separate AWS accounts, and thus separate ECRs
We have separe prod and non prod accounts. We use separate aws account that only has ECR. When we ready to deploy test image to prod we just change the tag in prod deployment
You can accomplish this by proper releasing…
I’m personally a fan of a single source of truth ECR repo and images get tagged etc as they get promoted through environments…
We have 2 repos. 1 for master builds, that is only for dev testing, and 1 for release candidates. The release candidates repos are in a single account that all other accounts can access
I did not know this was possible. You can set up cross-region/cross-account ECR access?
@joshmyers could you tell me how you tag your images? We also need a proper tagging strategy
Well technically you can do cross region. But usually you’ll want a repo in each region. Our build process for release candidates pushes the images to repos in every region where we have services
Our tags reflect tags from git. This way developers have more control over the tagging scheme
Decentralizing your container repos means you veer from the build one time only devops mantra. I’ve done this in my recent past due to there being a lack of centralized infrastructure in the environment (no one wanted to own it from a billing perspective). I survived the ordeal with clever pipeline as code until we finally settled on a single subscription for a container registry but I’ll never do that again if I can help it….
just my 2 cents
we tag with short hash of the commit that built the container
i think this is fine for most scenario for un published containers
semver is often an unneeded overhead
use a single repo. single artifact that is promoted. technically if you have multiple ecrs you have multiple artifacts
We’re excited to announce that npm will be joining GitHub.
Any lightweight release tools that you guys use? I’m guessing if we already use Jira then the best method would be to make sure we’re utilizing fields properly. What about change requests, e-mails, approvals, etc..? Making sure people own what they’re doing.
Hey everyone, I created a GitHub application that will check Terraform Formatting (0.12) and allows you to fix any issues by clicking a button in the GitHub UI. My goal was to not have to set this up on multiple repos and avoid the manual work of having to fix formatting when something slips through the cracks. In case this might help you as well, check it out: https://github.com/apps/terraform-format-0-12
GitHub is where people build software. More than 40 million people use GitHub to discover, fork, and contribute to over 100 million projects.