#spacelift (2022-06)
2022-06-01
CI/CD Pipeline : Everything You Need To Know with Examples What makes a good CI/CD pipeline? Take a closer look at the pipeline stages and some examples relating to Terraform.
2022-06-02
Drift Detection and How to Fix It With Different IaC Tools What is drift and why does it happen? See how IaC tools, including Spacelift, can help you with drift detection and remediation.
2022-06-08
Ansible Roles : Basics & How to Combine Them With Playbooks Explore the concept of Ansible roles, their structure, and how we can refactor our playbooks into roles or generate them from scratch.
2022-06-09
When to Run Terraform Apply – Quick Usage Examples Terraform apply is a core command in the Terraform workflow. Learn what this command is used for, what it does, the available options, and when to run it.
2022-06-14
Adventures in AWS Lambda Land – a Migration Gone Well See the approach we’ve taken that allowed us to simplify our setup and clarify common misconceptions you might have about AWS Lambda development.
2022-06-15
AWS IAM Policies : Best Practices & How to Create an IAM Policy Look into AWS IAM policies with some best practices. Learn how they are structured, how to create them, and how to assign necessary permissions.
2022-06-23
9 DevOps Best Practices – What You Should Do and Not-to-Do Explore different DevOps best practices and Anti-Patterns and see how you can incorporate them to accelerate your team performance and value creation.
2022-06-28
How to Use Terraform Join and Split Functions with Strings There are many built-in functions available for use with Terraform. Learn do the join and split functions in Terraform do and see examples.
After long thread with Andriy in atmos, about how to use atmos and considerations how to get it set up with GitLab CI/CD, and after seeing how greatly it plays with Spacelift, thankfully to SweetOps modules, I want to try Spacelift over the next few weeks.
Before that, I need to write-up a decent SPIKE Jira ticket and hopefully get that approved by my manager. I would like to get inputs from those who already have experience working with Spacelift about what excites them and how did it solve some of the day-to-day problems for Infrastructure work. I would kindly appreciate any info you can share - that would be valuable to corroborate my case on why we should adopt Spacelift.
Thank you!
some info here from Cloud Posse https://cloudposse.com/faqs/why-do-you-recommend-spacelift/
@Matt Gowie @Eric Berg
@Alex Jurkiewicz
@azec few things come to mind:
- It using OPA as a policy as code framework gives you endless flexibility with the whole platform. For example, the way your orchestrate approval workflows for your team and infra can be setup exactly how you want it and it’s really not that difficult.
- Being able to share a plan and discuss it as a team is huge for working with large infrastructure footprints.
- The organization of all of your root modules within their UI folders ends up being a big benefit for understand what is where.
- Their before / after hooks enable you to do all the hacky bits that you’d want to do with building your own which is huge. They work great.
Thank you all for following and contributing to this thread.
I am kinda soloing on this, so for yesterday and today, I just stacked ECR private registry and added repository for Spacelift Docker runner.
I guess I will be pivoting my focus from here to have at least somewhat useful Docker runner.
Just out of curiosity - @Andriy Knysh (Cloud Posse) or @Erik Osterman (Cloud Posse) - does CloudPosse have a public repo of Spacelift Docker runner image that comes with atmos and Terraform > 1.0 ?
I really want to prepare everything I can upfront, before I request SpaceLift trial. This includes two CP modules for day-1 spacelift stack that manages all other atmos stacks as well as module for worker pools.
we use geodesic which comes with terraform and other tools, but we install atmos on it
ATMOS_VERSION=1.4.22
# Using `registry.hub.docker.com/cloudposse/geodesic:latest-debian` as Spacelift runner image on public worker pool
apt-get update && apt-get install -y --allow-downgrades atmos="${ATMOS_VERSION}-*"
# If runner image is Alpine Linux
# apk add atmos@cloudposse~=${ATMOS_VERSION}
@azec This was something I just put together the other week for a client I have on public runners if it is of interest — https://github.com/masterpointio/spacelift-atmos-runner
If you’re going for private workers, then I would suggest building your own toolbox image from Geodesic like Andriy mentions.
Sorry, we don’t use Atmos so I can’t really comment on that. But I think the benefit of Spacelift over gitlab ci (or any generic ci platform) is that Spacelift is a) a deployment-aware tool, so dealing with things like failed runs is saner and b) Terraform-aware, so there are escape hatches for running one-off tasks, automatic Terraform version management/init /etc
Hey, @azec. Here’s my post-vacation reply. Hope its helpful.
We’re a start-up and pretty much green-field in terms of the technologies we use. We started with TF from the beginning, rolling in infrastructure repos (i.e., IAM, DNS, bastion access, etc.) as well as managing multiple dev and client environments. Knowing we were going in the TACOS direction, we built our processes around gitops, so by the time we got to Spacelift (plus the time it took to get my head around SL and then build out my TF SL implementation) integrating all of those stacks and multi-stacks (that deploy the same codebase to multiple environments) was a matter of updating YAML files.
A few things that thrill me about Spacelift:
- Auditing. It can be really helpful to have a list of runs with the plan and apply logs. a. It also makes it much easier to share plans, with links to the SL run
- The Github integration goes a long way to providing enough visibility into the runs triggered by PR checks, to make github the authoritative source for approving PRs. The main benefits here are that you don’t have to pass around plan files or even log into Spacelift, unless they see unexpected changes. Along the same lines, there are a number of ways to add the plan output to PRs as well.
- Gotta git it up for the Runs view. Once I accepted that time flows from right to left on the graph, I find it very satisfying to see my runs visually.
- Consistency. SL makes it easy to keep the TF run environments consistent. We have a Docker container we run on in the SL public worker pool, so every run runs in a consistent managed environment. This benefit of this was reenforced, when one of my devs scrogged his local env to the point that he couldn’t run TF, so he just used
local-previewto run it in SL, which unblocked him. - OPA. Lots there and it really gives you a lot of flexibility. We’re getting a lot of mileage out of policies using tag-based triggers WRT day-to-day solutions:
SL allowed me to complete the automation of our bastion-host access. After years of my approving PRs for FW access and then running local scripts to deploy those changes to multiple environments, it’s all in a multi-stack now and fully managed through PRs. When i first set this up, I had neglected to restrict merge access in that repo, so it was a surprise to see that the dev had gotten the PR approved and merged it, and that SL had taken care of the rest, deploying those changes to all client environments….completely without me. We do this now for app deploys, as well as changes to IAM/user mgmt, FW access, and DNS.
We deploy to multiple, essentially identical client accounts. Once i got my TF set up to deploy these “multi-stacks”, it became really easy to add additional stacks, by just copying and updating existing stack or muti-stack config files. I now have 4 multi-stacks that hit 8 environments. The multistacks break out very generally into app-related stuff, non-app infrastructure, non-app access, and account-level stuff. Without a TACOS solution, it would be difficult to manage all of these, both at the config level as well as in terms of running them.
Thank you for the feedback @Eric Berg. Highly appreciated and valued. It provides me with 1st-hand experience that I can benefit from especially as I start narrowing down focus of features of Spacelift I want to self-evaluate once I request the trial. I am wrapping up the documentation related to presenting Spacelift to my team this week and will try to have workers pools and runner by end of week - leveraging Cloudposse’s AWS modules in TF.
NP, @azec.
@azec one thing worth mentioning since you seem focused on a strong initial trial — The folks at SL are super friendly. I imagine they’ll give you a longer trial if you request and share your thoughts and direction why.
Good point, @Matt Gowie. The SL folks are very helpful. They especially love feature requests!
Thanks @Matt Gowie, I will def. have that in mind.
I have finished writeup of evaluation proposal yesterday. Moving forward with trying to get Spacelift runner ready and hopefully pushed to ECR by EOD tomorrow.
One blocker I have hit is how to safely save GitLab Personal Access Token to be accessible to the Docker runner image when I request Terraform stack plans (through atmos ).
The org allows only HTTPS not SSH.
See how to use netrc files
That is what we use
# The URL "[email protected]:" is used by `git` (e.g. `git clone`)
git config --global url."<https://github.com/>".insteadOf "[email protected]:"
# The URL "<ssh://[email protected]/>" is used by Terraform (e.g. `terraform init --from-module=...`)
# NOTE: we use `--add` to append the second URL to the config file
git config --global url."<https://github.com/>".insteadOf "<ssh://[email protected]/>" --add
this ^ can be added to .NETRC, see https://sweetops.slack.com/archives/C031919U8A0/p1657752871607329?thread_ts=1657751889.281139&cid=C031919U8A0
Long thread here, but I finally managed to build Spacelift runner Docker image in our GitLab CI/CD pipeline (which uses Kubernetes runners so I had to use Kaniko to avoid Docker-in-Docker problems with image builds).
This week (starting tomorrow) I am shifting focus to Terraform modules for Spacelift private workers.
My dream of trying to get this setup right for my team stops here today In the early morning in a call with sub. director of engineering I have learned that due to the org restructuring, my position is being eliminated at Verizon.
Ah so sorry to hear about that Amer! It seemed like you were well on your way and were doing great work.
Honestly, sounds to me like Verizon’s loss! I’m sure you’ll go and find an amazing role somewhere else where you can implement what you’ve been doing and be well appreciated for it!
That really sucks @azec! I agree with @Matt Gowie - sounds like you can easily bounce back from this and get a job somewhere else. Please post in #jobs!
Can Spacelift get ECR runner images if ECR has KMS key encryption enabled (under the assumption that IAM Role which Spacelift stack assumes has permissions for same KMS key)?
I believe so, yes, but I’d imagine this would only work on private workers.
Following up on this. Per https://docs.spacelift.io/integrations/docker#using-private-docker-images it seems that public worker pools only can work with public images, so I would probably have to host runner in public ECR registry.
@, would you know if private worker pools are available in 14-day trial of Spacelift ?
Indeed they are, the trial is all features of the enterprise plan
Thank you!