#general (2020-04)

I was raking my brain trying to remember this one; so relieved you reminded me of it. @Andriy Knysh (Cloud Posse) is filled with random trivia like this one and he told me about it a long time ago. I wanted to add it to our “guidlines when asking for help” section.

just did that!

https://sweetops.com/code-of-conduct/

Thanks @Arnaud Groussard ! Appreciate the shout out

I use rundeck. Here are a few ways we use it:

• Anytime we need a cron job, we set it up as a scheduled rundeck job. This gives us a central place to manage all jobs and easy access to logs for any engineer.

• Our CI pipelines hit rundeck to trigger a deploy of our apps and to cleanup test envs that are no longer needed.

• It’s nice to be able to give engineers access to specific jobs that they can run themselves. Their documentation has been getting better, but it’s still hit and miss for some things. They have a google group (which they said was going to be shut down in favor of using stack overflow, but hasn’t happened yet) and tags on stack overflow. They’ve got a guy who responds to pretty much all questions and is helpful in pointing people in the right direction.

I’ve filed a few bugs, one was confirmed and addressed almost immediately but others have not been acknowledged yet.

Overall I’m quite happy with rundeck, but it does have some rough edges here and there.

Do you self host? If so, how much effort is it to maintain? How do upgrades go?
Our CI pipelines hit rundeck to trigger a deploy of our apps and to cleanup test envs that are no longer needed. Ooh, that is quite nifty. We use hosted gitlab, and I’ve balked at storing deployment creds there, this seems like a nice way around that.

What’s the sharpest rough edge you’ve encountered?

Do you self host? Yes. We are using the open source version and hosting it ourselves. We’re currently in transition and have two deployments: one is Rundeck 2 on an aws instance, and we’re transitioning to rundeck 3 deployed in a k8s cluster.
how much effort is it to maintain? Very little, once setup I’ve not needed to do much maintenance on rundeck at all.

How do upgrades go? They publish upgrade guides to point out issues between upgrade versions, and as long as you read those first it’s usually not bad. I do recommend keeping up with the minor releases so when a new major version comes out (it’s only version 3 so it doesn’t happen often) it’s not as painful. The good news is that the projects and jobs that I’ve imported from 2 to 3 have transitioned without any issues.

We use hosted gitlab, and I’ve balked at storing deployment creds there, this seems like a nice way around that. We’re in the same boat. The other nice thing about this is if you need to do a manual deployment (say you change an env var in SSM and need a redeploy to get it) you can use the same rundeck job to do the deploy as your code-triggered deployments. That means you’ve got one place to do deployments and a single source of deploy logs, regardless of how they were triggered.

What’s the sharpest rough edge you’ve encountered? Hands down, documentation. I already mentioned it’s been improving and there’s a decent community (and someone who appears to be a dedicated at least part time to community support) so I wouldn’t consider it a show stopper, just go in with your eyes open and don’t hesitate to ask questions here or on stack overflow. The google group is also a good resource, especially since it’s searchable – https://groups.google.com/forum/#!forum/rundeck-discuss

Since I’d already mentioned that I’ll give another one: their ACL can be challenging to work with. They’ve recently made some pretty significant improvements to their documentation in this area and moved the ACL into the database so it can be modified more easily via the GUI (though it’s still editing YAML), so again, it’s improving.

I’ve setup our rundeck 3 instance to use gitlab for authentication, and it’s really nice. I’m using https://github.com/oauth2-proxy/oauth2-proxy to handle that.

Many thanks @bradym I am going to give it a spin in our dev environment

I haven’t started looking at docs yet, so maybe this would be answered easily: but does it support “groups” in a fine grained way? We work on a bunch of client projects with our won ops team and ops members of the clients. It would be critical that we could create “groups” for each client to keep their stuff separate from the rest, and of course put users in those groups so they can only see their own stuff.

You mention the ACL can be challenging to work with.. would this be possible?

Yes, groups work well with ACL. The ACL works well, it’s just not the most intuitive system for setting things up.

In our setup I have three groups: devs, senior devs, and ops. Members of the senior devs group can trigger prod deployments while devs can trigger deployments in stage and view prod deploy logs. Ops run any jobs.

What I’d probably do in your case is setup a different project for each client, and use group membership to limit what they can see / interact with to the projects they need.

my lerna.json:

{
  "npmClient": "yarn",
  "command": {
    "publish": {
      "registry": "<https://gitlab.com/api/v4/packages/npm/>"
    }
  },
  "packages": [
    "packages/*"
  ],
  "version": "independent"
}

And in my CI pipeline:

lerna --loglevel debug bootstrap --no-ci
lerna --loglevel debug run dist
lerna --loglevel debug version $VERSION_OPTIONS --yes --message "[skip ci] Publish"
lerna --loglevel debug publish from-package --git-head "$CI_COMMIT_SHA" --yes

$VERSION_OPTIONS is set based on the gitlab merge request title to one of: patch, minor, major

And when I make a change to a single package, lerna tries to publish all the packages.. which is currently failing as different packages have different version numbers (as you’d expect in independent mode)

Anyone have any insight?

i didn’t

linking since the straight google search was not helpful https://github.com/pomerium/pomerium

gomplate can do a lot of the same things

very well maintained by @hairyhenderson

https://gomplate.ca/

(I don’t know about “very well” - it’s been months since I’ve cranked out a release )

Haha, well, sometimes tools reach a level of maturity they don’t need constant updates. There’ll always be a sea of feature requests.

We’ve been using the tool for many years now and it “just works”

confd has been “just working” for us for years now too. except recently. i love tools that get mature, no recent commits isn’t a bad thing. but the author is MIA and not responding even to questions of whether the project will be updated ever, so we’re a little nervous

had no idea gomplate supported remote datasources like AWS param store, that’s great!

more remote datasources are coming… eventually I’ve been kinda spinning my wheels the past few months adding support for config files - almost finished on that, and then I’ll be able to focus on more datasources and functions

I want to get some better support for GCP too

I’m excited about config files! I think that will be a nice addition to gomplate

while were geeking out on cli tools, have you seen @mumoshu’s #variant?

https://github.com/mumoshu/variant2

gomplate is awesome. Just used a combination of SSM and yaml config file data sources to template out our packer.json.

@Erik Osterman (Cloud Posse) i’ve taken a cursory look at variant2, but I don’t grok it yet. Seems like a strange abstraction over a shell script. I guess I don’t have such hairy shell scripts that need it?

Can it be used to replace Makefiles?

I’ve heard of Mage for replacing makefiles - https://magefile.com - been considering switching to it myself…

you likely meant magefile.org

If you are a powershell nut I’d give invoke-build a look as well (https://github.com/nightroman/Invoke-Build). It’s a masterpiece of posh coding and build automation done in an exceptionally thoughtful and tight manner.

The outcome of variant is you have a cli that everyone in your company can use on any platform that has built in testing and none of the baggage of shell scripts

It’s a DSL (in HCL) for expressing all the commands a human would ordinarily run. Much like make you can pipeline and have dependencies.

It looks like a sweet wrapper that turns scripts that ‘feel’ like hacks into singular binary releases?

Make is very difficult to test. Makefiles become complicated as everything is passed with envs. Envs are hard to validate in make. As the project grows so does the complexity. We wrote the build-harness in make, but it’s pushing the limits I think.

I can speak from personal experience that you CAN push make to crazy limits but it is 50+ year old tech that probably shouldn’t be.

Bash, python, Zha, fish, ruby, go etc can also be used to write the cli. Ordinarily this reduces the the number of people who can contribute. It is expressed as a DSL to avoid complexity. Later, when you have proven your cli interface and have a few people to maintain it in go, you can rewrite it.

But chances are the quick wins with variant will be hard to beat. You can have your cli written in a day.

I use Make or Invoke-Build as ways to attempt to create declarative state tools in an imperative manner. I’ll still use them to get things done in the short term but am actively scoping out methods/tools out of such reliance.

I think variant is more of a competitor to “build your own cli” in one or these 12 dozen languages, rather than a Makefile killer. I think a lot of us have used make for operations because it exists everywhere and is simple to get started with. I think I would still use make to build my variant project :-)

But this reduces make a few targets and then my variant based cli will handle the rest

Plus mumoshu is a genius so if he is making tooling for something then it may be worth paying attention to. Just sayin’

Haha too true

we’ve done this, using cloudformation…

Type: AWS::SSM::Association
Properties:
  Name: AWS-JoinDirectoryServiceDomain
  Parameters:
    directoryId:
      - Ref: DomainDirectoryId
    directoryName:
      - Ref: DomainDnsName
  Targets:
    -
      Key: tag:aws:cloudformation:stack-id
      Values:
        - Ref: AWS::StackId

autoscaling also applies tags, so should be able to do the same using the autoscaling tag

can probably create the ssm association using terraform, if that’s your jam

aws:autoscaling:groupName

this implies that ssm associations are executed automatically when an asg launches an instance with the tag in that association’s targets

I was under the impression that the ssm associations only execute via the schedule or when manually started

all i can say is it works for us

interesting! I will experiment

SSM associations can be Cron based. However I just bootstrap with it. No schedule needed. On first creation or changing assocation it will run against all targeted instances as it checks missing assocation having been applied.

I just discovered the same! That’s great.

hey all. hope you’re safe and well during these crazy times.

Likewise

Thanks!

is great. could also be considered as “trends”

nice article. very helpful to discuss patterns like this. i think i consider ConfigOps to be GitOps? or at least, my interpretation of GitOps

I believe you could perform ConfigOps within GitOps. But you don’t need to be practicing wholescale GitOps to practice a ConfigOps pattern. Perhaps it is just a matter of nuance that doesn’t matter so much though and i’m being nit picky

I had several other terms I wanted to put in there. I sort of avoided GitOps as it was a bit more of an obvious pattern to me. Perhaps I’ll throw down my interpretation of that one as well to see how hard I get smacked down for it

yeah, i know GitOps started with containers specifically in mind. but i guess i generalize it to just keeping configuration in git and having pipelines that apply those configs

I had considered putting gitops as a term but it means too many different things for too many different people right now. <blank>Ops all have one thing in common, they start with version control and end with delivery of artifacts from it. I threw in ConfigOps specifically because I ran into a scenario where it would be disingenuous of me to call a solution I helped put into motion ‘GitOps’ as it was still not automating deployment in a watch/pull pipeline model as so many GitOps tools do (flux/argocd being the prime examples). Instead it was a repo constructed from the output elements of a large terraform manifest, one per environment which included multiple per-team and client sub-environment’s settings and included pipeline as code for then pushing all those settings into their pipeline system of choice (azure devops) for use in other deployment pipelines (that were also in that repo). Everything was still a push model (one of the deployments even built out and configured a kube cluster within the created environment) but changes to the deployment configuration were able to be controlled via git PRs

Perhaps a proper term for push vs. pull based devops solutions is in order

I’ve also seen a ton of work go into automating pipelines only to be undercut by manual changes of configuration elements for the pipelines

i think my idea of a pipeline also ends up being different i don’t really like the way, say, jenkins does things, or codepipeline. i like when the git repo owns everything

so i tend to build things in a way that events in the git repo drive the pipeline. which is more of that “push” approach

Partially why I hugely favor pipeline independence where I can make it happen

i like the pipeline independence term. i stress that with our team, have a makefile (or other command runner) that you can use locally to exercise a build, and also have the pipeline use that same runner

I try to do the same, it has been a life saver on more than a few occasions

@loren thanks for your feedback btw, I’ve come to respect your opinion quite a bit

anytime! i’ve learned so much from folks in this slack. it’s really great to be able to contribute back when i can

Glad you stopped by!

we’ll be in the #office-hours channel as well on Zoom tomorrow

Glad to have you here!

azure has their bastion service, is this the feature of session manager you’re referring to? https://azure.microsoft.com/en-us/services/azure-bastion/

and they have app configuration, which sounds some like the parameter store… https://azure.microsoft.com/en-us/services/app-configuration/

https://www.envkey.com/ (YC 2018)

Hey thats pretty cool

that tracks pretty closely with our (less formalized) requirements for open sourcing projects on our public github org

@sheldonh very intrigued by what you wrote here. looks interesting.

i remember this year at SCale 18x there were a few sessions on building and promoting open source projects. you should check out their youtube videos.

btw, do you mind sharing your blog url?

sure. sheldonhull.com

Its a good opportunity to do this, just have to approach it right as company from a company with no history tmk of open source contributions or regular blogging. any templates/or whatever welcome and appreciated

@sheldonh are you able to join us on #office-hours today? Maybe this is something we can discuss.

Hey thanks for the reminder. I might be able to listen in (in middle of some work.) That’s starting right now right?

wait that’s in an hour. Even better. I’ll try to make it. I really enjoyed the last one. I felt smarter after it

Yep, in 1 hour. Will you be able to join with mic?

I’ll do my best. Might get pulled into something different, but hopefully no conflicts

Thanks

I wouldnt call my self a DevOps guy yet. more of a developer trying to figure out how the hell some of this Ops related stuff works

You are at the right place then

Hello! I might be missing some context here. What’s up?

Glad to be here! Really nice community!

Welcome @Ayman!

Thanks @Erik Osterman (Cloud Posse)! Just stumbled on your company yesterday and I’m really impressed with your open philosophy! Trying to open source my knowledge/experience as well.. but there’s that time thing lol!

Too true. It’s a massive time commitment. But just place one grain of sand at a time and eventually it turns into a mountain

I found this community, no long ago. Happy to be here

Good to have you here

Check out aws-vault. Makes using a temp session virtually invisible

Thanks @roth.andy! will check that

performance reviews?

I’d push notifications to company Slack / Microsoft Teams > Email.

That’s true. I’ve been doing that by using the /feed to subscribe to githubs rss feed

It’s too bad we can’t do that as easily from email

you can send email to a slack channel… is that what you mean?

https://slack.com/slack-tips/send-email-to-slack

oh perhaps im misunderstanding. i was assuming @Matt Gowie meant subscribing to the rss feed from within slack. can you also subscribe an email list directly to slack if the email list doesnt have an rss feed ?

oh wow, thanks @loren!

all devops is sticky tape and bubble gum. might as well standardise the gum we’re chewing! looks cool

As long as I don’t have to use Ruby, blech

why is it always ruby

Any language that has whole articles on ‘truthiness’ scares me…

especially when it can be mostly summed up into one sentence

I had a leader use “devopsy stuff” when describing work I did today lol

I want that on my business card, I like it