#general (2020-05)

General conversations related to DevOps/Automation

General Discussions

2020-05-31

SweetOps avatar
SweetOps
07:00:05 PM

Hey everyone, give a warm welcome to our newest members!

  • @

Good to have you here =)

2020-05-30

SweetOps avatar
SweetOps
07:00:21 PM

Hey everyone, give a warm welcome to our newest members!

  • @Pyunix
  • @
  • @
  • @
  • @

Good to have you here =)

3

2020-05-29

muhaha avatar
muhaha

Guys? Is there any tool similar to https://github.com/korfuri/awesome-monorepo#build-systems--dependency-management-tools ? I have monorepo with separate Terragrunt projects, problem is that I need to:

  1. be able to select project to run in pipeline ( not so hard, I can pass $path variable to that project in gitlab )
  2. be able to determine which project was changed/impacted in PR ( this is hard, it can be probably done with git diff and bash scripting ) But something that can manage projects in my monorepo and also determine blast radius would be really nice ! Thanks
muhaha avatar
muhaha

Unfortunatelly Terragrunt can not manage 1. nor 2. , good example of using projects in monorepo is atlantis

Shikhar Goel avatar
Shikhar Goel

Is there a way in helmfile where i can stop it to upgrade job and stateful sets.Actually currently what is happening is that i have labels in helm charts but when i use helmfile to upgrade the deployed helm charts it is failing because job and statefull sets cannot be updated(i.e. cannot add labels in my case).

SweetOps avatar
SweetOps
07:00:21 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

2020-05-28

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

2

2020-05-27

SweetOps avatar
SweetOps
07:00:14 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @

Good to have you here =)

2

2020-05-26

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

4

2020-05-25

niek avatar

Last week I have create a new Terraform module to host GitHub self hosted action runners on AWS spot instances. For those who are interested the setup is explained in this post https://040code.github.io/2020/05/25/scaling-selfhosted-action-runners

Scaling GitHub Action Runners | 040code attachment image

This post explains how to run GitHub actions on self-hosted scalable runners on AWS spot instances. Introduction Last year GitHub released…

:--1:6
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

That’s great! I’ll share this in #office-hours announcement

Scaling GitHub Action Runners | 040code attachment image

This post explains how to run GitHub actions on self-hosted scalable runners on AWS spot instances. Introduction Last year GitHub released…

niek avatar

Cool, thx Erik

2020-05-24

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @

Good to have you here =)

Andy avatar

Hi, does anyone have experience with setting up fail2ban across multiple nginx boxes with a shared database for banned IPs?

2020-05-23

SweetOps avatar
SweetOps
07:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @caron.wills

Good to have you here =)

2020-05-22

gyoza avatar
gyoza
:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Jeremy (Cloud Posse)

Jeremy (Cloud Posse) avatar
Jeremy (Cloud Posse)

@gyoza Not sure what you mean by “remove” gravatar images. Do you mean to set disable_gravatar = true in .ini?

gyoza avatar
gyoza

probably

gyoza avatar
gyoza

alert

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

1
:--1:1
RB avatar

anyone ever use celery with sqs ? for some reason my tasks are sent to sqs but not taken from sqs

RB avatar

Fixed it. I was running flask with celery and had a route to add tasks using flask but didnt run the celery worker so it wasnt taking the tasks. Doh

2020-05-21

SweetOps avatar
SweetOps
07:00:08 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

4
RB avatar

requesting more slack emojis

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

sure thing! just link me to ‘em and I’ll add

2020-05-20

Harshal Vaidya avatar
Harshal Vaidya

Hello - Where can I get help for cloudposse terraform modules? I am facing some errors and need some help.

Christian Roy avatar
Christian Roy

I would say that #terraform would be a good place to start. Just expose your issue with what your trying to accomplish, what is happening that is unexpected, what you tried so far to fix it, and any specific errors messages your getting. and then be patient.

Harshal Vaidya avatar
Harshal Vaidya

I did that already .. but haven’t had any reply

Andrey Nazarov avatar
Andrey Nazarov

What do you think about “Managing stateful systems via container orchestration” blipping as Assess on the latest Technology Radar?

https://www.thoughtworks.com/radar/techniques/managing-stateful-systems-via-container-orchestration

Managing stateful systems via container orchestration | Technology Radar | ThoughtWorks attachment image

This Technology Radar quadrant explores the techniques being used to develop and deliver software

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

haha, sounds like another topic for #office-hours ?

Managing stateful systems via container orchestration | Technology Radar | ThoughtWorks attachment image

This Technology Radar quadrant explores the techniques being used to develop and deliver software

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

1

2020-05-19

SweetOps avatar
SweetOps
07:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

KEN avatar
Hello I am Ken! Our next online
@WebHackMeetup
is about building your own #Golang linter. And it is our first event on
@airmeet_com
https://webhack.connpass.com/event/175795/
WebHack#33 Write your own Golang Linter (2020/05/25 19:30〜) attachment image

WebHack If you believe building relationships with people who are in the same industry helps you know yourself and achieve what you couldn’t do alone, this month’s event is for you! This month, WebHack enables a better online meetup experience: you can select virtual tables to have deep face-…

sheldonh avatar
sheldonh

New to golang. Is this advanced or would it be useful to someone beginning too?

WebHack#33 Write your own Golang Linter (2020/05/25 19:30〜) attachment image

WebHack If you believe building relationships with people who are in the same industry helps you know yourself and achieve what you couldn’t do alone, this month’s event is for you! This month, WebHack enables a better online meetup experience: you can select virtual tables to have deep face-…

KEN avatar

It is a good beginning to work with Golang’s advanced features, for developing programming language tools.

2020-05-18

SweetOps avatar
SweetOps
07:00:22 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @Hari Krishna Ganji
  • @
  • @

Good to have you here =)

2020-05-17

SweetOps avatar
SweetOps
07:00:04 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @Fernando Torresan
  • @
  • @

Good to have you here =)

2020-05-16

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @

Good to have you here =)

2020-05-15

loren avatar
loren

joined another very active slack recently, didn’t realize just how much i appreciate the way this one uses threads appropriately

1
slack1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

ya, we’re been pretty good about that. also, a convention @sheldonh started that I like is prefixing questions with [thread] as a hint to others to start a thread.

loren avatar
loren

even more, i just appreciate that it seems to be part of the culture here. folks see messages with threads in pretty much every channel when they join, and they see others using threads to respond to new posts, and generally they seem to follow suit quickly. it’s glorious, even without a convention

Gowiem avatar
Gowiem

Agreed — I seriously appreciate threads and other slacks bog down my time when they don’t use them. I’m definitely going to steal @sheldonh’s [thread] idea going forward to push folks in the right direction. Thanks for mentioning @Erik Osterman (Cloud Posse)!

sheldonh avatar
sheldonh

I was in another community and got called out for using threads. Turns out they have major accessibility issues for screen readers. First time I’d ever heard of that. Pretty crazy that it hasn’t been dealt with. Unless somebody specifically mentions they’re having trouble reading it I’m not sure it makes sense to stop using them because context is lost for everybody else anyway. Just thought I’d mention that was the first time I’d heard anything negative about threads.

I have a PowerShell extension on a slack bot. It tracks channel postings and if unthreaded messages are posted more than say a couple times in 15 minutes it politely suggests using threads to help keep the noise down. I’m a big believer in keeping context

sheldonh avatar
sheldonh

Protip. When you post thread in the title I try to only post the main topic and maybe a mention, and then immediately post the main bulk of content in the thread. I found that that helps ensure the conversation immediately begins in the thread as somebody clicks to expand it. Just a little thing that seemed to help

:--1:2
Gowiem avatar
Gowiem

Good tips @sheldonh — Thanks! Will mention introducing the slack bot idea to one of my clients for sure — they could use it!

:--1:1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Clever hack!

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @contact871
  • @
  • @
  • @

Good to have you here =)

tomkinson avatar
tomkinson

Anyone really good at MongoDB here?

Kevin Chan avatar
Kevin Chan

Hi

2020-05-14

SweetOps avatar
SweetOps
07:00:05 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

2
1

2020-05-13

Rameez Iqbal avatar
Rameez Iqbal

Hi guys I have an issue with helm templates, wondering if someone can help. I am trying to pass private in values.yaml i.e.

ssh_key: |
    -----BEGIN RSA PRIVATE KEY-----
    ....
    -----END RSA PRIVATE KEY-----

And then I am using it in my template like

templates:
- |
  apiVersion: v1
  type: Opaque
  kind: Secret
  metadata:
    name: bitbucket
  stringData:
    username: git
    privateKey: {{ .Values.ssh_key  | indent 4 }}

But it keeps replacing newlines with spaces. Has anybody come across this issue?

Rameez Iqbal avatar
Rameez Iqbal

There seems to be something wrong with stringData encryption. I will just stick with data for the time being.

SweetOps avatar
SweetOps
07:00:17 PM

Hey everyone, give a warm welcome to our newest members!

  • @

Good to have you here =)

baris avatar
baris

Thanks I wasnt able to join yesterday as I had a problem with Zoom but i watched a recorded session. It is nice to be here :)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Awesome! Glad you were able to catch up on the recorded session.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

We also are syndicating them here: https://podcast.cloudposse.com

Cloud Posse DevOps "Office Hours" Podcast

Cloud Posse holds public DevOps “Office Hours” every Wednesday at 11:30am PST to answer questions on all things DevOps/Terraform/Kubernetes/CICD related. These “lunch & learn” style sessions are totally free and really just an opportunity to talk shop, ask questions and get answers.Register h…

2020-05-12

Shawn Petersen avatar
Shawn Petersen

Hi guys, I’m working with a small tech startup (less than 10 employees) and we’re trying to come up with a good IT policy for employee laptops. With the vast amount of experience in this Slack channel, I’m wondering what you guys think the pros/cons are for a BYOD policy, versus the company-paid-for laptop that the company owns, locks down and manages (this is what most IT departments I’ve been with in the past do, and still do). 

What do other cutting-edge tech startups do to manage their IT equipment now?  What is your experience with BYOD? Does it work well? How is it implemented? or is the “tried-and-true” company-owned laptops still the best policy? … or, are there any other options?

All opinions and/or examples greatly welcome…

roth.andy avatar
roth.andy

Do you have any regulatory requirements you have to be compliant with? Stuff such as HIPAA or DFARS?

Shawn Petersen avatar
Shawn Petersen

not sure about those, but probably want to be PCI and GDPR

Shawn Petersen avatar
Shawn Petersen

I should add, all our “data” is cloud-based (AWS/Gsuite), so the idea is lock those down and not worry about the client machines

conzymaher avatar
conzymaher

If using mac Jamf is a great solution

conzymaher avatar
conzymaher

Just to implement some sane defaults. Encryption enforced with an Institutional Recovery Key. Screen lock after 5 mins etc

conzymaher avatar
conzymaher

That simple thing will at least save you from the “Laptop left on the bus” scenario

conzymaher avatar
conzymaher

At the same time would I like the company to have these controls on a BYO device? No.. If the company is serious about security and implementing some controls there is no choice but to supply the equipment IMO

conzymaher avatar
conzymaher

Using a personal machine for business has many risks. What if you accidentally use the wrong AWS credentials? What if an employee installs a package for a personal project that exfiltrates their keys etc etc. Using a personal machine for business is crossing the streams and adds a huge amount of attack surface area

Shawn Petersen avatar
Shawn Petersen

Yeah I’ve considered that. The strategy currently is to NOT hand out any access keys, and only use SAML credentials (w/2factor) to gain temporary access via KMS… Again, the idea we have is to provide enough security into AWS that it shouldn’t matter what device you’re using.. but we still wonder if giving up that control is worth it or not..

Shawn Petersen avatar
Shawn Petersen

As a small company, we’d rather not have to worry about managing the equipment for employees, but we realize it might be a necessary evil…

Shawn Petersen avatar
Shawn Petersen

I’m hoping to get some examples of where BYOD works well for companies and how

Chris Fowles avatar
Chris Fowles

have a look at the principles of Beyond Corp

Chris Fowles avatar
Chris Fowles
BeyondCorp - Enterprise Security  |  Google Cloud attachment image

BeyondCorp is an enterprise security model that allows employees to work more securely from any location without the need for a traditional VPN. Learn more.

Chris Fowles avatar
Chris Fowles

like don’t try and go all in on it straight away - but if you make access control decisions with some of these principles in mind then you should end up in a pretty good place

Chris Fowles avatar
Chris Fowles

the key thing i would say is don’t try to boil the ocean at the start - but make any new decisions the right way

Shawn Petersen avatar
Shawn Petersen

Awesome information Chris! Thanks. I guess I’m looking to answer some issues like how to start or enforce a BYOD policy? Do companies have problems asking employees to use their own devices? Or do they end up just purchasing a laptop for the employee at hire like a hiring bonus to make sure they have a “good enough” workstation? And at that point, is it just better to say it is the company’s laptop in case they end up leaving a few days later etc?

Chris Fowles avatar
Chris Fowles

it really depends - i’ve seen companies do both - generally i’d prefer to be asked as an employee how they’d like to work

Chris Fowles avatar
Chris Fowles

i have a work laptop that was provided by work - but working from home i’d much rather use my own system as it’s much more powerful than i’d expect work to provide

Shawn Petersen avatar
Shawn Petersen

Thats the same as me at my previous job… I worked at my desk pc at home

Shawn Petersen avatar
Shawn Petersen

I’m trying to factor in the lowest denominator.. those who don’t have any laptop or pc at home at hire.. we don’t want to create an HR nightmare

Shawn Petersen avatar
Shawn Petersen

So what would a company do for them versus someone who says they’ll be happy to use their own? And how to make it “fair”?

Chris Fowles avatar
Chris Fowles

don’t worry about “fair” just focus on “comfortable for everyone”

Chris Fowles avatar
Chris Fowles

what one person values may not be the same as another

Chris Fowles avatar
Chris Fowles

and if you give them both the compromise no one is happy

Chris Fowles avatar
Chris Fowles

i’d generally figure out a spec that you find works and offer it to people if they want it - otherwise allow byod if you want to do that

the other option is to set a budget and allow people to decide what they want - this doesn’t scale in the late game though

Chris Fowles avatar
Chris Fowles

it can be a good perk for early talent though

Shawn Petersen avatar
Shawn Petersen

Yeah it seems different for a 10 person company from a 100 person company

Chris Fowles avatar
Chris Fowles

it’s very different and it’ll surprise you the things that become important as scale increases

Shawn Petersen avatar
Shawn Petersen

Maybe thats the key…

Chris Fowles avatar
Chris Fowles

my main advice would to be to just focus on the goals at each stage rather than the strategies because you’ll need to keep re-making decisions for different scales

Chris Fowles avatar
Chris Fowles

as you scale the cost/benefit equation changes

Chris Fowles avatar
Chris Fowles

like centralised management of devices is definitely overkill for 10 users - but for 1000 maybe that becomes worth while

Chris Fowles avatar
Chris Fowles

the trick is to figure out when to change things

Chris Fowles avatar
Chris Fowles

100? 500? 1000? who knows

Shawn Petersen avatar
Shawn Petersen

Yeah, but our thought is the early decisions really affects culture later, which is why we’re sweating it now, but I see your point

Chris Fowles avatar
Chris Fowles

culture is cultivated and it will change as well

Chris Fowles avatar
Chris Fowles

it’s not a thing that you set in place and it stays that way

Chris Fowles avatar
Chris Fowles

it’s kind of like a stew that you keep adding things to

Chris Fowles avatar
Chris Fowles

don’t sweat it now - just keep an eye on it as you cook it

Shawn Petersen avatar
Shawn Petersen

Yeah I can see that. I appreciate the perspective Chris. Thanks. I’ll read up on your links.

Chris Fowles avatar
Chris Fowles

figure out and make sure you can articulate your values early though - that’ll drive the culture. what’s really important to your founding group as people - as that will drive your culture more than any organised effort

Chris Fowles avatar
Chris Fowles

no worries - i really love this stuff and happy to share

Shawn Petersen avatar
Shawn Petersen

Yeah I love it too. Being in a startup has the advantage of being able to do things the way you want, but we don’t want to reinvent the wheel on what works. We’ve got most things figured out for now, but this has been going back and forth for a while, so its good to get some additional perspective.

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

:--1:2
1

2020-05-11

RB avatar

anyone here worked with celery ? currently on celery 2.x and looking to upgrade to the latest one but looking at the following strategies

• celery 2.x to celery 3.x to celery 4.x - ideal but seems like there are issues

• celery 4.x in tandum with celery 2.x and migration piece by piece - safer

• replacement for celery and migration piece by piece - possibly safer and better if the celery alternative has additional benefits All celery jobs are in python… most are running like a daily cron. Questions for you folks…

• have you done this ?

• any of the above strategies seem better ?

• any alternatives to celery you might recommend ?

chonan tsai avatar
chonan tsai

we are on celery 4. have been on it for some time. We had to migrate a while back. Can’t remember from what version but it addressed some issues we were all of sudden seeing. We migrated all at once and just did a lot of testing. We have been using SQS and we like it. May eventually moving everything to SQS.

RB avatar

that sounds awesome. what was your upgrade strategy? how many tasks did you have? we have around 500 so we need to be as careful as possible

chonan tsai avatar
chonan tsai

We have around 20 recurring tasks but all super critical. We took a very iterative approach and ran it in many different test env with prod db clones. Deciding when to do this is also critical for us since we didn’t want to disrupt any tasks half way. You are at a scale much larger than ours so I am not sure if our experience is any helpful to you. Another QC strategy we took, which was manual but helpful, was to trigger these events directly, rather than waiting for it to happen at the scheduled time.

Maciek Strömich avatar
Maciek Strömich

we did celery 3 to celery 4 a while back. basically it boiled down to changes in the django settings (in celery4 everything is prefixed with CELERY_ and there are some name chanages) and adding namespace kwarg to celery.config_from_object

joshmyers avatar
joshmyers

Not used Celery in a long time. What do the docs say about upgrade path?

RB avatar

there isn’t one

RB avatar

im considering dramatiq because thats the only thing that has an upgrade path (kind of)

SweetOps avatar
SweetOps
07:00:10 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

1

2020-05-10

SweetOps avatar
SweetOps
07:00:03 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

8
1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@ yo! nice to see you stopped by

Erick avatar
Erick

Thanks! My team loves SweetOps, so I made room in my left rail for another channel

cool-doge2

2020-05-09

SweetOps avatar
SweetOps
07:00:11 PM

Hey everyone, give a warm welcome to our newest members!

  • @

Good to have you here =)

:--1:1
1
Joey avatar

thanks for all of the warm welcomes guys

2020-05-08

HS avatar

Please can anyone recommend a good data masking tool to obfuscate production database in a non production environment?

conzymaher avatar
conzymaher

I dont know of an off the shelf tool to this because its not an easy problem. e.g will your application still work if you just randomly obfuscate every field? What I have done in the past on a ruby/rails project is use https://github.com/faker-ruby/faker

faker-ruby/faker

A library for generating fake data such as names, addresses, and phone numbers. - faker-ruby/faker

:--1:1
conzymaher avatar
conzymaher
joke2k/faker

Faker is a Python package that generates fake data for you. - joke2k/faker

conzymaher avatar
conzymaher

And then you can have ActiveRecord models that obfuscate fields with sensitive data

HS avatar

Thanks a lot @ This really helps, I just did a poc with it snd it worked as I wanted

HS avatar

:–1:

conzymaher avatar
conzymaher

e.g name, dob, and other sensitive details

conzymaher avatar
conzymaher

You can generate sensible fake names, addresses, social security numbers etc

conzymaher avatar
conzymaher

for most applications the obfuscated data needs to be plausible. e.g you cant just have a paragraph of lorem ipsum for date of birth or name

conzymaher avatar
conzymaher

So context and generating the correct kind of obfuscated data matters

conzymaher avatar
conzymaher

TLDR I dont think you will find an off the shelf tool that handles this. Build some custom-ish tooling to do this and run it in some automated process nightly / hourly etc

SweetOps avatar
SweetOps
07:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @

Good to have you here =)

2020-05-07

SweetOps avatar
SweetOps
07:00:14 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @
  • @
  • @

Good to have you here =)

1
4
Zachary Loeber avatar
Zachary Loeber

anyone happen to know offhand if there is a syntax for a gomplate file (like json or yaml) datasource to check for the existence of a key before attempting to use it?

hairyhenderson avatar
hairyhenderson

yup!

hairyhenderson avatar
hairyhenderson

coll.Has or just has will help with that

Zachary Loeber avatar
Zachary Loeber

Man, the author himself! I’m humbled good sir. Thanks for your help. I half expected no answers

1
hairyhenderson avatar
hairyhenderson

hairyhenderson avatar
hairyhenderson

I have a keyword set for gomplate in Slack, and I happened to notice

Zachary Loeber avatar
Zachary Loeber

my lucky day indeed then. It does not need to be said I think but I’m a huge fan of your work

hairyhenderson avatar
hairyhenderson

well thanks!

hairyhenderson avatar
hairyhenderson

I appreciate the encouragement! it’s hard to stay motivated, but comments like that keep me going!

Zachary Loeber avatar
Zachary Loeber

I’ll thank you by continuing to spread the word of your work like a good aspiring go disciple :D

2
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@btai is another new fan we just turned on to gomplate!

btai avatar

Erik speaks the truth!

hairyhenderson avatar
hairyhenderson

woohoo!

hairyhenderson avatar
hairyhenderson

there’s a totally unfounded rumour that a new gomplate release is coming out soon… but I have no idea where I heard that

cool-doge2
1
Zachary Loeber avatar
Zachary Loeber

I’ll just use your own personal emoji for that rumor

2020-05-06

SweetOps avatar
SweetOps
07:00:09 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

3

2020-05-05

SweetOps avatar
SweetOps
07:00:08 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

2020-05-04

Mikael Fridh avatar
Mikael Fridh

I’m looking to find some library (even shellscript…) to help write wrapper binaries/scripts for download-on-demand-and-re-execute binaries en masse… anyone seem something existing? I’ve written a couple of these things standalone.. shellscripts which curl from a “repo” and re-execs itself… trying to see if there’s anything out there already before I reinvent the wheel a 5th time.

whalebrew - wrapper for executing in docker for reference: https://github.com/whalebrew/whalebrew tfenv comes to mind as an inspiration as well. although that’s for one single tool and many versions.

whalebrew/whalebrew

Homebrew, but with Docker images. Contribute to whalebrew/whalebrew development by creating an account on GitHub.

Gowiem avatar
Gowiem

@Mikael Fridh — From your description, I’m not sure this fits, but because your mention of tfenv I figure this is worth mentioning: https://asdf-vm.com/#/

asdf vm - An extendable version manager

An extendable version manager

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

also check out #variant

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
mumoshu/variant2

Turn your bash scripts into a modern, single-executable CLI app today - mumoshu/variant2

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Though after re-reading your post
trying to see if there’s anything out there already before I reinvent the wheel a 5th time.
Variant wouldn’t give you this out of the box, but let you declaratively describe how you want to do it.

Variant works with mod to abstractly define dependencies. It can then automatically update those. It’s also designed to work in a gitops-fashion where it can PR the changes.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
variantdev/mod

Missing package manager for any task runners and build tools e.g. make and variant - variantdev/mod

1
Mikael Fridh avatar
Mikael Fridh

Thanks! Will check these out!

Zachary Loeber avatar
Zachary Loeber

asdf-vm is pretty sweet

1
joshmyers avatar
joshmyers
bitfield/script

Making it easy to write shell-like scripts in Go. Contribute to bitfield/script development by creating an account on GitHub.

Mikael Fridh avatar
Mikael Fridh

@Erik Osterman (Cloud Posse) mod - thanks! Remember seeing this way back and thinking I had a good use for it…

@joshmyers script looks interesting!! Could be a gateway drug to more Go :).

Mikael Fridh avatar
Mikael Fridh

I think I now know better how to express what I was after.

tfenv has a terraform shim, a script which when run, detects which version of terraform you want, and downloads that automatically and then just runs. No need to apt install / scoop install / brew install ..

So what I’m looking for is whether there are decent helpers out there already to allow you to “mass-shim” your environment to make the tools download on-demand only instead of baking everything inside a 2GiB tools image .

Mikael Fridh avatar
Mikael Fridh

Am already doing it with a couple of curl-based shell script shims. But they’re one-offs so far.

Gowiem avatar
Gowiem

Build it into a Make dependency?

Mikael Fridh avatar
Mikael Fridh

That + some shellscript could probably be a good framework actually, yes.

Gowiem avatar
Gowiem

Variant is a good use-case too. Likely just more work, but could have a more structured and streamlined approach for the end user.

Mikael Fridh avatar
Mikael Fridh

yes, the good thing about variant is that instead of having a whole bunch of loose tools available which you forget which was for what, having one single entrypoint, with possible informational output could help.

Mikael Fridh avatar
Mikael Fridh

like say you provide a “tools-cli” docker container for example…

Gowiem avatar
Gowiem

Yeah. I like the Variant approach. I’ve been wanting to use that tool since I learned about it.. I may try that out on my next client project to try to simplify client DevOPs on-boarding.

Gowiem avatar
Gowiem

Though ya, you could also build a “tools” docker image and then use whalebrew like you mentioned. That approach with mod would be pretty cool too.

Zachary Loeber avatar
Zachary Loeber

asdf with direnv

Zachary Loeber avatar
Zachary Loeber

there are those very basic requirements but if you are doing this from a local workstation then you are going to be using some base tooling anyway

Zachary Loeber avatar
Zachary Loeber

I need to figure out how to get variant to work properly for external dependencies, simply haven’t given it enough attention yet (but i bet it works well though)

Zachary Loeber avatar
Zachary Loeber

otherwise, I wrote a wrapper script for cloudposse’s package that I then pull in via makefiles to handle all that

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

And if you go the tools image approach, have a look at https://github.com/EnvCLI/EnvCLI

EnvCLI/EnvCLI

Don't install Node, Go, … locally - use containers you define within your project. If you have a new machine / other contributors you just have to install docker and envcli to get started. - …

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

this let’s you interact with the tools image as though all the commands were native (but they actually run in docker)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@Igor Rodionov is digging it

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@roth.andy have you seen envcli?

roth.andy avatar
roth.andy

No. Sounds cool though

Zachary Loeber avatar
Zachary Loeber

seems cool but what’s with dropping the releases on freaking bintray?

Mikael Fridh avatar
Mikael Fridh

Cool. EnvCLI seems like an even more specifically useful whalebrew

SweetOps avatar
SweetOps
07:00:06 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

5

2020-05-03

2020-05-02

HS avatar

Hi Everyone! I would love experts advice on scaling issues I’m having with a Monolith Application

The stack is Ruby, Mysql, Redis, RabbitMQ, and the webserver being used is Passenger, Scaling every other components (mysql, redis, rabbitmq) is not a problem, all of them are scaling awesomely, However, the application server is performing woefully, and the problem is with Passenger, the ruby app is a Legacy codebase in which every component of the application is bundle together in one place, including the mobile app graphql endpoints, and 20+ daemons that needs to run for the app to work.

I have tried scaling horizontally a couple of times which was not effective because I realized that the daemons works in a way that they can’t run concurrently on multiple servers, so that means I had to resolve to scaling vertically, now during peak periods, the server resources are not really used up, but the server keeps crashing, and the major reason is because Passenger(Ruby webserver) couldn’t handle some certain number of connections at once, so after thoroughly going through the passenger doc, I realized that there is a way to increase the maximum number of connection per process, this has been the temporary tweak that is keeping the server up and running for now.

However, We will be starting a campaign Next week, and we are expecting to have x5 the traffic requests we get during peak times, and I need advice on other approaches I can take to Autoscaling my Infrastructure

PS: 1. We are looking to re-architect the entire system by decoupling some services 2. We still run on Ruby 2.2, upgrading it will break our codebase (which is why we need to do 1 above)

I know this community consists of professionals that have handle way larger systems at scale, and I will appreciate everyone’s input on this

Thank you

Gemini avatar
Gemini

Hi @HS a couple thoughts to throw out there that might stir some ideas for you, the goal being to reduce, if possible, direct connections to your lone server ….

• put some cache server in front of it…

• add a CDN and/or WAF service e.g. cloudflare, akamai, others…

:--1:1
Gemini avatar
Gemini
How does Cloudflare work? attachment image

Learn how Cloudflare enhances a customer’s security and performance and the steps to get started with Cloudflare.OverviewMore than just Content Delivery (CDN) services, customers rely on Cloudflare…

:--1:1
Gemini avatar
Gemini
"Reduce the load on your origin infrastructure, especially during peak demand periods. Make your content always accessible using failover and origin health detection."

https://www.akamai.com/us/en/products/performance/web-performance-optimization.jsp#benefits

Ion Web Performance Optimization | Akamai attachment image

Akamai Ion is an integrated suite of mobile delivery, acceleration and situational performance technologies to deliver highly available, secure and scalable experiences to meet the new challenges of always-connected audiences.

maarten avatar
maarten

@HS Good luck on that, it doesn’t sound easy. If you can’t scale horizontally you can’t scale really. I found it really hard to scale ruby vertically in a way as I was facing massive memory leaks everywhere, especially with big monoliths. Also I think this is not a single mans Devops task, make sure a Sr. developer has hours allocated to help you.

  1. To repeat myself, scaling vertically is hard but FWIW you can go to the RUBY discord chat https://discord.gg/KbDHDez and I’m sure a few enthusiasts can help you.
  2. What @ says, caching is important, however, this is not something trivial and not always done in a week ahead especially when you are dealing with users and their sessions.
  3. Testing is important, if it is possible for you to duplicate the whole production enviornment into a second environment, then you can actually benchmark thoroughly, if this takes you too long, skip it.
  4. “However, the application server is performing woefully, and the problem is with Passenger, the ruby app is a Legacy codebase in which every component of the application is bundle together in one place, including the mobile app graphql endpoints, and 20+ daemons that needs to run for the app to work.” Here I think is where the opportunities are. If you can split off the ruby application from those daemons you might be able to scale the ruby application horizontally again. Depending what those daemons use for communication this might not work..
  5. Identify those daemons and configure them to listen on a tcp/udp port reachable from a different server, also firewalling…
  6. Create 2 or 3 new ruby app servers with none of those 20 daemons running, and configure the ruby apps to connect to the daemons of the earlier configured app server.
  7. If benchmarking shows one of the daemons is having performance issues, see if the daemon could be isolated to it’s own server or if it can scale to a group of servers.
  8. If on AWS and if having latency issues, you can use a single placment group for everything.
Join the Ruby Discord Server! attachment image

Check out the Ruby community on Discord - hang out with 1,285 other members and enjoy free voice and text chat.

:--1:2
HS avatar

Thanks for that @, I have caching setup on cloudflare and WAF rules on both AWS and Cloudflare and I have to say those are some of the things still keeping us alive for now

and also setting failover is hard to do with Cloudflare compared to AWS Route53 failover feature and since we migrated our DNS entirely (because we thought cloudflare can only work effectively in fighting DDOS for us if we completely handover our dns to clodflare to manage) to Cloudflare, we lost that aws failover feature

HS avatar

Thanjks @maarten True, I am currently working with a Senoir Engineer with some of these

on 4. those are one of the next steps we want to work on, by separating the daemons to run a service on their own

I want to also try and benchmark our entire app infrastructure, can you recommend any tool I can use for benchmarking?

Thanks, will also checkout the discord channel

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

@HS this would also be a great question for #office-hours. Join us on Wednesday 11:30am PST. There’ll be ~20 experienced devops on the call to answer questions.

HS avatar

Awesome! Will do

:100:1
HS avatar

Thanks @Erik Osterman (Cloud Posse)

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

btw, you can register here for the invite: https://cloudposse.com/office-hours/

HS avatar

I have ben a member since and I have attended a couple of #office-hours sessions too, I didn’t know why I didnt think about bringing it up in our sessions, but will def. do now

SweetOps avatar
SweetOps
07:00:19 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @

Good to have you here =)

:--1:1
Gemini avatar
Gemini

Thank you @Bot

2020-05-01

SweetOps avatar
SweetOps
07:00:07 PM

Hey everyone, give a warm welcome to our newest members!

  • @
  • @
  • @
  • @

Good to have you here =)

rms1000watt avatar
rms1000watt

I actually don’t know where else to ask the question. Does anyone have a preference of a connection pooler for postgres over:

• pgbouncer vs. anything else?

    keyboard_arrow_up