Hi @HS a couple thoughts to throw out there that might stir some ideas for you, the goal being to reduce, if possible, direct connections to your lone server ….

• put some cache server in front of it…

• add a CDN and/or WAF service e.g. cloudflare, akamai, others…

https://support.cloudflare.com/hc/en-us/articles/205177068-How-does-Cloudflare-work-

"Reduce the load on your origin infrastructure, especially during peak demand periods. Make your content always accessible using failover and origin health detection."

https://www.akamai.com/us/en/products/performance/web-performance-optimization.jsp#benefits

@HS Good luck on that, it doesn’t sound easy. If you can’t scale horizontally you can’t scale really. I found it really hard to scale ruby vertically in a way as I was facing massive memory leaks everywhere, especially with big monoliths. Also I think this is not a single mans Devops task, make sure a Sr. developer has hours allocated to help you.

1. To repeat myself, scaling vertically is hard but FWIW you can go to the RUBY discord chat https://discord.gg/KbDHDez and I’m sure a few enthusiasts can help you.
2. What @Gemini says, caching is important, however, this is not something trivial and not always done in a week ahead especially when you are dealing with users and their sessions.
3. Testing is important, if it is possible for you to duplicate the whole production enviornment into a second environment, then you can actually benchmark thoroughly, if this takes you too long, skip it.
4. “However, the application server is performing woefully, and the problem is with Passenger, the ruby app is a Legacy codebase in which every component of the application is bundle together in one place, including the mobile app graphql endpoints, and 20+ daemons that needs to run for the app to work.” Here I think is where the opportunities are. If you can split off the ruby application from those daemons you might be able to scale the ruby application horizontally again. Depending what those daemons use for communication this might not work..
5. Identify those daemons and configure them to listen on a tcp/udp port reachable from a different server, also firewalling…
6. Create 2 or 3 new ruby app servers with none of those 20 daemons running, and configure the ruby apps to connect to the daemons of the earlier configured app server.
7. If benchmarking shows one of the daemons is having performance issues, see if the daemon could be isolated to it’s own server or if it can scale to a group of servers.
8. If on AWS and if having latency issues, you can use a single placment group for everything.

Thanks for that @Gemini, I have caching setup on cloudflare and WAF rules on both AWS and Cloudflare and I have to say those are some of the things still keeping us alive for now

and also setting failover is hard to do with Cloudflare compared to AWS Route53 failover feature and since we migrated our DNS entirely (because we thought cloudflare can only work effectively in fighting DDOS for us if we completely handover our dns to clodflare to manage) to Cloudflare, we lost that aws failover feature

Thanjks @maarten True, I am currently working with a Senoir Engineer with some of these

on 4. those are one of the next steps we want to work on, by separating the daemons to run a service on their own

I want to also try and benchmark our entire app infrastructure, can you recommend any tool I can use for benchmarking?

Thanks, will also checkout the discord channel

@HS this would also be a great question for #office-hours. Join us on Wednesday 11:30am PST. There’ll be ~20 experienced devops on the call to answer questions.

Awesome! Will do

Thanks @Erik Osterman (Cloud Posse)

btw, you can register here for the invite: https://cloudposse.com/office-hours/

I have ben a member since and I have attended a couple of #office-hours sessions too, I didn’t know why I didnt think about bringing it up in our sessions, but will def. do now

Thank you @Bot

@mfridh — From your description, I’m not sure this fits, but because your mention of tfenv I figure this is worth mentioning: https://asdf-vm.com/#/

also check out #variant

https://github.com/mumoshu/variant2

Though after re-reading your post
trying to see if there’s anything out there already before I reinvent the wheel a 5th time. Variant wouldn’t give you this out of the box, but let you declaratively describe how you want to do it.

Variant works with mod to abstractly define dependencies. It can then automatically update those. It’s also designed to work in a gitops-fashion where it can PR the changes.

https://github.com/variantdev/mod

Thanks! Will check these out!

asdf-vm is pretty sweet

https://github.com/bitfield/script maybe?

@Erik Osterman (Cloud Posse) mod - thanks! Remember seeing this way back and thinking I had a good use for it…

@joshmyers script looks interesting!! Could be a gateway drug to more Go :).

I think I now know better how to express what I was after.

tfenv has a terraform shim, a script which when run, detects which version of terraform you want, and downloads that automatically and then just runs. No need to apt install / scoop install / brew install ..

So what I’m looking for is whether there are decent helpers out there already to allow you to “mass-shim” your environment to make the tools download on-demand only instead of baking everything inside a 2GiB tools image .

Am already doing it with a couple of curl-based shell script shims. But they’re one-offs so far.

Build it into a Make dependency?

That + some shellscript could probably be a good framework actually, yes.

Variant is a good use-case too. Likely just more work, but could have a more structured and streamlined approach for the end user.

yes, the good thing about variant is that instead of having a whole bunch of loose tools available which you forget which was for what, having one single entrypoint, with possible informational output could help.

like say you provide a “tools-cli” docker container for example…

Yeah. I like the Variant approach. I’ve been wanting to use that tool since I learned about it.. I may try that out on my next client project to try to simplify client DevOPs on-boarding.

Though ya, you could also build a “tools” docker image and then use whalebrew like you mentioned. That approach with mod would be pretty cool too.

asdf with direnv

there are those very basic requirements but if you are doing this from a local workstation then you are going to be using some base tooling anyway

I need to figure out how to get variant to work properly for external dependencies, simply haven’t given it enough attention yet (but i bet it works well though)

otherwise, I wrote a wrapper script for cloudposse’s package that I then pull in via makefiles to handle all that

And if you go the tools image approach, have a look at https://github.com/EnvCLI/EnvCLI

this let’s you interact with the tools image as though all the commands were native (but they actually run in docker)

@Igor Rodionov is digging it

@roth.andy have you seen envcli?

No. Sounds cool though

seems cool but what’s with dropping the releases on freaking bintray?

Cool. EnvCLI seems like an even more specifically useful whalebrew

yup!

https://docs.gomplate.ca/functions/coll/#coll-has

coll.Has or just has will help with that

Man, the author himself! I’m humbled good sir. Thanks for your help. I half expected no answers

I have a keyword set for gomplate in Slack, and I happened to notice

my lucky day indeed then. It does not need to be said I think but I’m a huge fan of your work

well thanks!

I appreciate the encouragement! it’s hard to stay motivated, but comments like that keep me going!

I’ll thank you by continuing to spread the word of your work like a good aspiring go disciple :D

@btai is another new fan we just turned on to gomplate!

Erik speaks the truth!

woohoo!

there’s a totally unfounded rumour that a new gomplate release is coming out soon… but I have no idea where I heard that

I’ll just use your own personal emoji for that rumor

https://github.com/joke2k/faker/

Thanks a lot @conzymaher This really helps, I just did a poc with it snd it worked as I wanted

@Erick yo! nice to see you stopped by

Thanks! My team loves SweetOps, so I made room in my left rail for another channel

we are on celery 4. have been on it for some time. We had to migrate a while back. Can’t remember from what version but it addressed some issues we were all of sudden seeing. We migrated all at once and just did a lot of testing. We have been using SQS and we like it. May eventually moving everything to SQS.

that sounds awesome. what was your upgrade strategy? how many tasks did you have? we have around 500 so we need to be as careful as possible

We have around 20 recurring tasks but all super critical. We took a very iterative approach and ran it in many different test env with prod db clones. Deciding when to do this is also critical for us since we didn’t want to disrupt any tasks half way. You are at a scale much larger than ours so I am not sure if our experience is any helpful to you. Another QC strategy we took, which was manual but helpful, was to trigger these events directly, rather than waiting for it to happen at the scheduled time.

we did celery 3 to celery 4 a while back. basically it boiled down to changes in the django settings (in celery4 everything is prefixed with CELERY_ and there are some name chanages) and adding namespace kwarg to celery.config_from_object

Do you have any regulatory requirements you have to be compliant with? Stuff such as HIPAA or DFARS?

not sure about those, but probably want to be PCI and GDPR

I should add, all our “data” is cloud-based (AWS/Gsuite), so the idea is lock those down and not worry about the client machines

If using mac Jamf is a great solution

Just to implement some sane defaults. Encryption enforced with an Institutional Recovery Key. Screen lock after 5 mins etc

That simple thing will at least save you from the “Laptop left on the bus” scenario

At the same time would I like the company to have these controls on a BYO device? No.. If the company is serious about security and implementing some controls there is no choice but to supply the equipment IMO

Using a personal machine for business has many risks. What if you accidentally use the wrong AWS credentials? What if an employee installs a package for a personal project that exfiltrates their keys etc etc. Using a personal machine for business is crossing the streams and adds a huge amount of attack surface area

Yeah I’ve considered that. The strategy currently is to NOT hand out any access keys, and only use SAML credentials (w/2factor) to gain temporary access via KMS… Again, the idea we have is to provide enough security into AWS that it shouldn’t matter what device you’re using.. but we still wonder if giving up that control is worth it or not..

As a small company, we’d rather not have to worry about managing the equipment for employees, but we realize it might be a necessary evil…

I’m hoping to get some examples of where BYOD works well for companies and how

have a look at the principles of Beyond Corp

https://cloud.google.com/beyondcorp

like don’t try and go all in on it straight away - but if you make access control decisions with some of these principles in mind then you should end up in a pretty good place

the key thing i would say is don’t try to boil the ocean at the start - but make any new decisions the right way

Awesome information Chris! Thanks. I guess I’m looking to answer some issues like how to start or enforce a BYOD policy? Do companies have problems asking employees to use their own devices? Or do they end up just purchasing a laptop for the employee at hire like a hiring bonus to make sure they have a “good enough” workstation? And at that point, is it just better to say it is the company’s laptop in case they end up leaving a few days later etc?

it really depends - i’ve seen companies do both - generally i’d prefer to be asked as an employee how they’d like to work

i have a work laptop that was provided by work - but working from home i’d much rather use my own system as it’s much more powerful than i’d expect work to provide

Thats the same as me at my previous job… I worked at my desk pc at home

I’m trying to factor in the lowest denominator.. those who don’t have any laptop or pc at home at hire.. we don’t want to create an HR nightmare

So what would a company do for them versus someone who says they’ll be happy to use their own? And how to make it “fair”?

don’t worry about “fair” just focus on “comfortable for everyone”

what one person values may not be the same as another

and if you give them both the compromise no one is happy

i’d generally figure out a spec that you find works and offer it to people if they want it - otherwise allow byod if you want to do that

the other option is to set a budget and allow people to decide what they want - this doesn’t scale in the late game though

it can be a good perk for early talent though

Yeah it seems different for a 10 person company from a 100 person company

it’s very different and it’ll surprise you the things that become important as scale increases

Maybe thats the key…

my main advice would to be to just focus on the goals at each stage rather than the strategies because you’ll need to keep re-making decisions for different scales

as you scale the cost/benefit equation changes

like centralised management of devices is definitely overkill for 10 users - but for 1000 maybe that becomes worth while

the trick is to figure out when to change things

100? 500? 1000? who knows

Yeah, but our thought is the early decisions really affects culture later, which is why we’re sweating it now, but I see your point

culture is cultivated and it will change as well

it’s not a thing that you set in place and it stays that way

it’s kind of like a stew that you keep adding things to

don’t sweat it now - just keep an eye on it as you cook it

Yeah I can see that. I appreciate the perspective Chris. Thanks. I’ll read up on your links.

figure out and make sure you can articulate your values early though - that’ll drive the culture. what’s really important to your founding group as people - as that will drive your culture more than any organised effort

no worries - i really love this stuff and happy to share

Yeah I love it too. Being in a startup has the advantage of being able to do things the way you want, but we don’t want to reinvent the wheel on what works. We’ve got most things figured out for now, but this has been going back and forth for a while, so its good to get some additional perspective.

Thanks I wasnt able to join yesterday as I had a problem with Zoom but i watched a recorded session. It is nice to be here :)

Awesome! Glad you were able to catch up on the recorded session.

We also are syndicating them here: https://podcast.cloudposse.com

ya, we’re been pretty good about that. also, a convention @sheldonh started that I like is prefixing questions with [thread] as a hint to others to start a thread.

even more, i just appreciate that it seems to be part of the culture here. folks see messages with threads in pretty much every channel when they join, and they see others using threads to respond to new posts, and generally they seem to follow suit quickly. it’s glorious, even without a convention

Agreed — I seriously appreciate threads and other slacks bog down my time when they don’t use them. I’m definitely going to steal @sheldonh’s [thread] idea going forward to push folks in the right direction. Thanks for mentioning @Erik Osterman (Cloud Posse)!

I was in another community and got called out for using threads. Turns out they have major accessibility issues for screen readers. First time I’d ever heard of that. Pretty crazy that it hasn’t been dealt with. Unless somebody specifically mentions they’re having trouble reading it I’m not sure it makes sense to stop using them because context is lost for everybody else anyway. Just thought I’d mention that was the first time I’d heard anything negative about threads.

I have a PowerShell extension on a slack bot. It tracks channel postings and if unthreaded messages are posted more than say a couple times in 15 minutes it politely suggests using threads to help keep the noise down. I’m a big believer in keeping context

Protip. When you post thread in the title I try to only post the main topic and maybe a mention, and then immediately post the main bulk of content in the thread. I found that that helps ensure the conversation immediately begins in the thread as somebody clicks to expand it. Just a little thing that seemed to help

Good tips @sheldonh — Thanks! Will mention introducing the slack bot idea to one of my clients for sure — they could use it!

Clever hack!

New to golang. Is this advanced or would it be useful to someone beginning too?

It is a good beginning to work with Golang’s advanced features, for developing programming language tools.

I would say that #terraform would be a good place to start. Just expose your issue with what your trying to accomplish, what is happening that is unexpected, what you tried so far to fix it, and any specific errors messages your getting. and then be patient.

I did that already .. but haven’t had any reply

haha, sounds like another topic for #office-hours ?

sure thing! just link me to ‘em and I’ll add

@Jeremy G (Cloud Posse)

@gyoza Not sure what you mean by “remove” gravatar images. Do you mean to set disable_gravatar = true in .ini?

probably

Fixed it. I was running flask with celery and had a route to add tasks using flask but didnt run the celery worker so it wasnt taking the tasks. Doh

I’ve seen a couple of examples online [example 1] [example 2] but am looking for something with more evidence of being production-grade

Have a look at ossec-hids as an alternative. It implements a controller/agent model.

That’s great! I’ll share this in #office-hours announcement

Cool, thx Erik

Unfortunatelly Terragrunt can not manage 1. nor 2. , good example of using projects in monorepo is atlantis