#random (2020-07)

Non-work banter and water cooler conversation

A place for non-work-related flimflam, faffing, hodge-podge or jibber-jabber you’d prefer to keep out of more focused work-related channels. Archive: https://archive.sweetops.com/random/

2020-07-31

roth.andy avatar
roth.andy

I’d like to put out a poll (completely anonymous, even I won’t know who answered what) on Employee Net Promoter Score. Reason for doing so here is to weigh my company’s score against a more broad score across many companies. As an encouragement to vote/bribe, here’s a cookie

roth.andy avatar
roth.andy
03:13:50 PM

cookie

roth.andy avatar
roth.andy

lol this was the first gif that popped up with /giphy, but It’s not really what I was shooting for… Kinda grungy

roth.andy avatar
roth.andy
03:21:49 PM

cookie

 avatar
03:15:48 PM

On a scale of 1-10, how likely are you to recommend working at your company to a friend?

roth.andy avatar
roth.andy

We can call it the “DevOps Industry NPS”

2020-07-29

rms1000watt avatar
rms1000watt

Shameless plug: Here’s my pops jamming and singing on guitar. Proud of him!! https://youtu.be/8HlNv39cluE

:--1:1

2020-07-27

adefemi171 avatar
adefemi171
Containerization |Part 2: From LXD to Kubernetes attachment image

In my last post (which was the part 1) where I introduced containerization and the types of containers I know, I said I will be writing on…

2020-07-23

Zaven avatar
Zaven

Does the archive.sweetops.com site have better search than native slack? or is the idea just to have google indexing?

loren avatar
loren

Think it was more that the free slack has limited history to 10,000 messages. Or did, until recently, but for the time being this slack workspace has free unlimited history…

1
Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Our team is temporarily upgraded for another month or so.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

Then it will go back to free plan and 10k messages.

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)

The slack archive contains history from the beginning of time (literally) even when our plan reverts to the free tier.

2020-07-21

sheldonh avatar
sheldonh

Anyone here generated their own docsets For Dash?

2020-07-20

Maciek Strömich avatar
Maciek Strömich

Any mysql admins with tables containing more than 1M rows here? We have a table that will grow 100k rows per day (already at 5M rows) and I wonder what should I start considering.

James Huffman avatar
James Huffman

what table engine are you using?

roth.andy avatar
roth.andy
Can MySQL reasonably perform queries on billions of rows?

I am planning on storing scans from a mass spectrometer in a MySQL database and would like to know whether storing and analyzing this amount of data is remotely feasible. I know performance varies …

roth.andy avatar
roth.andy

Things I gleaned from a quick read:

  1. Being able to normalize the data is huge
  2. The disk read time starts playing a major role when your DB is monolithic. This is starting to get into big data needs using distributed data stores and projects like Hadoop
  3. Spending time to carefully design the schema will be very valuable later on
Maciek Strömich avatar
Maciek Strömich

@James Huffman all tables are using innodb

Maciek Strömich avatar
Maciek Strömich

it’s fine because selects are faster and we have a batch of 100k writes happening within few hours from an import task and ~17k writes happening through out the day from user usage.

Maciek Strömich avatar
Maciek Strömich

table is rather simple 10 columns and no foreign keys and a autoincrement pk

Maciek Strömich avatar
Maciek Strömich

so the selects are also fine

Maciek Strömich avatar
Maciek Strömich

especially that we don’t do joins

vFondevilla avatar
vFondevilla

Do you need all data from the table to be quickly accessed? Or the query & fetch time is not so critical?

Maciek Strömich avatar
Maciek Strömich

it’s fine below few secs to retrieve the data

vFondevilla avatar
vFondevilla

awesome, in that case I think that it should be pretty straightforward while you have the data indexed

vFondevilla avatar
vFondevilla

We have some tables with more than 10M rows and the data need to be quickly accessed, so our main constraint is the innodb buffer pool, because retrieving data with a join from 2 tables with 3+M and 10+M takes about 40 seconds only from fetching from the storage.

Maciek Strömich avatar
Maciek Strömich

yeah with the current plan to ingest 100k new rows to this table per day we will hit 10M rows in less than 2 months from today

James Huffman avatar
James Huffman

things to consider:

  1. is your data well indexed?
  2. do you have enough RAM to keep a lot of the indexes in memory?
  3. is your data time-sensitive and do you actually need to hold all of that data in the same table/DB instance at all times? or can you archive some of it after it reaches a certain age?
vFondevilla avatar
vFondevilla

have you thought about doing data sharding?

:--1:1
Maciek Strömich avatar
Maciek Strömich

@James Huffman sadly this data can’t be cleaned because of the purpose. I need to take a closer look at the memory usage. maybe the innodb buffers are too low as well.

I mean it’s currently running on db.t3.small and the avg app time spent in the db is around 40ms (including connection time) and rises to ~250ms for use operations when the import job is running

Maciek Strömich avatar
Maciek Strömich

@vFondevilla that’s one of the ideas for the future, but now I want to know where my potential bottlenecks are before I will start thinking about changes to the database architecture

Maciek Strömich avatar
Maciek Strömich

thanks for the input tho. I’ve enough to look at and think about

:--1:1
Issif avatar
Issif

I managed a mysql instance with 5.10^9 rows, with 200-500k inserts/s and peaks at 800Mbit/s for write. Everything was overminded, we needed to use tokudb as engine and data was stored on a fusionIO flask drive. That was insane

2020-07-16

Maciek Strömich avatar
Maciek Strömich
05:38:08 PM

EU’s top court invalidates data-sharing pact with US | News | Al Jazeera https://www.aljazeera.com/news/2020/07/eu-top-court-invalidates-data-sharing-pact-200716091848578.html

EU's top court invalidates data-sharing pact with US attachment image

The ruling could require EU regulators to vet any new transfers due to concerns that the US can snoop on people’s data.

2020-07-13

2020-07-10

zadkiel.aharonian avatar
zadkiel.aharonian

We can’t send email more than 500 miles (2002) - http://web.mit.edu/jemorris/humor/500-miles

4

2020-07-09

Karoline Pauls avatar
Karoline Pauls

Issue: agent: none doesn’t work with post actions Jenkins: by design

To generalise:

Issue: A non-issue in any other mature CI is an issue on Jenkins Jenkins: by design

Erik Osterman (Cloud Posse) avatar
Erik Osterman (Cloud Posse)
How to track and display profile views on GitHub

GitHub has introduced READMEs for profiles that allow to display profile views

:--1:5
Karoline Pauls avatar
Karoline Pauls

That’s a tracking pixel. Gross! :D

How to track and display profile views on GitHub

GitHub has introduced READMEs for profiles that allow to display profile views

2020-07-08

MattyB avatar
MattyB

Anyone else downsizing on their cars? We barely have a need for 1 car so I think I’m going to sell mine and share my wife’s. I don’t see myself going into work for over a year so I’ll save on payments, insurance, and maintenance.

:--1:1
vFondevilla avatar
vFondevilla

Yup. I’m going to return mine to the automaker when the leasing ends. I work and live in Barcelona and I’m barely using it apart from some weekend trips (previously to the corona)

vFondevilla avatar
vFondevilla

I did some calculations and it’s cheaper to rent it 1 or 2 times per month than paying the leasing. So no brainer.

2020-07-07

Karoline Pauls avatar
Karoline Pauls

Sometimes I wonder where the random names for AWS API calls come from.

Please appreciate the maniac at AWS who was there to say:

  • “NO!!!!!!!! WE CANNOT NAME THE API CALL DELETE-IMAGE!!!! WE DO IT IN BATCHES SO IT HAS TO BE BATCH-DELETE-IMAGE!!!!!!”
  • “But that’s the only way to delete an image….”
  • “WHAT DO YOU EVEN KNOW ABOUT ENGINEERING?!”
Issif avatar
Issif

you want other jokes? Date Format are never the same. Depending on which endpoint you call, you have VpcID, VPCID, VpcID

Issif avatar
Issif

you have my total support, I face all the time that annoying ugly patterns

Karoline Pauls avatar
Karoline Pauls

yeah, but different conventions are from lack of coordination (i once had it too at a not-that-bad team actually), peculiar names are most likely from egomaniacs

Karoline Pauls avatar
Karoline Pauls

you’ve got my support for dealing with random capitalisation in response JSONs though

Issif avatar
Issif

and even better, with sdk-go I faced several times answers with missing fields in struct. Not even a nil pointer, just nothing.

Issif avatar
Issif

I was forced to defer + recover

Karoline Pauls avatar
Karoline Pauls

fortunately i only write synchronous code these days

wannafly37 avatar
wannafly37

My favorite is the unknown if an object uses name or id

Issif avatar
Issif

and that’s never name , it’s dbInstanceName , or else

Karoline Pauls avatar
Karoline Pauls

oh yes, names in create calls becoming ids here and there

Karoline Pauls avatar
Karoline Pauls

also the fact that aws cli writes to stdout but doesn’t take advantage of stdout being a stream and still forcing you to paginate… but that’s a more common problem

ikar avatar
ikar
05:52:09 AM
6
2

2020-07-04

Joe Niland avatar
Joe Niland

https://github.com/augmentable-dev/gitqlite Query git repo contents with SQL

augmentable-dev/gitqlite

Query git repositories with SQL. Uses SQLite virtual tables and go-git - augmentable-dev/gitqlite

    keyboard_arrow_up