#random (2020-07)

you want other jokes? Date Format are never the same. Depending on which endpoint you call, you have VpcID, VPCID, VpcID

you have my total support, I face all the time that annoying ugly patterns

yeah, but different conventions are from lack of coordination (i once had it too at a not-that-bad team actually), peculiar names are most likely from egomaniacs

you’ve got my support for dealing with random capitalisation in response JSONs though

and even better, with sdk-go I faced several times answers with missing fields in struct. Not even a nil pointer, just nothing.

I was forced to defer + recover

fortunately i only write synchronous code these days

My favorite is the unknown if an object uses name or id

and that’s never name , it’s dbInstanceName , or else

oh yes, names in create calls becoming ids here and there

also the fact that aws cli writes to stdout but doesn’t take advantage of stdout being a stream and still forcing you to paginate… but that’s a more common problem

Yup. I’m going to return mine to the automaker when the leasing ends. I work and live in Barcelona and I’m barely using it apart from some weekend trips (previously to the corona)

I did some calculations and it’s cheaper to rent it 1 or 2 times per month than paying the leasing. So no brainer.

That’s a tracking pixel. Gross! :D

what table engine are you using?

https://dba.stackexchange.com/a/20336

Things I gleaned from a quick read:

1. Being able to normalize the data is huge
2. The disk read time starts playing a major role when your DB is monolithic. This is starting to get into big data needs using distributed data stores and projects like Hadoop
3. Spending time to carefully design the schema will be very valuable later on

@James Huffman all tables are using innodb

it’s fine because selects are faster and we have a batch of 100k writes happening within few hours from an import task and ~17k writes happening through out the day from user usage.

table is rather simple 10 columns and no foreign keys and a autoincrement pk

so the selects are also fine

especially that we don’t do joins

Do you need all data from the table to be quickly accessed? Or the query & fetch time is not so critical?

it’s fine below few secs to retrieve the data

awesome, in that case I think that it should be pretty straightforward while you have the data indexed

We have some tables with more than 10M rows and the data need to be quickly accessed, so our main constraint is the innodb buffer pool, because retrieving data with a join from 2 tables with 3+M and 10+M takes about 40 seconds only from fetching from the storage.

yeah with the current plan to ingest 100k new rows to this table per day we will hit 10M rows in less than 2 months from today

things to consider:

1. is your data well indexed?
2. do you have enough RAM to keep a lot of the indexes in memory?
3. is your data time-sensitive and do you actually need to hold all of that data in the same table/DB instance at all times? or can you archive some of it after it reaches a certain age?

have you thought about doing data sharding?

@James Huffman sadly this data can’t be cleaned because of the purpose. I need to take a closer look at the memory usage. maybe the innodb buffers are too low as well.

I mean it’s currently running on db.t3.small and the avg app time spent in the db is around 40ms (including connection time) and rises to ~250ms for use operations when the import job is running

@vFondevilla that’s one of the ideas for the future, but now I want to know where my potential bottlenecks are before I will start thinking about changes to the database architecture

thanks for the input tho. I’ve enough to look at and think about

I managed a mysql instance with 5.10^9 rows, with 200-500k inserts/s and peaks at 800Mbit/s for write. Everything was overminded, we needed to use tokudb as engine and data was stored on a fusionIO flask drive. That was insane

Think it was more that the free slack has limited history to 10,000 messages. Or did, until recently, but for the time being this slack workspace has free unlimited history…

Our team is temporarily upgraded for another month or so.

Then it will go back to free plan and 10k messages.

The slack archive contains history from the beginning of time (literally) even when our plan reverts to the free tier.

lol this was the first gif that popped up with /giphy, but It’s not really what I was shooting for… Kinda grungy

cookie