Just a brief update and a thank you to all of our supporters.
I am still working on launching RDR2 RP tracking (and other changes, see previous blog for some more info). I have currently gotten sidetracked with rewriting the clip processing part of the backend java app. Despite all my best efforts of optomisiation on my home pc I was only able to process 100k clips an hour without noticeable impact on my pc / the test website. I had completely rewritten everything I could to eke out better performance when I decided to check the root of the issue. Turns out my local MySQL settings were pretty awful, only allocating 8mb for the inno db buffer. (How much ram of a specific table type could use as a buffer at a time).
This obviously was very minuscule amount but I had been suffering from ram issues for awhile, I have an old i7 920 pc and it had 6 gigs of ram total. I bumped it up to 80mb, this helped a fair bit but obviously was still rather low. I decided to finally bite the bullet and buy some more ram, it is something I have been meaning to do since I originally built the pc. I originally built it around 10 years ago and cheaped out on the ram with the intent to replace it / add more later. I never thought it would be 10 years later however.
Regardless, I bought 12 gigs of ram and installed it today. I then unleashed MySQL and it is currently eating 6gb of my ram but is it blazingly fast, I went from being able to process 100k clips an hour to 200k clips a minute. Not only that I was able to use my pc without it randomly freezing and crawling to a snails pace. Definitely something I should have done along time ago but money and other factors delayed it. I have spent today playing with MySQL settings and the performance is night and day. Everything is beyond insanely fast, I had to actually whip up a quick request throttler as Twitch only allows around 800 requests a minute and the backend end was doing 800 requests every 20 seconds.
To the point, I am still working on things and preparing a deployment plan as there is a fair amount of DB changes. When we do launch the new update there will probably be at least 4-5 hours of downtime though I might be able to just disable the clip generation stuff and do the bulk of that work while the site is available with limited functionality. We shall see.I hope to be doing the test deployment of the new version this week. Also from what I have learned from this experience I will be changing some things on the server to help improve the performance of the live MySQL server.
A thanks to all of our supporters
Without our supporters I would not have been able to afford that RAM, I used what was left in the site paypal to help pay for it. So a thank you to everyone that has given to the site who have enabled me to finally be able to work on the site again without insane performance issues. The database had gotten so large it was basically crippling my machine to the point where I wouldn't even use discord as I couldn't spare the ram. And it isn't just the amount of clips that can be processed an hour, so many aspects of the site (server info pages, sometimes streamer profiles etc) were basically just unusable on my dev pc.
TL;DR: I am re-energised to work on the site and have been doing so, the update is coming and hopefully soon. Still working on making things faster, it never ends. Also a thank you to everyone who supports the site, it is rare we get much feedback / love / support but it really means a lot to me and the rest of the staff that people appreciate and use the site.
Hope you are all enjoying the site.
ScottishRambler.