Resetting RaidStats Highscore/Ranking lists(?)

Reset RaidStats Highscore/Ranking lists? Will either happen 22/3 or 29/3.

Yes
49
56%
No
34
39%
Unsure
5
6%
 
Total votes : 88

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Dessel » 21 Mar 2016, 03:14

Sounds to me like auto-deletion of anything older than (a suitable number of) weeks/months is the most sustainable approach for the long run.
Rather lose some history than have the whole site go fubar because people want to sit and jack off to old raids.
Dessel
 
Posts: 1
Joined: 06 Aug 2015, 14:46

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Perr » 21 Mar 2016, 18:14

My suggestion would be to not record anything older than current content raids, and if that's still too much, reset the data every few months or so.
Perr
 
Posts: 1
Joined: 02 Dec 2015, 06:05

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Dilatazu » 23 Mar 2016, 00:49

Thanks for all the feedback everyone.

Even though there currently is more "yes" than "no" votes, the discussions this topic caused and the amount of "no" votes tells me that people really do not want to see a full database reset.
And therefor i will not do any database reset at this time.

Nuking everything older than X months and making it automatic seems to be the most popular opinion and also the one i would prefer. However it is not as quick and easy to do as it sounds. We will see what happens later.

Robustious wrote:How hard it is to master the coding of the raidstats? Would you also mind telling a magic number that you would require in order to keep this up? Like most end-game guilds would be able to donate that magic number for you for lets say y amount of months work. As i can totally understand your though regarding this.

I have tried to get people involved and so far i am still the only contributor. It is hard to get people involved and the codebase is very big and complicated. I made all the code available on github to try make it easier for people to get involved but no luck so far.

I think the way donations have been going last halfyear has been pretty good and it has now reached a point where it has supported all the server costs thus far.
The problem is that the website requires so much work just to have it running. This is partly my fault due to the way i coded it but also innevitable with a project this size. It is way harder to run a project like this than many think. The amount of traffic and uploaded data last halfyear has been insane and puts a lot of stress on the server. Stress the server couldnt handle with the way it was coded.

The service was designed to be extremely fast response times as that is what i valued the most back when i started. However the technique didnt scale when the amount of data reached the levels it is today(more than 200GB currently). Also my ambition with RaidStats being super good at filtering out unrealistic damage and fairly comparing raids between guilds(and even realms!) required a lot of work and nowadays is just a hassle everytime some edge case makes it break.

To get back to the point. The work required to just keep the service up is what is most annoying.
A recent example is on sunday night at around 22:00 i realised the service was not working. I looked into the issue and apparently the server had experienced its first "bluescreen of death". After i got it started up again i saw that there was some major issues with the entire OS and i had to work for around 4 hours and luckily i was able to get everything back to working normally again. On top of ruining the rest of the nights plans working on my other project, this made me only get 4 hours sleep for real work the next day.

Now ofcourse if i was compensated enough for the time i spent i would have no issues with it. But it is just not realistic asking for that kind of money in donations.
I have to see this as a fun hobby project, nothing else makes sense for me. Problem is this hobby project is demanding me to do work i dont enjoy and i cant move on.
Dilatazu
Site Admin
 
Posts: 759
Joined: 24 Feb 2014, 17:19
Location: Sweden

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Dilatazu » 23 Mar 2016, 01:09

As an extra note. I sometimes see some people talking about RealmPlayers armory data.
The stress on the service and issues with a lot of data is 99% because of RaidStats. No changes to RealmPlayers armory will make any difference. If i were to shutdown RaidStats section completely the service would probably be extremely stable and not require any work for a very long time.

RealmPlayers data is around total 1GB.
RaidStats is around 200GB(and it increases around 25GB every month right now).
Dilatazu
Site Admin
 
Posts: 759
Joined: 24 Feb 2014, 17:19
Location: Sweden

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Dzehn » 23 Mar 2016, 03:23

Hi

First of all, as stated by a lot of other people, Thanks for taking the time out of your life to give us this amazing tool.

Secondly, my five cents would be that all data older then 3 months gets removed. A few of the people posting in this thread was against that, or wanted season "hall-of-fames" -

But keeping the database "new" with old data getting removed ensures that people keep pushing themselves and their guilds to stay on top of the dps/heal meters.

I personally know absolutly nothing of coding, So on that part I am worthless.

The But would be that IF it is possible to setup, a self-deleting service would be the optimal

-Reduces overall stress on the server
-Keeps the Best dps/Hps lists up to date with current players
-reduces the chances of the server going haywire becuase of overload. (if you say 200GB is crashing it, and every month is ~25-30GB of data, 3month would mean ~80-100GB of data steadily on the server)

I, for one, would clap my hands if you say you need X in donations for your paycheck for 2 weekends while you set "something-like-this" up.

I cannot see the downside in doing this, and would applaud you greatly for finding your last ounce of strength, to set a self deleting service up.

//Zehn
Dzehn
 
Posts: 6
Joined: 19 Nov 2015, 01:09

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Istaria » 24 Mar 2016, 16:03

Not sure how the typical RealmPlayers user feels and how they use the site, but I personally use Armory at least 10 (probably 100) times more often than RaidStats and it has much more value to me. I can absolutely live without RaidStats being up 100% of the time but when I need to check items from Armory and it's down despite being a lightweight solution, that's kind of a bad thing. If it would be possible to separate the two even further so that RaidStats does not bring down Armory with it, that'd be awesome.

My random 2 cents are "purge everything that is not from this or last month".
Istaria
 
Posts: 5
Joined: 10 Jul 2015, 00:07

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby stivil » 24 Mar 2016, 21:15

Cant you safe all 5 last raids each from every active player and delete all the rest? and somehow keep speedclear times?
and make it so it automatically deletes the oldest raid (if at 6 BWL IDs for example) so we only keep 5 IDs to get average ratings etc ?
stivil
 
Posts: 1
Joined: 08 Jan 2016, 02:40

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Tinkertown » 25 Apr 2016, 16:16

I was wondering, since the Nost shutdown all the old logs are gone and they were the only record of our time together. I was wondering if they are lost forever. It seemed from this thread that the decition was not to purge them. Did something change?
Tinkertown
 
Posts: 2
Joined: 25 Apr 2016, 16:13

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Dilatazu » 26 Apr 2016, 00:38

Tinkertown wrote:I was wondering, since the Nost shutdown all the old logs are gone and they were the only record of our time together. I was wondering if they are lost forever. It seemed from this thread that the decition was not to purge them. Did something change?


Hey,

Sorry for not making it clear earlier. Since people did not want to see the data resetted i chose to just keep it running. As i said in the opening post the alternative to clearing the lists was: "Or should we wait for the day it blows up and i have no choice but to do it either way?". Well TL;DR: It blew up and i basically had no choice.

I knew the server had been struggling for a long time since the boost of users when Nostalrius posted link on homepage to RealmPlayers but also when it reached ZG and BWL content(lots of extra raids being uploaded). So i knew the best thing would be to just reset the lists since the whole service was slow and unstable because of it. I thought that if nothing was done it would eventually become so slow that everything stopped working completely. I thought atleast maybe it had another few months or so.

However the data was already at such extreme levels and at end of months it usually peaks a little bit extra before it separates data into different areas. So what happened was the server just completely shutdown just 22 hours after i made the post. This was the most brutal shutdown i had seen before, a "Bluescreen of Death" and after i restarted the server i was met with an operating system which refused to write files to the harddrive for some reason. A problem i have never seen occur before. After i let the server rest for a bit i removed some log files that had built up in some folders, turned out there was tons and i thought it might help the filesystem if i move them somewhere else. I was able to get everything working again after 4 hours. It was truely a sight of relief when it did.

After this i thought it maybe became more stable since i removed all the log files that had built up and cleaning in general etc. So i thought ok lets just keep it going, since people clearly was opposed to resetting the data.

But then 9 days after i made the original post. The server shutdown once again, this time even more brutal, both the servers(forum and website(hosted in a ESXi server VMWare environment)) completely stopped responding. The host ESXi server stopped responding which has pretty much never happened before and i ended up eventually having to cut the power as the only way to actually get the computer to restart. This is not a good way to restart a computer and i had a very bad feeling that this may be the end of RealmPlayers/RaidStats as i was almost sure something was going to be completely broken. I thought to myself, if i can even get this working again i will reset the database. It is better to have raidstats and realmplayers with a resetted database than no realmplayers/raidstats at all.

Luckily i was able to get everything working again and i decided to stick with what i thought earlier, it was just not worth the risk. So i resetted the summary database and after that everything seemed to be pretty stable finally again.

Anyways back to your question: All the data is still saved, actually the database that was reset is just generated using data from all the other databases. It would just take forever to generate it again(we are talking days or possibly weeks of procesisng power(ofcourse could be speed up a bit if optimized)). But i even saved the actual data so it is possible to revert the website back to exactly like it was back before the reset.

Only problem is that then the newer data would be gone. Best would be merging or treating old database differently(generating some static pages). However all this is work that requires some real time to be put into it.
Dilatazu
Site Admin
 
Posts: 759
Joined: 24 Feb 2014, 17:19
Location: Sweden

Re: Resetting RaidStats Highscore/Ranking lists(?)

Postby Tinkertown » 26 Apr 2016, 23:53

Thanks for the quick response!

I'm super happy that it potentially isn't permanently lost, even though getting it back will be hard. If you can find a way to get at least some of it back in some fashion that would be truly awesome. Regardless, thanks for all your great work. It is truly appreciated, always.
Tinkertown
 
Posts: 2
Joined: 25 Apr 2016, 16:13

PreviousNext

Return to General Discussions

Who is online

Users browsing this forum: No registered users and 1 guest

cron