This looks like a big computation mistake somewhere in the system, hope I'm wrong.

The rating system, for some reason, likes you.

Yes. This was another error which had gone unnoticed somehow. Thanks for spotting it. Those who scored equal to the test median score are affected. For example, you scored 145 in the last sudoku test which happened to be the median score as well. And the system was giving PS as 50 instead of 500. The correct ratings should be up soon.

Thanks for finding the mistake and correcting it. Still there remains a small(?) mistake since I have a sudoku rating of 688 now but should have 681.

My Normalized Scores (NS) given at my profile are correct, i.e. they are the same that I did compute myself.

But then, the rating must be:
(492+837+696+643*.8+721*.8+711*.6)/5.2=681

Since rounding errors are not sufficient to explain the difference of 7 points, there still must be a mistake.

Good luck for finding the error,
Stefan

P.S.: My prorated Score (PS) of the SomethingIsMissing test was always given as 500 at my profile. This proves that you have calculated the PS two times. One time to display it at my profile (PS=500) and one time to calculate my rating (there you used PS=50). Computing two times is always dangerous (if you adapt the rating in the future you will always have to do it twice...) but it might have some practical reasons. A similar mistake based on double computation must still be currently in place.

We are having an issue here - we've 2 different systems
1) Rakesh uses excel to compute all ratings and uploads after every test
2) What you see in profile page is directly computed from the score sheet

There are minor differences in computation in these 2 systems because of which we are seeing this difference.

As a first step, we are working on making sure that 1) and 2) are same.
As a future step, we are planning to get rid of 1) so that we've all the computation at one place. This will take a while because of the complexity involved.

The LMI Puzzle Ratings have been updated after Fillomino Fillia.

The Top 5: Motris, Deu, Uvo, MelloMelon and Nikola

Edited by rakesh_rai 2011-06-21 9:28 PM

The LMI Sudoku Ratings have been updated after L O G I D O K U.

The Top 5: Motris, Nikola, Jaku111, Misko and Deu.

Just in case that it is a still unknown bug:

My rank score (RS) for "Something is missing" has suddenly changed from 469 to 573. That would mean that the number of participants has jumped from 96 to 119,5 (my rank of #52 has not changed).
My rank score (RS) for "X'mas Special" has suddenly changed from 734 to 770. That would mean that the number of participants has jumped from 139 to 161 (my rank of #38 has not changed).

My prorated scores (PS) have not changed.

Stefan

No bug here. Number of participants stays the same.


This step is completed, that is the reason you are seeing these changes.

Probably the calculations are not so obvious from the given formula. For rating calculations, if two players have the same score, they will have the same RS, even if they are actually ranked N and N+1 respectively, in the test. So, the effective formula becomes

RS = (1 - [No of scores > your score] / [No of non zero scores])*1000

Interestingly, the significant variations that you see are because 13 other players scored the same as your score in XMas Special, and 12 other players scored equal to you in Something is Missing.

Thanks for the explanation, rakesh. Now I understand.

In the pdf explaining the new rating system (V2.0) it said:

"RS=[1-(No of players ranked above the player)/(Total players with non zero scores)]*1000"

Actually I find this original formula better than the current implementation because the rank score (RS) should depend on the rank only whereas the prorated score (PS) should depend on the points only. As you have implemented the formula now, two players with the same points get the same PS AND the same RS. With the original formula the player that sent his last answer earlier has the higher rank and a small benefit at his RS and thus at his normalized score (NS).

Of course I am well aware that one could argue that the time when one sends his/her last answer does not need to be very significant. But then, consequently, all players with the same score should be awarded the same rank in the result table. My rank in "Something is Missing" should then be 42 instead of 52. If the scoring system uses the rank 42 it should be shown in all statistics.

In my opinion, the current ranking should remain (i.e. rank 52 for my "Something is Missing") and the RS formula should use this rank. I know that this would take away RS (and NS) points from all players, nobody will increase his score.

Stefan

[edit:] Just to make sure that I am not misinterpreted. Please note that I am very happy with the ranking system and can easily accept everything that you want to implement. All my comments are opinions and suggestions only and in no way intended as negative criticism.

Edited by euklid 2011-06-24 2:28 PM

All your comments are really appreciated.
I think we should do the changes that you noticed. The logic in 'View Profile' page is rather easy to change. I guess Rakesh will take some time to update the excel sheets.

I also agree with the suggestions. This will be a definite improvement in the implementation part of the rating system.

Here I come.
Thank you for all the work you are doing and congratulation for an excellent rating system.
However I feel there is always room for improvement in every system.

I can't express myself in mathematical terms I will try to express my concept qualitatively.

In my opinion the current system can be affected negatively by some individual factors. For example, if a player gets a extremely high score (the case of deu in the Nikoli contest) all the score of the mid-range players will be low. in my case, I scored approximately 50% of his points, so although my score is higher than the median value my PS is relatively low compared to other contests. My RS is higher but this accounts only for 25%. The conclusion is that my NS for this particular contest, in which I believe I played better than my average performance, is in fact lower than my current rating, as opposed to other contests in which perhaps I playes more badly but either there was no such a uncommon performance from a single player or the mechanism of score was less rewarding towards high scorers.

As said I have no particular recipe or formula to express this, just bear this in mind next time you review the system. Of course this calls for a consistent method of scoring and of calculating bonus points, but I fully agree that in certain contest the uncommon way of granting bonuses was the "pepper" in the recipe and the inventiveness of authors must be preserved.

Stefano

This is precisely something Melon had brought up with me after Fillomino test (which had only normal bonus system).
Increasing the weightage for RS is an option, but not sure if that is desirable or if that is the only option.

I mostly agree with your point.

There are two things that the rating system should do here - (1) the normal players should not be heavily affected/penalized, and (2) the exceptional performer should also get something substantial out of it.

So, in the last review, we took two steps to address this - we introduced the median, so that all those who performed above the median get at least a 50% score in the PS. So, in the case of Nikoli Selection, a score of 150 (median score) is getting 500 as PS. The second step was to factor the ranks. If someone is 2nd by a big margin, he still gets a high RS. And when we calculate NS, we initially thought of doing it 50-50 between RS and PS. But after some sample runs, we found that the exceptional performer is not getting enough advantage. So we brought the RS weight down. So, if I take your example, in the earlier system you would have got a NS of about 488. Now you are getting a NS of 672.

We had similar scenarios earlier too - the most recent ones being those of deu in Fillomino Fillia/Puzzle Hybrids, Nikola in Logidoku (and, in the month of June generally) and motris in Twist/Prime Exotica. So we definitely need a well acceptable solution to this aspect.

Options:

(1) Increasing RS weight, as suggested by MellowMelon, is one option. With 50% weight, for example, forcolin will get a NS of ~716 for Nikoli Selection.

(2) Another thing which can be done is to introduce a couple of more "median points". So this will fix the PS of the 75th percentile performer at 750, 50th percentile at 500 and 25th percentile at 250. This way, for example, forcolin will get a NS of ~778 for Nikoli Selection. This method can protect a lot of normal players from the volatilities at the top.

What do you think?

Edited by rakesh_rai 2011-07-12 10:47 PM

Updated LMI Puzzle Ratings after Nikoli Selection (July 2011 LMI puzzle test #1) are now available.

In the current ratings, the RS implementation has been modified as suggested by euklid earlier.

Rakesh
what I think is that you have implemented a very good system.
There may be (small) margin of improvement, just bear my opinion in mind alongside with the one of others; and, if there is enough people which say a certain thing and justify a variation, do it.
But I also believe that you cannot change the mechanism after every contest, so once an equilibrium point has been achieved the system should be left running without major perturbations.
I also noticed that after Nikoli 2 my score has gone UP by about 10 points in spite of the score being lower than the previous average. Probably this happened because the new score has replaced old lower scores and consequently the average has gone up. So the compensation mechanisms in place work well after all.

Thank you again
STefano

Thats correct - since only the last 12 tests are considered for ratings, and you performed better in Nikoli 2 as compared to the one which went out of ratings calculations (EG2).

I think there is another option to eliminate negative effects of exceptional performances on the rankings.

(3) Base the calculation of NS on linear extrapolation between 0 and median point. It means to use the same calculation for all the players even above median.
E.g. median is 50 points and the top two players have 112,5 and 100 points. In the current system they would get NS 1000 and 900 respectively. So the second player would be heavily penalized. My suggestion would be to use the give them NS 1125 and 1000 instead.
So there would be no upper limit on NS and NS would be always calculated in relation to median performance.

The only condition to keep the system fair is that each competition should have no upper limit on points. So there should always be some time bonus for saved minutes.

But I think that the system is great even as it is and I agree with forcolin that it should not be changed too often. You may let it run for few more months and apply the changes (if desirable) e.g. from the beginning of new year.
Regards
Jiri

Updated LMI Sudoku Ratings after FiveFold test are now available.

motris is #1 with 993 rating points, followed by deu and jaku111.

Amongst the Top 10, deu, purifire, Kota and Ziti - all gained 3 ranks. In the top 100, ByronosaurusRex and xevs were the two biggest gainers with 100+ rating point gains.

I prefer the current rating system over all the changes (1),(2),(3) proposed above.

If there is a test like Nikoli Selection where the top solvers are exceptionally better than the rest, then this can result in average rating points even for the above-average solvers. No big problem with that.

Only if there were a test where all the top-solvers are absent (i.e. the maximum points are obtained by an average solver) then the results of this test would not be comparable to other tests. The rating system depends on the maximum points, thus it is important that there are competitors that constantly show the others what is theoretically possible. :-)

Stefan

P.S.: Thanks Rakesh that you are updating the rating numbers VERY fast after each test!

As mentioned sometime back by Deb, we have managed to automate the ratings calculation process now, as a result of which we are able to generate the ratings faster.

Thanks for sharing your views, Jiri.

We had thought of a similar system earlier. But, with no upper limit, it does not work out well in terms of consistent results. And, if we use the linear extrapolation, it will come up with randomly high numbers. For example, the median in Nikoli Selection was 150. So, a score of 492 can translate to something like 3000+ on a rating scale of 1000. With an upper limit in place, the results are better. It can also serve as a quantifiable goal/target for the top solvers. And, time bonus is (mostly) already included in scores. So we should not try to duplicate its effect.

As mentioned by all of you, we'll keep it stable for a long enough period. And review for further improvments after that.

You are right. I had a look at some previous competitions myself and I have to admit that calibrating points on median only is not enough. The top player points would be too volatile. There will always be some objections, but the current rating system looks fair enough.
Thanks
Jiri

Same problem in Magic Cube. motris winning by a huge margin