All puzzles are pinned and uploaded as a single pdf without password here

Thanks to all who participated, and to Deb for allowing me to contribute a puzzle and design the logo for this test.

If you have an account on my blog, The Griddle, and solved my braille word search in this test, enter the two missing words in your code entry page to unlock the new "Run For Your Life!" badge: http://thegriddle.net/home/badges/18

I noticed that some of the authors have participated as players too. They have been awarded an arbitrary 100 points for the puzzle they provided. I believe this is penalizing. In a competition in which the difference between players is given by the bonuses (which is good, and is the strongest point of this beautiful competition), it is equivalent to deny them the opportunity to earn a bonus in the puzzle they provided (for which they are supposed to be strong solvers too...). In my opinion it could be fair to award the author of a puzzle a bonus score equivalent to the average of their bonus scores obtained in the remaining puzzles solved (including zeroes, if any).

I agree that 100 points is not fair to all authors (it was not arbitrary though, 100 was assigned based on the assumption that an author should be able to solve his puzzles, and we just took the worst case scenario where there is no bonus). So, yeah, it is bit harsh.

In fact we started "best 9 of 10" to handle authors' cases. But later, 9 of 10 turned out to be very essential for this kind of contest even otherwise. So, we can give some bonus points based on forcolin's formula above. Or, we can say for authors, it should be best 8 of 9, and then multiply the score by 9/8.

Is it a good idea to have an option where a player can choose if he/she needs a hint to solve a puzzle at the cost of some X points depending on the time lapsed from the start of puzzle along with reduced bonus system (if applicable) ?
This might help few solvers to have the satisfaction of solving all the puzzles and create more interest in puzzles.

Regards,
Ravi

Can we make mine best 4 of 9 for this one and then multiply?

Anyway, on a serious note, I personally was fine with this format because I considered it a bonus in itself to co-author and participate in the same test. However, if everyone feels that what Stefano or Deb have suggested is fine to implement next time, then all the better!

I was surprised to be fastest on 2 puzzles. I don't generally top lists. As Palmer said he didn't finish any puzzle fastest in the playoffs, I didn't score any top 3 spots in any round, but still finished 5th overall after 2 days on the WPC before the playoffs.
I'm also surprised a bit to be in the top 5 as I've never considered myself good at solving big puzzles on speed. I usually make too many mistakes. It was fun, but I think I would have done better on the Samurai and Graffiti Snake if I had solved them in a row with the other 8 puzzles. I hasn't realised when I started all 10 puzzle types weren't listed to be honest. Will remember that for next time.

I think the scoring probably works better with a distance from the top time system. It seems a bit weird to be the fastest solver in a type and have to drop that puzzle.

Now that the full puzzle booklet is available, I am finding that it crashes my Adobe within a few seconds of opening it. Can we get individual puzzle versions?

Cyclone

Individual puzzle booklets without password uploaded at http://logicmastersindia.com/M201201P/

It doesn't make sense to me. It also is very impractical.

macherlakumar: I see some use in having easier puzzles available for people to try and to get started with, but to implement them in a test here is impractical, and this really doesn't seem the place for it. If there are any types that anyone wants to start on but where some simpler puzzles are needed, I strongly suggest contacting some puzzle authors in the community and requesting them. Speaking for myself, I'm happy to take requests, and many others in the community probably would be at least somewhat accommodating too.

Yes you are right I did not think through this thoroughly.

Regards,
Ravi

Time for some statistics -

Number of Puzzles Vs Starts / Finishes / Bonus

It is extremely pleasing to see that 124 players finished all 10 puzzles.
Also, on an average (of 259 players who submitted at least 1 puzzle), a player finished 7.4 puzzles.


And because of the interesting bonus system, here is how the 20th / 50th score compared against the top score.

From the feedback page :



There is a bit of story behind Graffiti Snake. The first version of IB had "Paint By Number". We wanted to include PNB because, as per Rohan's word, PNB is king of big puzzles. But before I asked Serkan to make PNB, I suggested that he should make a Graffiti Snake instead. (I get 1% credit for selecting the type, and Serkan gets 99% credit for making a beautiful puzzle)




Next 3 sets of graphs pretty much suggest that the corresponding aspects of the test were close to perfect.












Many players voted that they would use Online solving if provided. We will try to provide that next time, but no promises. We face enough issues while supporting both online and paper for Sudoku tests.




Of 149 players, 144 players voted that they would participate in future marathons, either as player or as player + author.

I found a technical issue at the result page(score.asp)
... columns of each puzzle are sorted improperly when they are displayed in 'Points'(not 'Submission Time').

They are to be sorted in numerical order(like 'Points' and 'Bonus' columns),
but they are sorted in character-based(ASCII) order ...

# I remember that some of them were correctly sorted a few days ago.

I'd really appreciate someone giving a break down of an entry point into Different Numbers - I really struggle with these and got absolutely nowhere with this particualr one.

And Serkan (Different Neighbours)

Why did I have to wait till someone point me that?

There is cheeky start to the Different Neighbours at the top right corner.
Note that X has to be 1 or 2, otherwise the top right is not solvable uniquely.

Then transferring the 4 we get that the 2X2 cell can only be 3.

Who are the test solvers ? I am not sure if it is mentioned somewhere and I missed it or it is not mentioned anywhere.

Regards,
Ravi

Oops... I didn't mention anywhere.
We had 3 live-testers this time - Branko, Prasanna and Rohan. While everyone else had 9 days to solve the puzzles at their own comfort timings, these guys had to solve all puzzles within two days time. Fortunately, we didn't find any issues during their solving and their scores remained official.

Yes, that is a bug. Too late to fix though

I have done some analysis of the score and bonus system.

First of all, in my opinion the scoring system in this contest was very good because every player had a realistic possibility of gaining bonuses on most puzzles, which means that the rank is very close to the sum of the times obtained in the individual puzzles. There is an exception, the Samurai Sudoku which was much more difficult than the remaining puzzles, and in which only 20 players were awarded bonus points, and which even the best solver (motris) had to drop as his worst result.

Overall, the number of players gaining a bonus was 1108 out of 1927 (57.5%), relatively high, and this is the distribution among the various puzzles.




If we consider the percentage of players which gained a bonus as a measure of the difficulty of the puzzle, we must conclude that the easiest of the puzzle was the Braille wordsearch, with 84% of the submissions gaining a bonus. This was originally indicated as an AVERAGE puzzle. Which means that an attempt to allocate different times (or different bonus thresholds) to puzzle of different difficulties as proposed by detuned, may be affected by wrong evaluation of the difficulty.

I have also analysed the proposal of awarding bonuses only to those players completing a puzzle within a fixed time (30 minutes) from the top solver. The total number of players earning a bonus in this case would be of 786, 40.8%, and the distribution is the following.



In my personal opinion, this system would be much worse. Not only the peculiarity of the Samurai sudoku is not solved (of course, 30 minutes margin on a very though puzzle means much less for a tough puzzle than for an easy one) but the total number of bonusus decreases dramatically, punishing the players earning 10-15 points with a solution time between 40 and 50 minutes. Overall, almost 350 submissions would earn no bonus at all, and this would be concentrated in the middle category solvers. Also, for those players, the average bonus would be reduced therefore the score would privilege a player with a very good time in just one puzzle against a player with decent times overall, and I do not think this is (or should be) the target of this competition.


I have tried to develop a different system. This is based on assigning to the top solver a bonus of, say, 50 points, to define a bonus threshold to n times the time of the top solver, and to calculate the bonus by linear interpolation between these two values. I have prepared 3 scenarios, with n= 4, 5 and 6 respectively. Which means that a player would earn a bonus if his/her time was 4, 5 or 6 times the time of the best solver, or better.


The following distributions are obtained








The total number of bonus scores is 905 (46.9%) for n=4, 1118 (58%) for n=5 and 1275 (66%) for n=6.

All these calculations give a better distribution of the scores among the puzzles, (the number of players earning a bonus on the samurai is now comparable to the other puzzles) and the situation which better approaches the system adopted is for n=5. Of course this system would give different results in terms of final ranking, benefitting mainly those players which had a good time on the Samurai, but not good enough to gain a bonus on it.

The negative consequence is that, with the system adopted for this competition, it was clear that after one hour from starting a player could put the puzzle in a corner to solve it the next day. With this system it could be possible (except for the very first player to start a puzzle) to show a “current bonus threshold” as an indication about when a player could give up, and also as an indication of the level of difficulty of the puzzle as required by Puzzlescot, but this indication may change with the time, as strong players will set up best times.


Overall, I think that the system adopted for Marathon number one has the advantage of being simple, and could be adopted again without variation if only the organizers will avoid to use puzzles with a remarkable difference in difficulty, such as the Samurai Sudoku. If a new system has to be adopted, a calculation based on a bonus threshold of minimum 5 or 6 times the time of the best solver can be an improvement and could allow to use puzzles of different level of difficulty, but I am convinced that very difficult puzzles requiring the average solver much more than an hour to be solved, should be avoided for practical reasons.


Excel Analysis : http://logicmastersindia.com/M201201P/MarathonSolvingTimes_forcolin...

Stefano,

Thanks for the details analysis and many insights. The %bonus per puzzle is indeed useful information.

1) About Braille Word Search - Yes, this puzzle was marked as AVERAGE difficulty. For some reason, this puzzle looked scary (to me, and I guess many others). As you can see this puzzle has least number of submissions, even compared to Graffiti which was uploaded 48 hours later.

2) About Samurai - A lot has been said in the forum about this puzzle. All I can repeat that it was a bad choice. It is doubly bad considering that I insisted all authors to make puzzles with 12-18 minutes target time for top solvers for each puzzle.
The low percentage for Kakuro is not really surprising. This is the only classic-Nikoli puzzle and we know that some players are extremely fast in those. (That is also the reason we had exactly 1 classic-Nikoli puzzle)

3) About 5XN or 6XN bonus system - This is really innovative. If the puzzle difficulties are varying a lot, we might have to follow something similar.
But as you mentioned, if there were no Samurai, there is little need for changing the current bonus system. The current bonus system has 2 major benefits
a) it is extremely simple
b) the target for each puzzle is published and is well known

So, in future marathons, we would first make sure that there are no Samurai like puzzles. That solves majority of the problems. It will be impossible to make all puzzles of similar difficulty. But as long as there is no puzzle extremely difficult, we should be ok.

There other points are
1) whether it is fair to compare scores by just adding up individual puzzle times of varying difficulties
2) whether ranks in individual puzzle should be given any importance (like LMI Ratings)
I think this post from motris briefs about these two, but it does not have specific formula.


Thanks once again for your analysis and your suggestions to improve everything that we should.

This is incredible data and I'm glad to finally have something like this in hand. I'm not sure what Stefano is trying to optimize (uniformity of percent achieving bonus? - is this really the relevant parameter?) and I haven't had time to dive too deep into the info myself. But I think the most fascinating graphs so far are just looking at the trends in time for each puzzle across the top 100 solvers and seeing how using the "nth solver" at any point is a good measure of the relative ranking of a puzzle's difficulty.

I've linked to two images, one with a view of the whole test and one with just a view of the first hour which cuts Pentomino, Kakuro, and Samurai from the top 100 solver graph but gives a much better picture of the other puzzles. I think the data establishes a clear order of Tapa < Loop The Loop < Braille/Small Regions < Diff Neighbors/Graffiti < Black and White Loop < Pentomino/Kakuro < Samurai.

Notice that top time is probably the worst of the 100 choices for ranking the difficulty of the puzzles (and therefore the worst to use to normalize by multiplication or other means). Looking at the 10th solver (95th percentile) seems much better though. The top time suggests Pentomino is slightly easier than Black and White Loop. The 10th place (or any spot from 10-100) shows it is a 20-25% harder puzzle than the Black and White Loop for the vast majority of solvers.

These graphs also show me good characteristics to fit to either a rank-based or a normalized scoring model to get all the puzzles back on par with each other. The linear nature actually suggests rank may be best, with perhaps 150 to the top solver, 149 to the second, down to 100 for the 51st and later. For only Samurai, which we agree is too hard, would this system break down at one hour. But I do think you need to treat Kakuro and Pentomino differently from the easiest 6, and maybe even Black and White Loop as well. I disagree that only Samurai was an outlier on this test, and I'll let these graphs speak for themselves on that point.


Edited by motris 2012-02-02 10:14 AM








Attachments
----------------
top100.png (60KB - 0 downloads)
top100-zoom.png (87KB - 1 downloads)

First of all thank you for the nice contest.

Beside any data analysis: for me (as mediocre solver) the Sudoku was the only puzzle, that felt a little bit like marathon. All the other ones are just a little bit bigger than usual, which didn't matter, because I could solve one or two per day. If you want to test endurance, I would suggest the following: Give an even longer general time window (maybe four weeks), so that much people can find the time to participate. In this window, you can start the contest at any time, which gives you a 24h-window working like the last contest.

I think a general problem of this contests is the time difference between a top solver and an average solver. In a 2 hour contest, which the best solvers hardly finish, I will get 1/3-1/2 of the points and need maybe 2-3 more hours, if I want to finish all puzzles. If the contest should feel like a marathon for the best, this would mean at least 4-5h for them. But if it aims for "time needed for a fixed amount of puzzles" instead of "finished puzzles in a fixed time" that would mean something like 15 hours for me, which isn't suitable. And if I can do it in different sessions it's not really a marathon for me.

(This reminds me of an interview with an hobby-marathonist which I read a while ago. He said things get easier after you can beat the 3h-mark, because you don't have to run so long, if you're fast enough ;-) )