How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Fairly Nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	A bit skewed
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	A bit too many puzzles

How balanced do you think the puzzle types of this test were?	A bit skewed
What was your opinion of the distribution of easy/hard puzzles?	Too many easy puzzles
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	Perfectly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

I think your question should be focused more on the spread of difficulty of the puzzles than on the specific scoring. In other words, the proper constructive spirit of your points are: Should a test have a puzzle that takes five times as long as another? Should it then be scored so that it is worth five times the other?

The first is really the question of test balance. The second is just the organizers determining what is fair. There is a lot of variation in how people view the first question but I don't believe there is much variation with the second. If, in a timed heat, something takes twice as long to do as another task it is most often awarded twice the value without any question. The "points per minute" value is meant to be close to constant.

All test authors make their best attempts to score a test based on the solving data they get from both internal and external test-solvers. While these scores and entry rows are not completely based on my testing, they were definitely influenced by my testing results. So let's just say each puzzle is worth in points about 10x my solving time in minutes. If a puzzle takes me about 3 minutes (quadruple) and another takes me over 15 minutes (Little Killer), then scores of 32 and 157 reflect the spread in the difficulty of those puzzles for a top-level solver. Similarly the 36 point Renban took me close to 3.5 minutes, "surprise" and all. I was not surprised. Perhaps my prior knowledge of tennis and my expectation the Serbian test would use it a lot meant I more quickly grasped that A, T, P made sense as the three regions. Perhaps I'm just good at testing surprise puzzles. The Indian Grand Prix test similarly had complaints about the Even/Odd/Big/Small sudoku. It was not a surprise to me. The score given was pretty reflective of my time and other testers.

Perhaps more testers would help. But having so many solvers want to compete in the Grand Prix means there are few options out there. And recognize that even with a lot of testers, the scoring will reflect how solvers gets through the puzzles on average but cannot possibly represent how you individually will solve the puzzles as you will have your own strengths and weaknesses. If you keep finding that you are spending too much time on under-valued puzzles and never solving the over-valued ones, you are probably taking the wrong approach to these tests and could improve simply by changes in test-taking strategy. Understand, in other words, there is a difference between "over-valued in general" and "over-valued for you".



As a sports fan in addition to a sudoku solver, I can't let this go undiscussed either. Your Olympics analogy seems flawed in that it treats all events as equals. Some competitions involve performing 1 thing for 30 seconds and others involve performing 10 things over 2 days. I do view a gold medal in team basketball quite differently from a gold medal in the 50m butterfly and both of those quite different from a gold medal in the Decathlon. This is why I do not put much credance in the "nation weighting of medals" as there is no real way to equate all medals from quite different competitions. What is essentially different here is we are not comparing the results of an archer with those of a marathon runner.

No, here we are comparing the results of several hundred solvers on a 16 sudoku competition for which they have 2 hours to compete, and the puzzles have varying difficulty indicated in a fair -- albeit impossible to be perfect -- way with variable scoring made known to all solvers in advance.


Edited by motris 2013-04-21 10:48 PM

How balanced do you think the puzzle types of this test were?	Perfectly balanced
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	Most puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Fairly Nice
How did you feel about the length / time limit for this test?	Way too many puzzles (too little time)
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	Most puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

Dear Serbian friends, thank you for a such great puzzles! I really liked the tennis theme. During solving I discovered about 10 secrets, and have a great time after competition when discovering the other ones. Converting the ideas to ALL (!!!) puzzles is really unbelievable, specially letters ATP in movable digits. Thank you once again.

In the name of all authors, thank you for kind words.
Great reward for our work.

Branko

Jan,
I sent to you PM.
Please respond.

Branko

Since Thomas' post covered the rest of your feedback, I'll just address this part. The time frame is again (usually) decided on the basis of the testers' timings. Although, here, the time frame's been set to 2 hours for uniformity and I assume the tester's times were used to confirm that the set falls within that time frame. It is fine tuned (usually) so that the top solvers can come close to or just finish the set. Looking at the score page, this set has achieved that goal. I don't see why puzzles being worth more points means the test should go for longer. But I didn't see why the test is imbalanced either. There's all kinds of points from 20, to 30s, to 40s, to 60s, to 70s, and I think the really tough ones should be seen more as a way to differentiate the top solvers, unless they're particularly your strength. The last sentence, well, I'm sorry you couldn't submit on time, but that happens to many solvers in every test, no matter what the time constraints, so that isn't really something time frame will change.

And, thanks to the Serbian authors for a beautifully themed set of puzzles. I had the unfortunate experience of breaking the 2 highest pointers pretty late, and so skipped those two, but overall I really enjoyed the test. I'll post a few more detailed comments after I redo those and others so I know where I went wrong.

How balanced do you think the puzzle types of this test were?	Perfectly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	Most puzzles were worth the right amount

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Fairly Nice
What was your opinion of the booklet formatting and printing?	Just right

Do not really feel qualified to judge contest. I thought I was pretty good at Sudoku until starting the Grand Prix. I enjoy doing it but seem to be completely out-classed. Where can you get experience (try, fail, learn) on these type of 'non-standard' - at least compared to "expert" book/ newspaper puzzles without quitting my day job? This is not a complaint against any contest. I appreciate the challenge being offered - and obviously others do not find it too difficult. Keep up the good work challenging us.

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	Too many hard puzzles
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	A bit too many puzzles
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	A bit too many puzzles
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	Most puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

I would have liked to have more than 2 hours. I felt like I was often deciding whether to try a puzzle, or which ones to come back to, based on the very large point values of some puzzles and wanting to finish them in the time limit.

Of course I look forward to more time spent solving the remaining puzzles later!

These were quite difficult and I feel that I was lucky to solve as many as I did during the test.

The tennis theme was very nicely done! I suspect there are more tennis messages for me to find if I keep looking for them.

How balanced do you think the puzzle types of this test were?	Perfectly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	Perfectly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Way too many puzzles (too little time)
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	Many puzzles were worth too much or too little
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	Just right
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	Most puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

How balanced do you think the puzzle types of this test were?	Perfectly balanced
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Just right
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right

Here on LMI there's been many such contests authored by various brilliant authors from around the world. Just access the monthly tests page. Also, you can check the links page for more sites/blogs where you can find Sudokus/other puzzles of such quality/difficulty. Hope that helps.

How balanced do you think the puzzle types of this test were?	Very skewed
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Fairly Nice
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	Most puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	I have a different complaint

Perhaps this feedback reflects the imeediate frustrations with my solving experience - so please take with a pinch of salt.

I wonder what the point of having a competition with at least 3 puzzles which dwarf the rest in terms of difficulty and points distribution. It forces the solver to try the harder puzzles, and means that if you get stuck on just one of these harder puzzles it ruins the result of the solver. The easier, less weighted puzzles become neither here nor there in terms of their importance in the competition. I have strong feelings that if you are going to have more than 1 or 2 10, 20, 30 minute puzzles in a test, then all the puzzles shoud be of this much harder level.

As a set of individual puzzles I feel they were all good quality puzzles and I enjoyed solving them in isolation. However I am quite frustrated solving them in the context of a competition.

I should also mention that I didn't attempt the shape puzzle due to ambiguities in the presentation. As it turned out, there were only 4 shapes, but this could easily be interpreted as having 8x L-triominoes and 4 x /-triominoes which obviously makes things far more complicated in theory.

I thought given the relatively low value of the puzzle it wasn't worth the time trying to work out that it was the easier option.

EDIT: the tennis racquet theme has become more apparent to me now. But I see I'm not the only one to have missed it so perhaps it is still a point still worthy of consideration. It wouldn't have been hard in this case to have provide an outline bordering the shapes to clear up the ambiguity.

Edited by detuned 2013-04-22 7:59 PM

How balanced do you think the puzzle types of this test were?	Very skewed
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Way too many puzzles (too little time)
Of the puzzles you solved/attempted, how well did their point values reflect the difficulty/time spent?	All puzzles were worth the right amount
What was your opinion of the booklet formatting and printing?	Just right