Agree to Neeraj,
Overall i think pencil marks was very much needed, we can do away with UNDO option if one of them is to be exlcuded
Nice variety of puzzles , covered a large no of variants

Edited by utkaarsh 2012-12-26 7:51 PM

First, I would like to thank you for this first sudoku screen test. It was interesting, fun, well organized and with nice sudokus.

I would like to give my opinion on 2 points: The number of puzzles and pencilmarks:

About number of puzzles: I asked myself if it's really useful to ask this question to players. Is it really a subjective topic? If you ask me this question, my first reaction will be to look at the ranking and analize it.
So for this test, what I see is that 5 players submitted all puzzles (without leaving some of them after just a few seconds). Nobody has the perfect score, without mistake; misko didn't make mistake, but he lacked just around 1 minute to finish the test. The other 4 finished the test before the time limit, but with a few mistakes. Janoslaw spared more than 5 minutes, but without the possibility to check the puzzles a perhaps fix some mistakes.
My point is that, with less puzzles, more players would have been able to submit all, but the ranking would not have been fair, because one or 2 little mistakes would have resulted in a lot of places lost, thus perhaps not reflected the performance. For this test, in my opinion, the number of puzzles was just right.
The number of puzzles is always related to the fairness of ranking and scoring. On normal tests (where we can choose the order in which we solve the puzzles): if there are too little puzzles, one mistake can cost a lot, on the other side, with too many puzzles, you need a perfect evaluation of each puzzle's difficulty to be sure that the ranking is fair, what you can reach only by having at least 10-15 testers in my opinion (what is generally not the case).
When I organize a tournament, I have the answer to the "number of puzzles/general difficulty" question when I look at the ranking.

The other point is pencilmarks for this test:

I understand the decision not having the possibility to write pencilmarks for this screen test. However I noticed 2 things:
1. Writing pencilmarks is a matter of habit. Some players like to write lot of them, others prefer to write only a few. In that sense, the ranking would not have been the same if we could write pencilmark. Therefore it's an important decision.
2. Not being allowed to write pencilmarks, we were tempted to use more bifurcation/guesses than logical work at some points (on hard sudokus). I don't like to do some guesses to solve sudokus, normally I try to avoid them when it's possible. I did more guesses than usually on this test.

I would have preferred to have the possibility to write pencilmarks, even if I'm not a big fan of writing lot of them, and even if I'm not sure that my ranking would have been as good.

On a personal note, I'm happy to have made top10 ranking on the last 3 monthly contest, I hope I'll continue the same way in 2013

Fred

How balanced do you think the puzzle types of this test were?	A bit skewed
What was your opinion of the distribution of easy/hard puzzles?	Too many hard puzzles
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	Way too many puzzles (too little time)
What is your opinion about rules, points and scoring for this test?	Fair
What is your opinion about the solving interface?	Average

How balanced do you think the puzzle types of this test were?	Fairly balanced
What was your opinion of the distribution of easy/hard puzzles?	A bit hard
What did you think about the puzzle quality of the test?	Very nice
How did you feel about the length / time limit for this test?	A bit too many puzzles
What is your opinion about rules, points and scoring for this test?	Exteremely unfair
What is your opinion about the solving interface?	Very nice

Hi Deb.

I was on travel during the weekend and hence could not take part in the test.

Can you post the unofficial link of the Screen Test, which was given as reply to john_reid as a Bold or Bigger font so that people who are interested in giving it a try can do it. Also can you extend this till weekend if possible.

Hi Varun, It is extended till weekend.
Here is the link again.

Unofficial link

Thanks for all the feedback Fred. I'd like to address those two topics too.

When Deb first explained the concept to me, I personally was pretty excited for it. It seemed like a fun, light way to end the year. I'm glad a lot of others enjoyed it too.

A few of the decisions/features/format of the test are of course debatable, but overall I at least thought it was a fun experience (including the minute-refresh feature which can only be used best in such a test :P ).

To Fred's point about number of puzzles, I look for the same thing when evaluating number of puzzles. Every test is ideally planned so that the top 4-5 scorers can finish it and the rest can come close. As Deb mentioned, Branko was the first one to attempt this test, and then I did it. Seeing my score, both me and Deb were unsure of whether to reduce the time limit to 50 minutes. In the end we went with the belief that I'd had a good day and kept it at 60, and thankfully so, as this turned out to be the ideal limit for the first few to solve, as Fred said. When discussing this test, we thought at length about the beginners and that is why there was the categorization A, B and E in the IB, and I think Deb's already explained about that. So, we tried our best to keep a target for everyone, at the same time keeping enough to give the top players a challenge. At least going statistically, that worked ok.

The point about pencil marks has been debated by everyone, and most seem to feel that its required. When planning the test, I was in favor of disabling pencil marks because of a few reasons -
A. I know most people use notations to proceed, but I've seen it being very much possible to go ahead without them, even on harder grids (There's a site or two that have online solving without pencil marks too, and Fred frequents one of those ;) ). I just felt it was more fun this way.
B. After attempting the test myself, I had solved 29 puzzles logically, guessed on one, and I went back and solved that one too pretty easily after the test. So I told Deb as much, and the system seemed good to go with.

I think Tom's point is good about the Undo button contradicting the decision to disable pencil marks, and here again, it didn't strike me as I didn't use the button during my solves (I used Clear All once I think). We concentrated more on some other features and workings of the page instead, like the warning if an incomplete puzzle is submitted.

I'm glad that some people have commented that they enjoyed it overall, and with a few changes I do think we can have this format again.

With Christmas getting in the way I hadn't gotten around to commenting.

I think much is already said. I, like Fred also opted far more often to use guessing instead of logic on these puzzles than normal. This was mostly caused by using a lot of notation while solving and I just noticed I missed a lot in the beginning and making silly errors as i misremembered things, which led me to place the wrong digits. These would all probably not have happened if I had made notations. The high amount of guessing was mostly the reason I voted a bit lower than normal on the enjoyability of the test. Resorting to guessing always makes a puzzle les enjoyable for me and with the fixed time limit I have to try to get it out somehow. So I would prefer pencilmarks over an undo button, as I never use those anyway. I guess in another colour and find it far easier to just remove the digits of a single colour using the space bar than use the undo button.

I also think there should be a higher cutoff point for how quickly you get points for a puzzle. I would say you need to have at least half of the grid filled to receive points, because I don't think people should be able to get points for looking at a grid for 15 seconds and placing 1 or 2 digits. I also think a wrong digit should count differently than an empty cell. I feel you should score for the amount of points for the total of (correct digits-wrong digits), with at least half the grid filled, with the extra rule that you can't get any negative points for a puzzle, so when the amount of wrong digits is higher the amount of correct digits is should score 0. I get Deb's argument that someone who guesses and then it gets autosubmitted would get less points if it's wrong, even though they did the same parts logically, except he could also have guessed correctly and get the extra points. And as most guesses are a 50-50 chance of being correct, this would in the long run just even out.

I thought the amount of puzzles was good. The best ones managed to get through it completely, without anyone having a lot of time left over. I think the rating might be partly caused by the fact that many people though, "it's only 6x6 sudokus, that's going to be really easy", as most 6x6 puzzles that you can find are just really easy. And then when a 6x6 puzzles stumps you, like some of these puzzles definitely will beginners, it might be demoralising. But that happens to the best of us. I think this test deserves a repeat, and maybe with everyone now better knowing what to expect the ratings will probably go up.

One final comment. I would have liked it if the letter and number of the puzzle were listed somewhere on the puzzle page while solving (in front of the puzzle name for example). I find it handy to know how far along I am in the puzzle set.

Edited by Para 2012-12-28 4:20 AM

Thanks guys for the in-depth analysis. I think it is safe to assume that for a similar test next time, pencil marks should be enabled, if the sudoku difficulties are more or less same. At the same time, I feel adding pencil marks is not going to help much to beginners. (for example : take the Consecutive Sudoku. I see lot of beginners struggling in that. If you miss the 4 (out of 5 possible) dots in row 2, you are never going anywhere. Not sure if pencil mark would have helped here.)

Another theoretical thought I had about pencil marks is that, LMI's pencil marking is very unique. It is not certainly easy to use, especially if you would be using it for the first time. So I was of the opinion that having pencil marks significantly benefits players who are used to LMI's interface. So disabling pencil marks was one way to make it more balanced to paper players.

Anyway, given that almost all comments suggest that pencil marks should be enabled, we should just do it accordingly next time.

About undo : I remember now that during my initial visualization of the test, I had thought of limiting Undo to 2 steps. (Just enough steps to undo mis-types). But I had forgotten about it when the test took the final shape. So, yeah, I agree that not having pencil marks and enabling undo is a perfect recipe for guessing.

Will do next time.

On the contrary, I feel LMI's online interface is the best I have found till now. It's way-way-way better than any other interface out there. And only because of LMI's interface, I have started using Pencil Marks - earlier I was used to solving Sudokus with No Pencil Marks at all. (I still don't use Pencil Marks when solving on paper).

I kept putting off and forgot to officially do this test (would have been my second, I think) despite doing the demo. I think the reason may have something to do with the fact that a puzzle can be skipped as incomplete, and with the appearance as a whole of a 30 puzzle test. You can't please everyone, and if the screen test had been one hour for 12 puzzles - something I don't consider too out of the ordinary, as those not as skilled can then spend five minutes on a puzzle without stressing - then some others would have said "Where's the challenge?" With a test of 30 puzzles, and the ability to skip puzzles, I don't blame someone who skipped over 5 puzzles they couldn't do and otherwise solved 18 (right or wrong, don't matter) before running out of time for saying there was way too many puzzles. Personally, I thought it was too many; you have to literally solve each in, on average, two minutes. However, if the goal had been to solve any 20 and skip any six, that would give a little more time and allow people to customize their test for the best result. Obviously, bonus points could then be awarded for correct solves of 21-30 (not by puzzle number but by number of correct puzzles), and the best performances thus would well exceed the otherwise expected maximum test score based on 20 puzzles (as an example). For 24 puzzles, for instance, that shaves off 12 minutes that can be allocated to other puzzles, giving 2:30 to each puzzle instead of 2:00, and skip any six you don't like the look of.

Just my opinion.

Also, a booklet of the puzzles now that the test is complete might be nice. :)

Cyclone


Edited by Cyclone 2013-01-04 3:48 AM