There is probably also information in how long it takes to submit the correct answer after the initial mistake. If someone has simply typoed, they'd likely input the correct solution in less than 30 seconds. If someone has a large mistake in the puzzle, they'd certainly need more time to fix it before re-entry.

Edited by motris 2011-06-08 5:10 AM

It seems to me that so long as the points awarded decreases with each error that this is true only if the chance to correct it provides information that helps you solve the puzzle - if for example you can narrow a puzzle down to two or three likely options and it's more points-per-time effective to run through those option and see which are correct than to actually solve. For most puzzles and answer keys this probably isn't much of an issue, assuming you are given no feedback as to what part of your key is wrong.

Other than that, what's wrong in principle with losing points and taking time to re-solve the puzzle? Losing points on resubmitting discourages you from guessing, and if there are a sufficiently large number of options then you can't use it to do something that might be called cheating. If you need to resolve the puzzle as opposed to fix a typo you lose both time and points, which seems a suitable penalty in any case - so you'd naturally be penalised an amount proportional to "how wrong" you are as you spend time checking and correcting or even re-solving from scratch.

On the other hand for those who've made either a typing/key calculation error or a small mistake when solving the puzzle it offers the chance to reward you for what you actually have succeeded in doing. Compared with someone who doesn't solve the puzzle at all, isn't that actually eminently reasonable?

It also means tests can contain bigger point puzzles which take longer without them being quite so risky if you fail to get the points due to a small mistake.

So I don't really see a downside with the concept, but technical issues with live validation might be more of a problem. For example, what if you submit a correct solution that is mis-formatted and then waste time re-solving, not realising the problem is with the key? You'd lose out compared to the current system where it would presumably be manually fixed for no penalty.

Edited by Gareth 2011-06-08 6:44 AM

Gareth: I might be an idiot, but isn't the information on whether your answers are right or wrong withheld from you until the test is over? That means you can't just submit one answer, see whether it's right or not, and then try another answer, because the only way to determine that your answer is wrong before the opportunity to change your answer disappears, is to solve the puzzle.

Yes, currently. I was talking about the possible change discussed above (motris's post 4744) whereby you are told immediately if your answer is wrong and are given a chance to resubmit for less points.