Snowie ER and XG elo - the new Performance rating (PR) system
Posted By: Ian Shaw In Response To: Snowie ER and XG elo - the new Performance rating (PR) system (Jason Lee)
Date: Monday, 23 November 2009, at 10:00 a.m.
In Response To: Snowie ER and XG elo - the new Performance rating (PR) system (Jason Lee)
PR = -(equity error per decision)*500
Why 500 and not 1000? 1000 makes more sense... then your PR is millipoints of EMG loss per decision.
I agree that dividing by 500 is horrible. It is much more satisfying that the error rate has a proper meaning, such as mppm (millipoints per move).
The trouble is that people have got used to quoting error rates in Snowie terms, and Snowie uses the sum of both players' moves as the divisor. Therefore, Snowie reports an error rate approximately half that reported by gnubg and XG, both of which use only the count of one player's moves.
There are other differences, too. Gnubg and XG discount forced moves; Snowie (I've heard) discounts bearin plays because it plays them badly (at least, it did at one time).
Dividing by 500 is an effective conversion to Snowie mppm error rate; divide by 1000 to get millipoints, then multiply by 2 because Snowie is counting twice the moves of gnubg/XG.
To most backgammon players, the gnubg/XG method is more intuitively correct, but they are used to the Snowie numbers.
Douglas Zare wrote an article at GammonVillage Normalizing Errors in which he argued that the Snowie method was superior. I'll just quote one paragraph here, for those who are not GV subscribers.
"Suppose you get to a classical high anchor holding game. Perhaps it is a small error to double now, since you are not quite far enough ahead in the race. On the other hand, if you don't double now, you will be faced with a series of increasingly complicated cube decisions, e.g., you might lose ground in the race, but your opponent might have to leave the anchor with one checker, or might have to kill some checkers. Will that be strong enough to double? To minimize your total error, and your Snowie-method error rate, it may be right to double now, making a small investment of equity to simplify your future decisions. However, this will greatly decrease the number of unforced cube decisions, so it might increase your gnu-method cube error rate. Trying to minimize your Snowie-method error rate will lead you to minimize your total errors. Trying to minimize your gnu-method error rate may not." (My italics)
It would be less confusing for the bg community if the bot developers and players came to a consensus on an error reporting method. The proposed XG method looks like a nasty half-way-house between two methods, with no inherent theory behind it. It offends my inner-scientist! Personally, I would prefer that they adopt either the Gnubg or Snowie method in full, though I acknowledge that they are free to do as they please.
Gnubg presently quotes both error rates. However, Douglas' argument does not seem to have been persuasive, and Snowie looks like it is receeding in the bg standings. Snowie development seems to have stopped, so maybe it's time to move away from their method as the de facto standard. So far, I've left BgBlitz out of this, but that's only because of my ignorance of it's methods. It's certainly one of the the active, strong bots, so its error rate reporting is of interest.
Messages In This Thread
BGonline.org Forums is maintained by Stick with WebBBS 5.12.