| |
BGonline.org Forums
What is the purpose of EMG? Of Snowie error rate? [longish]
Posted By: dlevy In Response To: Cube error rates (Joe Russell)
Date: Wednesday, 22 July 2009, at 3:02 p.m.
I have been following the discussion about error rates with interest. Error rates, whether expressed in terms of match winning chances (MWC) or equivalent to money games (EMG) are used for (too?) many purposes and the statistics are not useful for all of the intended purposes.
Error rates expressed as MWC have a very useful purpose when combined with luck expressed as MWC. A match starts with each player 50% to win. The net luck and net error rate will by definition be 50%. An example from a recent match:
P1 error rate -12% MWC; luck +37% MWC
P2 error rate -17% MWC; luck -8% MWC
Net error rate 5% MWC; luck +45% MWC total +50% MWC
This is a matter of definition and will always hold true if you use the same neural net and the same parameters for error rates and luck. [Aside: gnubg defaults to 0-ply for luck and I use 2-ply for a first look at a match, so the numbers can be off, sometimes by a lot. When you roll out a decision, gnubg updates the error rate but not luck, potentially introducing additional discrepancy. You can set gnubg to use 2-ply for luck, but since that is figured over all possible rolls, it takes a lot of time for no benefit.]
Now these numbers are very useful in terms of understanding why you won or lost a match (P1 was outplayed a bit, but substantially out-rolled], but are very unsatisfying for all the reasons that led to the EMG calculation. EMG attempts to normalize error rates to overcome at least two perceived problems. First, the "same" error with a higher cube results in a greater loss of MWC than with a lower cube. Second the "same" error later in a match results in a greater loss of MWC than earlier.
So, EMG is useful for asking the question "what were my n biggest errors in the match." The student can study these errors and improve play.
Then we average EMG over the number of moves to get a Snowie error rate (whether calculated over all moves as Snowie does or unforced moves as gnubg does) and get a number. And we tend to use that number to express how good a player is. "X plays at error rate 2 or 5 or 10" each says something about player X that we all understand intuitively. While over a huge number of matches, that is a good approximation, in any one match, it doesn't necessarily work.
I think what is behind Joe's original complaint about getting "ding[ed] for cumulative missed doubles." Let's assume the missed doubles were not consecutive, but in different games in the same match. Is there are problem about multiple dinging? If they were in different matches? I suspect Joe is thinking "I am a Snowie 3 player, but these multiple dings gave me an error rate of 6 and I don't believe it." The Snowie error rate in any match reflects how well the player understands the positions that came up in the match and the frequency those positions came up. If a poorly-understood position comes up a lot (cumulative missed doubles), the error rate is higher. I have a great error rate in a match consisting only of five-anchor holding games. But I know I'm not any better after seeing that low error rate.
Moral: Don't expect too much of the Snowie error rate, particular for a single match.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.