| |
BGonline.org Forums
"Easy" improvement to PR calculations
Posted By: Henrik Bukkjaer In Response To: "Easy" improvement to PR calculations (Timothy Chow)
Date: Friday, 12 April 2013, at 10:54 p.m.
>> In order to decide whether an improvement has been made, one needs to have a clear idea of what the goal is.
This is indeed very true. Fail to see why it should be more true for minutiae though.
If we look at current measurements of backgammon errors, and their use, ER/PR differentiates itself in particular from MWC-Loss and various luck-based calculations.
Where MWC-Loss is used to compare two players head to head performance in a given match, or to measure the significance of individual errors in a match, ER/PR are used "in the long run", to measure development in your game, and used to "estimate" the strength of a player (expected strength in his next game), based on previous analyzed games/matches. It is also the preferred measurement of choice, if you are to compare different efforts.
For all the purposes stated above, PR must be an improvement on ER, since it will come out measuring the same hing in the long run, but with less variance - thus higher expected precision in any sized sample.
And so would my proposal take even more steps in the same direction.
Think about the background for the measurement. How much equity will this player, on average, through away per decision. Originally the unit were millipoints per decision if I recall correctly.
Snowie ER measures sum(err) / [no of turns]. Gnubg ER / XG PR measures sum(err) / [no of decisions].
The latter must be preferred, given the objective. Even though than [no of turns] and [no of decisions] correlate heavily, and in the long run will even out.
Now, think about a decision! What is a decision? In the formula for PR, a decision would be to chose a play in a given turn. If we think checkerplay, it would be to complete a move of 1, 2, 3 or 4 checkers given a roll.
But how about thinking of a decision being moving one checker one number. Thus any non-forced checkerplay would consist of 1-4 decisions, depending on the roll and position.
That would be an improvement over the current PR thinking, even though, in the long run, it would yield the same results.
Basically, that is the basis for my suggestion - to rethink "what is a decision". I'm not messing around with the error part or the sum(err) for that matter. An error is an error. I'm just trying to give ourselves a better term for "average" error, by considering what a decision is, and trying to differentiate different plays into more or less decisions (or big/small decisions if you like).
The result would be that a given player would end up having less variance in his PR from game to game, and from match to match, and that is would be easier to compare PRs between matches. A more useful measurement for the purposes that we would typically use PR for.
Granted, it would still have flaws, and it would still be wrong to conclude that Player A with a PR 2.65 in match 1 "played better" than Player C with a 3.4 in match 2... And there would still be conclusions that could not be made on this measurement, since it is still using normalized errors (such as in a given match, the player with the lowest PR would have the best chance to win - for that calculation, use MWC-Loss, not PR).
Would you agree to this?
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.