| |
BGonline.org Forums
A take you can't pass
Posted By: Mike Clapsadle In Response To: A take you can't pass (christian munk-christensen)
Date: Saturday, 21 September 2013, at 1:00 p.m.
http://www.simborgbackgammonlessons.com/bob-koca--welcome-to-the-giants
Many have suggested looking at a bot calculated error rate or performance rate to decide a contest but there are many problems with that approach. If one wants to go that route I suggest using variance reduced results instead.
What is so bad about error rates? What do you mean exactly by variance reduced results?
Error rates are a useful tool but they have limitations that many don’t think about. Here is a list; some of these points apply to using an ER to decide a contest, others to ranking players according to an ER, and some apply to both:
1) You cannot just assume that the bot is perfect (even with rollouts)
2) Playing to the opponent is part of the skill of a game also. The bot may penalize you for making the play that is correct considering your opponent.
3) Exactly how should an ER be calculated? Should forced moves be included? How about potential cube actions?
4) Certain styles of play will fare better than they really should. For example suppose that a player’s weakness is that he takes races that he should be dropping. He will be penalized for the incorrect take but then will get to play out a race in which the plays are relatively simple. He may very well get back what was lost by getting to play out the race.
5) The goal of the game is to defeat your opponent not play with the lowest ER possible. This is related to points 2 and 4 above but there is another factor here. Suppose that a player has a choice between two plays that are about equal in equity, For example maintaining contact or racing. If your goal is to play with a low ER you should race. After the other play you will probably make more errors but your opponent will also. Thus players who play a complicated style may have a worse ER than someone who plays a simple style yet be more likely to win matches.
6) There are “tricks” that can be used to artificially improve an ER. Some of these work only with ER calculated in a certain way. Suppose for example that close or actual cube decisions get added to unforced checker plays in the denominator. Then delaying the optional double at 3a – 1a post Crawford helps one’s error rate. Suppose that someone suggests that if it only counts in the ER calculation if it was not optional. OK, but now suppose that it shouldn’t count in the ER calculation. O,K. but suppose that your best roll followed by your opponent’s worst loses your market by a little. Then not doubling is an error. But it is such a small error that if your goal is to minimize your ER that you shouldn’t double,
Here is another trick. Suppose that the ER calculation does not include checker plays in which the move is forced. If you have one checker on each of the ace, two and three points and roll a 61 then your best play for ER is 3/2/off. How is that any better than 3/off 1/off? Suppose your next roll is an ace other than 11. Your play would be forced if you had played 3/off 1/off but it is not forced if you had played 3/2/off .You will probably play that roll correctly and get credit for making the choice correctly,
Variance reduction is calculated independently of what your play is. The bot considers whether you got a good roll or bad roll and adjusts the results appropriately and in a fair way. It is essentially the same idea used when doing rollouts. For example suppose you lost a match but got unlucky to an amount equaling 40%. Instead of losing the match 0 – 1 you lose it .4 to .6. If you were gambling for $100 on the match you would lose $20 instead of $100. If you were playing a Swiss tournament you would credit with .4 wins and .6 losses. The big advantage of variance reduction as compared to an ER approach is that you just make the play you think gives the best chance to win the match. Considerations of what the bot would do are not relevant.
There are two disadvantages. One is that some just don’t understand or accept that variance reduction is fair. I would ask those though if they know exactly how the ER system they like calculates the ER. The other is that it does not give an objective number to compare different players who have not played each other. If incorporated into an ELO system ratings would converge to an accurate value much more quickly though.
Both systems have the disadvantage that the entire game must be recorded.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.