I have proposed that a feature be added to the bots to report actual statistics in addition to pseudocubeless ones. (By the way, to clear up one point that seems to have confused some people: implementing my proposal would be completely transparent to those who like the bots the way they are now. Rollout speed would not be affected and the old statistics would be available as always. I am just requesting an additional feature.) GNU Backgammon already does part of what I want, but its support for this feature is poor and IMO needs improvement.
Here I would like to illustrate, with an explicit example (Problem 3 from my Checkerplay Challenge #2), why we need actual statistics. First, here is the position and the XG rollout.
                         |
| ||||
| XGID=-ABBBCA-A--Aa----a-cBbcbc-:1:1:1:22:0:0:3:0:10 | |||||
| Blue to play 22 | |||||
| 1. | Rollout1 | 20/12* | eq: -0.357 | |||
| ||||||
| 2. | Rollout1 | 11/9 8/6 5/1 | eq: -0.498 (-0.141) | |||
| ||||||
| 3. | Rollout1 | 20/18(2) 11/9 8/6 | eq: -0.504 (-0.147) | |||
| ||||||
| 4. | Rollout1 | 20/18(2) 8/6 5/3 | eq: -0.507 (-0.149) | |||
| ||||||
| 1 1296 Games rolled with Variance Reduction. Dice Seed: 2 Moves and cube decisions: 3 ply | ||||||
eXtreme Gammon Version: 1.21
One thing I've taken away from Stick's DQRs is the importance of developing the skill of estimating wins and gammons. In the above position, for example, how would we decide OTB whether to hit with 20/12* or to play something like 20/18(2) 11/9 8/6 (my play)? The only way I can see how to do this is as follows:
1. Estimate games won and gammons lost (and maybe backgammons lost, if you're ambitious) after 20/12*.
2. Estimate games won after 20/18(2) 11/9 8/6 by assuming that hitting on our next turn wins, and that otherwise we have a straight race. (Assume gammons are negligible.)
Now, if I can't come up with a good estimate, I would at least like the bot rollout to give me this information so that I can do a better job next time. But now we see a problem. The bot appears to say that after 20/12*, I win 43% of the time and I win 8.4% gammons. But it's clear that this can't be right. After 20/12*, if White dances it's D/P, so I must win at least 44.4% of the time. Furthermore, I hardly ever win 4 points because if White doesn't dance, she hits me and I surely win less than 8.4/55.5 of those games, let alone win an unredoubled gammon or a RD/T single.
Similarly, after 20/18(2) 11/9 8/6, White gives us 11 hitters after 43 61 62 63 64 65, 12 hitters after 22 42, 14 hitters after 33, 15 hitters after 21 31, 17 hitters after 66, and 20 hitters after 11, so we hit immediately in 21.5% of all games. Thus XG's figure of 22.48% total wins looks rather low.
The problem is that the bot is reporting (pseudo)cubeless statistics, and what I need for my OTB decision are actual statistics, telling me how often I actually win, and how often I actually lose 4 points. How does one get this information from the bot?
The answer is that we can't, at least not from XG. With GNU, it is possible to tease out the necessary information with some difficulty. I took the position after 20/12*, except that I put the cube in the center, and then I asked GNU to roll out White's cube decision from the bar (because GNU will give us actual statistics only for cube action rollouts). I rolled out about 19k trials without variance reduction (because variance reduction doesn't help for actual statistics), and clicked "View Statistics."
GNU tells me that I win 52.5% of the time, including 49.3% single games and 2.9% double games (i.e., gammons with the cube on 2 or single wins with the cube on 4). On the losing side, I lose about 35.3% double games and 3.6% triple games. Putting these figures together gives me an equity of about –0.40 after 20/12*. This isn't quite right because I'm ignoring GNU's statistics for games with higher cube values, but it's not too far off, and in particular it is giving us a much more accurate picture of what's going on than the pseudocubeless numbers are. There's a pretty large difference between winning 52.5% of the time and winning 43% of the time.
I did a similar rollout for 20/18(2) 11/9 8/6. There's less variance here so I stopped after about 11k trials. GNU tells me I win about 24.6% and lose 75.4%, for an approximate equity of –0.51. As a bonus, GNU tells me that I manage to hit one of White's checkers 23.4% of the time. This makes much more sense in view of the 21.5% immediate hits we calculated manually. And indeed, –0.51 is very close to the XG rollout equity.
Of course, if we are unable to estimate OTB that (for example) we win 3% double games and lose 35% double games after 20/12*, then all this information is of little practical use. However, I think that with practice, one can get good at estimating these numbers. But a necessary first step is to get the bots to tell us the right numbers in the first place; otherwise, there's no hope of learning them. This is why I continue to beat the drum for actual statistics.