| |
BGonline.org Forums
Another caveat about stopping rollouts at 3 jsds
Posted By: Timothy Chow
Date: Sunday, 4 April 2010, at 11:06 p.m.
As I understand it, a popular strategy for doing long rollouts is to fix a minimum and a maximum number of trials, and then roll out until the difference between the top two plays is 3 jsds. Then we conclude that there is less than a 1% chance that the top two plays are actually equally good.
I want to point out that as computers and bots get faster and rollouts get longer, there is a subtle problem with the above procedure that will start to become apparent. Namely, even if two plays are in reality equally good, if you roll them out long enough, eventually one of them will surge ahead of the other by 3 jsds just by random chance. For the math geeks out there, this follows from something called the "law of the iterated logarithm."
I wrote a quick simulation to get a feeling for how big an effect this is. I simulated two plays at DMP that are actually equally good, and yield exactly 50% MWC. Then I simulated rolling them out (with no variance reduction) for a minimum of 108 trials and a maximum of 46656 trials, stopping early if one of them surged ahead of the other by 3 jsds. I found that the rollout stopped early about 7% of the time. That is, 7% of the time, the procedure yielded a verdict "A is better than B" even though A and B were actually equally good. Obviously, 7% is a lot bigger than 1%.
I just threw the code together this afternoon so it's quite possible there's a bug, so I wouldn't take the 7% too seriously until someone else independently confirms it. However, in principle the issue is definitely there. The good news is that it's a fixable problem because the correct probabilities in question can be calculated, although off the top of my head I'm not sure of the best way to do the calculation.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.