| |
BGonline.org Forums
Minimum # of Trials and Confidence
Posted By: higonefive In Response To: Minimum # of Trials and Confidence (Stick)
Date: Saturday, 16 October 2010, at 11:25 a.m.
Bill Robertie did an experiment years ago with snowie. He wrote to a question at problem #10 of the week:
"Question:
What sort of rollout settings do you recommend? Do you vary them by position?
This is obviously a hugely important question. A few years ago (around the time Snowie 4.0 came out) I decided to set up an experiment to answer this question. I selected a suite of 50 positions from various types of games, ranging from opening to back games to races. In each position, the object was to calculate Black's winning chances as closely as possible, as though Black were on roll and thinking about doubling. I decided to accept the results of a Snowie 3-ply, 1296 game rollout as the "correct" answer for each position, and then did rollouts using different numbers of trials and different play levels to see which level gave me the best answer in the least time.
The whole experiment took about six months, with two computers running all night, every night to get all the positions solved for all the ply levels and trial possibilities. But the result yielded a pretty clear winner: 2-ply and 648 trials is "good enough". Running more trials and/or going to 3-ply was overkill, while dropping to fewer trials or going to 1-ply resulted in quickly increasing discrepancies with the "correct" answer.
As I say, this was done some years ago using Snowie 4.0 or 4.1. Snowie is now up to 4.7, but I'd wager a similar result would still hold true. If I were doing a backgame, I'd crank the rollout up to 3-ply and keep the trial number at 648. That might be unnecessary because Snowie 4.7 is noticeably better at playing backgames than the early versions were, but it's good insurance. For simple positions like races and blitzes, 1-ply rollouts are fine -- actually unnecessary because the evaluations are extremely accurate as well for these positions.""
Xavier Dufaure de Citres wrote to me in this question via email:
"648 trials is not enough, no offense to Robertie. My recommend setting for serious players is 3-ply (checker and cube) and 1296 trials minimum. If time allows 3-ply checker and 4-ply cube is a very strong level (still using at least 1296 games).
I’ll pass your request to Neil J. I don’t think strong player have any trouble sharing this information. But if you asked Neil he will tell you he wants to use XGRoller+ within rollout (which will be implemented in the next version). But not many people are willing to spend a week rolling a position.
Neil often rolls first in 2-ply (checker and cube) as it take just a few minutes and then decide if it is worth using stronger settings."
So it is time that one of you sharks tell the fishes, how to use the bots properly. I think, that is even i nice sujet for a new book, like Kit Woolseys "How to play Tournament Backgammon" it could be titled "how to set up a bot for research". Or is there reluctance to grind this bastion?
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.