| |
BGonline.org Forums
Somewhat disturbing GNU 4-ply error
Posted By: Daniel Murphy In Response To: Somewhat disturbing GNU 4-ply error (Timothy Chow)
Date: Sunday, 11 October 2009, at 7:13 p.m.
Well, I think
(1) it's a leap to look at one grossly misevaluated position and think that the bot is 'probably' misevaluating many others. Although, it probably is. In fact, we know it is. But
(2) it doesn't matter how far off the evaluation is as long as gnubg makes the best decision. OR, it doesn't matter much if the evaluation is wrong but the real error is small. We know that Gnubg makes plenty of errors in its opening roll rollouts. We can know this simply by running 2-ply and 3-ply analysis and seeing that often they disagree; when they do, one or the other or both must be wrong.
In this case, the actual error on the take (if it is an error) is probably less than 0.020. Even if this is a position that may be reached at Move X from the opening position (it's not clear to me if it was, or if you constructed it), the impact of that one error among 1296 trials is very small.
--
The bots are a lot better than they used to be. But there's still a lot of handwaving in accepting how accurate they now are (talking opening roll positions). One handwave is the idea that the bot makes errors for both sides and somehow they even out. Another is the idea that the bots are so much better now in certain types of positions that they used to be bad in that their opening rollouts are now more accurate. I think they are better in those types of positions, and their rollouts are more accurate, and errors do tend to even out, but I also think that one reason experts are more accepting of some of the most recent results is that those results are more compatible with expert expectations, and another is that expert opinion is that overall the latest bots play very well.
However, although we know the bots will misplay some types of positions, we (a) don't know enough to quantify the effect of that misplay on opening rollouts, and (b) don't even know enough about how often Play A or B will lead to misplayed positions (I'm thinking of "slotting leads more often to this and splitting leads more often to that" -- really ? Does it ? And how often?)
My feeling is that the bots are now good enough (that is, have much less systematic error than before) that if they say a 1st, 2nd or 3rd roll play choice is very close, it's likely to be very close, and if a play is clearly an error (-0.030?), it very likely is. Is that good enough until Godbot comes along? What else can we do?
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.