| |
BGonline.org Forums
More data to support making the 2 pt with opening 64
Posted By: Daniel Murphy In Response To: More data to support making the 2 pt with opening 64 (Timothy Chow)
Date: Wednesday, 10 February 2010, at 9:14 p.m.
No, you made the assumption that it is equally likely that the error manifests in either direction. This is stronger than assuming only that the error could go in either direction. And this is the assumption that Daniel Murphy says might be true, but is an assumption.
Aye.
Another approach: So you have a rollout that says Play A is 0.004 better than Play B, with, say, 95% confidence interval of +- 0.002. We can be pretty confident that with the settings used in the rollout that Play A is better than Play B (call the plays "tied" if you like, by better I mean that if we do another rollout with the same settings, we're likely to get a result very close to the result we have. But we don't just want to know if the Play is better with those settings. We want to know if Play A is better, period, with perfect play. And there may be a lot of uncertainty about that. The closer to the opening roll, the more uncertainty. The greater the likelihood that the bot will reach positions (or already is in one) that it doesn't understand, the greater the uncertainty. The greater the likelihood that the two plays we are comparing will lead to divergent positions which the bot may play unequally perfectly, the greater the uncertainty.
So suppose we're talking about a opening-game rollout. Did the bot play perfectly? That we can answer: clearly not. But there are other questions that don't have easy answers.
How imperfectly did it play?
If the rollout found that the equity for Play A was 0.050, what's the true equity of Play A? 0.050? 0.052? 0.048? 0.060? 0.040? Greater or lesser? And the same for Play B -- is it 0.046? 0.0.48? 0.044? More? Less? Do you know? If so, how?
If we've rolled out two plays, there are four sources of possible error:
(1) After Play A, the bot made mistakes for Player 1.
(2) After Play A, the bot made mistakes for Player 2.
(3) After Play B, the bot made mistakes for Player 1.
(4) After Play B, the bot made mistakes for Player 2.
So then, another difficult question -- how bad were the errors from source (1) (2) (3) and (4)? What's the likelihood that each of these was no more than 0.0005, or 0.001, or 0.002, or 0.005? Or 0.010? Or 0.030? And in which direction? Pick a number if you can; how do justify that number? If you assume that the four sources of error somehow balance out, so that the rollout result is more likely than not to have found the best play, what's the justification for that assumption? Alternatively, if you assume that even if the four sources of error don't balance out precisely, the magnitude of each source is small (where "small" is small enough not to have affected the reliability of the rollout), what's the justification for that assumption? And note that if the difference between plays in your rollout is only 0.004, the bot's deviation from perfect play in case (1) (2) (3) and (4) has to be quite small simply to dismiss the possibility that the rollout has not found the best play, especially when we are uncertain both of the direction and of the magnitude of the error in each of the four cases.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.