[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums

Is XG confidence confident?

Posted By: Timothy Chow
Date: Thursday, 6 May 2010, at 6:25 p.m.

In Response To: Is XG confidence confident? (mtuhtan)

The intended meaning of 95% confidence is that if the two plays were actually equal then only 5% of the time would you see a difference as large as actually observed due to random fluctuation.

But how does XG know this? It doesn't. It has to make an assumption that the rollouts are randomly sampling from a Gaussian distribution. Then it has to estimate the variance of the Gaussian distribution from the random samples.

The problem is that the distribution is not, in reality, Gaussian. Especially when the cube is in play, the deviation from Gaussian can be substantial. This creates two sources of error: first, the estimated variance may be quite different from the true variance, and second, the translation between variance and % confidence can be off.

This is not to say that the 95% confidence intervals are useless, but they should be taken with a grain of salt. They're really 95% only under certain optimistic assumptions. Unfortunately there's no easy way to compensate for the violation of the assumptions. We just have to live with the uncertainty. In general, though, I would say that when the bot says "95% confidence" you should read that as "at most 95% confident." The more volatile the position and the more likely the cube is going to be turned multiple times in the subsequent play, the less you should trust the 95% figure.

Messages In This Thread

 

Post Response
Your Name:
Your E-Mail Address:
Subject:
Message:

If necessary, enter your password below:

Password:
 

 

[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums is maintained by Stick with WebBBS 5.12.