| |
BGonline.org Forums
A better approch?
Posted By: Timothy Chow In Response To: A better approch? (Tom Keith)
Date: Tuesday, 3 November 2009, at 1:29 a.m.
I wrote: I claim that what you really want to know is which play the bot thinks is better.
Tom wrote: I assume you mean which play will come out best in a rollout with enough trials to clearly distinguish the candidates. (We already know which play the bot thinks is best.)
You're right that I chose my words poorly. What I meant was that we want to know which play would come out best if we could roll out every possible eventuality.
I wrote: If you knew for sure that the bot preferred Play A to Play B, would you really care to know that there is a 0.1% probability that extending the rollout would erroneously score Play B higher than Play A? Not really; if you knew for sure that Play A was better than Play B, then you wouldn't care about the chances that the bot would randomly make a mistake if you extended the rollout.
Tom wrote: I'm not sure how this is pertinent.
It's pertinent because that's what you're calculating if you assume that the estimated means and s.d.'s are the correct means and s.d.'s, and calculate the probability of getting a result that ranks the plays in the incorrect order. Conceptually, that is the wrong thing to calculate.
Tom wrote: I would like to know that there is, say, an 85% chance that the current leader is the true leader.
Exactly. And what I'm saying is that the only way you can attach a number to that is to assume a prior probability and calculate a posterior probability, based on the observational evidence.
I wrote: This is not the same as taking the estimated means and standard deviations and calculating tail probabilities.
Tom wrote: Isn't this an implementation issue? Is your point that the usual method of calculating such probabilities might not work very well in this application?
No, it's not an implementation issue; it's a conceptual issue. Think carefully about the meaning of the probabilities you're calculating according to the current approach. They're not giving you what you say you're interested in.
For simplicity, consider the case when there are just two plays. The j.s.d. thing is telling you the probability that you would observe an equity difference as large as (or larger than) what you actually observe, under the null hypothesis that the plays have the same equity. This is conceptually quite distinct from the probability that the higher-scoring play is really the better play.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.