BGonline.org Forums

DQ: Consider the bar point? *** Rollout ***

Posted By: Timothy Chow
Date: Wednesday, 23 July 2014, at 10:12 p.m.

In Response To: DQ: Consider the bar point? *** Rollout *** (Tom Keith)

I didn't notice this post until just now.

I'm not sure I understand your objection. There are, as you point out, two separate issues.

1. Whether the so-called "probability of being best" really is that.

2. Whether that number is useful.

I did point out that the answer to #1 is no, strictly speaking. I also think that the answer to #2 is "not particularly," but my reasons for that have nothing to do with my view of #1. So the baby/bathwater analogy doesn't make any sense to me.

Let me accept the prevalent view that the answer to #1 is yes, for the sake of argument. In most cases, I just don't care that much about the probability that the top play is the best one. I care more about the equity difference between the plays. I believe that that's actually what most others are interested in too. But they are distracted by the probability of being best. So they terminate certain rollouts when the probability of being best reaches 100%, and then they go away with the impression that the computer is 100% confident that the second-best play is a whopper, when the computer says nothing of the sort. Or they run a rollout for days to drive the probability to 100% when a short rollout already tells them the top two plays are essentially tied, which is all we really care about in almost all cases.

There's a whole book called "The Cult of Statistical Significance" that shows how this kind of misguided attitude leads to all kinds of problems in the medical world and elsewhere in science. Now, that book is very ranty and I don't agree with everything in it, but the basic point that there is far too much attention paid to the binary question of whether X is equal or not equal to Y, rather than the amount by which they are different, is a valid one IMO.

Post Response

Subject:
Message: