[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums

Minimum # of Trials and Confidence

Posted By: Timothy Chow
Date: Sunday, 17 October 2010, at 4:53 p.m.

In Response To: Minimum # of Trials and Confidence (Daniel Murphy)

The specific thing that I thought was an overstatement was the claim that a single trial with variance reduction might be enough.

It's important to distinguish between confidence in what the right play is and confidence in what the bot thinks the right play is. Your example of an opening 31 looks to me like a conflation of these two questions. If you only care about the right play and you're 100% confident, sans bot, that you know what the right play is, then you certainly don't "need" 324 trials to obtain 100% confidence. You don't need any trials at all. In fact, when your mind is already made up, doing trials with the bot carries a slight risk of confusing you with facts.

But if you talk about a single variance-reduced trial with the bot, then my presumption, and I think the tacit presumption of many people, is that you are trying to find out what the bot thinks. If you then claim that a single trial is "enough" based on the reported confidence numbers, then my response is that the estimated confidence is just that: an estimate. It's an estimate based on a very limited number of queries to the bot and the use of the normal approximation to convert standard deviations into percentages. It could be significantly far off from the truth. Whatever your threshold happens to be (90% or 95% or 99% or whatever), I think you're making a mistake by assuming that just because your threshold happens to be exceeded after a single variance-reduced trial, then that's "enough."

Another way to put the point is that when you do a rollout, not only is the equity an estimate, so is the confidence percentage that the bot reports. A single variance-reduced trial is not enough to make me, or many other people, confident (in the ordinary sense of the word) that the bot's estimated confidence percentage is accurate.

Messages In This Thread

 

Post Response
Your Name:
Your E-Mail Address:
Subject:
Message:

If necessary, enter your password below:

Password:
 

 

[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums is maintained by Stick with WebBBS 5.12.