BGonline.org Forums

Minimum # of Trials and Confidence

Posted By: Tom Keith
Date: Saturday, 16 October 2010, at 1:31 p.m.

In Response To: Minimum # of Trials and Confidence (Timothy Chow)

All those confidence percentages are implicitly based on the normal approximation. The normal approximation is correctly only asymptotically, i.e., when the number of trials is large.

I don't agree with the second sentence. Even a single (variance reduced) trial is normally distributed about the rollout mean, or close to it.

The problem with having a very low number of trials is that the bot's estimate of the standard deviation may be off. And if the estimate of SD is off, then the estimate of confidence will also be off.

(Maybe this is what you meant. But I thought it was worth emphasizing the point.)

Messages In This Thread

• Minimum # of Trials and Confidence
  Stick -- Friday, 15 October 2010, at 10:21 p.m.
  • Minimum # of Trials and Confidence
    Timothy Chow -- Friday, 15 October 2010, at 11:08 p.m.
    • Minimum # of Trials and Confidence
      Tom Keith -- Saturday, 16 October 2010, at 1:31 p.m.
  • Minimum # of Trials and Confidence
    Daniel Murphy -- Saturday, 16 October 2010, at 2:22 a.m.
    • Minimum # of Trials and Confidence
      Stick -- Saturday, 16 October 2010, at 5:32 a.m.
      • Minimum # of Trials and Confidence
        Daniel Murphy -- Saturday, 16 October 2010, at 6:34 p.m.
        • Minimum # of Trials and Confidence
          Neil Robins -- Sunday, 17 October 2010, at 5:32 p.m.
          • Minimum # of Trials and Confidence
            Stick -- Sunday, 17 October 2010, at 6:19 p.m.
            • Minimum # of Trials and Confidence
              eXtreme Gammon -- Sunday, 17 October 2010, at 7:27 p.m.
              • Minimum # of Trials and Confidence
                Daniel Murphy -- Wednesday, 20 October 2010, at 5:55 p.m.
    • Minimum # of Trials and Confidence
      Timothy Chow -- Sunday, 17 October 2010, at 4:53 p.m.
      • In defense of single trial rollouts
        Tom Keith -- Sunday, 17 October 2010, at 7:46 p.m.
        • In defense of single trial rollouts
          Maik Stiebler -- Tuesday, 19 October 2010, at 11:42 a.m.
          • In defense of single trial rollouts
            Tom Keith -- Tuesday, 19 October 2010, at 2:54 p.m.
            • In defense of single trial rollouts
              Maik Stiebler -- Tuesday, 19 October 2010, at 3:56 p.m.
            • In defense of single trial rollouts
              Bob Koca -- Tuesday, 19 October 2010, at 7:18 p.m.
              • In defense of single trial rollouts
                Daniel Murphy -- Tuesday, 19 October 2010, at 8:42 p.m.
              • In defense of single trial rollouts
                Tom Keith -- Wednesday, 20 October 2010, at 11:17 p.m.
    • Minimum # of Trials and Confidence
      Tom Keith -- Sunday, 17 October 2010, at 8:19 p.m.
  • OT Minimum # of Trials and Confidence
    svilo -- Saturday, 16 October 2010, at 8:46 a.m.
  • Minimum # of Trials and Confidence
    higonefive -- Saturday, 16 October 2010, at 11:25 a.m.
  • 324 is basically my minimum.
    neilkaz -- Tuesday, 19 October 2010, at 4:50 p.m.