BGonline.org Forums

How to design a statisctally significant bot head-to-head?

Posted By: Ian Shaw
Date: Friday, 8 January 2010, at 4:23 p.m.

I've a question for the stats experts.

If I want to play off two bots in a head-to-head contest, how do I choose the right number of games to play, given the following assumptions?

They will play cubeless money games.
I believe that the two bots are very close in strength.
I have the time to play lots of games.

My statistical knowledge is very limited, but I want the test to be rigorous. As I understand it, one starts with the null hypothesis that "The two bots are equal in strength". Then one should run a test with enough samples to give a statistically significant result; you are not supposed to keep extending the test until one bot is far enough in front. If the result is significant, you reject the null hypothesis.

In the end, of course, I want to know which bot is better. Will the test to see whether they are equal answer this question.

Is the design of the test affected if I have a benchmark result which implies that Bot A is x ppg better than Bot B?

For speed, can I run a shorter test simultaneously on several processors, and combine the results by summing them, weighted by the number of games completed, of course?

Messages In This Thread

• How to design a statisctally significant bot head-to-head?
  Ian Shaw -- Friday, 8 January 2010, at 4:23 p.m.
  • How to design a statisctally significant bot head-to-head?
    Timothy Chow -- Friday, 8 January 2010, at 10:18 p.m.
    • How to design a statisctally significant bot head-to-head?
      Bob Koca -- Sunday, 10 January 2010, at 6:45 p.m.
  • How to design a statisctally significant bot head-to-head?
    eXtreme Gammon -- Sunday, 10 January 2010, at 7:58 p.m.
    • Follow up for match play
      eXtreme Gammon -- Sunday, 10 January 2010, at 8:08 p.m.
    • How to design a statisctally significant bot head-to-head?
      Bob Koca -- Monday, 11 January 2010, at 12:49 a.m.
      • How to design a statisctally significant bot head-to-head?
        eXtreme Gammon -- Monday, 11 January 2010, at 1:02 a.m.
        • How to design a statisctally significant bot head-to-head?
          Bob Koca -- Monday, 11 January 2010, at 1:06 a.m.
    • How to design a statisctally significant bot head-to-head?
      Matt Cohn-Geier -- Monday, 11 January 2010, at 9:06 a.m.
    • How to design a statisctally significant bot head-to-head?
      Rich Munitz -- Monday, 11 January 2010, at 2:29 p.m.
  • How to design a statisctally significant bot head-to-head?
    Ian Shaw -- Monday, 11 January 2010, at 2:04 p.m.
    • How to design a statisctally significant bot head-to-head?
      Maik Stiebler -- Monday, 11 January 2010, at 4:08 p.m.