[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums

How to design a statisctally significant bot head-to-head?

Posted By: Ian Shaw
Date: Monday, 11 January 2010, at 2:04 p.m.

In Response To: How to design a statisctally significant bot head-to-head? (Ian Shaw)

Thanks to everybody who's replied for their responses.

In my example, I have a estimate that Bot A is 0.005 ppg better than Bot B. So its probability of winning is 50.0025. Putting this into Timothy's formula I get 4/(50.0025 - 50)^2 = 640,000 games. In fact, I need more because the variance for cubeless money games exceeds 1. So I probably need at least 1 million games to identify which is better. Fortunately, I'm only running a static evaluation (gnubg 0-ply, Sn & XG 1-ply) , so the timescale is manageable.

Bob's idea of matched pairs is also interesting. A lot of games might cancel out if the bots are equally strong. Any idea of how many games this might take to get a result? Is this a calculation I can readily find on the net, or can you give a quick example?

Messages In This Thread

 

Post Response
Your Name:
Your E-Mail Address:
Subject:
Message:

If necessary, enter your password below:

Password:
 

 

[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums is maintained by Stick with WebBBS 5.12.