| |
BGonline.org Forums
New GnuBg - DMP Battles
Posted By: playBunny
Date: Tuesday, 7 May 2013, at 3:42 p.m.
Michael Petch: One of the reasons I have held back from officially releasing Windows builds with the new neural net was to determine whether there was a sizable improvement in strength.
I've been doing a big DMP test, churning out a million and half games last month. The test covers both versions (2012, 2013) and all plies (0 to 3), each engine against the other (16 battles) and each against itself where plies differ (2 x 6 battles). All games have been played with an individual seed, although each battle has used the same set of seeds so that they are comparable. Each engine has played both sides of each seed in each battle so that the luck is somewhat balanced out.
Having the engines play both sides of each seed allows comparison between the set of games where Engine A plays the opening roll and those where it's Engine B. As a further examination of the seeds, I've had both engines play themselves at the same ply (2 x 4 battles) to see what the wins/losses split is.
One result of using the same set of seeds in each battle is that it's possible to enumerate how many wins the starter can achieve with a given seed. Some of the seeds are unwinnable when played at bot level and some are unlosable. I think that these should be discounted.
The hardest part is presenting the results. I have a mind-boggle of grids and possible graphs. Any encouragement would be appreciated as I have a highly distractable mind and staying focused can be a significant challenge. Think ADD. I've been working on the report sporadically for the last week and I'm struggling to stick with it. I don't like admitting this but it's the reason that I'm posting this now instead of posting the results when they are ready. I fear that they may not become ready and the effort will be wasted. :-/
I'd also appreciate ideas as to what to show. I'm really very much a novice statistician, having started with "Statistics for Dummies" only a couple of months ago. ;-)
One question is what is the value to use? If Engine A wins 48.2% and Engine B wins 51.8%, is the figure of interest the 1.8% deviation from the expected 50:50 share, the 3.6% difference, the proportion 48.2/51.8 or something else? So far I've been using the 1.8% in my examination of the results.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.