BGonline.org Forums

For Bot developers........

Posted By: Frank Berger
Date: Saturday, 23 July 2011, at 11:56 a.m.

Hi,

as mentioned some time ago, I got the files of the 2010 Benchmark from Michael Depreli and analyzed the data to construct an XML file that contains all positions that were differently played with their rollout data. I carefully checked the data for consistency (I got a pretty good idea how much work it has been to collect this data manually. Kudos to Michael) added rollouts and together with Michael we looked at the problems and fixed them. Feel free to add some additional quality control.

The idea of having such a file is to have a benchmark that allows quick (BGB needs just half an hour) and verifiable results. How difficult it has been to estimate bot strength in the past? Remember the Big-Bot-Shootout? 6000 25-point matches, month of computing time but for statistical relevant results (other than that JF is worse) it is still to few. With the Depreli 2010 benchmark this should have been improved vastly now.

Naturally through the selection of positions, the rollout method etc. there might be deviations from the "truth", but it is better by far than anything we had before.

My expectation is to establish a file format for benchmarks, so we get more of this data in the future. I invite anyone to add suggestions, rollouts (or rollout data with a different bot.. Xavier?) etc., but to have one "master" copy of this file I ask you to send modifications to me.

I would be glad if further benchmarks would be created, or where SW is developed where the creation of such an benchmark is simplified etc. That's just a start I hope.

You find the files here: http://www.bgblitz.de/Depreli2010/dep_2010_id.xml.zip

The sgf files of the matches are here: http://www.bgblitz.de/Depreli2010/sgf.zip The excel file is here: http://www.bgblitz.de/Depreli2010/BOT%20SHOOTOUT500R8.xls (I asked Michael for permission to make them available)

I would have liked to add an part of the XML file, but the layout is destroyed completely so I omitted it here.

ciao Frank