[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums

Dear Mr. Depreli

Posted By: Frank Berger
Date: Sunday, 14 April 2013, at 8:30 p.m.

In Response To: Dear Mr. Depreli (Christian Sorensen)

I have seen some of your studies and i think some of your conclusions are wrong. In the Michael Depreli study from 2012, which is posted on http://www.extremegammon.com/studies.aspx . There is no way that GNU and XG can be compared by speed and strength. The neural networks that are used to create the bots are different so they will get a different output. So analysing GNU and XG, by making a rollout will always favor XG in the end because they have the same neuralnet. they will have the same errors or imperfections and will favor eachother, while GNU is screaming in the background that they are wrong.

Hm... So you say that if you roll out a position up to 45000 times with an good bot and high settings the result is of no value?

If you take the setup: We have 5000 difficult positions and look at the score the bots achieve. This is something not unusual in other games, e.g. chess.

So actually the results of the studies are complete rubbish.

I personally would be very cautious, giving such a statement if I haven't carefully studied the approach, especially given that the people involved may know what they are doing.

The only way to check which bot is stronger is to let them play against eachother, and let those bots use the same time of processing power. It may take a while and more than a I7 core, but the results will be fair. We would only have to check the win-rate of the bots, it will take a LOT of matches but it will be indepent. Not only one-sided "study".

A really great idea and never heard before. Do you want to volunteer? There is currently no way to do a automatic bot-vs-bot gameplay for the relevant bots. So it has to be done manually. Do you have an idea how many matches you need to get something statistically significant?
And even if you have a bot: Thorsten Schoop has done this some years ago. 1000 25-pt matches between each bot supported by the "Dueller". He devoted 2-4l computers and about half a year time and the only really statistically significant result was that Jellyfish is weaker than Gnubg, Snowie and BGBlitz. So how any years you're willing to invest?

And with the next release, of a bot (with an AI change) you have to do part of the effort again. And again. And again. Don't you think that it is much more practical to run the positions through the bot and half an hour later you have a number?

I really don't know which bot is best. but what i do know is that there needs to be done real research into which bot is best. Assuming that XG is the best isn't enough, there should be proof, real proof. Currently no one knows this for *sure*. Feel free to surprise us with an elegant and realistic approach to measure the bots. This surely will deserve a round of applause. But before I suggest you're be a little more reluctant judging ideas and work of others.

Messages In This Thread

 

Post Response
Your Name:
Your E-Mail Address:
Subject:
Message:

If necessary, enter your password below:

Password:
 

 

[ View Thread ] [ Post Response ] [ Return to Index ] [ Read Prev Msg ] [ Read Next Msg ]

BGonline.org Forums is maintained by Stick with WebBBS 5.12.