| |
BGonline.org Forums
Understanding the Bots
Posted By: Phil Simborg
Date: Tuesday, 2 March 2010, at 10:53 a.m.
I was asked some insightful questions about bots and ER's and gave the answer my best shot. Perhaps someone here can provide a better answer or confirm my answers.
The question was:
"One thing I have always found with bot analysis is that, if you play OK in a game where you have a series of very tough decisions, you'll end up with a low rating. Conversly, you can be awarded world champ status in a game where you've basicallyhad nothing to do.
I understand why this happens and obviously the error analysis should be absolute, at least if it is to be used as a performance measure over a variety of games. But I was wondering if there had been any attempts to measure the difficulty of backgammon decisions? It would be useful to have some sort of rough stat for a game, indicating the 'tarriff', if you like. It might also be interesting to see what a typical players difficulty profile is - i.e. do players generally perform worse on tough decisions, or are there some freaks out there who get bored but manage to raise their game for the tough stuff?"
And my response was:
You raise some very interesting points that I have discussed for several years, as the same issues certainly arose when Snowie was the bot of choice.
To my knowledge, there is no way to determine how difficult a given play is other than seeing how many humans miss the play and how many good players have trouble with a given play. So the bot cannot know this. I guess it is logical to assume that the worse one's PR or Elo, the more tough plays there were.
You are absolutely correct that generally, the worse you played a given game, the more likely it is that you simply had more difficult decisions to make that game. That explains why I might have a PR of .0 one game and 11.0 the next.
As for whether some players do better with tougher decisions, that's pretty much a self-cancelling, or oxymoron statement, because the better they do the easier it was FOR THEM. The point is that no matter what your level of play, everyone has their own demons as far as what is tough and what is easy.
A very good example is a 15 question quiz that Falafel gave in Istanbul last weekend. I got a copy and I missed 5 of the questions. Matt Cohn-Geier, one of the best players in the world, missed 4, and Joe Russell, another one of the best players in the world, missed 3, as did Kit Woolsey. The interesting thing is there was only 1 question that all three of us missed, and the rest were scattered. So even though these three players are far better than I, there were some I got right that they missed.
Lucky guess could be one reason; the quiz factor could be another; or simply different knowledge and skill traits could be the answer. Also note that with them missing 3 and 4 and me 5, it would appear our skill levels are close, but overall, they probably have an average PR rating 3 or 4 points lower than mine...so the other factor could be that a 15 question quiz is simply not a long enough sample from which to draw any meaningful conclusions.
So the answer is that you have to use a very large sample and not get hung up on how you rate in a single match or game, and you really have to compare apples to apples to see differences in skill. That is one reason why I have been giving quizzes with identical positions to people to see how they do.
I gave a 10 question cube quiz in Pittsburgh and a 10 question checker play quiz in New Mexico. Kit Woolsey won the quiz in New Mexico, though he missed a question that most intermediates got right and handled all the very tough ones well.
In the doubling cube quiz it was completely unpredictable which questions the top players missed. So I cannot tell you which ones were the hardest (for top players), though I can predict fairly easily which ones intermediates would most likely miss.
There are other, far better players with more experience than I who might have a different answer to your question, so I will post this on the BG forum and let you know if I get more insights for you or a different answer.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.