| |
BGonline.org Forums
XG Total cost and Should have won and some stats
Posted By: Christian Plenz
Date: Friday, 8 March 2019, at 2:01 a.m.
Inspired by Matthias and the 'should-have-won' or 'xx.xx% favorite' in the Match Summary that I think has been inherited from Snowie's 'you were xx.xx% favorite' I finished a deeper test on my matches. I wanted to know a little bit more how good the XG estimates are for a single match and for a series of matches and what does it mean at all. And I raised the bar pretty high I guess in checking it's probabilistic estimates ex-post against the actual outcome (won/lost) for accuracy match by match. Regarding the last point I am not sure if this attaches too much of importance to any single XG match analysis since there is so much luck and variance in backgammon and this probably applies as well to the luck how we measure it. Bots are providing their 'guesses' after the match, so why not raise the bar? As a side note in football analytics xG becomes more and more popular. This is 'expected goals' or 'should have scored' and if you derive some winning probabilities from xG it's some kind of equivalent to the 'should-have-won'. Backgammon bot developers were definitely ahead of the times and the accuracy winning award xG vs. XG goes to XG :)
- 1957 matches from online. ELO ratings of players tracked as well for another comparison of the ELO accuracy
- Only complete matches. I had a few more with last games only but skipped them to not have a player starting with a score and luck-handicap.
- MWC cost taken and calculated from the Profile export, opponents just have an entry and for player the sum of move+double+take+resign+take_resign (and double checked some with the match summary to be sure) Cost luck also is given directly
- MWC and luck cost I have capped at 99%, scoring rule cannot deal >100%, anyway only a few and almost can be neglected
- The two groups I separated them later is arbitrary, thought it makes sense to have more or less even matches in one and matches with a favorite 55+% in the other group (no matter who was the favorite - opponent or me)
What are my findings?
- my 'should have won' is pretty close to the actual won matches which is nice
- Luck accuracy is outperforming MWC accuracy which also is nice
- bias towards the underdog
- bias can even out or partially even out for the player-should-have-won since sometimes I was favorite and sometimes underdog according to XG
all 1957 matches
..........................MWC / Luck / ELO
Favorite exp. 58.41% / 58.97% / 57.23%
Favorite act. 52.78% / 56.72% / 55.90%
(MWC this is -5 St.dev(!))
So the Luck accuracy not looking that bad here, 2 St.dev. still ok, also ELO is a bit below expectation. MWC is a big surprise and hard to believe or even trust the result, at least the expectation of MWC and Luck is comparable. No-clue-50:50-guesses would have been more accurate than MWC here, Luck barely managed to outperform 50:50-guess (Brier Scores: 0.2536, 0.2489 and 0.2435. 0.25 would be the 50:50-guess always, lower score is the goal here)
and now the bias..
Group 1 matches (55% or more favo)
..........................MWC / Luck / ELO
matches.... 1065 / 1126 / 1020
Favorite exp. 63.53% / 63.91% / 61.70%
Favorite act. 56.43% / 59.50% / 59.51%
Group 2 matches (50 <55% favo)
..........................MWC / Luck / ELO
matches.... 892 / 831 / 937
Favorite exp. 52.28% / 52.28% / 52.37%
Favorite act. 48.43% / 52.95% / 51.97%
The 48.43% isolated in 892 matches can happen however Luck and ELO managed to be close in Goup 2. The Favo must have been unlucky in the MWC matches sample. The bias is in Group 1 so probably XG is overvaluing the favorites and probably this correlates with how big the favorite is favorite. Luck is -3 St.dev. in Group 2, again better than MWC and worse than ELO
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.