| |
BGonline.org Forums
The statsig police
Posted By: Timothy Chow
Date: Saturday, 17 October 2009, at 9:00 p.m.
Although Michael Depreli specifically asked the statsig police to stay away from his excellent bot comparison project until later, I wanted to make a comment now because it concerns something that may affect the way the project is carried out. It's possible that people already understand the point that I'm about to make, but since it's an easy thing to overlook, I want to say it just in case.
To state the punchline first, my suggestion is that rollouts be continued until the j.s.d. is equal to (or proportional to) the square root of the equity difference (measured in millipoints) and not until the equity difference is some constant multiple of the j.s.d., as is often done.
Here's the reason. The project is 6% done and we're observing error totals in the 2000 to 3000 millipoints range. Let's suppose that when we're done we'll have error totals around 40000 millipoints. We'd like to be able to say that the error total is 40000 plus or minus some relatively small number, say 200 millipoints. I've picked 200 here because, conveniently, it's the square root of 40000; the importance of this fact will become clear shortly.
How are we getting the error total X = 40000? Well, we're adding up a bunch of individual errors: X = X1 + X2 + ... + Xn. Each Xi has some mean mi and variance si2. The variance s2 of X is the sum of the si2. If we roll things out until each si2 equals mi (i.e., until the [estimated] j.s.d. equals the square root of the [estimated] equity difference), then s2 will be equal to the sum of the mi (i.e., the standard deviation of the final error total will be [approximately] equal to the square root of the final error total).
In contrast, if we roll things out until the equity difference is some constant multiple of the j.s.d., then the standard deviation of the total error will be messy—it will be proportional to the square root of the sum of the squares of the individual j.s.d. values, and so the larger equity differences will contribute unduly to the uncertainty of the final error total.
Implementing my suggestion will probably increase the computation time. If at present, one is stopping when the equity difference is 3 j.s.d.s, then for an equity difference of 81 millipoints one would stop when the j.s.d. gets down to 27, whereas I'm suggesting stopping only when the j.s.d. gets down to 9, the square root of 81. However, if we want the final total (40000 according to my estimate above) to be "correct" within plus or minus 200 millipoints, then I don't think there is any way around doing more computation.
Finally, note that instead of the square root, one can of course take some constant multiple of the square root. Stopping when we reach k times the square root of the equity difference for each of the individual rollouts will give us an overall standard deviation that is (about) k times the square root of the error total.
| |
BGonline.org Forums is maintained by Stick with WebBBS 5.12.