BGonline.org Forums

The statsig police

Posted By: Timothy Chow
Date: Sunday, 18 October 2009, at 2:34 p.m.

In Response To: The statsig police (Maik Stiebler)

Maik Stiebler wrote:

I'm afraid I don't see the point. If a play is rolled out where the error magnitude is huge, why do we want its contribution to total variance to be proportional to the error magnitude, if we can, with comparatively little effort, reduce it even further?

Of course, if we have unlimited computational resources, then there is no need to worry about this issue; simply roll everything out until the j.s.d. drops below a millipoint, or a tenth of a millipoint, or whatever.

My assumption, though, is that under the current strategy as you describe it, there will be cases where after 1296 trials, the j.s.d. will still be larger than the square root of the estimated equity difference. For example if the equity difference is 81 millipoints, then the j.s.d. might be 20 millipoints after 1296 trials, which is larger than sqrt(81) = 9. Even worse, you might stop before 1296 trials, when you reach a j.s.d. of 81/2.33 = 35 millipoints.

I'm suggesting rolling things out at least until the square-root threshold is reached. If you have the resources to do more then that's fine. The square-root threshold ensures that if we get a final total of 40000 millipoints then the standard deviation will be at most 200 millipoints. With a fixed j.s.d. threshold, or a fixed number-of-j.s.d.s threshold, you won't get this kind of a priori guarantee of the accuracy of the final total.

Messages In This Thread

• The statsig police
  Timothy Chow -- Saturday, 17 October 2009, at 9:00 p.m.
  • The statsig police
    Maik Stiebler -- Sunday, 18 October 2009, at 11:01 a.m.
    • The statsig police
      Timothy Chow -- Sunday, 18 October 2009, at 2:34 p.m.
      • The statsig police
        Maik Stiebler -- Sunday, 18 October 2009, at 2:56 p.m.
        • The statsig police
          Timothy Chow -- Monday, 19 October 2009, at 1:48 a.m.
          • The statsig police
            Maik Stiebler -- Monday, 19 October 2009, at 4:21 p.m.
            • The statsig police
              Timothy Chow -- Tuesday, 20 October 2009, at 2:19 a.m.
      • PS
        Maik Stiebler -- Sunday, 18 October 2009, at 3:10 p.m.
    • The statsig police
      Philippe Michel -- Sunday, 18 October 2009, at 3:57 p.m.
      • The statsig police
        Maik Stiebler -- Sunday, 18 October 2009, at 4:37 p.m.
        • The statsig police
          Philippe Michel -- Sunday, 18 October 2009, at 7:23 p.m.
  • The statsig police
    neilkaz -- Sunday, 18 October 2009, at 5:29 p.m.
    • The statsig police
      Timothy Chow -- Monday, 19 October 2009, at 1:54 a.m.
  • Another example, to clarify
    Timothy Chow -- Monday, 19 October 2009, at 2:15 a.m.
    • Another example, to clarify
      Timothy Chow -- Monday, 19 October 2009, at 2:21 a.m.
  • The statsig police
    Michael Depreli -- Monday, 19 October 2009, at 7:16 a.m.
    • The statsig police
      Timothy Chow -- Tuesday, 20 October 2009, at 2:10 a.m.
      • The statsig police
        MaX -- Thursday, 22 October 2009, at 12:19 p.m.
        • The statsig police
          Timothy Chow -- Friday, 23 October 2009, at 3:35 p.m.
          • The statsig police
            MaX -- Monday, 26 October 2009, at 4:32 p.m.
        • The statsig police
          Daniel Murphy -- Friday, 23 October 2009, at 3:50 p.m.