BGonline.org Forums

The statsig police

Posted By: Maik Stiebler
Date: Monday, 19 October 2009, at 4:21 p.m.

In Response To: The statsig police (Timothy Chow)

If the equity difference is small, say 1 millipoint, then following my prescription, one would stop the rollout when the j.s.d. was 1 millipoint. One wouldn't bother continuing until the j.s.d. dropped to 1/2.33 millipoints.

Sure. But with the current method, one usually wouldn't bother either, because one is supposed to do a maximum of 1296 trials.

So in some cases you would be doing less work than the current method, rather than more.

I never wrote anything to the contrary.

you can also apply the square-root threshold when deciding how to allocate resources during the second pass.

Our disagreement seems to lie in the question what our goals should be when allocating resources. Let's consider the following example: We only have two play decisions with two play options each. After spending a total of 2 days of computational resources, one day and 1296 trials on each decision, we learn that for the first decision the equity difference is 100 millipoints, jsd 20 millipoints (I'm aware that's not possible with the current method), while for the second decision, the equity difference is 10 millipoints, jsd also 20 millipoints. We decide that we are not happy with the jjsd of sqrt(2)*20 millipoints, so we decide to spend more computational resources totalling one day. How would you spend it?

With a square-root threshold, the size of the uncertainty depends only on the size of the final error total, not on how many errors were made or how big they were

Yes, that's cute.

Messages In This Thread

• The statsig police
  Timothy Chow -- Saturday, 17 October 2009, at 9:00 p.m.
  • The statsig police
    Maik Stiebler -- Sunday, 18 October 2009, at 11:01 a.m.
    • The statsig police
      Timothy Chow -- Sunday, 18 October 2009, at 2:34 p.m.
      • The statsig police
        Maik Stiebler -- Sunday, 18 October 2009, at 2:56 p.m.
        • The statsig police
          Timothy Chow -- Monday, 19 October 2009, at 1:48 a.m.
          • The statsig police
            Maik Stiebler -- Monday, 19 October 2009, at 4:21 p.m.
            • The statsig police
              Timothy Chow -- Tuesday, 20 October 2009, at 2:19 a.m.
      • PS
        Maik Stiebler -- Sunday, 18 October 2009, at 3:10 p.m.
    • The statsig police
      Philippe Michel -- Sunday, 18 October 2009, at 3:57 p.m.
      • The statsig police
        Maik Stiebler -- Sunday, 18 October 2009, at 4:37 p.m.
        • The statsig police
          Philippe Michel -- Sunday, 18 October 2009, at 7:23 p.m.
  • The statsig police
    neilkaz -- Sunday, 18 October 2009, at 5:29 p.m.
    • The statsig police
      Timothy Chow -- Monday, 19 October 2009, at 1:54 a.m.
  • Another example, to clarify
    Timothy Chow -- Monday, 19 October 2009, at 2:15 a.m.
    • Another example, to clarify
      Timothy Chow -- Monday, 19 October 2009, at 2:21 a.m.
  • The statsig police
    Michael Depreli -- Monday, 19 October 2009, at 7:16 a.m.
    • The statsig police
      Timothy Chow -- Tuesday, 20 October 2009, at 2:10 a.m.
      • The statsig police
        MaX -- Thursday, 22 October 2009, at 12:19 p.m.
        • The statsig police
          Timothy Chow -- Friday, 23 October 2009, at 3:35 p.m.
          • The statsig police
            MaX -- Monday, 26 October 2009, at 4:32 p.m.
        • The statsig police
          Daniel Murphy -- Friday, 23 October 2009, at 3:50 p.m.