BGonline.org Forums

Confidence Interval Overlap vs Joint Standard Deviation (long)

Posted By: neilkaz
Date: Saturday, 31 October 2009, at 5:13 p.m.

There's been some discussion and concern here since GNU rollouts use JSD and since XG rollouts like Snowie rollouts report as a 95% confidence interval.

Way back when Snowie was the only really good bot on the scene (Was this before or after the Norman Conquest?..I dunno...too many pain pills here to recall that long ago) I used feel really confident about a rollout if the confidence intervals didn't overlap. Knowing that each specific interval gave 95% confidence made me feel quite certain that if those intervals didn't overlap, or even if they just touched, that the RO result was giving me almost 100% stat sig.

Noting that I only took a couple college prob/stat courses 30 years ago, I'll do my best with some basic stat now. Those here who are better than me in math can please correct, or tell me that I'm far wrong.

It all starts with standard deviation. GNU reports as std dev. XG and Snowie report as 95% CI. A 95% CI is a range of data that is +/- 1.96 SD from the mean.

Lets say that a RO resulted in a position being a .96 take with a .005 SD. The 95% CI for this RO is .96 +/- .0098. I'll simplify and call the CI .96 +/- .01 since multiplying by 2 is almost the same as multiplying by 1.96. I'll use .005 as SD and .010 as 95% CI for the remainder of this discussion.

OK now lets say that we are rolling out one play vs another, and play A results in .92 and play B results in .94 emg. Both plays have a SD of .005.

The CI for both plays is therefore ~.01. So XG would show play A confidence as (.91 to .93) and Play B confidence as (.93 to .95). Does the fact that these CI touch make you feel uncomfortable that Play A may actually be superior to play B? Well, it shouldn't !

"Joint Standard Deviation The standard deviation of the difference between two rollouts: JSD = sqrt(SD1*SD1 + SD2*SD2). A measure of how statistically significant the result is." comes from Tom's site.

Lets calculate JSD here. Sqrt(.025+.025)= .00707...or about 1.4 times SD when SD's are the same.

Hmmm...Play A was .92 and Play B was .94 emg. They differ by .020 which is almost 3 times the JSD of .007 ! ie this is more than 99% confidence that play B would finish ahead of play A if we could extend the rollout until the end of time. 99+% confidence is certainly good enough for me.

What we see here is that if the two 95% CI's don't overlap or if they even touch we have at least 99% confidence.

What more do we need? XG is blazing fast for RO's and I am very comfortable with any results that don't have CI overlap and I think all should be comfortable with this method as well.

Who needs JSD? ..well not me... neilkaz ..

Messages In This Thread

• Confidence Interval Overlap vs Joint Standard Deviation (long)
  neilkaz -- Saturday, 31 October 2009, at 5:13 p.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Timothy Chow -- Saturday, 31 October 2009, at 5:30 p.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Tom Keith -- Saturday, 31 October 2009, at 5:51 p.m.
    • Confidence Interval Overlap vs Joint Standard Deviation (long)
      Rich Munitz -- Sunday, 1 November 2009, at 3:19 a.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Bob Koca -- Saturday, 31 October 2009, at 7:30 p.m.
    • Confidence Interval Overlap vs Joint Standard Deviation (long)
      neilkaz -- Saturday, 31 October 2009, at 9:20 p.m.
      • Confidence Interval Overlap vs Joint Standard Deviation (long)
        Bob Koca -- Saturday, 31 October 2009, at 9:31 p.m.
      • Confidence Interval Overlap vs Joint Standard Deviation (long)
        Timothy Chow -- Sunday, 1 November 2009, at 12:21 a.m.
  • A better approch?
    eXtreme Gammon -- Sunday, 1 November 2009, at 4:02 a.m.
    • A better approch?
      neilkaz -- Sunday, 1 November 2009, at 4:32 a.m.
    • A better approch?
      MaX -- Sunday, 1 November 2009, at 1:19 p.m.
    • A better approch?
      MaX -- Sunday, 1 November 2009, at 1:22 p.m.
    • A better approch?
      Timothy Chow -- Sunday, 1 November 2009, at 5:37 p.m.
      • A better approch?
        eXtreme Gammon -- Sunday, 1 November 2009, at 6:12 p.m.
        • A better approch?
          Bob Koca -- Sunday, 1 November 2009, at 7:36 p.m.
          • A better approch?
            Tom Keith -- Sunday, 1 November 2009, at 8:38 p.m.
            • A better approch?
              Chuck Bower -- Monday, 2 November 2009, at 1:20 a.m.
              • A better approch?
                Jason Lee -- Monday, 2 November 2009, at 2:24 a.m.
                • A better approch?
                  Chuck Bower -- Monday, 2 November 2009, at 3:06 p.m.
              • A better approch?
                Timothy Chow -- Monday, 2 November 2009, at 6:56 p.m.
        • A better approch?
          Maik Stiebler -- Sunday, 1 November 2009, at 10:05 p.m.
        • A better approch?
          MaX -- Monday, 2 November 2009, at 10:15 a.m.
          • A better approch?
            Timothy Chow -- Monday, 2 November 2009, at 6:47 p.m.
            • A better approch?
              MaX -- Tuesday, 3 November 2009, at 8:18 a.m.
          • A better approch?
            eXtreme Gammon -- Monday, 2 November 2009, at 7:26 p.m.
            • A better approch?
              MaX -- Thursday, 12 November 2009, at 12:56 p.m.
        • A better approch?
          Timothy Chow -- Monday, 2 November 2009, at 6:40 p.m.
          • A better approch?
            Tom Keith -- Monday, 2 November 2009, at 7:28 p.m.
            • A better approch?
              Timothy Chow -- Monday, 2 November 2009, at 10:10 p.m.
              • A better approch?
                Tom Keith -- Tuesday, 3 November 2009, at 12:14 a.m.
                • A better approch?
                  Timothy Chow -- Tuesday, 3 November 2009, at 1:29 a.m.
                  • A better approch?
                    Maik Stiebler -- Tuesday, 3 November 2009, at 7:18 a.m.
                    • A better approch?
                      Timothy Chow -- Tuesday, 3 November 2009, at 4:17 p.m.
                      • A better approch?
                        Tom Keith -- Tuesday, 3 November 2009, at 5:23 p.m.
                        • A better approch?
                          Timothy Chow -- Tuesday, 3 November 2009, at 5:37 p.m.
                          • Aren't you two just particpating in the 250 year old fracas?
                            Chuck Bower -- Tuesday, 3 November 2009, at 6:19 p.m.
                            • Aren't you two just particpating in the 250 year old fracas?
                              Timothy Chow -- Tuesday, 3 November 2009, at 7:55 p.m.
                              • Aren't you two just particpating in the 250 year old fracas?
                                Bob Koca -- Tuesday, 3 November 2009, at 8:29 p.m.
                            • Aren't you two just particpating in the 250 year old fracas?
                              Tom Keith -- Wednesday, 4 November 2009, at 12:11 p.m.
                      • A better approch?
                        Maik Stiebler -- Wednesday, 4 November 2009, at 8:40 p.m.
                        • A better approch?
                          Timothy Chow -- Wednesday, 4 November 2009, at 10:09 p.m.
                          • A better approch?
                            Maik Stiebler -- Wednesday, 4 November 2009, at 10:51 p.m.
                            • A better approch?
                              Maik Stiebler -- Wednesday, 4 November 2009, at 11:57 p.m.
                            • A better approch?
                              Timothy Chow -- Thursday, 5 November 2009, at 6:52 a.m.
                              • A better approch?
                                Maik Stiebler -- Thursday, 5 November 2009, at 7:25 a.m.
                                • A better approch?
                                  Timothy Chow -- Thursday, 5 November 2009, at 3:37 p.m.
                                  • A better approch?
                                    Maik Stiebler -- Thursday, 5 November 2009, at 9:19 p.m.
                                    • A better approch?
                                      Maik Stiebler -- Thursday, 5 November 2009, at 9:42 p.m.
          • A better approch? [LONG]
            MaX -- Thursday, 5 November 2009, at 3:04 p.m.
            • A better approch? [LONG]
              Timothy Chow -- Thursday, 5 November 2009, at 3:47 p.m.
            • A better approch? [LONG]
              Maik Stiebler -- Thursday, 5 November 2009, at 10:01 p.m.
              • A better approch? [LONG]
                MaX -- Friday, 6 November 2009, at 3:31 p.m.
                • A better approch? [LONG]
                  Maik Stiebler -- Saturday, 7 November 2009, at 5:52 p.m.
                  • A better approch? [LONG]
                    MaX -- Monday, 9 November 2009, at 12:39 p.m.
                    • A better approch? [LONG]
                      Maik Stiebler -- Monday, 9 November 2009, at 10:59 p.m.
                      • A better approch? [LONG]
                        MaX -- Tuesday, 10 November 2009, at 1:33 p.m.
        • A better approch?
          Tom Keith -- Thursday, 5 November 2009, at 10:23 p.m.
          • A better approch?
            MaX -- Friday, 6 November 2009, at 12:35 p.m.
    • A better approch?
      Bob Koca -- Sunday, 1 November 2009, at 7:45 p.m.
      • A better approch?
        eXtreme Gammon -- Sunday, 1 November 2009, at 10:03 p.m.
        • A better approch?
          Bob Koca -- Sunday, 1 November 2009, at 10:30 p.m.
    • A better approch?
      Mislav Radica -- Monday, 2 November 2009, at 12:59 a.m.
      • A better approch?
        eXtreme Gammon -- Monday, 2 November 2009, at 1:58 a.m.
        • A better approach?
          eXtreme Gammon -- Monday, 2 November 2009, at 2:00 a.m.
        • A better approch?
          MaX -- Monday, 2 November 2009, at 10:19 a.m.
        • A better approch?
          Mislav Radica -- Monday, 2 November 2009, at 12:55 p.m.
          • The different approachs-give us a choice
            Mislav Radica -- Monday, 2 November 2009, at 1:18 p.m.
        • A better approch?
          misja -- Monday, 2 November 2009, at 7:36 p.m.
    • A better approch?
      Neil Robins -- Tuesday, 3 November 2009, at 9:39 p.m.