BGonline.org Forums

A better approch?

Posted By: Timothy Chow
Date: Monday, 2 November 2009, at 10:10 p.m.

In Response To: A better approch? (Tom Keith)

I wrote:

You want to know, what is the probability that I would see the results I am actually observing, given that this lower-scoring play is in fact the best one?

Tom Keith wrote:

Why do you want to know that? I want to know whether it is worthwhile extending the rollout, so I'd like to know what are the chances I'll get a different answer if I do extend the rollout.

I claim that what you really want to know is which play the bot thinks is better. If you knew for sure that the bot preferred Play A to Play B, would you really care to know that there is a 0.1% probability that extending the rollout would erroneously score Play B higher than Play A? Not really; if you knew for sure that Play A was better than Play B, then you wouldn't care about the chances that the bot would randomly make a mistake if you extended the rollout.

What you want to know is how much evidence you currently have that the highest-scoring play is the play the bot really prefers (i.e., that it would prefer if you had infinite computing resources). You want to be able to say something along these lines: It is extremely unlikely that the bot truly prefers Play B to Play A, because if it did, then the probability that Play A would outscore Play B as much as it has so far would be exceedingly small.

This is not the same as taking the estimated means and standard deviations and calculating tail probabilities. That amounts to saying, let's assume that Play A has this mean and s.d. and Play B has this mean and s.d.; what's the probability that if I continue rolling out, I'll see Play B outscore Play A? As I said before, this is not the question you are really interested in.

Messages In This Thread

• Confidence Interval Overlap vs Joint Standard Deviation (long)
  neilkaz -- Saturday, 31 October 2009, at 5:13 p.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Timothy Chow -- Saturday, 31 October 2009, at 5:30 p.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Tom Keith -- Saturday, 31 October 2009, at 5:51 p.m.
    • Confidence Interval Overlap vs Joint Standard Deviation (long)
      Rich Munitz -- Sunday, 1 November 2009, at 3:19 a.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Bob Koca -- Saturday, 31 October 2009, at 7:30 p.m.
    • Confidence Interval Overlap vs Joint Standard Deviation (long)
      neilkaz -- Saturday, 31 October 2009, at 9:20 p.m.
      • Confidence Interval Overlap vs Joint Standard Deviation (long)
        Bob Koca -- Saturday, 31 October 2009, at 9:31 p.m.
      • Confidence Interval Overlap vs Joint Standard Deviation (long)
        Timothy Chow -- Sunday, 1 November 2009, at 12:21 a.m.
  • A better approch?
    eXtreme Gammon -- Sunday, 1 November 2009, at 4:02 a.m.
    • A better approch?
      neilkaz -- Sunday, 1 November 2009, at 4:32 a.m.
    • A better approch?
      MaX -- Sunday, 1 November 2009, at 1:19 p.m.
    • A better approch?
      MaX -- Sunday, 1 November 2009, at 1:22 p.m.
    • A better approch?
      Timothy Chow -- Sunday, 1 November 2009, at 5:37 p.m.
      • A better approch?
        eXtreme Gammon -- Sunday, 1 November 2009, at 6:12 p.m.
        • A better approch?
          Bob Koca -- Sunday, 1 November 2009, at 7:36 p.m.
          • A better approch?
            Tom Keith -- Sunday, 1 November 2009, at 8:38 p.m.
            • A better approch?
              Chuck Bower -- Monday, 2 November 2009, at 1:20 a.m.
              • A better approch?
                Jason Lee -- Monday, 2 November 2009, at 2:24 a.m.
                • A better approch?
                  Chuck Bower -- Monday, 2 November 2009, at 3:06 p.m.
              • A better approch?
                Timothy Chow -- Monday, 2 November 2009, at 6:56 p.m.
        • A better approch?
          Maik Stiebler -- Sunday, 1 November 2009, at 10:05 p.m.
        • A better approch?
          MaX -- Monday, 2 November 2009, at 10:15 a.m.
          • A better approch?
            Timothy Chow -- Monday, 2 November 2009, at 6:47 p.m.
            • A better approch?
              MaX -- Tuesday, 3 November 2009, at 8:18 a.m.
          • A better approch?
            eXtreme Gammon -- Monday, 2 November 2009, at 7:26 p.m.
            • A better approch?
              MaX -- Thursday, 12 November 2009, at 12:56 p.m.
        • A better approch?
          Timothy Chow -- Monday, 2 November 2009, at 6:40 p.m.
          • A better approch?
            Tom Keith -- Monday, 2 November 2009, at 7:28 p.m.
            • A better approch?
              Timothy Chow -- Monday, 2 November 2009, at 10:10 p.m.
              • A better approch?
                Tom Keith -- Tuesday, 3 November 2009, at 12:14 a.m.
                • A better approch?
                  Timothy Chow -- Tuesday, 3 November 2009, at 1:29 a.m.
                  • A better approch?
                    Maik Stiebler -- Tuesday, 3 November 2009, at 7:18 a.m.
                    • A better approch?
                      Timothy Chow -- Tuesday, 3 November 2009, at 4:17 p.m.
                      • A better approch?
                        Tom Keith -- Tuesday, 3 November 2009, at 5:23 p.m.
                        • A better approch?
                          Timothy Chow -- Tuesday, 3 November 2009, at 5:37 p.m.
                          • Aren't you two just particpating in the 250 year old fracas?
                            Chuck Bower -- Tuesday, 3 November 2009, at 6:19 p.m.
                            • Aren't you two just particpating in the 250 year old fracas?
                              Timothy Chow -- Tuesday, 3 November 2009, at 7:55 p.m.
                              • Aren't you two just particpating in the 250 year old fracas?
                                Bob Koca -- Tuesday, 3 November 2009, at 8:29 p.m.
                            • Aren't you two just particpating in the 250 year old fracas?
                              Tom Keith -- Wednesday, 4 November 2009, at 12:11 p.m.
                      • A better approch?
                        Maik Stiebler -- Wednesday, 4 November 2009, at 8:40 p.m.
                        • A better approch?
                          Timothy Chow -- Wednesday, 4 November 2009, at 10:09 p.m.
                          • A better approch?
                            Maik Stiebler -- Wednesday, 4 November 2009, at 10:51 p.m.
                            • A better approch?
                              Maik Stiebler -- Wednesday, 4 November 2009, at 11:57 p.m.
                            • A better approch?
                              Timothy Chow -- Thursday, 5 November 2009, at 6:52 a.m.
                              • A better approch?
                                Maik Stiebler -- Thursday, 5 November 2009, at 7:25 a.m.
                                • A better approch?
                                  Timothy Chow -- Thursday, 5 November 2009, at 3:37 p.m.
                                  • A better approch?
                                    Maik Stiebler -- Thursday, 5 November 2009, at 9:19 p.m.
                                    • A better approch?
                                      Maik Stiebler -- Thursday, 5 November 2009, at 9:42 p.m.
          • A better approch? [LONG]
            MaX -- Thursday, 5 November 2009, at 3:04 p.m.
            • A better approch? [LONG]
              Timothy Chow -- Thursday, 5 November 2009, at 3:47 p.m.
            • A better approch? [LONG]
              Maik Stiebler -- Thursday, 5 November 2009, at 10:01 p.m.
              • A better approch? [LONG]
                MaX -- Friday, 6 November 2009, at 3:31 p.m.
                • A better approch? [LONG]
                  Maik Stiebler -- Saturday, 7 November 2009, at 5:52 p.m.
                  • A better approch? [LONG]
                    MaX -- Monday, 9 November 2009, at 12:39 p.m.
                    • A better approch? [LONG]
                      Maik Stiebler -- Monday, 9 November 2009, at 10:59 p.m.
                      • A better approch? [LONG]
                        MaX -- Tuesday, 10 November 2009, at 1:33 p.m.
        • A better approch?
          Tom Keith -- Thursday, 5 November 2009, at 10:23 p.m.
          • A better approch?
            MaX -- Friday, 6 November 2009, at 12:35 p.m.
    • A better approch?
      Bob Koca -- Sunday, 1 November 2009, at 7:45 p.m.
      • A better approch?
        eXtreme Gammon -- Sunday, 1 November 2009, at 10:03 p.m.
        • A better approch?
          Bob Koca -- Sunday, 1 November 2009, at 10:30 p.m.
    • A better approch?
      Mislav Radica -- Monday, 2 November 2009, at 12:59 a.m.
      • A better approch?
        eXtreme Gammon -- Monday, 2 November 2009, at 1:58 a.m.
        • A better approach?
          eXtreme Gammon -- Monday, 2 November 2009, at 2:00 a.m.
        • A better approch?
          MaX -- Monday, 2 November 2009, at 10:19 a.m.
        • A better approch?
          Mislav Radica -- Monday, 2 November 2009, at 12:55 p.m.
          • The different approachs-give us a choice
            Mislav Radica -- Monday, 2 November 2009, at 1:18 p.m.
        • A better approch?
          misja -- Monday, 2 November 2009, at 7:36 p.m.
    • A better approch?
      Neil Robins -- Tuesday, 3 November 2009, at 9:39 p.m.