BGonline.org Forums

A better approch?

Posted By: Tom Keith
Date: Tuesday, 3 November 2009, at 12:14 a.m.

In Response To: A better approch? (Timothy Chow)

I claim that what you really want to know is which play the bot thinks is better.

I assume you mean which play will come out best in a rollout with enough trials to clearly distinguish the candidates. (We already know which play the bot thinks is best.)

If you knew for sure that the bot preferred Play A to Play B, would you really care to know that there is a 0.1% probability that extending the rollout would erroneously score Play B higher than Play A? Not really; if you knew for sure that Play A was better than Play B, then you wouldn't care about the chances that the bot would randomly make a mistake if you extended the rollout.

I'm not sure how this is pertinent.

What you want to know is how much evidence you currently have that the highest-scoring play is the play the bot really prefers (i.e., that it would prefer if you had infinite computing resources).

Agreed.

You want to be able to say something along these lines: It is extremely unlikely that the bot truly prefers Play B to Play A, because if it did, then the probability that Play A would outscore Play B as much as it has so far would be exceedingly small.

I would like to be able to attach numbers to these likelihoods. I would like to know that there is, say, an 85% chance that the current leader is the true leader. Then I can decide whether it is worth my time to extend the rollout and try for a higher number.

This is not the same as taking the estimated means and standard deviations and calculating tail probabilities.

Isn't this an implementation issue? Is your point that the usual method of calculating such probabilities might not work very well in this application? How big an error are we looking at? Is it a difference between 90.1% and 90.2%? (Something I don't care about.) Or more like the difference between 85% and 95%? (Something I definitely do care about!)

Messages In This Thread

• Confidence Interval Overlap vs Joint Standard Deviation (long)
  neilkaz -- Saturday, 31 October 2009, at 5:13 p.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Timothy Chow -- Saturday, 31 October 2009, at 5:30 p.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Tom Keith -- Saturday, 31 October 2009, at 5:51 p.m.
    • Confidence Interval Overlap vs Joint Standard Deviation (long)
      Rich Munitz -- Sunday, 1 November 2009, at 3:19 a.m.
  • Confidence Interval Overlap vs Joint Standard Deviation (long)
    Bob Koca -- Saturday, 31 October 2009, at 7:30 p.m.
    • Confidence Interval Overlap vs Joint Standard Deviation (long)
      neilkaz -- Saturday, 31 October 2009, at 9:20 p.m.
      • Confidence Interval Overlap vs Joint Standard Deviation (long)
        Bob Koca -- Saturday, 31 October 2009, at 9:31 p.m.
      • Confidence Interval Overlap vs Joint Standard Deviation (long)
        Timothy Chow -- Sunday, 1 November 2009, at 12:21 a.m.
  • A better approch?
    eXtreme Gammon -- Sunday, 1 November 2009, at 4:02 a.m.
    • A better approch?
      neilkaz -- Sunday, 1 November 2009, at 4:32 a.m.
    • A better approch?
      MaX -- Sunday, 1 November 2009, at 1:19 p.m.
    • A better approch?
      MaX -- Sunday, 1 November 2009, at 1:22 p.m.
    • A better approch?
      Timothy Chow -- Sunday, 1 November 2009, at 5:37 p.m.
      • A better approch?
        eXtreme Gammon -- Sunday, 1 November 2009, at 6:12 p.m.
        • A better approch?
          Bob Koca -- Sunday, 1 November 2009, at 7:36 p.m.
          • A better approch?
            Tom Keith -- Sunday, 1 November 2009, at 8:38 p.m.
            • A better approch?
              Chuck Bower -- Monday, 2 November 2009, at 1:20 a.m.
              • A better approch?
                Jason Lee -- Monday, 2 November 2009, at 2:24 a.m.
                • A better approch?
                  Chuck Bower -- Monday, 2 November 2009, at 3:06 p.m.
              • A better approch?
                Timothy Chow -- Monday, 2 November 2009, at 6:56 p.m.
        • A better approch?
          Maik Stiebler -- Sunday, 1 November 2009, at 10:05 p.m.
        • A better approch?
          MaX -- Monday, 2 November 2009, at 10:15 a.m.
          • A better approch?
            Timothy Chow -- Monday, 2 November 2009, at 6:47 p.m.
            • A better approch?
              MaX -- Tuesday, 3 November 2009, at 8:18 a.m.
          • A better approch?
            eXtreme Gammon -- Monday, 2 November 2009, at 7:26 p.m.
            • A better approch?
              MaX -- Thursday, 12 November 2009, at 12:56 p.m.
        • A better approch?
          Timothy Chow -- Monday, 2 November 2009, at 6:40 p.m.
          • A better approch?
            Tom Keith -- Monday, 2 November 2009, at 7:28 p.m.
            • A better approch?
              Timothy Chow -- Monday, 2 November 2009, at 10:10 p.m.
              • A better approch?
                Tom Keith -- Tuesday, 3 November 2009, at 12:14 a.m.
                • A better approch?
                  Timothy Chow -- Tuesday, 3 November 2009, at 1:29 a.m.
                  • A better approch?
                    Maik Stiebler -- Tuesday, 3 November 2009, at 7:18 a.m.
                    • A better approch?
                      Timothy Chow -- Tuesday, 3 November 2009, at 4:17 p.m.
                      • A better approch?
                        Tom Keith -- Tuesday, 3 November 2009, at 5:23 p.m.
                        • A better approch?
                          Timothy Chow -- Tuesday, 3 November 2009, at 5:37 p.m.
                          • Aren't you two just particpating in the 250 year old fracas?
                            Chuck Bower -- Tuesday, 3 November 2009, at 6:19 p.m.
                            • Aren't you two just particpating in the 250 year old fracas?
                              Timothy Chow -- Tuesday, 3 November 2009, at 7:55 p.m.
                              • Aren't you two just particpating in the 250 year old fracas?
                                Bob Koca -- Tuesday, 3 November 2009, at 8:29 p.m.
                            • Aren't you two just particpating in the 250 year old fracas?
                              Tom Keith -- Wednesday, 4 November 2009, at 12:11 p.m.
                      • A better approch?
                        Maik Stiebler -- Wednesday, 4 November 2009, at 8:40 p.m.
                        • A better approch?
                          Timothy Chow -- Wednesday, 4 November 2009, at 10:09 p.m.
                          • A better approch?
                            Maik Stiebler -- Wednesday, 4 November 2009, at 10:51 p.m.
                            • A better approch?
                              Maik Stiebler -- Wednesday, 4 November 2009, at 11:57 p.m.
                            • A better approch?
                              Timothy Chow -- Thursday, 5 November 2009, at 6:52 a.m.
                              • A better approch?
                                Maik Stiebler -- Thursday, 5 November 2009, at 7:25 a.m.
                                • A better approch?
                                  Timothy Chow -- Thursday, 5 November 2009, at 3:37 p.m.
                                  • A better approch?
                                    Maik Stiebler -- Thursday, 5 November 2009, at 9:19 p.m.
                                    • A better approch?
                                      Maik Stiebler -- Thursday, 5 November 2009, at 9:42 p.m.
          • A better approch? [LONG]
            MaX -- Thursday, 5 November 2009, at 3:04 p.m.
            • A better approch? [LONG]
              Timothy Chow -- Thursday, 5 November 2009, at 3:47 p.m.
            • A better approch? [LONG]
              Maik Stiebler -- Thursday, 5 November 2009, at 10:01 p.m.
              • A better approch? [LONG]
                MaX -- Friday, 6 November 2009, at 3:31 p.m.
                • A better approch? [LONG]
                  Maik Stiebler -- Saturday, 7 November 2009, at 5:52 p.m.
                  • A better approch? [LONG]
                    MaX -- Monday, 9 November 2009, at 12:39 p.m.
                    • A better approch? [LONG]
                      Maik Stiebler -- Monday, 9 November 2009, at 10:59 p.m.
                      • A better approch? [LONG]
                        MaX -- Tuesday, 10 November 2009, at 1:33 p.m.
        • A better approch?
          Tom Keith -- Thursday, 5 November 2009, at 10:23 p.m.
          • A better approch?
            MaX -- Friday, 6 November 2009, at 12:35 p.m.
    • A better approch?
      Bob Koca -- Sunday, 1 November 2009, at 7:45 p.m.
      • A better approch?
        eXtreme Gammon -- Sunday, 1 November 2009, at 10:03 p.m.
        • A better approch?
          Bob Koca -- Sunday, 1 November 2009, at 10:30 p.m.
    • A better approch?
      Mislav Radica -- Monday, 2 November 2009, at 12:59 a.m.
      • A better approch?
        eXtreme Gammon -- Monday, 2 November 2009, at 1:58 a.m.
        • A better approach?
          eXtreme Gammon -- Monday, 2 November 2009, at 2:00 a.m.
        • A better approch?
          MaX -- Monday, 2 November 2009, at 10:19 a.m.
        • A better approch?
          Mislav Radica -- Monday, 2 November 2009, at 12:55 p.m.
          • The different approachs-give us a choice
            Mislav Radica -- Monday, 2 November 2009, at 1:18 p.m.
        • A better approch?
          misja -- Monday, 2 November 2009, at 7:36 p.m.
    • A better approch?
      Neil Robins -- Tuesday, 3 November 2009, at 9:39 p.m.