What we have to keep in mind is that the classical modeling is so vast* and full of possibilities that many types of bizarre conclusions can arise when we are not concerned about optimal properties. We could restrict ourselves only to tests based on the likelihood ratio statistics or other optimal procedure.

Notice that, Bayesians use the likelihood function and, in addition, represent prior information by probability distributions, so the set of possibilities to build a test is restrict. On the other hand, classical statisticians also have the option of using the likelihood function (but is not mandatory), they implicitly represent prior information by possibility distributions, so the set of possibilities to build a test is sensibly bigger than the Bayesian procedure.

*We can model by using likelihoods, pseudo-likelihoods, matching moments, estimating equations and so forth.

H0: The Sun didn’t explode,
H1: The Sun exploded.

Then,

p-value = P(“the machine returns yes”, when the Sun didn’t explode).

Now, note that the event

“the machine returns yes”

is equivalent to

“the neutrino detector measures the Sun exploding AND tells the true result” OR “the neutrino detector does not measure the Sun exploding AND lies to us”.

Assuming that the dice throwing is independent of the neutrino detector measurement, we can compute the p-value. First define:

p0 = P(“the neutrino detector measures the Sun exploding”, when the Sun didn’t explode),

then the p-value is

p-value = p0*35/36 + (1-p0)*1/36

=> p-value = (1/36)*(35*p0 + 1 – p0)

=> p-value = (1/36)*(1+34*p0).

If p0 = 0, then we are considering that the detector machine will never measure that “the Sun just exploded”. The value p0 is obviously incomputable, therefore, a classical statistician that knows how to compute a p-value would never say that the Sun just exploded. By the way, the cartoon is funny.

Suppose x(1)<.. <x(m-1)<T<x(m)<y(1)< .. <y(n), then AUC=1 but the prediction is not perfect. So yes, a perfect model has AUC=1 but not the other way around. Thus I wonder why we do not use RMSE or OOB that attain 100% only when the prediction is perfect.

(I have a die with 2′s on three faces and 5′s on the other. I plan to roll this one for my big Stats class and see how long it takes for them to catch on….)