How to bootstrap the AUC on a data set?
Rather than just doing one AUC calculation on your full data and saying the AUC is .77, you may end up finding your AUC is .75 + / − .03, which is much more reliable to make a claim on. Now in hindsight you can see your single AUC measure was a little optimistically high.
How to derive the AUC of a classifier?
Derivation: Here I derive the AUC of a random classifier on a dataset with an arbitrary class imbalance. Assuming that there is a fraction x of positive cases and a fraction 1 − x of negative cases, and that our classifier consists of randomly assigning the positive class with probability ρ and the negative class with probability 1 − ρ.
What is the bootstrapped 95% confidence interval for 10, 000 AUCs?
4) Now you have 10,000 AUCs. If you take the 5th percentile and 95th percentile, you have the bootstrapped 95% confidence interval. The big benefit here is now you can take the mean of those 10,000 AUCs, and give the standard error ( s d / s q r t ( n)) to give some idea of the variability.
How is the ROC curve used in effects?
To obtain the whole ROC curve, we have to vary the probability with which we assign the positive class, from 0 to 1. So in effects, the ROC curve is a graphical evaluation of the performance of infinitely many classifiers! Each one of these random classifiers with a different probability will have a different expected confusion matrix.
When is ROC AUC sensitive to class imbalance?
It depends how you mean the word sensitive. The ROC AUC is sensitive to class imbalance in the sense that when there is a minority class, you typically define this as the positive class and it will have a strong impact on the AUC value. This is very much desirable behaviour.
Can a model with AUC of 1 be 0?
Image 2 — A model with AUC = 1 (image by author) You can see that this yields an AUC score of 1, indicating that the model classifies every instance correctly. Can AUC be 0? Yes — it means the model is reciprocating the classes.