Type: Journal Publication
Abstract: Error estimation is a crucial part of classification methodology and it becomes problematic with small samples. We demonstrate here that the complexity of the decision boundary plays a key role on the performance of error estimation methods. First, a model is developed which quantifies the complexity of a classification problem purely in terms of the geometry of the decision boundary, without relying on the Bayes error. Then, this model is used in a simulation study to analyze the bias and root-mean-square (RMS) error of a few widely used error estimation methods relative to the complexity of the decision boundary: resubstitution, leave-one-out, 10-fold cross-validation with repetition, 0.632 bootstrap, and bolstered resubstitution, in two- and three-dimensional spaces. Each estimator is implemented with three classification rules: quadratic discriminant analysis (QDA), 3-nearest-neighbor (3NN) and two-layer neural network (NNet). The results show that all the estimation methods lose accuracy as complexity increases.
Cited as: E. Atashpaz-Gargari, C. Sima, U.M. Braga-Neto and E.R. Dougherty, "Relationship Between the Accuracy of Classifier Error Estimation and Complexity of the Decision Boundary", Pattern Recognition, Volume 46, Issue 5, May 2013, Pages 1315-1322, doi:10.1016/j.patcog.2012.10.020.