Enlighten Publications

In this section

Area Under the Distance Threshold Curve as an Evaluation Measure for Probabilistic Classifiers

Williams, S. , Harris, M., Furst, J. and Raicu, D. (2013) Area Under the Distance Threshold Curve as an Evaluation Measure for Probabilistic Classifiers. In: 9th International Conference on Machine Learning and Data Mining in Pattern Recognition, New York, NY, USA, 19-25 July 2013, pp. 644-657. ISBN 9783642397127 (doi: 10.1007/978-3-642-39712-7_49)

Full text not currently available from Enlighten.

Abstract

Evaluation for probabilistic multiclass systems has predominately been done by converting data into binary classes. While effective in quantifying the classifier performance, binary evaluation causes a loss in ability to distinguish between individual classes. We report that the evaluation of multiclass probabilistic classifiers can be quantified by using the area under the distance threshold curve for multiple distance metrics. We construct our classifiers for evaluation with data from the National Cancer Institute (NCI) Lung Image Database Consortium (LIDC) for the semantic characteristic of malignancy. We conclude that the area under the distance threshold curve can provide a measure of the classifier performance when the classifier has more than two classes and probabilistic predictions.

Item Type:	Conference Proceedings
Keywords:	Machine learning, medical informatics, probabilistic classifier, ROC curve, K-Nearest neighbor.
Status:	Published
Refereed:	Yes
Glasgow Author(s) Enlighten ID:	Williams, Dr Sydney
Authors:	Williams, S., Harris, M., Furst, J., and Raicu, D.
Subjects:	Q Science > QA Mathematics > QA75 Electronic computers. Computer science T Technology > T Technology (General)
College/School:	College of Medical Veterinary and Life Sciences > School of Psychology & Neuroscience
Research Group:	DePaul University Medical Informatics
ISBN:	9783642397127
Copyright Holders:	Springer Link

University Staff: Request a correction | Enlighten Editors: Update this record

References

1. Bradley, A.P.: The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recogn. 30, 1145–1159 (1997), doi:10.1016/j.bbr.2011.03.031 2. Drummond, C., Holte, R.: Cost curves: an improved method for visualizing classifier performance. Mach. Learn. 65, 95–130 (2006) 3. Zinovev, D., Furst, J., Raicu, D.: Building an ensemble of probabilistic classifiers for lung nodule interpretation. In: Tenth Intern. Conferen. on Mach. Learn. and App (ICMLA 2011), pp. 151–167. IEEE Press (December 2011), doi:10.1109/ICMLA.2011.44 4. Amor, N.B., Benferhat, S., Elouedi, Z.: Information-based evaluation functions for probabilistic classifiers. In: Eleventh Internat. Conferen. on Infor. Processing and Management of Uncertainty in Knowledge-Based Systems (IPMU 2006), pp. 428–433 (July 2006) 5. Ling, C.X., Huang, J., Zhang, H.: AUC: a statistically consistent and more discriminating measure than accuracy. In: Proc. of Eighteenth Internat. Conf. on Artifical Intelligence (IJCAI 2003), pp. 519–526 (August 2003) 6. Provost, F., Fawcett, T.: Analysis and visualization of classifier performance: comparison under imprecise class and cost distributions. In: Proc. Third Internat. Conf. on Knowledge Discovery and Data Mining (KDD 1997), pp. 43–48. AAAI Press (August 1997) 7. Fawcett, T.: ROC graphs: notes and practical considerations for data mining. Technical report, HPL-2003-4 8. Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. 27, 861–874 (2006) 9. Hérnandez-Orallo, J., Flach, P., Ferri, C.: Brier curves: a new cost-based visualization of classifier performance. In: Proc. Twenty-Eighth Internat. Conf. on Mach. Learn (ICML 2011), pp. 585–592 (June 2011) 10. Hand, D.J., Till, R.T.: A simple generalization of the area under the ROC curve for multiple class classification problems. Machine Learning 45, 171–186 (2001) 11. Jain, P., Kapoor, A.: Active learning for large multi-class problems. Comp. Vision and Pattern Recogn (CVPR 2009), 762–769 (June 2009) Area under the Distance Threshold Curve as an Evaluation Measure 657 12. Liu, H., et al.: Comparing dissimilarity measures for content-based image retrieval. In: Li, H., Liu, T., Ma, W.-Y., Sakai, T., Wong, K.-F., Zhou, G. (eds.) AIRS 2008. LNCS, vol. 4993, pp. 44–50. Springer, Heidelberg (2008) 13. Rubner, Y., Tomasi, C., Guibas, L.J.: A metric for distributions with applications to image databases. In: Sixth Internat. Conf. Comp. Vis. (ICCV 1998), pp. 59–66. IEEE Press (January 1998), doi:10.1109/ICCV.19 14. Raicu, D.S., Varutbangkul, E., Furst, J.D., Armato III, S.G.: Modeling semantics from image data: opportunities from LIDC. Internat. Jour. of Biomed. Eng. and Tech. 3(30:1-2), 83–113 (2009)

Deposit and Record Details

ID Code:	223851
Depositing User:	Dr Sydney Williams
Datestamp:	07 Oct 2020 09:05
Last Modified:	07 Oct 2020 09:05
Date of acceptance:	1 November 2012
Date of first online publication:	19 July 2013