Issue 7, 2015

Assessment of the statistical significance of classifications in infrared spectroscopy based diagnostic models

Abstract

Fourier transform infrared (IR) spectroscopy in combination with multivariate data analysis is a versatile tool that can be applied to disease diagnosis. However, a rigorous validation of the obtained models is necessary in order to obtain robust results. This work evaluates the advantages of the use of permutation testing for determining the statistical significance of the misclassification errors obtained from IR based diagnostic models through cross validation (CV). The model performance, estimated by CV, is compared to a distribution of CV-performance values obtained using randomly permuted class labels. The distribution of ‘random CV-values’ is considered as a null distribution and used to establish the significance of the model estimators obtained using real class labels. ATR-FTIR spectra of serum samples were classified using random forest (RF) classifiers according to two criteria, the tag number (a randomly assigned pseudo class membership) and the level of urea (real class). CV errors obtained were compared to the null distribution of CV errors from a permutation test and an independent validation set. The procedure was evaluated testing typical conditions leading to overoptimistic estimations provided by the CV like e.g. the size of subsamples used during CV, variable selection and the use of replicates. Results show that for the tag number (pseudo class), CV indicated classification errors between 23 and 33% depending on the subsample size employed. Those values were even lower when variable selection or replicates were used. However, permutation testing indicated that those CV errors were non-significant. In contrast, for sample classification according to their levels of urea, all cross validation errors were found to be significant. Although the proposed method is computationally intensive, it provides a simple way of calculating an empirical p-value of the CV-estimator, thus establishing the statistical significance and providing a feasibility indicator especially useful for studies where the number of samples is limited.

Graphical abstract: Assessment of the statistical significance of classifications in infrared spectroscopy based diagnostic models

Article information

Article type
Paper
Submitted
03 Oct 2014
Accepted
31 Oct 2014
First published
31 Oct 2014

Analyst, 2015,140, 2422-2427

Author version available

Assessment of the statistical significance of classifications in infrared spectroscopy based diagnostic models

D. Pérez-Guaita, J. Kuligowski, S. Garrigues, G. Quintás and B. R. Wood, Analyst, 2015, 140, 2422 DOI: 10.1039/C4AN01783H

To request permission to reproduce material from this article, please go to the Copyright Clearance Center request page.

If you are an author contributing to an RSC publication, you do not need to request permission provided correct acknowledgement is given.

If you are the author of this article, you do not need to request permission to reproduce figures and diagrams provided correct acknowledgement is given. If you want to reproduce the whole article in a third-party publication (excluding your thesis/dissertation for which permission is not required) please go to the Copyright Clearance Center request page.

Read more about how to correctly acknowledge RSC content.

Social activity

Spotlight

Advertisements