1.

Introduction

Coronavirus 2019 (SARS-CoV-2 or COVID-19) is an infectious disease affecting the respiratory system, which has caused a global pandemic and widespread morbidity and mortality in humans. It has mild to severe symptoms similar to those of severe acute respiratory syndrome (SARS) and Middle East respiratory syndrome (MERS), resulting in multiorgan damage.¹ Currently, two tests are available for detection of COVID-19 in the affected patients: diagnostic tests (current infection) and antibody tests (past infection). Diagnostic tests, such as reverse transcription polymerase chain reaction (RT-PCR) and antigen tests, are used for rapid diagnosis of COVID-19. Since false positives (FPs) are more common in antigen tests, RT-PCR is used as the gold standard for diagnosis of the disease. RT-PCR tests require an intensive lab work to acquire the results,² and the cost of the test is a major concern in many countries that have a private health system. Although the PCR and antigen test can now provide a rapid diagnosis, the assessment of the lungs using medical imaging will provide information on disease burden. Also, faster and earlier detection of COVID-19 would help in isolating the affected patients sooner to alleviate the disease spread.

Chest radiography (CXR) and computed tomography (CT) images are the conventional medical imaging modalities used in lung disease diagnosis.^3,4 Though CT images are extensively used in the COVID-19 diagnosis,^5–7 cost⁸ and radiation exposure are major concerns. CXR are preferred over CT images as they have less exposure to radiation and extensively available.⁹ Hence, in this study, CXR images are used for automatic diagnosis of COVID-19.

Deep learning techniques are widely used in various fields, such as computer vision, machine vision, and speech recognition, among which computer vision is one of the most popular fields in which promising results to have been obtained in image classification tasks.^10–12 In medical image analysis, deep learning has been widely investigated for computer-aided diagnosis and treatment.^13,14 Several state-of-art methods have been proposed for the diagnosis of COVID-19 using CXR images^15–19 and CT images^5,20,21 based on deep learning techniques. The transfer learning approaches based on deep learning have been preferred in detection of COVID-19 due to the limited available dataset. Several studies have been implemented for COVID-19 diagnosis using the pretrained model as a feature extractor by implementing transfer learning techniques.^22–24

Table 1 summarizes recent studies describing methods developed for detecting COVID-19 from CXR images; the overall datasets used for training and testing the model, accuracy, and the sensitivity are provided. Ozturk et al.¹⁸ developed the DarkCovidNet model for the detection of COVID-19 using multiclass classification. The model was developed using end-to-end architecture without adding any feature extraction techniques. Sensitivity of 85.35% was achieved by the proposed model. Wang and Wong²⁵ proposed COVID-Net, a deep convolutional neural network (DCNN) developed by adopting machine-driven design exploration strategy. A multiclass classification was performed using CXR images in identifying COVID-19 from normal and non-COVID disease (i.e., pneumonia). Both qualitative and quantitative analysis were performed to determine the success of the model. The developed model was able to achieve a sensitivity of 91% for detecting COVID-19. Panwar et al.²⁶ introduced nCOVnet, an algorithm developed based on a transfer learning model for faster diagnosis of COVID-19 from CXR images. The proposed model is built using VGG 16 as the base model. The model was able to achieve 97.62% in detecting COVID-19. Sethy et al.²³ described a comparative study on Xception, ResNet18, ResNet50, Resnet 101, Inceptionv3, Inceptionresnetv2, GoogleNet, Densenet201, VGG16, VGG19, and AlexNet using the deep features extracted from the CNN layers and fed to the SVM classifier for classification. ResNet50 performed better compared with other classification models on a binary classification between viral pneumonia and COVID-19 with a classification accuracy of 95.38%. Jain et al.²⁷ utilized ResNet50 model that was used in stage-I network model for distinguishing viral (including COVID-19) from bacterial pneumonia and normal cases using CXR images. Based on the result obtained, ResNet101 was used in stage-II network model to classify COVID-19 from other viral pneumonia. The model achieved an accuracy of 97% in COVID-19 detection. Narin et al.²⁸ proposed three CNNs for the detection of COVID-19 through CXR images. The pretrained models chosen for comparative study were ResNet50, InceptionV3, and Inception-ResNetV2. A binary classification was performed between normal and COVID-19. ResNet50 model outperformed all the other models with the highest classification accuracy of 98%. Ioannis et al.²² compared VGG 19, Mobile Net, Inception, Xception, and Inception ResNetv2 pretrained models for the automatic detection of COVID-19 using CXR images. They implemented multiclass classification among COVID-19, normal, and pneumonia. MobileNetV2 achieved highest sensitivity rate of 98.6%. Asnaoui et al.²⁹ put forward a comparative study on the pretrained DCNN models, namely VGG16, VGG19, Inception-ResNetV2, InceptionV3, ResNet50, DenseNet201, and MobileNetV2 for the classification of CXR into normal, bacteria, and coronavirus (multiclass classification). Inception-ResNetV2 model provided a sensitivity rate of 92.11% in detecting coronavirus. Hemdan et al.³⁰ proposed COVIDX-Net a deep learning framework based on seven DCNNs namely, VGG19, Xception, ResNetV2, InceptionV3, Inception-ResNetV2, DenseNet201, and MobileNetV2 for the diagnosis of COVID-19 using x-ray images. The VGG19 and DenseNet201 achieved an accuracy of 90% compared with other models with $F1$-score of 0.89 for normal and 0.91 for COVID-19.

Table 1

Methods described for diagnosis of COVID-19 from CXR and CT images.

One of the major limitations of the previous studies is the relatively smaller test dataset employed for the classification. Moreover, an imbalanced realistic representation of the patient population was not considered in the testing phase. Furthermore, some of the methods were designed to differentiate COVID-19 from normal cases by ignoring the pneumonia cases, or by combining bacterial, viral pneumonia, and COVID-19 under one class. To address these shortcomings and to further enhance the diagnosis of COVID-19, we propose a fully automated deep learning-based method for the detection of COVID-19 from CXR images. The proposed method was based on a transfer learning approach using dense convolutional neural network 201 (DenseNet201)³¹ pretrained model. Our model was trained and tested using a dataset that reflected realistic class imbalance in the actual setting. As a benchmark, we compared results of our method to those of several state-of-the-art CNN architectures, including VGG16,³² VGG19,³² DenseNet121,³¹ ResNet50,³³ ResNet101,³³ MobileNetv2,³⁴ and Inceptionv3.³⁵

2.

Materials and Methods

2.1.

Dataset

Our dataset comprised 8644 CXR images of normal, respiratory distress syndrome (ARDS), COVID-19, MERS, pneumonia, and SARS from five open-access GitHub repositories^36–40 used in Wang e al.²⁵ It consisted of 4000 normal cases, 4000 pneumonia cases, and 644 COVID-19 cases. Out of 644 images of COVID-19, 20% were taken from COVID-19 Image Data Collection,³⁶ 5% from the Figure 1 COVID Chest X-ray Dataset,⁴⁰ 25% from the Actualmed COVID-19 Chest X-ray Dataset,³⁹ and 50% of images were obtained from the COVID-19 Radiography Database.³⁸ About 5% of pneumonia images were taken from COVID-19 Image Data Collection³⁶ and the remaining 95% from the RSNA Pneumonia Detection Challenge dataset.³⁷ All the normal images were obtained from the RSNA Pneumonia Detection Challenge dataset.

Fig. 1

Examples of (a) normal, (b) pneumonia, and (c) COVID-19 in the dataset.

CXR images of COVID-19 affected patients have patchy and hazy lungs compared with normal, healthy lungs. Certain characteristic findings can be observed on the lower lobes and periphery of the lungs affected with COVID-19. Examples of normal, pneumonia, and COVID-19 are shown in Fig. 1. Since the purpose of this paper is to correctly classify COVID-19 cases, both bacterial and viral pneumonia are grouped under one label (pneumonia).

The CXR images are resized to $224\times 224$ to be compatible with the pretrained model used in this study. Except for Xception where the default input size of $299\times 299$ is used. All the images in the dataset used for training, validation, and testing are normalized between 0 and 1. Since the dataset is imbalanced, data augmentation techniques were incorporated to increase the training dataset. The training data of each class are increased to 4000 images summing up to 12,000 images in total. The images were horizontally flipped, magnified, and rotated for augmentation. The augmentation techniques were not applied on the validation and test data set. The 6223 (80%) of the dataset were used for training the model and the remaining 1729 (20%) of the dataset were used for testing the model. From the 80% of training dataset, 692 images (10%) were used for validation. A detailed description on the number of datasets used for training, validation, and test is provided in Fig. 2.

Fig. 2

Distribution of the dataset among different classes.

3.

Results

The automated diagnostic tool developed using the DenseNet201 model for the detection of COVID-19 evaluated using the test dataset yielded an overall accuracy of 92% with sensitivity of 94%. The model was also able to classify 764 normal and 709 pneumonia out of 800 cases. Considering COVID-19, it was able to classify 121 cases out of 129 cases. Only eight cases were misclassified. Most of the misclassification occurred between normal and pneumonia cases due to the early onset of pneumonia. The confusion matrix and the ROC curve for the proposed model are shown in Fig. 3. The proposed model was able to achieve area under the curve (AUC) of 0.99, 0.97 and 0.97 for COVID-19, normal and pneumonia classes, respectively.

Fig. 3

Confusion matrix and the ROC curve for the proposed model.

We compared our method to seven state-of-the-art pretrained models. All the models were able to classify all the three classes (normal, pneumonia, and COVID-19) with a relatively high accuracy. Among 129 COVID-19 cases considered in the test dataset, all the models were able to classify more than 100 COVID-19 cases. Our proposed model based on DenseNet201 classified 121 cases bagging the first place, while ResNet50 was able to detect 117 cases holding the second place. When considering TN cases, VGG16, DenseNet121, DenseNet201, and ResNet101 were able to identify more than 700 cases correctly for both normal and pneumonia cases. VGG16 shows the least FP with two normal cases misidentified as COVID-19 and no FP for pneumonia cases. The least detection of TP was provided by VGG16 and highest FP was provided by the MobileNetV2 with least TN classification. The confusion matrix for all the seven pretrained models is summarized in Fig. 4.

Fig. 4

Confusion matrix for all the seven models used for analytical analysis.

The ROC curve for all the eight models used in this study is provided in Fig. 5, and AUC for all the three classes (normal, pneumonia, and COVID-19) were calculated and provided in the ROC curve. When the AUC for COVID-19 is taken into consideration, VGG16, VGG19, DenseNet121, and proposed model achieved the highest value of 0.99, and the least value of 0.95 was achieved by MobileNetV2. When the normal and pneumonia are examined, DenseNet121 obtained 0.98 for both the cases. A summarized AUC for all the classes provided in Table 2 shows that the DenseNet121 achieved higher AUC for all the three classes with 0.99 for COVID-19, 0.98 for normal, and 0.98 for pneumonia, and MobileNetV2 showed a lower result compared with the other models with 0.95 for COVID-19, 0.96 for normal, and 0.97 for pneumonia.

Fig. 5

ROC curve for all the seven models used for analytical analysis.

The evaluation metrics such as sensitivity, specificity, accuracy, and $F1$-score are summarized in Table 3. Considering the overall classification accuracy, DenseNet201model was able to classify COVID-19 from normal and pneumonia with highest classification accuracy of 92% among all the other models considered. The DenseNet121 model was able to achieve the same classification accuracy of 92% as DenseNet201 model. VGG16, VGG19, ResNet101, and InceptionV3 obtained a classification accuracy of 90%, which is 2% less than the highest accuracy proposed.

Table 2

AUC for all three classes for all the eight models (the model with best result is highlighted in bold).

The sensitivity and specificity rate are some of the important metrics to be considered when the dataset is imbalanced. Based on sensitivity rate, proposed model achieved highest rate of 0.94 and specificity of 0.99. DenseNet121 with less deep model compared with DenseNet201 achieved the same specificity rate but a lower sensitivity rate of 0.89. ResNet50 was able to achieve 0.91 which is second to the proposed model. Though ResNet50 achieved a higher sensitivity rate when compared with ResNet101, we can imply that the network depth can improve the overall classification accuracy but a reduction in the sensitivity is observed. VGG16 and VGG19 show higher accuracy as they are large network and provide better performance on the larger dataset used in this work. The classification accuracy remains the same for both VGG16 and VGG19, whereas the sensitivity rate is higher for VGG19, and VGG16 has the least sensitivity rate in detecting COVID-19.

$F1$-score is included in the metrics to provide a balance between the exactness and completeness of the model. Considering the $F1$-score achieved by each model, VGG19 achieved a higher score of 0.91 and DensNet201 acquired 0.90, following VGG19. An exceptionally low $F1$-score of 0.66 was achieved by the MobileNetV2 model.

Table 3

Evaluation metrics for eight pretrained models (model that achieved best result under each metric is highlighted in bold).

Also, to quantitatively justify, our proposed model was compared with a lightweight CNN designed from scratch. The lightweight CNN contains fewer parameters resulting in a faster CNN and provides a similar performance to the state-of-art pretrained models considered. The CNN architecture is made up of repeated blocks of convolution, batch normalization, and pooling as shown in Fig. 6. The architecture consists of five convolution and pooling layers and a fully connected classification layer. The convolutional and the hidden layers are followed by the ReLU activation function. The convolutional layer comprises 64, 128, and 256 neurons with convolution kernel of size $3\times 3$ with stride length of 1. The pooling layer comprises $2\times 2$ kernel with stride length 2. Before convolution, the inputs are padded using “same” padding in Tensorflow.

Fig. 6

Lightweight CNN model architecture.

The results obtained by training and evaluating the model using the dataset are provided in Table 4. The model was able to achieve higher sensitivity and specificity value with an overall classification accuracy of 89.7%. Using the transfer learning approach, our proposed model was able to outperform in terms of overall classification accuracy, sensitivity, specificity, and $F1$-score, providing a more accurate diagnosis.

Table 4

Evaluation metrics for proposed model and lightweight CNN.

4.

Discussion

In this study, we have proposed a fully automated diagnostic tool for the classification of COVID-19 from normal and pneumonia cases. A large, imbalanced test dataset (at least 78% more than that of previous studies)^25,28,30 was used for evaluation of the model, and deeper CNN models were included as a benchmark analysis.²⁴ In comparison to the method proposed by Ozturk et al.,¹⁸ our model reported an accuracy and sensitivity of 92% and 94% (versus 87% and 85.53%), respectively, on a relatively larger dataset (eight times larger). Also, when the specificity and $F1$-score are considered, 7% and 3% increase were observed, respectively. Wang et al.²⁵ used a large dataset for model evaluation and achieved high performance for the classification. However, the test dataset may not reflect the actual clinical setting where small number of COVID-19 cases in the testing pool creates a huge class-imbalance. Compared with the sensitivity of 91% and accuracy of 93.3% reported, our model yielded a sensitivity and accuracy that is 3% and 0.7% higher, whereas the same class imbalance exists in the dataset is kept in the test dataset. Panwar et al.²⁶ reported an overall accuracy of 88% (4% less than our proposed model) based on a binary classification. Though their model achieved 97% in detecting COVID-19 which is 3% higher than our proposed model, their specificity rate is 78% (21% lower) which is an important metric to be considered in classification. Our model was able to achieve higher overall accuracy and specificity rate than those of previous methods.^26,29

Asnaoui et al.²⁹ performed a comparative study of various pretrained models using three-class classification. Since they used a fairly large dataset, looking into the results obtained by DenseNet201 model shows that an overall accuracy of 88.10%, sensitivity of 75.14%, and $F1$-score of 82.04% (4%, 19%, and 8% lower than our proposed model, respectively) were reported. Hemdan et al.³⁰ proposed a DCNN model using DenseNet201 and achieved an overall accuracy of 90% on a balanced test dataset. Based on the sensitivity rate, Hemdan et al.³⁰ and Narin et al.²⁸ reported a sensitivity rate of 1 but only 5 COVID-19 and 10 normal cases were included for test dataset, and no pneumonia cases were included for classification. In comparison, our dataset was 173 times larger and included an imbalanced test dataset. Since the CXR images were collected from various resources, a binary classification was performed using the developed model between normal and pneumonia cases obtained from the same source to eliminate the bias between the two classes (normal and pneumonia). We were able to achieve a classification accuracy of 95.75%, which shows that having data from different sources or the same sources does not influence the results, and our model is robust for images collected from different sources using different machines, but there is still a possibility for bias between COVID-19 and other classes decision boundary. Also, since we are dealing with x-ray images, they are more standardized compared with the other medical imaging modalities for detection of lung diseases.^45,46

Our fully automated deep learning-based model based on DenseNet201 was able to detect COVID-19 with a higher accuracy based on a large and imbalanced dataset using CXR images. Considering the results obtained, the proposed model was able to outperform the other tested models in terms of overall accuracy, sensitivity, and specificity in identifying COVID-19 cases from normal and pneumonia cases. The analysis of the proposed model using higher number of pretrained models provided an insight on different model performance for medical image classification by implementing transfer learning approach using a dataset that is a large realistic representation of the patient population. Hence, the developed model utilizing CXR images will assist in the initial diagnosis of COVID-19 by the radiologists and will aid in subsequent analysis for clinical prognosis, including determining the severity of the disease affecting the lungs.