Risk prediction and effect evaluation of complicated appendicitis based on XGBoost modeling

Abstract

Purpose

The distinction between complicated appendicitis (CAP) and uncomplicated appendicitis (UAP) remains challenging. The purpose of this study was to construct a safe and economical diagnostic model that can accurately and rapidly differentiate between CAP and UAP.

Methods

Patient data from 773 appendectomies were retrospectively collected, important features were selected using random forests, and the data were divided into training and test sets in a 3:1 ratio. An integrated learning algorithm, Extreme Gradient Boosting (XGBoost), was introduced to predict the risk of CAP and compared with Support Vector Machine (SVM), Random Forest (RF), and Decision Tree (CART) algorithms. A comprehensive comparison of the four algorithms was performed using model performance metrics such as the area under the receiver’s operating characteristic curve (AUC), sensitivity, specificity, accuracy, precision, negative predictive value(NPV), positive predictive value(PPV),calibration curves, and clinical decision curve analysis (DCA).

Result

The results show that all four prediction models exhibit some predictive ability. The XGBoost model showed the best prediction with AUC, accuracy, sensitivity, specificity,NPV and PPV of 0.914, 0.855, 0.865, 0.846,0.848 and 0.897, respectively, followed by the SVM model with results of AUC, accuracy, sensitivity, specificity,NPV and PPV of 0.882, 0.819, 0.865, 0.779, 0.770 and 0.871, respectively. XGBoost and SVM models show very good calibration. The XGBoost model showed better net clinical benefit compared to the DCA curves of the other models.

Conclusion

Predictive models based on the XGBoost algorithm have good performance in predicting the risk of acute appendicitis progressing to complicated appendicitis, which helps to optimize clinical decision making.

Introduction

Acute appendicitis is one of the most common acute abdominal surgical disorders globally, and the lifetime risk of appendicitis in the United States is estimated to be approximately nine percent, with a prevalence of 16–40% for complicated appendicitis [1]. In uncomplicated appendicitis, conservative antibiotic therapy is both effective and safe; in contrast, most patients with complicated appendicitis require emergency appendectomy, and delayed removal of complicated appendicitis leads to additional complications [1, 2]. Although the mortality rate for uncomplicated appendicitis is about 0.1%, the risk of death for complicated appendicitis caused by a perforated or gangrenous appendix is significantly higher, up to five to six percent [3, 4]. Therefore, it is very important to differentiate between these two diseases in time.

The preoperative diagnosis of appendicitis is mainly based on medical history, physical examination, laboratory tests, and imaging. Through these diagnostic methods, more than 90% of acute appendicitis patients can achieve early and accurate diagnosis [5], However, these methods will be challenging to differentiate between complicated appendicitis and uncomplicated appendicitis. The AIRS and Alvarado scoring systems used to diagnose acute appendicitis are constructed on the basis of clinical presentation and laboratory findings [6, 7].These scoring systems are simple to use but are highly subjective and do not effectively differentiate between complicated and uncomplicated appendicitis [8]. There are also several CT-based scoring systems that improve sensitivity and specificity to about 80% [9, 10], CT examinations increase radiation exposure and economic burden for patients, and the diagnostic results obtained in exchange for increased radiation exposure still need to be further improved.The scoring systems of Appendix Severity Index(APSI) and Atema et al. combined clinical use of CT imaging findings for the prediction of complicated appendicitis, and although the results of the two studies possessed a high specificity of 93% and a sensitivity of 90.2%, respectively [11, 12], a study validating these two studies showed that both scoring systems performed poorly overall [13].

In recent years, machine learning techniques have been widely used in clinical medicine, especially in predictive models, which improve the accuracy of predictions by learning from data and experience [14]. Some researchers, tried to apply machine learning to the prediction of complicated appendicitis.Tuong-Anh Phan-Mai et al. combined clinical data and ultrasound results to predict the risk of complicated appendicitis using various machine learning algorithms.The gradient boosting model showed a strong performance (AUC 0.890, accuracy 0.810) [15]. Hilmi Yazici et al., on the other hand, evaluated the performance of multiple machine learning algorithms for the prediction of complex appendicitis in conjunction with clinical data and abdominal CT results, with multiple algorithms demonstrating more than 90% accuracy and specificity, but only 60% sensitivity [16]. Currently relevant predictive models are very dependent on imaging and there is still room to increase their performance. Therefore, a new approach to rule out complicated appendicitis is urgently needed. Therefore, an urgent need exists for a new method to rule out complicated appendicitis.

XGBoost is a relatively novel and efficient machine learning method that combines multiple decision trees in a boosted manner, with the advantages of saving resources, less training time, and higher accuracy, developed by Chen and Guestrin in 2016, and was first used in fields such as transportation and electronics [17,18,19] and is now gradually being applied to many aspects of the medical field, e.g., the prediction of early-stage lung cancer, type 2 diabetes, myocardial infarction, etc., all of which have demonstrated good prediction results [20,21,22]. In this study we attempted to use the XGBoost algorithm for risk assessment of complicated appendicitis using only clinical data with the aim of improving the accuracy of identifying non-complicated appendicitis from complex appendicitis, which is an innovative application of the XGBoost algorithm in the field, and to compare it with other currently dominant machine learning models.The model is finally interpreted through the SHapley's method of additive interpretation (SHAP).

Methods

Patient and data

This retrospective study included all patients who underwent appendectomy between January 1, 2020, and December 30, 2023, at Wenzhou Central Hospital (Zhejiang Province, China). Inclusion criteria were all patients with histologically reported appendicitis, and patients who met one of the following criteria were excluded from the study: 1. patients who were not ultimately diagnosed with acute appendicitis (e.g., histology reported as normal appendix as well as appendiceal tumor); 2. samples with obvious errors in the data (e.g., erythrocyte count of 0.1 × 10¹²/L); and 3. samples with greater than 50% missing data. After the above inclusion and exclusion criteria, a total of 773 patients were finally enrolled in the study (Supplementary Fig. 1). For the small amount of missing data in this group of patients, we utilized the missForest package to fill in the missing values, the median to fill in the continuous variables, and the plurality to fill in the categorical variables.Data on all patients were collected through electronic medical records, including demographic data (gender, age, area of residence, underlying disease), vital signs (temperature, heart rate, systolic blood pressure, diastolic blood pressure), preoperative clinical manifestations (appetite, nausea or vomiting, diarrhea, right lower abdominal tenderness, rebound tenderness and myalgia, metastatic right lower abdominal pain, whether the time between pain and surgery was greater than 24 h, and the degree of pain), laboratory findings (white blood cell, red blood cell count, neutrophil, lymphocyte, monocyte, and eosinophil percentages, hemoglobin, erythrocytes, platelets, C-reactive protein, glucose level, blood urea nitrogen, blood creatinine, and total bilirubin), and the presence or absence of fecaliths. Underlying diseases include diabetes, hypertension, myocardial infarction, heart failure, chronic kidney disease, chronic liver disease, and chronic lung disease. Based on the VRS score, the pain level was categorized as mild, moderate, and severe. Based on the surgical reports and histopathologic findings of the appendix, the patients were divided into two groups: the uncomplicated appendicitis group and the complicated appendicitis group. Patients were considered to have CAP if they had (1) perforated appendicitis, (2) gangrenous appendicitis, or (3) complications such as diffuse peritonitis and pathological findings of abscess formation [2, 5]. Simple and suppurative appendicitis were categorized as UAP [2, 5]. The study follows the revised Declaration of Helsinki and was approved by the the ethics committee of Wenzhou CentralHospital.

Feature selection

For predictive models, too many features increase the risk of model overfitting and can predict diseases more cumbersome in clinical work. To reduce the number of features to be included, we also investigate the importance of predictor variables, in this paper, we use the Mean Decrease Accuracy function provided by RF for feature selection, which ranks the importance of the features by directly measuring the impact of each feature on the accuracy of the model, and use a bar chart to visualize the obtained importance ranking of the features, and then exclude unimportant, This operation does not significantly affect the accuracy of the model, and we finally include the top 15 features for model training.

Model development

Machine learning models were developed and validated using R software (version 4.1.4, https://www.rproject.org/). The XGBoost model and three other common machine learning predictive models were constructed by dividing the data into training and test datasets in a 3:1 ratio, with “whether or not it is complicated appendicitis” as the main output of the model, and accuracy and AUC value as the endpoint variables. The models were also evaluated by calculating AUC, accuracy, sensitivity, specificity, precision, recall, positive predictive value(PPV) and negative predictive value(NPV).

XGBoost model: XGBoost is a powerful integrated learning algorithm, a modification of gradient-boosted Decision Trees (GBDT), which integrates many weak classifiers to form a single strong classifier. The algorithm has good learning results and efficient training speeds and is able to produce prediction accuracies that match many state-of-the-art supervised learning techniques. In this study, we use the “XGBoost” package for model training, and use cross-validation and grid search to optimize the XGBoost model parameters in the training set, and finally we obtain the optimal model parameters, eta, nrounds, max_depth, gamma. subsample, which are 0.3, 1000, 2, 0.001 and 0.7 respectively, subsample are 0.3, 1000, 2, 0.001, and 0.7, respectively, which control the learning rate of the model, the number of decision trees, the maximum depth of the decision tree, the smallest sub-weight in the decision tree, the minimum loss reduction (gamma) required for partitioning, and the number of columns to be subsampled when building the tree. Finally, the model is retrained using the optimal hyperparameters described above and final predictions are made on the test set.

SVM model: SVM is a widely used supervised machine learning algorithm with the goal of creating a hyperplane that can effectively partition a given dataset into different classes. The"e1071"package was used for training, and the Gaussian kernel function was used to establish the nonlinear decision boundary, and the hyperparameters cost and gamma were set to 1 and 0.1, respectively. Cost is a penalty parameter in SVM that controls how much the model is penalized for misclassification and indirectly controls the generalization ability of the model. Gamma controls the complexity of the model by regulating the distribution of the data in the feature space.

RF model: Random forest is a commonly used integrated learning algorithm in the field of machine learning, which consists of a number of decision trees, by combining the predictions of multiple decision trees and thus improving the performance of the model. The random forest model is trained using the"randomForest"package in the R software, using two hyperparameters, ntree, and mtry. ntree is used to set the tree of the decision tree in the random forest, and mtry is used to set the number of variables that can be selected at each node, and the original parameters are set to 500 and 6, respectively, and then determined the values of ntree and mtry when the error value is minimized.

Decision tree model: A decision tree consists of root nodes, branch nodes, and leaf nodes, which can progressively partition the data set based on the input features, and ultimately be able to predict the class of the target variable based on the values of the features. We use the"rpart"package to train the decision tree model, using two hyperparameters, cp, and split. cp is the complexity parameter of the tree, which can be used as a penalty factor to control the size of the tree, and split controls the splitting rule of the tree."gini"is set as the splitting rule and the result will be constructed as a CART decision tree model. The initial cp value is set to 0.001, then the optimal cp value is found and pruned based on the optimal cp value to finally output the decision tree model.

Model evaluation

To evaluate the predictive models using the test dataset, we plotted receiver operating characteristic curves(ROC), calculated the corresponding AUC, accuracy, sensitivity, specificity, precision, recall, PPV and NPV to evaluate the model discrimination. Brier scores and calibration curves were used to evaluate the calibration of the model, and clinical decision curve analysis (DCA) was used to evaluate the net clinical benefit of the model.

Model interpretation

Understandability of predictive models is important in clinical practice.SHAP theory, derived from cooperative game theory, provides a rigorous and highly interpretable tool. We analyze and discuss models with the highest overall performance using SHAP-based explanatory methods to reveal their predictive mechanisms and characteristic contributions.

Statistical analysis

All statistical analyses were performed using R software (version 4.1.4, https://www.rproject.org/),Python(version 3.9.12) software,and SPSS (version 24.0, IBM).

Continuous variables: different representations are used depending on the type of their distribution. For normally distributed data, mean ± standard deviation (SD) is used; for non-normally distributed data, median and interquartile range (IQR) are used.

Categorical variables: expressed as frequencies and percentages. In the table, these data will be presented as “n (%)”, where “n” denotes the number of samples in a category and “%” denotes the proportion of the total sample in that category.

Between-group comparisons: for normally distributed data, the t-test was used for between-group comparisons; for non-normally distributed data, the Mann–Whitney U-test was used for between-group comparisons. Between-group comparisons for categorical variables were tested using the chi-square test. The original hypothesis (H0) was that the distribution of the two groups of data was the same, and the alternative hypothesis (H1) was that the distribution of the two groups of data was different. The two-sided significance level for all tests was set at five percent (p < 0.05).

Results

Baseline characteristics of participants

A total of 773 patients were enrolled in the study, out of which 357 patients were diagnosed with complicated appendicitis (46.2%). In the CAP group, age, body temperature, heart rate, eosinophil percentage, platelets, CRP, blood glucose, and urea nitrogen were significantly higher than in the UAP group, and systolic blood pressure, monocyte percentage, erythrocyte percentage, and total bilirubin were significantly lower in the CAP group than in the UAP group. Also compared to the UAP group, 89% of patients in the CAP group had WTOG24 h. Other key demographics, clinical presentation, and laboratory findings are summarized in Table 1.

Table 1 Baseline characteristics of patients

Distribution of variable screening results and data sets

Appendicitis type was used as an outcome variable, and there were 31 predictor variables. The results of feature selection performed by Mean Decrease Accuracy are shown in Fig. 1. We included the top 15 predictor variables in the subsequent model construction, which were, in order, CRP、WTOG24 h、Muscular tone、HR、Platelet、Body temperature、UN、Eosinophils、SBP、Blood glucose、Monocytes、Age、Neutrophils、Lymphocytes and Erythrocyte. After removing redundant variables, patients were divided into a training set (n = 580) and a test set (n = 193) in a ratio of 3:1, and all features of the samples were comparable in the training and test data sets (Supplementary Table 1).

Fig.1

Variable importance graph. The larger the Mean Decrease Accuracy value corresponding to the predictor variable, the greater the impact on the accuracy of the model's predictions

Comparison with other prediction models

First, we train the model on XGBoost, RF, SVM, and CART in the training dataset, respectively. In the training set, XGBoost, RF, and SVM perform very well in terms of discrimination with AUCs of 0.996, 1.0, and 0.951 and accuracies of 96.7%, 99.7%, and 91.2%, respectively, while possessing very good sensitivity, specificity, NPV,and precision, while performing the worst in the CART model with AUCs and accuracies of 0.82 and 80.3% (Table 2). Although XGBoost, RF, and SVM models perform well in the training set, it does not mean that they have better generalization ability.

Table 2 Diagnostic performance of different machine learning models

Subsequently, we validated the model in the test dataset, and the results show that the XGBoost model exhibits fairly good discrimination, with AUC, sensitivity, specificity, precision, PPV, NPV, and accuracy of 0.914, 86.5%, 84.6%, 82.8%, 84.8%, 89.7%,and 85.5%, respectively, in comparison to RF, which, although it performs the best in the training set, shows a significant decrease in the test set in terms of specificity, PPV, and accuracy both decreased significantly, 69.2%, 71.2%, and 78.2%, respectively. The diagnostic performance of the SVM model also showed good discrimination in the test dataset, with AUC, sensitivity, NPV, and accuracy of 0.882, 86.5%, 87.1%, and 81.9%, respectively, and specificity and accuracy showing lower 77.9% and 77%, which is still a little less than XGBoost's performance in general. The corresponding ROC curves for the four models are shown in Fig. 2. To further assess the value of these predictive models for clinical applications, we plotted DCA curves (Fig. 3) using the test dataset for evaluating the expected net benefits of the models at the corresponding risk thresholds. The results showed that the XGBoost, RF, and SVM models all showed substantial net benefits, with the XGBoost model showing superior net clinical benefit.

Fig. 2

ROC curves for XGBoost, RF, CART, and SVM models in the test dataset

Fig. 3

DCA curves for (a) XGBoost, (b) RF, (c) SVM, and (d) CART models in the test dataset

To understand the accuracy of the model predictions, we used calibration curves in our test dataset to show the gap between the actual probabilities and the predicted probabilities as shown in Fig. 4, it is obvious that the XGBoost and SVM models show the best calibration ability, which indicates that the predictions of the models are highly consistent with the actual values. Finally, we also calculated the Brier scores for the XGBoost, RF, SVM, and CART models, which were 0.118, 0.151, 0.202, and 0.139, respectively, indicating that all four prediction models have some predictive ability.

Fig. 4

Calibration curves for (a) XGBoost, (b) RF, (c) SVM, and (d) CART models in the test dataset

Feature-based model interpretation

We calculated and visualized the SHAP values for each feature in the Xgboost model. The swarm plot (Fig. 5A) shows an overview of the contribution of all patient features. In the swarm plot, features are sorted by the sum of the magnitude of the SHAP values of all samples, and the SHAP values are used to show the distribution of the impact of each feature. The bar graph shown in Fig. 5B illustrates the mean absolute value of the SHAP values for each feature. The graph shows that CRP,heart rate and WTOG24 h have higher mean values (Fig. 6B).

Fig. 5

The SHAP plots illustrated the feature-based model interpretation process. A The beeswarm plot used SHAP values to show the distribution of each feature’s impacts. B The standard bar plot demonstrated the mean absolute value of the SHAP values for each feature

Fig. 6

SHAP plots demonstrated SHAP values from a case-based perspective. Sampled by model output, the overall SHAP plot (A) showed the decision process of all patients. The force plot (B) and the waterfall plot (C) demonstrated the proportion and absolute SHAP value of various features in the decision-making process for a single patient

In addition Figs. 6A and B show the specific contribution of each feature in a single sample to the model prediction from two perspectives. Among them, CRP and WTOG24 h have the most significant impact on model prediction. low values of CRP and WTOG24 h significantly reduce the probability of model prediction in the negative category. Other characteristics such as age and systolic blood pressure also had an effect, but it was relatively small. Overall, model predictions were primarily driven by CRP and WTOG24 h, two features that are closely associated with disease onset and progression in clinical practice.

Discussion

In this study, XGBoost was introduced for the first time for the prediction of complicated appendicitis. The study focuses on building an XGBoost prediction model for the prediction of complicated appendicitis by combining clinical and laboratory findings and comparing the results with several other traditional machine learning models.Based on the order of importance of the variables, we finally included 9 laboratory indicators and 6 other clinical indicators, all of which were readily available before undergoing appendectomy, the top 5 predictors of which were CRP, WTOG24 h, abdominal muscle tension, heart rate, and platelet count, which is slightly different from other similar studies, which may be related to the methodology of the study and the various parameters included, but similarly, CRP levels were significant in several studies [11, 23,24,25]. In addition, in order to provide clinicians with a deeper understanding of the model, we introduced SHAP into the Xgboost prediction model to further reveal the decision-making mechanism of the model, and analyzed the total contribution of each feature in the model, and the results showed that CRP and WTOG24 h occupied the most important position in the model's contribution, which was in perfect agreement with our Mean Decrease Accuracy feature selection results. This is fully consistent with our Mean Decrease Accuracy feature selection results.

CRP is a non-specific acute phase reactant that primarily stimulates cell-mediated immunity and chemotaxis in inflammation. It has been shown that CRP levels were significantly higher in perforated appendicitis cases than in non-perforated cases, which is similar to our findings, in addition serum CRP concentration and severe appendicitis were correlated, and elevated CRP was an independent risk factor for predicting perforation in acute appendicitis [26, 27]. Nina A Bickell et al. explored the risk of future appendiceal rupture in patients as the time since symptom onset without treatment increased, showing that the risk was negligible in the first 24 h of symptomatic untreated life, began to rise between 24 and 36 h, and then climbed to six percent in patients who had not been treated 36 h after symptom onset, and the guidelines recommend that for Treatment of patients with appendicitis should be limited to as few as 24 h after the onset of symptoms as possible [28, 29].

Several of the models we built were then internally validated in a separate cohort, and the XGBoost prediction model showed optimal predictive power, which cannot be separated from the superiority of the algorithm itself.Similarly, Qingqing Li et al. [30] constructed a variety of machine-learning models based on gene expression, including XGBoost, DT, SVM, K-Nearest Neighbors (KNN), LR, and RF for breast metastasis-assisted identification, with the XGBoost classifier achieving an overall higher average AUC (0.82); In addition, Tingting Fan et al [31] extracted patient data from the MIMIC-IV database and constructed an XGBoost model for predicting the risk of diabetic ketoacidosis-associated acute kidney injury in ICU patients and compared it with seven other machine learning models, and the XGBoost model performed the best among the eight machine learning models, with AUCs for the training and validation sets of 0.835 and 0.800 for the training and validation sets, respectively. The research on XGBoost will bring this excellent machine-learning method to the attention of more researchers and has many implications for future research. It is highly anticipated that in the future even better researchers will make more in-depth studies on XGBoost in the prediction of complex appendicitis, and even extend XGBoost to more medical fields.

Given the numerous adverse complications of complicated appendicitis, early and accurate diagnosis of the disease can greatly reduce complications. This study is dedicated to determining whether a patient has progressed to complicated appendicitis using only the patient's preoperative vital signs, laboratory tests, and clinical presentation, The constructed predictive model improves clinical decision-making in patients with suspected complex appendicitis and can be targeted to change high-risk patients to a higher level of care with early preoperative preparation as well as surgical interventions, yet antibiotic therapy is also a safe and effective option for low-risk patients, which can save low-risk patients from acute surgical risks. Dan Liang et al. [32] incorporated clinical features, CT visual features, deep learning features, and radiomics features to develop a model using the CatBoost algorithm for the prediction of complicated appendicitis, and the model was validated at three other medical centers, and the results demonstrated moderate predictive performance with AUCs of 0.836, 0.793, and 0.72, respectively. Also for deep learning radiomics features, the model uses a manual depiction of regions of interest(ROI), which will be challenging to diagnose complicated appendicitis in emergency situations. Hui-An Lin et al. [33] utilized an ANN model to evaluate the diagnostic performance of nine different variable sets in a clinical model, and the results showed that Lin et al.'s model exhibited the most outstanding diagnostic performance, with a sensitivity and specificity of 85.7% and 91.7% in the test dataset, respectively, and that higher sensitivity and specificity could reduce the possibility of leakage and misdiagnosis, and that the high performance of the model was attributed to the incorporation of the periappendiceal fluid and fat stranding signs, which are two very important radiologic Characterization. In contrast, our study demonstrated a very high level of accuracy even without relying on radiologic imaging, and high accuracy means that the diagnostic test correctly identifies the patient's disease state in most cases, whether it is complex or uncomplicated appendicitis, which is critical for clinical decision making.This initiative will bring numerous benefits, first, by reducing the financial burden on patients while eliminating the time spent waiting for CT results; second, by reducing unnecessary radiation exposure to patients; and lastly, by allowing for adjunctive diagnostics in the absence of specialized equipment and personnel to make decisions for surgical decision making.

There are some limitations of our study. First, limited by the research platform and funding, our study was a single-center retrospective study with cases from the same medical institution and a limited number of cases, which may not be representative of a broad population, and we need to obtain a large sample in the future to further validate our findings; second, retrospective studies may have potential bias, which will result in a lack of extrapolation of our study, and we look forward to adopting a multicenter in our future studies, prospective design to improve the generalizability of the findings.

Conclusion

In this study, we developed and validated the XGBoost prediction model for risk assessment of complex appendicitis and compared it with the current mainstream algorithms (SVM, RF and CART). The results show that XGBoost has the strongest discrimination and calibration and also has good generalization ability. The predictive model developed in this study based on key clinical features and laboratory tests improves the ability to discriminate between complex and uncomplicated appendicitis, thus optimizing clinical decision-making. In current diagnostic practice, reliance on subjective clinical presentation and imaging may lead to missed or delayed diagnosis of complicated appendicitis, increasing the risk of complications such as perforation and abscess. This study provides a rapid and reliable diagnostic tool by integrating easily accessible clinical indicators, which fills the gaps in existing diagnostic methods and can reduce unnecessary imaging tests and decrease the risk of complications, thus improving patient prognosis and the efficiency of healthcare resource utilization.