- Research
- Open access
- Published:
Development and validation of prediction models for nosocomial infection and prognosis in hospitalized patients with cirrhosis
Antimicrobial Resistance & Infection Control volume 13, Article number: 85 (2024)
Abstract
Background
Nosocomial infections (NIs) frequently occur and adversely impact prognosis for hospitalized patients with cirrhosis. This study aims to develop and validate two machine learning models for NIs and in-hospital mortality risk prediction.
Methods
The Prediction of Nosocomial Infection and Prognosis in Cirrhotic patients (PIPC) study included hospitalized patients with cirrhosis at the Qingchun Campus of the First Affiliated Hospital of Zhejiang University. We then assessed several machine learning algorithms to construct predictive models for NIs and prognosis. We validated the best-performing models with bootstrapping techniques and an external validation dataset. The accuracy of the predictions was evaluated through sensitivity, specificity, predictive values, and likelihood ratios, while predictive robustness was examined through subgroup analyses and comparisons between models.
Results
We enrolled 1,297 patients into derivation cohort and 496 patients into external validation cohort. Among the six algorithms assessed, the Random Forest algorithm performed best. For NIs, the PIPC-NI model achieved an area under the curve (AUC) of 0.784 (95% confidence interval [CI] 0.741–0.826), a sensitivity of 0.712, and a specificity of 0.702. For in-hospital mortality, the PIPC- mortality model achieved an AUC of 0.793 (95% CI 0.749–0.836), a sensitivity of 0.769, and a specificity of 0.701. Moreover, our PIPC models demonstrated superior predictive performance compared to the existing MELD, MELD-Na, and Child-Pugh scores.
Conclusions
The PIPC models showed good predictive power and may facilitate healthcare providers in easily assessing the risk of NIs and prognosis among hospitalized patients with cirrhosis.
Graphical abstract
Introduction
Liver cirrhosis represents the 11th most common cause of death and accounts for over one million deaths worldwide [1]. Infections are the most frequent and severe complications, affecting 20–46% of patients with cirrhosis [2], particularly those with decompensated cirrhosis. Patients with cirrhosis are at an increased risk of infections due to intestinal dysbiosis, impaired intestinal barrier, continuous translocation of pathogens, cirrhosis-related immune dysfunction, and portal shunting [3, 4]. An important aspect of infection in patients with cirrhosis is the development of nosocomial infections (NIs), with or without multidrug resistance (MDR), given the frequency of hospital admissions and antibiotic prophylaxis administered for bacterial peritonitis [5]. NIs in patients with cirrhosis are particularly dangerous [6,7,8] and can lead to prolonged hospitalization and an increase in the risk of death [9].
NIs frequently occur and adversely impact outcomes for hospitalized patients with cirrhosis [10]. The development of an NI complicates the condition of patients with cirrhosis and predisposes them to various complications, such as hepatic encephalopathy (HE), acute-on-chronic liver failure (ACLF), and acute kidney injury (AKI) [11, 12]. Moreover, NIs may occur during the progression of these complications, leading to further decompensation and potentially increasing the short-term mortality rate by 2–4 times [13]. Even when the infection resolves, survival continues to be compromised [14]. Given the generally poor prognosis for patients with cirrhosis with NIs, early identification of high-risk individuals is essential for their prevention, treatment, and management [2].
Antibiotic prophylaxis is recommended for high-risk populations because it reduces the incidence of infections and improves survival for patients with cirrhosis. However, the prescription of antibiotic prophylaxis is usually empirical, which may increase the risk of MDR and worsen the prognosis of patients with cirrhosis [15]. The identification of populations at high risk who will benefit from antibiotic prophylaxis needs to be further refined. Several models have been proposed for risk prediction of NIs in patients with cirrhosis. However, these models have generally relied on traditional statistical modeling, lacked external validation, and showed poor performance [10, 12]. Currently, there is a noticeable absence of widely utilized, NI-specific models in patients with cirrhosis. This gap underscores the necessity of employing advanced approaches and developing effective predictive models in this field. Therefore, our study compared a range of machine learning methods, with the goal of developing a robust model for risk prediction of NIs and concurrently evaluating the risk of in-hospital mortality in patients with cirrhosis. This endeavor holds the potential for early identification of high-risk groups among patients with cirrhosis, offering a valuable foundation for clinical intervention and informed decision-making.
The patterns and pathogens of infections in patients with cirrhosis are evolving rapidly, partly due to concomitant medications and antibiotic overuse [2]. There is an urgent need for effective management strategies to improve the outcomes for cirrhotic patients who develop infections [2]. Integrating the NI prediction model into clinical decision support systems can provide personalized risk assessments by analyzing large volumes of patient data, allowing for tailored preventive and therapeutic measures for each patient. Additionally, implementing the NI prediction model will help hospitals accumulate more data for continuous analysis and improvement. This ongoing optimization will enhance infection control, reduce infection rates, and create a virtuous cycle.
Materials and methods
Study design and participants
We conducted a retrospective observational study to develop and validate prediction models for NIs and in-hospital mortality in hospitalized patients with cirrhosis. The deviation cohort consists of a retrospective cohort of patients hospitalized with cirrhosis in the Qingchun branch of the First Affiliated Hospital of Zhejiang University, from Jan 1 to Dec 31, 2022. All adult patients hospitalized with cirrhosis (≥ 18 years old) were included. For patients with multiple hospitalizations during the study period, only the first hospitalization was included. Exclusion criteria were patients with incomplete or missing medical records, patients with hepatocellular carcinoma, patients with previous liver transplantation, and patients with a hospital stay of < 48 h. The external validation cohort of the models were obtained from the Zhijiang branch of the First Affiliated Hospital of Zhejiang University during the same period. Figure 1 presents a detailed flowchart of inclusion and exclusion of participants. The study protocol conforms to the principles of the Declaration of Helsinki and has been approved by the Ethics Committee of the First Affiliated Hospital of Zhejiang University.
Definitions
The diagnosis of cirrhosis was established independently by two experienced hepatologists, relying on either liver histology or a combination of distinctive clinical, biochemical, and imaging features [16]. In cases of discrepancies, resolution was achieved through panel discussion. Non-infectious complications of cirrhosis, such as ascites, hepatorenal syndrome (HRS) and HE, were identified in patients according to guidelines established by the European Association for the Study of the Liver and International Ascites Club [17].
Infections in patients with cirrhosis refer to various bacterial or fungal infections that occur due to immune dysfunction and bacterial translocation during the course of the disease [2]. In this study, we focus on common infections in cirrhosis patients, including pneumonia, SBP, spontaneous bacteremia, UTI, C. difficile infection, soft tissue/skin infection, intra-abdominal infection, secondary bacterial peritonitis, infections of unknown sites, and other infections. These infections are identified as the most common in cirrhosis patients based on previous studies [10, 18, 19]. Detailed diagnostic criteria for these infections are provided in the Diagnostic Criteria of Infections section of the Supplementary Methods. All infections that occurred, both upon admission and during hospitalization, were documented according to the standard criteria. Specific data regarding the site of infection and the resistance profiles of infections were collected. Following the guidelines of the European Society of Clinical Microbiology and Infectious Diseases (ESCMID) and consensus definitions, MDR was defined as insensitivity to at least one agent from three or more antimicrobial classes [20].
Following previous studies [2, 4, 19, 21], infections are classified based on their source into community-acquired (CA), healthcare-associated (HCA), and NIs. Specifically, CA was defined as those diagnosed within 48Â h of admission and without any hospital stays in the prior 6 months; HCA infection was defined as those diagnosed within 48Â h of admission in patients who had been hospitalized for at least 2 days in the preceding 6 months; and NIs were defined as those identified more than 48Â h after admission.
Predictors and data collection
Data collection was conducted by senior medical students who were specially trained to use the electronic medical records (EMRs) system. Standardized protocols were developed to ensure data quality and consistency, with supervising faculty members performing regular quality control by sampling and reviewing the data. Inter-rater reliability assessments were also conducted to ensure consistency among different data collectors. Our candidate predictive variables were determined based on previous published studies, expert opinions, and consensus statements [7, 19, 22]. Variables at admission included demographic variables (sex, age), date of hospitalization, ward, cause of liver disease, laboratory test values, underlying diseases according with the Charlson Comorbidity Index (CCI) [23], and the source of infection and its susceptibility pattern. A total of 44 variables were finally identified for inclusion in our study. This study followed the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) reporting guideline [24].
Data processing and variables selection
Table S2 shows the specific missing data rates for each variable. The table reveals that, apart from variables with less than 10% missing data, the remaining variables exhibit substantial missingness, with the lowest being 41.94%. This high level of missing data is likely due to these variables not being routinely collected in patients with cirrhosis. Therefore, we excluded the 11 variables with more than 10% missing values. For variables with a missing rate of 10% or less, we applied multiple imputation (MI) using predictive mean matching (PMM) for imputation [25]. Following the processes of data cleaning, sampling, and preprocessing, 33 variables were retained for analysis [26]. Furthermore, continuous variables were fitted using Decision Tree (DT) models to account for nonlinearities, and these variables were categorized into risk categories based on their relationship to NIs and in-hospital mortality, respectively (Supplementary Methods). Considering that the imbalance of sample categories could impair the model’s ability to predict minority classes accurately, we employed the Synthetic Minority Over-sampling Technique (SMOTE) to synthesize samples for the minority classes, thereby enhancing the predictive precision of the models (the detailed mechanism of SMOTE is described in the Supplementary Methods) [27].
Prior to the modeling process, we used Random Forest (RF), Extremely randomized Tree (ET), and Ridge regression to rank the importance of variables. For each run the data were identically encoded but due to SMOTE (under 10-way cross-validation) each run differs slightly. Therefore, each classifier was applied 100 times to ensure robustness. Subsequently, the variable importance of the three classifiers were visualized using matplotlib [28]. In determining the variables ultimately included in the model, we followed a systematic process. First, we conducted a comprehensive comparison of three different methods, selecting variables that consistently ranked high among them. Second, we referenced published literature to consider variables that, while initially ranked lower in importance, demonstrated significant correlations with clinical outcomes, such as NIs or in-hospital mortality, in prior studies. Finally, we integrated the performance of the selected variables from the first two steps to finalize the variables included in the model development.
Development, validation, and evaluation of the models
Owing to the sample size, we did not use split-sample internal validation techniques because this may lead to instability of the predicted results [29]. We followed the TRIPOD recommendations and internally validated our prediction models with 1,000 bootstrapping of the training data. Bootstrapping is a process that involves random sampling and substitution of the original dataset, which is essential for obtaining reliable statistical inferences. By averaging performance metrics over multiple replicated experiments, the bootstrap method is more effective at reducing overfitting bias than other methods of internal validation, providing stable and low-biased estimates, and is a reliable method for assessing internal validity [30, 31]. Six classical machine learning models; RF, eXtreme Gradient Boosting (XGBoost), Neural Network (NN), Logistic Regression (LR), Adaptive Boosting (Adaboost), and DT were developed to predict NIs and in-hospital mortality. We chose these models because they are commonly used for clinically relevant predictions, have easily interpretable clinical outcomes, and consistently produce the best predictions [32, 33]. Random grid search and cross-validation analysis were performed to determine the optimal parameters.
We evaluated the performance of different algorithms by comparing their area under the curve (AUC) results in internal and external validation. In addition to the AUC metric, for a comprehensive assessment of the predictive capabilities of the algorithms, we also reported six evaluation metrics: sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR) and negative likelihood ratio (NLR). When various modes exhibit comparable predictive performance we chose the most parsimonious model [27]. To enhance user accessibility, we also developed an online calculator. By inputting data from the EMRs upon admission, this calculator automatically predicts the risk of NIs and in-hospital mortality during hospitalization.
To assess the clinical utility of the models, we performed a decision curve analysis using the external validation cohort to compare the net benefit of the best performing model with a treat-all strategy. Within a range of probability thresholds, a robust model should outperform the treat-all strategy, identifying those at risk while avoiding unnecessary interventions for those unlikely to develop NIs or experience in-hospital mortality. An ineffective model would show no, or lower net benefit compared with the treat-all strategy [33]. Moreover, we used the calibration curve with Brier score to assess model calibration. Brier score can be interpreted as the distance between the predicted and observed outcomes. A lower score indicates a better calibration [34]. To further assess the robustness, we compared the models with several established risk prediction scores for liver disease: the Model for End-stage Liver Disease (MELD), MELD-Na, and Child-Pugh score. Subgroup analyses were conducted regarding age (< 60 or ≥ 60 years), body mass index (BMI, < 25 or ≥ 25 kg/m²), hypertension (with or without hypertension), diabetes (with or without diabetes), and complication (with or without ascites, HE, HRS, or gastrointestinal bleeding complications).
Statistical analysis
Categorical variables were shown as frequencies and proportions. For continuous variables, those following a normal distribution were expressed as means and standard deviations, while those not normally distributed were presented as medians and interquartile ranges. In univariate analyses, categorical variables were analyzed using either the Chi-square or Fisher’s exact test. For normally distributed continuous variables the two-tailed Student’s t-test was used, and the Mann-Whitney U test was used for continuous variables that were not normally distributed. In all statistical analyses, significance was set at p < 0.05. Analyses were conducted with R software version 4.3.0 (The R Foundation, Vienna, Austria) and Python software version 3.10.7 (Python Software Foundation, Wilmington, USA) [35].
Results
Patient characteristics
Our study initially enrolled 5,479 hospitalized patients with cirrhosis. Of these, 1,793 were included in the final analysis, including 1,297 in the derivation cohort and 496 in the external validation cohort. Baseline characteristics of patients included in both cohorts during hospitalization are shown in Table S1. In the derivation cohort, patients were younger compared to those in the external validation cohort (mean age, 61 vs. 63 years, respectively, p < 0.01). The proportion of viral hepatitis as an etiology was higher in the external validation cohort than in the derivation cohort (63.3% vs. 59.4%, p < 0.01). Furthermore, the rate of intensive care unit (ICU) admission was higher in the external validation cohort than in the derivation cohort. No significant differences were observed in terms of length of hospitalization, the proportion of NIs, or in-hospital mortality, between the two cohorts.
NIs profiles
Table 1 shows the occurrences of NIs in the derivation cohort, with 108 patients (8.3%) reporting NIs. Demographic characteristics, such as sex and BMI, were similar between patients with and without NIs. However, patients with NIs were more frequently admitted with infections, developed ascites and HE, and had higher CCI and MELD scores. Compared to patients without NIs, those with NIs had higher rates of ICU admission, longer hospital stays, and an increased in-hospital mortality rate.
Table 2 shows the details of NIs in the derivation and external validation cohorts. Among NIs, the proportions of spontaneous bacterial peritonitis (SBP), intra-abdominal infection, and fungal infection were similar in the derivation and external validation cohorts. By contrast, the external validation cohort had a higher proportion of pneumonia, urinary tract infections, and MDR. More individuals in the derivation cohort tended to be spontaneous bacteremia, spontaneous empyema, and other bacterial infections, though these were not statistically different from those in the external validation cohort.
Correlation analysis and variable importance ranking analysis
To reduce the complexity of the prediction models and avoid including variables with high correlations, we analyzed Pearson correlation coefficients using heat maps. Fig. S1 shows high correlations between aspartate aminotransferase (AST) and alanine aminotransferase (ALT) (r = 0.83) and between international normalized ratio (INR) and prothrombin time (PT) (r = 0.87). However, the other candidate predictors did not show significant correlations, indicating their independence from each other.
In the process of variable selection, three ranking methods; RF, ET, and ridge regression, were used to rank the importance of variables (Fig. S2 and Fig. S3). The variables included in the model were selected based on the results from variable importance ranking results as well as a priori knowledge of the clinical significance of the variables. For NIs, we included eight variables: C-reactive protein (CRP), albumin (ALB), ALT, glomerular filtration rate (GFR), INR, CCI, BMI, and ICU admission. For in-hospital mortality, we included ten variables: ALB, GFR, white blood cell (WBC) count, CCI, INR, complication, sex, NI, high-density lipoprotein (HDL), and serum sodium (Na).
Performance of the machine learning models in predicting NIs
A summary of the performance of our machine learning models is reported in Table S2. Among the six classification algorithms for predicting NIs, RF performed best. In the internal validation cohort, the RF model predicted NIs with an AUC of 0.784 (95% confidence interval [CI] 0.741–0.826), a sensitivity of 0.712, and a specificity of 0.702 (Fig. 2a and Table S2). In the external validation cohort, the RF model predicted NIs with an AUC of 0.728 (95% CI 0.655–0.798), a sensitivity of 0.692, and a specificity of 0.716 (Fig. 2b and Table S2). Further, the calibration plot for the external validation confirmed the consistency between the risks of NIs predicted by the RF algorithm and the actual observed risks of NIs (Fig. 2c). Given that the event occurrence rate was 11%, models with a Brier score less than 0.10 were considered informative, and our model achieved a score of 0.08. Next, we conducted decision curve analysis to assess the net benefit of the developed prediction model. The analysis showed that the net benefit of the NIs prediction model was significantly higher compared to the strategies of ‘treat none’ and ‘treat all’ (Fig. 2d).
Performance of the machine learning models in predicting in-hospital mortality
As shown in Table S3, the performance of RF in predicting in-hospital mortality was better than that of other machine learning models. In the internal validation cohort, the RF model predicted in-hospital mortality with an AUC of 0.793 (95% CI 0.749–0.836), a sensitivity of 0.769, and a specificity of 0.701 (Fig. 3a and Table S3). In the external validation cohort, the RF model predicted in-hospital mortality with an AUC of 0.751 (95% CI 0.666–0.834), a sensitivity of 0.727, and a specificity of 0.706 (Fig. 3b and Table S3). In the calibration curve, considering a 7% event rate, a model with a Brier score of less than 0.07 was informative, and our model reached a score of 0.05, indicating that it had reliable predictive power (Fig. 3c). Decision curve analysis showed that the in-hospital mortality prediction model had considerable clinical benefit (Fig. 3d). Additionally, the feature importance ranking plots for the models were provided in Fig. S4 and Fig. S5. An online application to estimate the predicted NI and in-hospital mortality risk is available at the website: https://pipcmodel.streamlit.app/.
Comparison of model performance with clinical existing models
Our PIPC-NI model demonstrated superior predictive performance compared to the MELD, MELD-Na, and Child-Pugh scores. It achieved an AUC of 0.784 (95% CI 0.741–0.826) in the internal validation, exceeding the best performing MELD score of 0.687 (95% CI 0.634–0.741). In the external validation, the PIPC-NI model’s AUC was 0.728 (95% CI 0.655–0.798), superior to the highest Child-Pugh score of 0.660 (95% CI 0.582–0.727) (Fig. S6). Similarly, our PIPC-mortality model also outperformed the above scores in both internal and external validations. It recorded an AUC of 0.793 (95% CI 0.749–0.836) in the internal validation, and an AUC of 0.751 (95% CI 0.666–0.834) in the external validation (Fig. S7). The DeLong’s test for comparing AUCs and the McNemar test for comparing sensitivity and specificity are presented in Tables S7 and S8.
Subgroup analysis
We evaluated our models across different subgroups of patients. Generally, an AUC of 0.50–0.70 is regarded as low accuracy, 0.70–0.80 as acceptable, 0.80–0.90 as excellent, and above 0.90 as outstanding. In our study, the AUC for the PIPC-NI model ranged from 0.737 to 0.827 in the internal validation. In the external validation, the PIPC-NI model ranged from 0.654 to 0.849 across different subgroups (Table S4). For the in-hospital mortality model, the AUC ranged from 0.715 to 0.817 in the internal validation, and from 0.675 to 0.820 in the external validation (Table S5).
Discussion
Using retrospective data, we developed two new prediction models for NIs and in-hospital mortality in adults with cirrhosis. The RF models demonstrated better performance in predicting NIs (AUC: 0.784, 95% CI 0.741–0.826) and in-hospital mortality (AUC: 0.793, 95% CI 0.749–0.836) when contrasted with other machine learning models (AUC ranging from 0.707 to 0.751 and AUC ranging from 0.716 to 0.762, respectively). Our PIPC models (PIPC-NIs and PIPC-mortality) had been tested to be robust through internal and external validation processes. Additionally, based on these models, we developed an easily accessible website tool. Given that all variables were derived from routinely collected clinical variables, these models can be easily applied to patients hospitalized with cirrhosis, thereby facilitating the identification of high-risk groups, and enhancing clinical decision-making.
NIs are prevalent and detrimental in individuals with cirrhosis, as it can compromise the functionality of both hepatic and extra-hepatic organs, which can be associated with a very poor prognosis. The available literature suggests that the mortality rate within one month for patients with cirrhosis with NIs reaches 28%, compared to only 8% for those without NIs [10]. Our study aligns with previous findings, indicating that the in-hospital mortality rate for patients with cirrhosis with NIs stands at 19%, in stark contrast to the lower rate of 8% observed in individuals without NIs. These findings underscore the importance of primary prophylaxis against NIs in this specific population. Identifying high-risk patients for early intervention is imperative, with antibiotic prophylaxis emerging as the preferred strategy, particularly for those high-risk individuals, such as patients with acute gastrointestinal bleeding, low levels of ascitic fluid proteins, and those with a history of spontaneous bacterial peritonitis [36].
Despite the demonstrated efficacy of antibiotic prophylaxis in reducing infection rates and mortality among patients with cirrhosis, prolonged use presents significant drawbacks, notably the emergence of pathogens with MDR. This critical concern emphasizes the necessity for more nuanced and tailored approaches in antibiotic administration [37]. Predictive models may play a pivotal role by accurately identifying high-risk patients, enabling more personalized therapy [38]. Our PIPC-NIs models have been developed to ensure that the appropriate antibiotic treatment is delivered to the high-risk individuals at the optimal time. This targeted approach minimizes unnecessary antibiotic exposure, thereby preserving their effectiveness for future needs. Adopting this precision approach in managing infections among patients with cirrhosis is not only a move towards individualized care but also a significant step in improving public health outcomes.
There are many scoring methods for liver disease prognostic assessment in clinical settings, including MELD, MELD-Na, and Child-Pugh, but prediction models for NI-specific outcomes in patients with cirrhosis are still lacking in the literature. Our study underscores the superior performance of the PIPC-NIs and PIPC-mortality models through comparisons with existing models in predicting the risk of NIs and prognosis in patients with cirrhosis. The enhanced effectiveness of our model, compared to previous predictive scores, can be attributed to a meticulous approach to variable selection and modeling strategies. Our predictive variable selection was firstly guided by previous research, expert opinions, and consensus statements, which helped to narrow down the pool of candidate predictors from the complex array of clinical variables. Subsequently, three variable importance ranking methods were used to select the most important predictive variables. By eliminating these superfluous variables and focusing on key variables, we can reduce data complexity and enhance model efficiency [39].
Moreover, in contrast to prior scores that heavily relied on statistical regression methods for variable selection and model training [10], we embraced a series of machine learning methods to construct the prediction model, characterized by a reduced reliance on assumptions and an enhanced capacity to handle numerous predictors and complex interactions. This methodological shift contributes significantly to the improved performance of our predictive model [10, 33]. In addition, we integrated our models into a web-based tool, designed to aid clinicians in assessing the risks of NIs and in-hospital mortality in patients with cirrhosis. Compared with the time-consuming pathogen culture method, our web tool is easy-to-apply and promptly estimates the risks of NIs. This opens up the possibility of early detection of high-risk patients with cirrhosis for NIs, thus enabling timely intervention, potentially reducing hospitalization duration and healthcare costs. Additionally, when integrated into clinical decision support systems, our PIPC model can automatically assess the infection status of patients, providing comprehensive information to doctors, aiding in the development of personalized treatment plans and dynamic adjustment of therapeutic strategies. Prediction models also drive continuous data-driven improvement by accumulating and analyzing vast amounts of patient data, allowing for ongoing optimization of the models and infection control strategies. Overall, our prediction models have considerable potential to enhance the treatment effectiveness and quality of life for patients with cirrhosis.
The PIPC-NI model has substantiated the impact of eight key variables, including CRP, ALB, ALT, GFR, INR, BMI, CCI, and ICU admission, on NIs outcomes in patients with cirrhosis. Specifically, our model confirms the predictive value of ALB, CRP, diabetes, BMI, and ICU admission, which aligns with previous studies [10, 22, 40, 41]. In prior research, lower levels of ALB have been correlated with higher rates of infection, reflecting compromised health and immune function. Elevated CRP levels have been consistently linked to an increased risk of NIs, as they indicate an ongoing inflammatory response and potential underlying infection. Patients in the ICU are often exposed to invasive procedures, prolonged hospital stays, and the use of broad-spectrum antibiotics, all of which increase the risk of developing NIs. Additionally, the model identifies that ALT, GFR, and INR were also major factors in terms of NIs risk prediction. In clinical practice, elevated ALT levels usually reflect the aggravation of inflammation or damage of the liver, and the decline of liver function will affect its ability to clear bacteria and toxins, thereby increasing the risk of infection. Moreover, HRS is a serious complication of end-stage cirrhosis marked by increased splanchnic blood flow, hyperdynamic state, reduced central volume, activation of vasoconstrictor systems, and extreme renal vasoconstriction, which together result in a significant decrease in GFR [42]. Impaired kidney function will further lead to the accumulation of toxins, forming a vicious circle. In addition, INR is an indicator used to evaluate blood coagulation time. When liver function is compromised, the ability to synthesize coagulation factors decreases, and the normal coagulation process plays a crucial role in local control of infection [43]. The PIPC-mortality model includes ALB, GFR, WBC, CCI, INR, complication, sex, NIs, HDL, and Na as predictors. As previously reported, NIs were associated with an increased risk of mortality, and its inclusion in the PIPC-mortality model underscore the importance of NIs for patients with cirrhosis. Additionally, patients with a lower ALB had a higher risk of mortality that may be due to compromised nutritional status, impaired immune response, and increased susceptibility to infections [44]. Furthermore, the PIPC-mortality model also confirmed that comorbidities and complications of cirrhosis could increase the risk of death in hospitalized patients [22].
Strengths
First, this study is a large, clinical cohort that incorporates routinely available clinical and demographic variables to predict NIs and prognosis of patients with cirrhosis. This implies that it can be directly applied to clinical practice and is readily available for further external validation in countries with routine data for such purposes. Second, we evaluated various types of machine learning models to compare their performance in modeling the same dataset, guiding us in selecting the final machine learning model for clinical applications. Third, we have developed an online calculator for established prediction models, which effectively overcomes the applicability challenges of machine learning models and makes them easy for clinical use.
Limitations
The study is subject to several limitations that warrant consideration. Firstly, the PIPC models are static models that assesses the risk of NIs and in-hospital mortality in hospitalized patients with cirrhosis based on data at the time of admission. However, a patient’s health status may improve or deteriorate during their hospital stay, potentially affecting the likelihood of NIs and in-hospital mortality. The current models do not account for these dynamic changes in health status, which limits their predictive capability. Therefore, a future research direction could be to develop a dynamic prediction model capable of integrating dynamic parameters to reflect the real-time risk of patients more accurately during hospitalization. Secondly, the models were developed based on learning from input variables, and the presence of unknown or unregistered variables may impact results. Thirdly, due to the nature of machine learning algorithms, our predictive models do not establish causality. Therefore, interpreting the effects of individual factors in isolation is cautioned, as they are integral components within the intricate interactions of the algorithmic prediction process. Fourthly, the model has not yet been validated on a broader external dataset, which limits our ability to generalize the findings to more diverse patient populations and different clinical settings. Lastly, the retrospective design of this study introduces inherent biases, such as incomplete records and inconsistent data quality. Future research would benefit from a prospective design, allowing for more controlled and systematic data collection.
In conclusion, our study indicates that our machine learning-driven prediction models exhibit superior performance compared to previous models in predicting NIs and prognosis in patients with cirrhosis. The integration into an online calculator enhances accessibility, enabling healthcare providers to easily assess the risk of NIs and prognosis in hospitalized patients with cirrhosis.
Data availability
The data that support the findings of this study are not openly available due to reasons of privacy and are available from the corresponding author upon reasonable request.
Abbreviations
- ACLF:
-
Acute-on-chronic liver failure
- AKI:
-
Acute kidney injury
- ALB:
-
Albumin
- ALT:
-
Alanine aminotransferase
- AST:
-
Aspartate aminotransferase
- AUC:
-
Area under the curve
- Adaboost:
-
Adaptive Boosting
- BMI:
-
Body mass index
- BUN:
-
Blood urea nitrogen
- CA:
-
Community-acquired infection
- CCI:
-
Charlson Comorbidity Index
- CDC:
-
Centers for Disease Control and Prevention
- CI:
-
Confidence Interval
- CRP:
-
C-reactive protein
- DT:
-
Decision Tree
- EMRs:
-
Electronic medical records
- ESCMID:
-
European Society of Clinical Microbiology and Infectious Diseases
- ET:
-
Extremely randomized Tree
- GFR:
-
Glomerular filtration rate
- GGT:
-
Gamma-glutamyl transferase
- HCA:
-
Health care-associated infection
- HDL:
-
High-density lipoprotein
- HE:
-
Hepatic encephalopathy
- HRS:
-
Hepatorenal syndrome
- ICU:
-
Intensive care unit
- INR:
-
International normalized ratio
- LDL:
-
Low-density lipoprotein
- LR:
-
Logistic Regression
- MAP:
-
Mean arterial pressure
- MDR:
-
Multidrug Resistance
- MELD:
-
Model for End-stage Liver Disease
- NI:
-
Nosocomial infection
- NLR:
-
Negative likelihood ratio
- NN:
-
Neural Network
- NPV:
-
Negative predictive value
- PIPC:
-
Prediction of Nosocomial Infection and Prognosis in Cirrhotic patients
- PLR:
-
Positive likelihood ratio
- PLT:
-
Platelet
- PPV:
-
Positive predictive value
- PT:
-
Prothrombin time
- RF:
-
Random Forest
- ROC:
-
Receiver operating characteristic curve
- Scr:
-
Serum creatinine
- SpO2:
-
Oxygen saturation
- TBIL:
-
Total bilirubin
- TC:
-
Total cholesterol
- TG:
-
Triglycerides
- TRIPOD:
-
Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis
- WBC:
-
White blood cell
- XGBoost:
-
eXtreme Gradient Boosting
References
Asrani SK, Devarbhavi H, Eaton J, et al. Burden of liver diseases in the world. J Hepatol. 2019;70(1):151–71.
Bajaj JS, Kamath PS, Reddy KR. The evolving challenge of infections in cirrhosis. N Engl J Med. 2021;384(24):2317–30.
Fernandez J, Acevedo J, Wiest R, et al. Bacterial and fungal infections in acute-on-chronic liver failure: prevalence, characteristics and impact on prognosis. Gut. 2018;67(10):1870–80.
Wong F, Piano S, Singh V, et al. Clinical features and evolution of bacterial infection-related acute-on-chronic liver failure. J Hepatol. 2021;74(2):330–9.
Bonnel AR, Bunchorntavakul C, Reddy KR. Immune Dysfunction and infections in patients with cirrhosis. Clin Gastroenterol Hepatol. 2011;9(9):727–38.
Schultalbers M, Tergast TL, Simon N, et al. Frequency, characteristics and impact of multiple consecutive nosocomial infections in patients with decompensated liver cirrhosis and ascites. United Eur Gastroenterol J. 2020;8(5):567–76.
European Association for the Study of the Liver. Electronic address eee, European Association for the study of the L: EASL Clinical Practice guidelines for the management of patients with decompensated cirrhosis. J Hepatol. 2018;69(2):406–60.
Liao WC, Chung WS, Lo YC, et al. Changing epidemiology and prognosis of nosocomial bloodstream infection: a single-center retrospective study in Taiwan. J Microbiol Immunol Infect. 2022;55(6 Pt 2):1293–300.
Dionigi E, Garcovich M, Borzio M, et al. Bacterial infections change natural history of cirrhosis irrespective of Liver Disease Severity. Am J Gastroenterol. 2017;112(4):588–96.
Bajaj JS, O’Leary JG, Tandon P, et al. Nosocomial infections are frequent and negatively impact outcomes in hospitalized patients with cirrhosis. Am J Gastroenterol. 2019;114(7):1091–100.
Griemsmann M, Tergast TL, Simon N, et al. Nosocomial infections in female compared with male patients with decompensated liver cirrhosis. Sci Rep-Uk. 2022;12(1):3285.
Bajaj JS, Reddy KR, Tandon P, et al. Association of serum metabolites and gut microbiota at hospital admission with nosocomial infection development in patients with cirrhosis. Liver Transpl. 2022;28(12):1831–40.
Vazquez C, Gutierrez-Acevedo MN, Barbero S, et al. Clinical and microbiological characteristics of bacterial infections in patients with cirrhosis. A prospective cohort study from Argentina and Uruguay. Ann Hepatol. 2023;28(4):101097.
Kimmann M, Tergast TL, Schultalbers M, et al. Sustained impact of nosocomial-acquired spontaneous bacterial peritonitis in different stages of decompensated liver cirrhosis. PLoS ONE. 2019;14(8):e0220666.
Fernandez J, Tandon P, Mensa J, et al. Antibiotic prophylaxis in cirrhosis: good and bad. Hepatology (Baltimore MD). 2016;63(6):2019–31.
Bartoletti M, Giannella M, Caraceni P, et al. Epidemiology and outcomes of bloodstream infection in patients with cirrhosis. J Hepatol. 2014;61(1):51–8.
European Association for the Study of the L. EASL clinical practice guidelines on the management of ascites, spontaneous bacterial peritonitis, and hepatorenal syndrome in cirrhosis. J Hepatol. 2010;53(3):397–417.
Cazzaniga M, Dionigi E, Gobbo G, et al. The systemic inflammatory response syndrome in cirrhotic patients: relationship with their in-hospital outcome. J Hepatol. 2009;51(3):475–82.
Piano S, Singh V, Caraceni P, et al. Epidemiology and effects of bacterial infections in patients with cirrhosis Worldwide. Gastroenterology. 2019;156(5):1368–e13801310.
Magiorakos APSA, Carey RB, Carmeli Y, et al. Multidrug-resistant, extensively drug-resistant and pandrug-resistant bacteria: an international expert proposal for interim standard definitions for acquired resistance. Clin Microbiol Infect. 2012;18:268–81.
Bajaj JS, O’Leary JG, Reddy KR, et al. Second infections independently increase mortality in hospitalized patients with cirrhosis: the north American consortium for the study of end-stage liver disease (NACSELD) experience. Hepatology (Baltimore MD). 2012;56(6):2328–35.
Fernandez J, Acevedo J, Castro M, et al. Prevalence and risk factors of infections by multiresistant bacteria in cirrhosis: a prospective study. Hepatology (Baltimore MD). 2012;55(5):1551–61.
Charlson MEPP, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–83.
Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med. 2015;13:1.
Lee J, Westphal M, Vali Y, et al. Machine learning algorithm improves the detection of NASH (NAS-based) and at-risk NASH: a development and validation study. Hepatology (Baltimore MD). 2023;78(1):258–71.
An Z-Y, Wu Y-J, Hou Y, et al. A life-threatening bleeding prediction model for immune thrombocytopenia based on personalized machine learning: a nationwide prospective cohort study. Sci Bull. 2023;68(18):2106–14.
Yuan S, Sun Y, Xiao X et al. Using machine learning algorithms to Predict Candidaemia in ICU patients with New-Onset systemic inflammatory response syndrome. Front Med 2021, 8.
Tacconelli E, Göpel S, Gladstone BP, et al. Development and validation of BLOOMY prediction scores for 14-day and 6-month mortality in hospitalised adults with bloodstream infections: a multicentre, prospective, cohort study. Lancet Infect Dis. 2022;22(5):731–41.
Hernaez R, Karvellas CJ, Liu Y, et al. The novel SALT-M score predicts 1-year post-transplant mortality in patients with severe acute-on-chronic liver failure. J Hepatol. 2023;79(3):717–27.
Hirota Y, Shin JH, Sasaki N, et al. Development and validation of prediction models for the discharge destination of elderly patients with aspiration pneumonia. PLoS ONE. 2023;18(2):e0282272.
Steyerberg EWHFJ, Borsboom GJ, Eijkemans MJ, et al. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol. 2001;54(8):774–81.
Jones GD, Kariuki SM, Ngugi AK, et al. Development and validation of a diagnostic aid for convulsive epilepsy in sub-saharan Africa: a retrospective case-control study. Lancet Digit Health. 2023;5(4):e185–93.
Park H, Lo-Ciganic WH, Huang J, et al. Machine learning algorithms for predicting direct-acting antiviral treatment failure in chronic hepatitis C: an HCV-TARGET analysis. Hepatology (Baltimore MD). 2022;76(2):483–91.
Kanwal F, Taylor TJ, Kramer JR et al. Development, Validation, and evaluation of a simple machine learning model to Predict Cirrhosis Mortality. JAMA Netw Open 2020, 3(11).
Fernandez J, Prado V, Trebicka J, et al. Multidrug-resistant bacterial infections in patients with decompensated cirrhosis and with acute-on-chronic liver failure in Europe. J Hepatol. 2019;70(3):398–411.
Ferrarese A, Passigato N, Cusumano C, et al. Antibiotic prophylaxis in patients with cirrhosis: current evidence for clinical practice. World J Hepatol. 2021;13(8):840–52.
Dirchwolf M, Marciano S, Martinez J, et al. Unresolved issues in the prophylaxis of bacterial infections in patients with cirrhosis. World J Hepatol. 2018;10(12):892–7.
Konig IR, Fuchs O, Hansen G et al. What is precision medicine? Eur Respir J 2017, 50(4).
Su M, Guo J, Chen H, Huang J. Developing a machine learning prediction algorithm for early differentiation of urosepsis from urinary tract infection. Clin Chem Lab Med. 2023;61(3):521–9.
Deschênes MVJ. Risk factors for the development of bacterial infections in hospitalized patients with cirrhosis. Am J Gastroenterol. 1999;94(8):2193–7.
Huttunen RSJ. Obesity and the risk and outcome of infection. Int J Obes (Lond). 2013;37(3):333–40.
Francoz C, Durand F, Kahn JA, et al. Hepatorenal Syndrome. Clin J Am Soc Nephrol. 2019;14(5):774–81.
Colling ME, Tourdot BE, Kanthi Y. Inflammation, infection and venous thromboembolism. Circ Res. 2021;128(12):2017–36.
Akirov A, Masri-Iraqi H, Atamna A, et al. Low albumin levels are Associated with Mortality Risk in Hospitalized patients. Am J Med. 2017;130(12):1465.e1411-1465 e1419.
Acknowledgements
We highly appreciate the financial support from National Natural Science Foundation of China (72374179,71904170), the Fundamental Research Funds for the Central Universities (2022ZFJH003), Zhejiang University K. P. Chao’s High Technology Development Foundation(2022RC017), Mega-Project of National Science and Technology for the 13th Five-Year Plan of China (2018ZX10721102-003-006, 2018ZX10715013-003-003), and Zhejiang Province Healthcare Innovation Talent Program.
Funding
This study was funded by National Natural Science Foundation of China (72374179,71904170), the Fundamental Research Funds for the Central Universities (2022ZFJH003), Zhejiang University K. P. Chao’s High Technology Development Foundation(2022RC017), Mega-Project of National Science and Technology for the 13th Five-Year Plan of China (2018ZX10721102-003-006, 2018ZX10715013-003-003), and Zhejiang Province Healthcare Innovation Talent Program.
Author information
Authors and Affiliations
Contributions
JW designed the study. SL, YZ, YL accessed and verified all the data. SL, LZ and KF analyzed the data and interpreted the results. SL, YZ, YL and JW wrote the manuscript. All authors revised the manuscript from the preliminary draft to submission. JW supervised the whole study. JW is responsible for the decision to submit the manuscript.
Corresponding author
Ethics declarations
Ethics approval
The study protocol has been approved by the Ethics Committee of the First Affiliated Hospital of Zhejiang University.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it.The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Li, S., Zhang, Y., Lin, Y. et al. Development and validation of prediction models for nosocomial infection and prognosis in hospitalized patients with cirrhosis. Antimicrob Resist Infect Control 13, 85 (2024). https://doi.org/10.1186/s13756-024-01444-y
Received:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13756-024-01444-y