Maddalena Giannella,Manuel Huth,Elda Righi,Jan Hasenauer,Lorenzo Marconi,Angelina Konnova,Akshita Gupta,An Hotterbeekx,Matilda Berkell,Zaira R. Palacios-Baena,Maria Cristina Morelli,Mariarosa Tamè,Marco Busutti,Luciano Potena,Elena Salvaterra,,Giuseppe Feltrin,Gino Gerosa,Lucrezia Furian,Patrizia Burra,Salvatore Piano,Umberto Cillo,Mara Cananzi,Monica Loy,Gianluigi Zaza,Francesco Onorati,Amedeo Carraro,Fiorella Gastaldon,Maurizio Nordio,Samir Kumar-Singh,Jesús Rodríguez Baño,Tiziana Lazzarotto,Pierluigi Viale,Evelina Tacconelli
Using machine learning to predict antibody response to SARS-CoV-2 vaccination in solid organ transplant recipients: the multicentre ORCHESTRA cohort
Antibody response,COVID-19,Machine learning,SARS-CoV-2,Solid organ transplantation,Vaccination
Objectives The study aim was to assess predictors of negative antibody response (AbR) in solid organ transplant (SOT) recipients after the first booster of SARS-CoV-2 vaccination. Methods Solid organ transplant recipients receiving SARS-CoV-2 vaccination were prospectively enrolled (March 2021–January 2022) at six hospitals in Italy and Spain. AbR was assessed at first dose (t0), second dose (t1), 3 ± 1 month (t2), and 1 month after third dose (t3). Negative AbR at t3 was defined as an anti-receptor binding domain titre <45 BAU/mL. Machine learning models were developed to predict the individual risk of negative (vs. positive) AbR using age, type of transplant, time between transplant and vaccination, immunosuppressive drugs, type of vaccine, and graft function as covariates, subsequently assessed using a validation cohort. Results Overall, 1615 SOT recipients (1072 [66.3%] males; mean age±standard deviation [SD], 57.85 ± 13.77) were enrolled, and 1211 received three vaccination doses. Negative AbR rate decreased from 93.66% (886/946) to 21.90% (202/923) from t0 to t3. Univariate analysis showed that older patients (mean age, 60.21 ± 11.51 vs. 58.11 ± 13.08), anti-metabolites (57.9% vs. 35.1%), steroids (52.9% vs. 38.5%), recent transplantation (<3 years) (17.8% vs. 2.3%), and kidney, heart, or lung compared with liver transplantation (25%, 31.8%, 30.4% vs. 5.5%) had a higher likelihood of negative AbR. Machine learning (ML) algorithms showing best prediction performance were logistic regression (precision-recall curve-PRAUC mean 0.37 [95%CI 0.36–0.39]) and k-Nearest Neighbours (PRAUC 0.36 [0.35–0.37]). Discussion Almost a quarter of SOT recipients showed negative AbR after first booster dosage. Unfortunately, clinical information cannot efficiently predict negative AbR even with ML algorithms.
Using machine learning to predict antibody response to SARS-CoV-2 vaccination in solid organ transplant recipients: the multicentre ORCHESTRA cohort

The study aim was to assess predictors of negative antibody response (AbR) in solid organ transplant (SOT) recipients after the first booster of SARS-CoV-2 vaccination.

Solid organ transplant recipients receiving SARS-CoV-2 vaccination were prospectively enrolled (March 2021–January 2022) at six hospitals in Italy and Spain. AbR was assessed at first dose (t0), second dose (t1), 3 ± 1 month (t2), and 1 month after third dose (t3). Negative AbR at t3 was defined as an anti-receptor binding domain titre <45 BAU/mL. Machine learning models were developed to predict the individual risk of negative (vs. positive) AbR using age, type of transplant, time between transplant and vaccination, immunosuppressive drugs, type of vaccine, and graft function as covariates, subsequently assessed using a validation cohort.

Overall, 1615 SOT recipients (1072 [66.3%] males; mean age±standard deviation [SD], 57.85 ± 13.77) were enrolled, and 1211 received three vaccination doses. Negative AbR rate decreased from 93.66% (886/946) to 21.90% (202/923) from t0 to t3. Univariate analysis showed that older patients (mean age, 60.21 ± 11.51 vs. 58.11 ± 13.08), anti-metabolites (57.9% vs. 35.1%), steroids (52.9% vs. 38.5%), recent transplantation (<3 years) (17.8% vs. 2.3%), and kidney, heart, or lung compared with liver transplantation (25%, 31.8%, 30.4% vs. 5.5%) had a higher likelihood of negative AbR. Machine learning (ML) algorithms showing best prediction performance were logistic regression (precision-recall curve-PRAUC mean 0.37 [95%CI 0.36–0.39]) and k-Nearest Neighbours (PRAUC 0.36 [0.35–0.37]).

Almost a quarter of SOT recipients showed negative AbR after first booster dosage. Unfortunately, clinical information cannot efficiently predict negative AbR even with ML algorithms.

Solid organ transplant (SOT) recipients are at higher risk for a complicated course of COVID-19 [1,2] and considered a priority setting for vaccination in several countries [3]. When testing was performed for research purposes, the immune response to vaccination in SOT recipients, in particular antibody response (AbR), was lower than that observed in immunocompetent patients [4]. Also among vaccinated individuals, SOT recipients are likely to have a higher risk for hospitalization and death compared with immunocompetent individuals [5,6]. Based on this evidence, booster dosages in SOT recipients have been recommended. However, studies have shown that although an increase in AbR could be observed after the third, fourth, or even fifth dosage, a negative or low-level AbR may still persist in a percentage of patients. ranging from 10% to 30%, and that the impact of further booster doses after the first one is limited [[7], [8], [9], [10]]. To increase protection against COVID-19 in this population, additional strategies have been proposed, such as the modulation of immunosuppressive therapy near the administration of booster doses [11], and/or the pre-exposure treatment with monoclonal antibodies [12]. Although the implementation of these strategies usually has been subordinated to the assessment of AbR, international transplant societies have discouraged the routine use of such practices (https://tts.org/tid-about/tid-officers-and-council?id=749, accessed in August 2022). Hesitance to use anti-spike antibody levels as a marker for either vulnerability to or protection from SARS-CoV-2 infection is due to several reasons, including variability in antibody assays, lack of an antibody threshold associated with protection in immunocompromised patients, potential for protective cellular responses, logistic issues, and costs [13]. However, recent data have shown that there is a relationship between non-high level AbR and increased risk of breakthrough infection (BI) after three mRNA SARS-CoV2 vaccine doses in SOT recipients; as well as that the probability of reaching immunization is inversely related to that of developing BI, mainly for some type of grafts as heart transplant recipients [14]. On this background, we deemed that a tool able to predict a negative AbR after at least a booster dose of mRNA SARS-CoV2 vaccine in SOT recipients could be useful to stratify patients in order to personalize antibody testing in this setting. In this regard, machine learning (ML) methodology has recently been reported as a very useful tool to predict AbR after two doses of SARS-CoV2 vaccine in SOT recipients [15]. Thus, we have used ML models, including traditional logistic regression analysis, to build a predictive binary-response model to identify SOT recipients at higher risk of a negative AbR after the first booster of SARS-CoV-2 vaccination (Fig. S1).

We used the multicentre prospective longitudinal cohort of SOT recipients within the Horizon 2020 ORCHESTRA project-work package 4 (https://orchestra-cohort.eu/), which aims to create a new pan-European cohort to rapidly advance the knowledge on the COVID-19 infection. The study was approved by the Agenzia Italiana del Farmaco (AIFA) and the Ethics Committee of Istituto Nazionale per le Malattie Infettive (INMI) Lazzaro Spallanzani (document n. 359 of Study's Registry 2020/2021) and registered at ClinicalTrials.gov with the number NCT05222139. Informed consent was obtained from all the enrolled patients. The cohort runs at six hospitals (five in Italy — Bologna, Verona, Padova, Vicenza, and Treviso — and one in Seville, Spain). Participants were enrolled from 1 March, 2021 to 31 December, 2021 and followed up until 31 January, 2022. The database was locked on 1 March, 2022 after careful revision for incongruent or missing data. Data sources were clinical charts and hospital electronic records. All data were gathered anonymously and managed using the REDCap electronic data capture tools hosted at the Interuniversity Consortium CINECA (https://redcap-dev.orchestra.cineca.it/) [16]. SOT recipients undergoing SARS-CoV-2 vaccination during the enrolment period and who accepted to participate into the ORCHESTRA project were prospectively enrolled. As previously described [17], patients were assessed for AbR to SARS-CoV-2 vaccination at pre-defined timepoints: first dose (t0), second dose (t1), 3 ± 1 month after the first dose (t2), and at 1 month after the third dose (t3). All patients had a minimum follow-up of one month after the third dosage.

The primary endpoint was AbR at t3. The response was stratified into non-reactive (<5.58 BAU/mL), inconclusive (5.58–<45 BAU/mL), positive-low (45–<205 BAU/mL), positive-mild (205–<817 BAU/mL), and positive-high (>817 BAU/mL) according to WHO International SARS-CoV-2 Antibody Standards criteria. For the purpose of the study, a negative AbR was defined as an anti-receptor binding domain (RBD) titre <45 BAU/mL (including non-reactive and inconclusive results). Exposure variables collected at t0 included age, sex, comorbidities other than the cause of transplant according to the Charlson index, and type and date of transplant. Data on immunosuppressive regimen, receipt of induction regimen in the past 6 months, and graft function defined as good, impaired, or failure according to the judgement of attending physicians were collected at each timepoint.

The Elecsys® Anti-SARS-CoV-2 ECLIA assay and V-PLEX SARS-CoV-2 Panel 6 Kit (IgG) from Meso Scale Discovery (MSD, MD, USA) were used to detect AbR according to the manufacturer instructions and as previously described [17].

The distribution of age is reported by mean and standard deviation (SD). Due to the censored structure caused by the detection thresholds of the serology tests, we refrain from reporting AbR titres by means (SD) or median and interquartile ranges. Patients with a previous history of documented SARS-CoV2 infection or with positive anti-N antibodies before or between doses were excluded from the analysis. The dataset used in the statistical analysis consisted of 14 binary covariates encoded with 0 and 1, and age as the only non-binary variable, reported as integers. These variables were chosen based on univariate statistical analysis (p < 0.1) (age, type of transplant, time between transplant and vaccination, and immunosuppressive drugs), and on clinical relevance (type of vaccine and graft function) (see Table S1). An ordinal logistic regression analysis was performed to identify risk factors, utilizing the five ordinal antibody levels (non-reactive, inconclusive, positive-low, positive-mild, and positive-high) as the outcome variable. The objective of this model was to determine the magnitude and direction of the covariates' effect. The analysis was executed using R version 4.1.3, and the MASS package was used to train the ordinal logistic regression. The ML model training and validation methods are described in Supplementary Text and Tables S1–S3.

The study cohort consisted of 1615 SOT recipients of kidney (n = 886), liver (n = 350), heart (n = 340), and lung (n = 56) transplants, with 17 patients having multiple organ transplantation (liver-kidney, n = 10; liver-heart, n = 2; kidney-heart, n = 2; kidney-lung, n = 2; and lung-heart, n = 1). The type and number of SOT recipients enrolled by each centre are detailed in Table S4. The majority of the study population consisted of males (n = 1072), and the mean (SD) age was 57.85 (13.77) y. Time from transplant to vaccination onset was less than 1 year, between 1 and 3 years, and more than 3 years in 6, 47, and 870 individuals, respectively. Graft failure occurred in 35 patients. During the study period, 1211 patients received three doses of SARS-CoV2 vaccine. This was mRNA based in all but 11 patients, who received a viral vector vaccine either as the first, second, or third dose (see Table S5). In 318 out of 1211 patients (26.2%), a change in the types of vaccine received between the initial vaccination schedule (first two dosages) and the booster dosage was reported. In the majority of cases (n = 301), it consisted of shifting from BNT162b2 (Pfizer) to mRNA-1273 (Moderna). The mean (SD) time between the second and the third dose was 190.35 (34.26) days.

Overall, 946 participants were assessed for AbR at first dose, 975 at second dose, 1363 at 3 ± 1 month after the first dosage, and 923 at one month after the third dosage. The rate of patients with anti-RBD levels ≥45 BAU/mL progressively increased from 6.34% (60/946), 14.05% (137/975), 50.92% (694/1363) to 78.11% (721/923) at each timepoint. The rate of individuals with a high AbR (>817 BAU/mL) increased from 1.80% (17/946), 5.95% (58/975), 20.00% (273/1363), to 63.20% (583/923), whereas the number of negative responses (<45 BAU/mL) decreased from 93.80% (886/946), 85.90% (889/975), 49.20% (669/1363), to 21.90% (202/923). For patients with multiple consecutive assessments, transition of the AbR from t0 to t3 is shown in Fig. 1 .

Univariate analysis showed that kidney, heart, or lung transplant recipients had a higher likelihood of a negative AbR compared with liver transplant recipients. Furthermore, older patients, those taking anti-metabolites and/or steroids, and patients with recent transplant (<3 years) appeared to have an increased probability of a negative AbR (Table 1 ). The ordinal logistic regression (see Table S6) showed a significant negative influence of age (log odds ratio, -0.03) and anti-metabolites (-1.10) on the AbR. In addition, the analysis showed that patients with heart (-1.72), kidney (-1.59), or lung (-2.25) transplants were more likely to have a lower AbR than patients with a liver transplant. The type of vaccine, the time from transplant to the vaccination, and a graft failure did not seem to influence AbR after the booster dose. Parameter estimates and 95% confidence intervals are reported in Fig. 2 . The predictive power of the ML models in the validation cohort was assessed using the balanced accuracy (BA) (Fig. S2) and the area under the precision-recall curve (PRAUC) (Fig. S3) as decision criteria. Further evaluation measures, such as accuracy, sensitivity, and specificity, are depicted in Figs. S4–S7. The results showed that relative performance of different models is almost independent of the AbR threshold. At an AbR level of 45, the top three performers were logistic regression (LR) (BA, 0.66 [0.65, 0.67]; PRAUC, 0.37 [0.36, 0.39]), k-Nearest Neighbours (KNN) (BA, 0.65 [0.64, 0.66]; PRAUC, 0.36 [0.35, 0.37]), and ordinal logistic regression (OLR) (BA, 0.63 [0.62, 0.64]; PRAUC, 0.34 [0.33, 0.35]). However, when examining the average specificities (Fig. S4) and sensitivities (Fig. S5) at this threshold, it was found that LR and the OLR performed well in predicting SOT recipients with negative AbR (OLR, 0.71 [0.68, 0.73]; LR, 0.70 [0.68, 0.72]), but performed worse in predicting those with positive AbR (OLR, 0.56 [0.54, 0.58]; LR, 0.62 [0.61, 0.64]). Tree-based methods, such as the Bagged tree and the Gradient Boosting Machine (GBM), performed well in predicting patients with a positive AbR (BT, 0.83 [0.82, 0.84]; GBM, 0.78 [0.77, 0.8]), but poorly in predicting those with negative AbR (BT, 0.30 [0.28, 0.32]%; GBM, 0.35 [0.33, 0.37]). The tree-based methods, such as the GBM, appear to have overfitted the training data, as demonstrated by an average BA of 0.94 [0.94, 0.95] and an average PRAUC of 0.94 [0.93, 0.95] (Figs. S8 and S9). On the other hand, other methods, such as LR, did not exhibit such overfitting in the training set (BA, 0.70 [0.70, 0.71]; PRAUC, 0.44 [0.44, 0.45]). For the area under the receiver operating characteristic curve (AUROC) — a metric which we report for comparison with prior research — we found that the LR model achieved an average AUROC of 0.72 [0.71–0.73], the k-nearest neighbour algorithm had an average AUROC of 0.73 [0.72–0.74], and the OLR model had an average AUROC of 0.68 [0.67–0.69] (Fig. S6).

Our data confirm the persistence of lack of response in almost one fourth of the patients after booster dose. Using this data, we aimed to develop a prediction models based on easy-to-obtain clinical covariates, such as age, type of transplant, time from transplant to first dosage, types of immunosuppressive drugs, type of mRNA vaccine received, and graft failure. Unfortunately, the best ML model we found only reached a moderate prediction accuracy. This suggests that the clinical covariates provide only limited information. Our results are consistent with those obtained by Alejo et al. [15], who developed and validated a ML model to predict AbR to two doses of SARS-CoV-2 mRNA vaccines using a nationwide cohort of 1031 SOT recipients, and an external single-centre cohort of 512 SOT recipients in the United States. The authors used 19 clinical factors very similar to those used in our models. Indeed, Alejo et al. found that mycophenolate mofetil use, a shorter time since transplant, and older age were the strongest predictors of a negative AbR. The performance of the model was good in the training set (AUROC, 0.79) and moderate in the external test set (AUROC, 0.67). The main difference between the U.S. cohort and our cohort is the definition of negative AbR (which is < 0.8 U/mL if assessed by Roche and ≤1.1 AU if assessed by EUROIMMUN) used in the U.S. cohort [15]. Alejo et al. used a GBM to predict antibody responses. We found that LR analysis was most accurate in predicting a negative AbR, while tree-based ML models performed worse. A possible explanation for this is that the tree-based methods might have overfitted the training data, as indicated by the Figs. S8–S13, despite being optimized through cross-validation of the hyperparameters (Table S1). This overfitting results in poor generalization performance when applied to the unseen validation cohort in contrast to the not overfitting other models. Our study has limitations. First, due to the censored structure caused by the detection thresholds of the serology tests, we refrained from reporting and assessing quantitative antibody levels. Second, regarding the type of SARS-CoV2 vaccines (BNT162b2, mRNA-1273, and ChAdOx1), the predictive power of ChAdOx1 could not be assessed due to limited number of subjects exposed. Third, we did not analyse cellular immune response that is an essential component in the clinical protection of SOT recipients from clinically relevant SARS-CoV2 infections. Finally, we developed the model with AbR assessed one month after the first booster dosage, while currently most fragile patients should have received several booster dosages. However, it has been shown that the impact of further booster dosages on AbR may be limited, with lower than 50% of seronegative patients achieving a positive AbR or showing a significant increase in antibody levels [18,19]. Thus, we deem that our model could be valid also in patients exposed to more than one booster dosage. Although booster dosage in SOT recipients is associated with a progressive increase in AbR, one fourth of this population remains negative or with suboptimal antibody levels. Unfortunately, clinical characteristics are of limited values in developing high performing predictive models of negative AbR.

Natascia Caroccia, Beatrice Tazza, Cecilia Bonazzetti, Francesca Fanì, Michela Di Chiara, Maria Eugenia Giacomini, Oana Vatamanu, Zeno Pasquini, Renato Pascale, Matteo Rinaldi, Clara Solera Horna, Caterina Campoli, Giacomo Fornaro, Fabio Trapani, Luciano Attard, Antonio Gramegna, Valeria Cesari, Simona Varani, Elena Rosselli Del Turco, Sara Tedeschi, Kristian Scolz, Gaetano La Manna, Valeria Grandinetti, Marcello Demetri, Simona Barbuto, Chiara Abenavoli, Giovanni Vitale, Laura Turco, Matteo Ravaioli, Matteo Cescon, Valentina Bertuzzo, Angela Lombardi, Alessandra Trombi, Marco Masetti, Paola Prestinenzi, Mario Sabatino, Laura Giovannini, Aloisio Alessio, Antonio Russo, Maria Francesca Scuppa, Laura Borgese, Giampiero Dolci, Gianmaria Paganelli, Giorgia Comai, Liliana Gabrielli, Chiara Gamberini, and Marta Leone. IRCCS Policlinico Sant’Orsola, University of Bologna, Bologna, Italy. Simona Granata, Alberto Verlato, and Rossella Elia. Renal Unit, Department of Medicine, University Hospital of Verona, Verona, Italy. Alex Borin. Liver Transplant Unit, Department of Surgery and Dentistry, University and Hospital Trust of Verona, Verona, Italy. Livio San Biagio, Alessandra Francica, and Ilaria Tropea. Division of Cardiac Surgery, University of Verona, Verona, Italy. Maria Mongardi, Mariana Nunes Pinho Guedes, Gaia Maccarone, Concetta Sciammarella, Massimo Mirandola, Lorenzo Maria Canziani, Chiara Konishi, Chiara Perlini, and Giulia Rosini. Division of Infectious Diseases, Department of Diagnostics and Public Health, University of Verona, Verona, Italy. Francesca Russo and Michele Mongillo. Direzione Prevenzione sicurezza alimentare veterinaria, Regione Veneto, Italy. Maria Giulia Caponcello, Natalia Maldonado, Paula Olivares, David Gutiérrez-Campos, Ana Belén Martín-Gutiérrez, Ana Silva, Virginia Palomo, Almudena Serna, and Marta Fernández-Regaña. Infectious Diseases and Microbiology Unit, Hospital Universitario Virgen Macarena, and Department of Medicine, University of Sevilla/Biomedicines Institute of Sevilla, Sevilla, Spain. Ana Belén Hidalgo, Ioana Hrom, Myriam Adorna, Rubén Murillo, and Ma Isabel García-Sánchez. Biobank Nodo Hospital Virgen Macarena (Biobanco del Sistema Sanitario Público de Andalucía) integrated in the Spanish National biobanks Network (PT20/00069). Paolo Angeli, Alessandra Brocca, Nicola Zeni, Roberta Gagliardi, and Simone Incicco. Unit of Internal Medicine and Hepatology (UIMH), Department of Medicine - DIMED, University of Padua, Padua, Italy. Andrea Carraro, Josè Igeno San Miguel, Emanuele Vianello, and Susanna Negrisolo. Paediatric Nephrology, Dialysis and Transplant Unit, Department of Women's and Children's Health, Padua University Hospital, Padua, Italy. Enrico Gringeri, Patrizia Boccagni, Francesco Enrico D'amico, Riccardo Boetto, and Lara Borsetto. Department of Surgery, Oncology and Gastroenterology, Hepatobiliary Surgery and Liver Transplantation Unit, Padua University Hospital, Padua, Italy. Erica Nuzzolese, Marianna Di Bello, and Caterina Di Bella. Kidney and Pancreas Transplantation Unit, Department of Surgical, Oncological and Gastroenterological Sciences, University of Padua, Padua, Italy. Paola Gaio. Unit of Paediatric Gastroenterology, Digestive Endoscopy, Hepatology and Care of the Child with Liver Transplantation, Department of Women's and Children's Health, University Hospital of Padua, Padua, Italy. Luigi Garufi and Chiara Tessari, SaimaImran. Cardiac Surgery Unit, Department of Cardiac, Thoracic, Vascular Sciences and Public Health, University of Padua, Padua, Italy. Debora Bizzarro and Francesco Paolo Russo. Unit of Gastroenterology and Multivisceral Transplant, Department of Surgery, Oncology and Gastroenterology, University Hospital of Padua, Padua, Italy. Lorena Brunello, Marta Tenan, and Monica Rizzolo. Nephrology Unit, Treviso Hospital, Treviso, Italy. Carlotta Caprara, Grazia Maria Virzì, and Matteo Marcello. Department of Nephrology, Dialysis and Transplantation, San Bortolo Hospital, Vicenza, Italy.

Salvatore Piano reports consulting fees from Plasma Protein Therapeutics Association (PPTA) and Resolution Therapeutics and participation in Advisory board for Mallinckrodt Inc. Pierluigi Viale reports consulting fees from bioMérieux, Mundipharma, AstraZeneca, Tillots Pharma, Gilead, Shionogi, Sobi, Advanzpharma, MSD, Angelini and Pfizer. All other authors report no potential conflicts of interest.

This work was supported by the European Union's Horizon 2020 Research and Innovation Program, grant agreement 101016167, within the ORCHESTRA project (EU H2020 ORCHESTRA).

The authors confirm contribution to the paper as follows: study conception and design: Giannella M, Tacconelli E, Huth M, Hasenauer J; clinical data collection: Marconi L, Palacios-Baena ZR, Morelli MC, Tamè M, Busutti M, Potena L, Salvaterra E, Feltrin G, Gerosa G, Furian L, Burra P, Piano S, Cillo U, Cananzi M, Loy M, Zaza G, Onorati F, Carraro A, Righi E, Gastaldon F, Nordio M; immunological analysis and interpretation of results: Konnova A, Gupta A, Hotterbeekx A, Berkell M, Lazzarotto T, Kumar-Singh S; statistical analysis and interpretation of results: Huth M, Hasenauer J, Giannella M, Tacconelli E; draft manuscript preparation: Giannella M, Tacconelli E, Huth M, Hasenauer J; draft manuscript revision: Righi E, Rodríguez Baño J, Viale P. All authors reviewed the results and approved the final version of the manuscript.