Pilot Study for the Psychometric Validation of the Sheffield Profile for Assessment and Referral to Care (SPARC) in Korean Cancer Patients
2021

Jung Hye Kwon
Department of Internal Medicine
Chungnam National University College of Medicine
Daejeon

Sun Kyung Baek
Department of Internal Medicine
Kyung Hee University School of Medicine
Seoul

Do Yeun Kim
Department of Internal Medicine
Dongguk University Ilsan Hospital
Goyang

Yu Jung Kim
Department of Internal Medicine
Seoul National University Bundang Hospital
Seoul National University College of Medicine
Seongnam

Myung Ah Lee
Division of Medical Oncology
Department of Internal Medicine
College of Medicine
Seoul St. Mary's Hospital
The Catholic University of Korea
Seoul

Hye Jin Choi
Department of Internal Medicine
Yonsei University College of Medicine
Seoul

Ja Min Byun
Department of Internal Medicine
Seoul National University Hospital
Seoul National University College of Medicine
Seoul

Jin Young Jeong
Hallym Research Institute of Clinical Epidemiology
Hallym University
ChuncheonKorea

Sam H Ahmedzai
Clinical Research Network -Cancer
National Institute for Health Research
University of Leeds
LeedsUK

Geun-Doo Jang
Division of Hemato-Oncology
Department of Internal Medicine
Department of Internal Medicine
Kangdong Sacred Heart Hospital
SeoulKorea

Chungnam National University College of Medicine
266 Munhwa-ro, Jung-gu35015DaejeonKorea

Jung Hye Kwon
Jung Hye Kwon
Sun Kyung Baek
Kwon Jh
Baek Sk
Kim Dy
Kim Yj
Lee Ma
Choi Hj
Byun Jm
Jeong Jy
Ahmedazai Sh
Jang Gd
Kwon Jh
Baek Sk
Kim Dy
Kim Yj
Lee Ma
Choi Hj
Byun Jm
Jeong Jy
Ahmedazai Sh
Jang Gd
Pilot Study for the Psychometric Validation of the Sheffield Profile for Assessment and Referral to Care (SPARC) in Korean Cancer Patients

Cancer Res Treat
531202110.4143/crt.2020.235Received March 23, 2020 Accepted August 28, 2020 Published Online August 31, 202025 │ https://www.e-crt.org │ Original Article Cancer Res Treat. 2021;53(1):25-31 Correspondence: contributed equally to this work. Supplementary materials are available at Cancer Research and Treatment website (https://www.e-crt.org). Ethical Statement This pilot study was reviewed and approved by the institutional review boards at Kangdong Sacred Heart Hospital (2018-08-014). All patients provided written informed consent; the informed con-sent form was approved by the institutional review board. Author Contributions Conceived and designed the analysis: Collected the data: Kwon JH, Ahmedazai SH. Contributed data or analysis tools: Kwon JH, Jeong JY. Performed the analysis: Kwon JH, Jeong JY. Wrote the paper: Translation: Baek SK, Kwon JH, Kim YJ, Lee MA. Back translation: Byun JM. Conflicts of Interest Conflicts of interest relevant to this article was not reported.Sheffield Profile for Assessment and Referral to CareCancerHolisticPalliative careKorean
Purpose This study aimed to validate the Sheffield Profile for Assessment and Referral to Care (SPARC) as an effective tool for screening palliative care needs among Korean cancer patients.Materials and MethodsThe English version of the SPARC was translated by four Korean oncologists and reconciled by a Korean language specialist and a medical oncologist fluent in English. After the first version of the Korean SPARC (K-SPARC) was developed, back-translation into English was performed by a professional translator and bilingual oncologist. The back-translated version was reviewed by the original author (S.H.A.), and modifications were made (ver. 2). The second version of the K-SPARC was tested against other questionnaires, including the Functional Assessment of Cancer Therapy-General (FACT-G) and the Edmonton Symptom Assessment System (ESAS).Results Thirty patients were enrolled in the pilot trial. Fifteen were male, and the median age was 64.5 years. Six patients had an Eastern Cooperative Oncology Group performance status of 2 or more. All patients except one were receiving chemotherapy. Regarding internal consistency, the Cronbach's α scores for physical symptoms, psychological issues, religious and spiritual issues, independency and activity, family and social issues, and treatment issues were 0. 812, 0.804, 0.589, 0.843, 0.754, and 0.822, respectively.   The correlation coefficients between the SPARC and FACT-G were 0.479 (p=0.007) for the physical domain and -0.130 (p=0.493) for the social domain.Conclusion This pilot study indicates that the K-SPARC could be a reliable tool to screen for palliative care needs among Korean cancer patients. A further study to validate our findings is ongoing.

Introduction

Palliative care is an approach that improves the quality of life of patients and their families who are facing problems associated with life-threatening illness. Palliative care prevents and relieves suffering through the early identification, correct assessment, and treatment of pain and other problems, whether physical, psychosocial, or spiritual [1]. Moreover, early palliative care, when combined with usual oncology care, led to the improvement of mood and longer survival of non-small-cell lung carcinoma patients [2]. Based on many randomized clinical trials, several guidelines recommend the initiation of palliative care in advanced cancer patients at the early stage of their disease trajectories and the integration of concurrent palliative care into disease-directed, life-prolonging therapies [3][4][5].

To implement early referral to palliative care, there is a need to assess symptoms and identify the unmet needs of advanced cancer patients [6]. In addition, palliative care teams also need comprehensive assessments of patients' needs to provide holistic care. The Sheffield Profile for Assessment and Referral for Care (SPARC) was developed over a 5-year period by the Academic Unit of Supportive Care at the University of Sheffield. The SPARC is a holistic multidimensional screening questionnaire regarding supportive and palliative care needs regardless of the diagnosis. The SPARC is a selfrated questionnaire with a total of 45 questions under the following eight subscales: communication and information issues, physical symptoms, psychological issues, religious and spiritual issues, independence and activity, family and social issues, treatment issues, and personal issues. The SPARC helps healthcare professionals to identify patients and families that would benefit from a referral to specialist supportive and palliative care [7]; it was translated into Polish and then validated by Leppert et al. [8]. The aim of this study was to develop a Korean version of the SPARC (K-SPARC) and validate the K-SPARC for the evaluation of palliative care needs among Korean cancer patients.

Materials and Methods

In this study, we translated and back-translated the SPARC and then conducted a pilot study of the K-SPARC by conducting semistructured debriefing interviews. To evaluate the reliability of the K-SPARC, we measured the internal consistency by calculating the Cronbach's alpha coefficients of six of the subscales. We evaluated the criterion validity through correlation analysis of the K-SPARC with other tools such as the physical well-being and social well-being domains of the Functional Assessment of Cancer Therapy-General (FACT-G).

Translation and back-translation process

Ahmed et al. [7] provided approval for the translation of the SPARC into Korean. The translation and back-translation processes are detailed in Fig. 1. Four Korean oncologists fluent in English (S.K.B., J.H.K., Y.J.K., M.A.L.) independently translated the 45 items of SPARC, including the response options and questionnaire instructions. The expert panel which was consisted of five oncologists including four primary translators reviewed each translation and reconciled the translations to create the first version of the K-SPARC. The first version of the K-SPARC was reviewed by a Korean language specialist (J.C.S.), and the items were modified to reflect the layperson's perspective in the second version of the K-SPARC. The expert panel reviewed the second version of the K-SPARC, and a professional translator and a bilingual oncologist back-translated it into English. This translation was reviewed by S.H.A. and was found to be consistent with the original English: a few minor discrepancies were discussed and allowed for linguistic or cultural differences and for variations in healthcare systems.

Pilot study

The pilot study was performed with the translated version of the SPARC (Supplementary Material), and two rounds of semistructured debriefing interviews were conducted with 15 patients in each round. Patients were asked if the instructions were clear and easy to understand, if there were any word that were difficult to understand, if the order of items was appropriate, if the questionnaire was easy to complete, if they needed help completing the survey and if there were additional suggestions. The debriefing interviews were performed by a palliative care nurse who had been trained for interviewing.

Patients

Patients who were 20 years old or older, who had advanced cancer and who understood written Korea were included. Patients with symptomatic brain metastasis, a psychologic condition that could not be controlled or that affected cognition, or cognitive problems such as dementia or delirium were excluded. The following patient demographics were collected from the patient's medical record upon enrollment: age, sex, Eastern Cooperative Oncology Group (ECOG) performance status, diagnosis, stage of cancer, current cancer treatment, duration of disease, admission status, and education level. Patients completed the initial questionnaires including the SPARC, Edmonton Symptom Assessment System (ESAS), and FACT-G, on the same day.

Instruments used in the study

The instruments used in this study were the Korean version of the ESAS (K-ESAS), the Korean version of the FACT-G (FACT-G Korean), and the ECOG performance status; these instruments were used as comparative tools, as well as to test the feasibility of survey set.

Cancer Res Treat. 2021;53(1):25-31 Reconciliation by expert panel

1) Sheffield Profile for Assessment and Referral for Care (SPARC)

The SPARC is a self-rated questionnaire to screen for supportive and palliative care needs and contains a total of 45 questions under in the following eight subscales: communication and information issues (1 question), physical symptoms (21 questions), psychological issues (9 questions), religious and spiritual issues (2 questions), independence and activity (3 questions), family and social issues (4 questions), treatment issues (2 questions), and personal issues (3 questions). Among the 45 items, four are yes/no questions, and the other items are scored on a four-point Linkert scale with the following options: not at all, a little bit, quite a bit, and very much. For the Polish version of the SPARC, the Cronbach's alpha coefficients for physical symptoms, psychological issues, religious and spiritual issues, independence and activity, and family and social issues were 0.68, 0.86, 0.65, 0.77, 0.80, and 0.62, respectively [8].

2) Other instruments to evaluate the quality of life of cancer patients

The instruments used in this study were the K-ESAS, the FACT-G Korean, and the ECOG performance status; these instruments were used as comparative tools, as well as to test the feasibility of the survey set. The ESAS uses numerical scales ranging from zero to 10 to assess the average intensity (10 indicates highest intensity) of symptoms experienced during the previous 24 hours, namely, pain, fatigue, nausea, depression, anxiety, drowsiness, loss of appetite, decreased well-being, dyspnea, and sleep disturbance [9,10]; the K-ES-AS has been validated [11].

The FACT-G scale, which was developed and validated by Cella et al. [12] in the United States, consists of a total of 27 Likert-type items formulated into separate subscales, i.e., physical (seven items), emotional (six items), social/family (seven items) and functional (seven items) well-being; it is widely used to measure health-related quality of life in cancer patients [12]. The FACT-G has also been translated into Korean and validated [13].

Physical activity was evaluated by the ECOG performance status. The performance status is indicated on a scale from 0-5 according to a chart of activities of daily life and is an indicator that is widely used by researchers and clinicians.

Statistical analysis

Descriptive statistics were used to summarize the data. The reliability of the K-SPARC was measured in this study based on the internal consistency of the SPARC using Cronbach's alpha coefficients, and it will be assessed in a future study based on the test-retest reliability. The internal test-retest reliability of the Cronbach's alpha coefficients for 41 items under six factors will be evaluated with the correlation coefficients between the measured values at the beginning of the first seven days and 3 days later. Cronbach's alpha is the most commonly used measure of internal consistency. Internal consistency is generally considered good if the coefficient is 0.60 or higher [14]. Validity was evaluated in terms of the construct validity and the criterion validity. The construct validity was evaluated for six areas (physical symptoms, psychological issues, religious and spiritual issues, independence and activity, family and social issues, and treatment issues) from the English version. We analyzed the factors of the K-SPARC by conducting a factor analysis of the translated Korean version. A correlation analysis with the FACT-G, as the gold standard, was also conducted. Of the four factors of the FACT-G (physical well-being, social/family well-being, emotional well-being, and functional well-being), correlation analysis of the physical well-being and social well-being dimensions with the SPARC was conducted. The statistical analysis was performed with the SPSS ver. 23.0 statistical software package (IBM Corp., Armonk, NY). A p-value less than 0.05 was considered significant in the statistical analysis.

Results

Modification of items based on cultural differences

In the translation and back-translation process, the modification of the items was not permitted except for changing "b. community nurse" and "c. hospital nurse" to "nurse" in the communication and information issues section because of the differences in the healthcare system between the UK and Korea. Several items also needed to be modified due to the constraints of the Korean language. It was difficult to find an appropriate translation for "sore mouth" in Korean. We decided to modify this phrase as "oral pain" (Gugang Tongjeung) in the physical symptoms section. The phrase "feeling as if you are in a low mood" was used in the original version to avoid making respondents feel negative or embarrassed; however, in Korean feeling depressed (WooUlham) is different from depression (WooUljeung) itself, so we decided to use "WooUlham". No patients expressed embarrassment or annoyance with the word "WooUlham". The phrase "feeling that life is not worth living" was translated into "feeling that life is meaningless" because the expert panel decided that "not worth living" (Sal Ga Chiga Eopseum) seemed more aggressive than "meaningless" (Saneun Geosi Uimi Eopneun Geotgateum) and more related to suicide in Korean. The phrase "thoughts about ending it all" is a soft expression for suicide in the original version, and the expert panel had difficulty determining a soft, metaphorical expression in Korean. We ultimately decided on "Modeun Geoseul Kkeutnaeya getdaneun Saenggak". Finally, we discussed the term "distress" in Korean at great length. There is no exact word for distress in Korean, and a previous validation of the Korean version of the Distress Thermometer used the English term itself [15]. We also decided to use the English term "distress" itself and included a description of the term as referring to total pain (Jeonbanjeokin Gotong) in parentheses next to it.

Two rounds of a pilot study of the SPARC survey

A total of 30 patients were included in the pilot trial between 10 December 2018 and 21 February 2019. In the first round of the study, 15 patients were enrolled. Only one patient expressed difficulties with several words, including urinary problems (Banggwang Munje), sex life (SeongSaenghwal), sore mouth (Gugang Tongjeung), and feeling week (Soeyakgam), and the other patient expressed difficulty with the word "distress." Others had no difficulties to understand each item. We conducted a second round with an additional 15 patients. The pilot study was complete after the second round because patients no longer expressed difficulties in completing the survey. Of all the patient, 19 indicated that the difficulty level of the questions was easy and clear, and 10 patients indicated a moderate difficulty level. Twenty-seven patients agreed with the order of the questionnaire. The expert panel reviewed the questionnaire, and the response of one patient was concluded to be an outlier. Two patients expressed additional problems, including with information on patient advocacy, cancer rehabilitation, and new cancer drugs. The median time to respond to the survey, including the SPARC, FACT-G, and K-ESAS, was 20 minutes (Q1-Q3, 20-35 minutes), and the median time to complete only the SPARC was 5 minutes (Q1-Q3, 10-15 minutes; minimummaximum, 5-25 minutes). The median difficulty score of the SPARC was 2 (on a Likert scale from 1, very easy, to 5, very difficult; Q1-Q3, 2-3).

Of the 30 patients in the study, 50% were men, and the mean age was 61.6 years (range, 52 to 69.3 years). Gastrointestinal tumors were the most common primary cancers, followed by head and neck cancers, breast cancer, and lymphoma/myeloma. Of all the patients, 80% had stage IV cancer, and 97% were receiving chemotherapy. The patients' demographics are detailed in Table 1 Table 3. Correlation between the physical domain of the SPARC and FACT-G SPARC was assessed by Cronbach's alpha coefficients with the following results: 0.812 for physical symptoms, 0.804 for psychological issues, 0.589 for religious and spiritual issues, 0.843 for independence and activity, 0.754 for family and social issues, and 0.822 for treatment issues. Table 2 shows the comparison of the Cronbach's α scores of the Korean and Polish versions of the SPARC. Table 3 shows the correlation between the physical and social domains of the SPARC and FACT-G. The correlation coefficient was significant at 0.479 (p=0.007); however, the functional well-being domain of the FACT-G and the independence and activity domain of the SPARC (r=0.307, p=0.099) were not significantly correlated, and neither were the psychological domain of SPARC and the emotional well-being domain (r=0.175, p=0.356).

Discussion

The SPARC is a holistic multidimensional screening instrument for supportive and palliative care needs. It has also been used to capture the holistic needs of newly diagnosed lung cancer patients [16]. In this study, we developed a Korean translation of the SPARC and conducted a linguistic validation for cancer patients. The Korean version was first translated using the forward-back procedure. After the first version of the K-SPARC was created, it was back-translated into English and reviewed; minor variations from the original English version were discussed and agreed for linguistic and cultural reasons and because of difference in healthcare systems. Subsequently, the validity and reliability of the K-SPARC (Supplementary Material) were assessed, and the tool was used with a sample of 30 patients with advanced cancer. During the pilot study, participants expressed an accurate understanding of the concepts.

In this pilot study, the internal consistency of the total scale and the subscales was assessed, with Cronbach's alpha values for the subscales ranging from 0.589-0.843. The Cronbach's alpha values of 5 of the subscales, including physical symptoms, psychological issues, independence and activity, treatment issues, and family and social issues, were over 0.75, and good or acceptable internal consistency was found when comparing the Korean and Polish versions. This finding indicates that the K-SPARC has high structural validity. However, although it was not unacceptable, the Cronbach's alpha value of the religious and spiritual issues subscale was poor at 0.589. The religious and spiritual issues subscale comprises two questions with conflicting thoughts about death and dying and religious or spiritual needs not being met. In Korea, there are many obstacles and taboos regarding discussions of death, as in other Asian nations. There are differences among patients, family members, and physicians not only in terms of the disclosure of an incurable cancer diagnosis but also in terms of end-of-life discussion when death is imminent [17,18]. For example, a study used semistructured interviews to compare meaning-making in coping among cancer patients in Sweden and South Korea [19]. The study found that Korean patients hardly mentioned "spirituality" in the interviews, whereas cancer patients in Sweden were familiar with the construct of spirituality and used the term "spirituality" in their daily communication. This finding was attributed to Koreans' difficulty in expressing issues related to spirituality in the Korean language rather than to Koreans' intrinsic lack of spiritual consciousness. There is no consensus about the definition of spirituality among the general public in Korea, and spirituality can be defined in one's own way. As a result, it is not unexpected that the Cronbach's score would be poor for religious and spiritual issues, which included two questions about death and spirituality that Koreans are not used to discussing due to cultural obstacles to talking about death and unclear definitions of spirituality.

The correlations between the K-SPARC and FACT-G varied. The K-SPARC was moderately correlated with the physical well-being domain of the FACT-G but weakly correlated with the social domain of the FACT. As mentioned above, this finding might be a result of cultural differences among nations. In addition, there were some differences between the FACT-G and K-SPARC in how social questions were asked. Questions from the social domain of the FACT-G focus on intimacy with and the care of family members or friends due to disease. However, the K-SPARC tends to focus on the impact of disease on family members and patient concerns. The low correlation between the K-SPARC and FACT-Social might be related to cultural differences between nations or the emphasis of questions, but this was difficult to conclude due to the small number of patients. An ongoing study is expected to clarify this issue.

There are several limitations to our study. First, our sample was small (30 patients). Second, we did not evaluate testretest reliability. Finally, this study was a pilot study providing preliminary evidence of the validity of the K-SPARC. However, this study represents a fundamental step in the development of a Korean version of the SPARC, and future research to support further reliability and validity evidence is ongoing.

The present study developed K-SPARC and validated linguistically. However, there are possible differences in culture or language use, and some problematic translations might be present. We are conducting another study to validate the psychometric properties of the K-SPARC, including the testretest reliability, convergent/divergent validity, and prognostic value for Korean cancer patients.

Fig. 1 .
1The translation and back-translation processes.

FACT
-G, Functional Assessment of Cancer Therapy-General; K-SPARC, Korean SPARC; SPARC, Sheffield Profile for Assessment and Referral to Care.

. The reliability of theTable 1. Demographic characteristics of the patientsCancer Res Treat. 2021;53(1):25-31

Characteristic
No. (%)

Age, median (range, yr)
64.5 (52-69.3)
Sex
Male
15 (50)
Female
15 (50)
Performance status
0, 1
24 (80)
2, 3
6 (20)
Diagnosis
Gastrointestinal tumor
8 (27)
Head and neck cancer
7 (23)
Breast cancer
5 (17)
Lymphoma/Myeloma
4 (13)
Lung cancer
2 (7)
Hepatobiliary tumor
2 (7)
Other
2 (7)
Stage
I/II/III
6 (20)
IV
24 (80)
Metastasis
Present
18 (60)
Absent
12 (40)
Current treatment
Chemotherapy
29 (97)
Palliative care
1 (3)
Duration of disease, median (range, mo)
9 (5-23)
Setting
Inpatient
26 (87)
Outpatient
4 (13)
Education
Less than high school
10 (33)
High school or more
19 (63)
Other
1 (3)

Table 2 .
2Internalconsistency of each domain using Cronbach's
α coefficients

Cronbach's α score

Characteristic
Current
Polish
pilot study
version

Physical symptoms
0.812
0.68
Psychological issues
0.804
0.86
Religious and spiritual issues
0.589
0.65
Independence and activity
0.843
0.77
Family and social issues
0.754
0.80
Treatment issues
0.822
0.62

AcknowledgmentsThe research has been supported by Korean Society of Medical Oncology. The authors would like to appreciate clinical investigators, patients, and their families. We also would like to express our special thanks to Dr. Shin JC, a Korean language specialist to help us in the translation process.Electronic Supplementary Material
Geneva: World Health Organization. World Health OrganizationWorld Health Organization. Palliative care [Internet]. Geneva: World Health Organization; 2020 [cited 2020 Feb 19]. Available from: https:/ /www.who.int/news-room/fact-sheets/detail/ palliative-care.

Early palliative care for patients with metastatic non-small-cell lung cancer. J S Temel, J A Greer, A Muzikansky, E R Gallagher, S Admane, V A Jackson, N Engl J Med. 363Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, et al. Early palliative care for patients with meta- static non-small-cell lung cancer. N Engl J Med. 2010;363:733- 42.

Integration of palliative care into standard oncology care. B R Ferrell, J S Temel, S Temin, E R Alesi, T A Balboni, E M Basch, J Clin Oncol. 35American Societyof Clinical Oncology clinical practice guideline updateFerrell BR, Temel JS, Temin S, Alesi ER, Balboni TA, Basch EM, et al. Integration of palliative care into standard oncology care: American Society of Clinical Oncology clinical practice guideline update. J Clin Oncol. 2017;35:96-112.

European Society for Medical Oncology (ESMO) position paper on supportive and palliative care. K Jordan, M Aapro, S Kaasa, C I Ripamonti, F Scotte, F Strasser, Ann Oncol. 29Jordan K, Aapro M, Kaasa S, Ripamonti CI, Scotte F, Strasser F, et al. European Society for Medical Oncology (ESMO) posi- tion paper on supportive and palliative care. Ann Oncol. 2018; 29:36-43.

National Comprehensive Cancer Network. InternetNational Comprehensive Cancer Network [Internet].

PA: National Comprehensive Cancer Network. Plymouth Meeting, Plym- outh Meeting, PA: National Comprehensive Cancer Network;

Systematic review of the problems and issues of accessing specialist palliative care by patients, carers and health and social care professionals. N Ahmed, J C Bestall, S H Ahmedzai, S A Payne, D Clark, B Noble, Palliat Med. 18Ahmed N, Bestall JC, Ahmedzai SH, Payne SA, Clark D, Noble B. Systematic review of the problems and issues of ac- cessing specialist palliative care by patients, carers and health and social care professionals. Palliat Med. 2004;18:525-42.

The use of cognitive interviewing methodology in the design and testing of a screening tool for supportive and palliative care needs. N Ahmed, J C Bestall, S A Payne, B Noble, S H Ahmedzai, Support Care Cancer. 17Ahmed N, Bestall JC, Payne SA, Noble B, Ahmedzai SH. The use of cognitive interviewing methodology in the design and testing of a screening tool for supportive and palliative care needs. Support Care Cancer. 2009;17:665-73.

The adaptation of the Sheffield Profile for Assessment and Referral for Care (SPARC) to the Polish clinical setting for needs assessment of advanced cancer patients. W Leppert, M Majkowicz, S H Ahmedzai, J Pain Symptom Manage. 44Leppert W, Majkowicz M, Ahmedzai SH. The adaptation of the Sheffield Profile for Assessment and Referral for Care (SPARC) to the Polish clinical setting for needs assessment of advanced cancer patients. J Pain Symptom Manage. 2012;44: 916-22.

The Edmonton Symptom Assessment System (ESAS): a simple method for the assessment of palliative care patients. E Bruera, N Kuehn, M J Miller, P Selmser, K Macmillan, J Palliat Care. 7Bruera E, Kuehn N, Miller MJ, Selmser P, Macmillan K. The Edmonton Symptom Assessment System (ESAS): a simple method for the assessment of palliative care patients. J Palliat Care. 1991;7:6-9.

A comprehensive study of psychometric properties of the Edmonton Symptom Assessment System (ESAS) in Spanish advanced cancer patients. A Carvajal, C Centeno, R Watson, E Bruera, Eur J Cancer. 47Carvajal A, Centeno C, Watson R, Bruera E. A comprehensive study of psychometric properties of the Edmonton Symptom Assessment System (ESAS) in Spanish advanced cancer pati- ents. Eur J Cancer. 2011;47:1863-72.

Validation of the Edmonton Symptom Assessment System in Korean patients with cancer. J H Kwon, S H Nam, S Koh, Y S Hong, K H Lee, S W Shin, J Pain Symptom Manage. 46Kwon JH, Nam SH, Koh S, Hong YS, Lee KH, Shin SW, et al. Validation of the Edmonton Symptom Assessment System in Korean patients with cancer. J Pain Symptom Manage. 2013; 46:947-56.

The Functional Assessment of Cancer Therapy scale: development and validation of the general measure. D F Cella, D S Tulsky, G Gray, B Sarafian, E Linn, A Bonomi, J Clin Oncol. 11Cella DF, Tulsky DS, Gray G, Sarafian B, Linn E, Bonomi A, et al. The Functional Assessment of Cancer Therapy scale: development and validation of the general measure. J Clin Oncol. 1993;11:570-9.

Validation of the Functional Assessment of Cancer Therapy-General (FACT-G) scale for measuring the health-related quality of life in Korean women with breast cancer. E H Lee, M Chun, S Kang, H J Lee, Jpn J Clin Oncol. 34Lee EH, Chun M, Kang S, Lee HJ. Validation of the Functional Assessment of Cancer Therapy-General (FACT-G) scale for measuring the health-related quality of life in Korean women with breast cancer. Jpn J Clin Oncol. 2004;34:393-9.

Scale development: theory and applications. R F Devellis, Applied social research methods series. Newbury Park, CASage Publications26DeVellis RF. Scale development: theory and applications. App- lied social research methods series, Vol. 26. Newbury Park, CA: Sage Publications; 1991.

Ministry of Health and Welfare. Recommendations for distress management toward improvement of quality of life in cancer patients. Seoul: Ministry of Health and Welfare. Ministry of Health and Welfare. Recommendations for dis- tress management toward improvement of quality of life in cancer patients. Seoul: Ministry of Health and Welfare; 2009.

Holistic needs of people with thoracic cancer identified by the Sheffield profile for assessment and referral to care questionnaire((c)). A Wilcock, A Hussain, M Maddocks, J Palliat Med. 22Wilcock A, Hussain A, Maddocks M. Holistic needs of peo- ple with thoracic cancer identified by the Sheffield profile for assessment and referral to care questionnaire((c)). J Palliat Med. 2019;22:1120-3.

Discrepancies among patients, family members, and physicians in Korea in terms of values regarding the withholding of treatment from patients with terminal malignancies. D Y Oh, J E Kim, C H Lee, J S Lim, K H Jung, D S Heo, Cancer. 100Oh DY, Kim JE, Lee CH, Lim JS, Jung KH, Heo DS, et al. Dis- crepancies among patients, family members, and physicians in Korea in terms of values regarding the withholding of treatment from patients with terminal malignancies. Cancer. 2004;100:1961-6.

Experiences and attitudes of patients with terminal cancer and their family caregivers toward the disclosure of terminal illness. Y H Yun, Y C Kwon, M K Lee, W J Lee, K H Jung, Y R Do, J Clin Oncol. 28Yun YH, Kwon YC, Lee MK, Lee WJ, Jung KH, Do YR, et al. Experiences and attitudes of patients with terminal cancer and their family caregivers toward the disclosure of terminal illness. J Clin Oncol. 2010;28:1950-7.

Meaning-making coping among cancer patients in Sweden and South Korea: a comparative perspective. F Ahmadi, J Park, K M Kim, N Ahmadi, J Relig Health. 56Ahmadi F, Park J, Kim KM, Ahmadi N. Meaning-making cop- ing among cancer patients in Sweden and South Korea: a com- parative perspective. J Relig Health. 2017;56:1794-811.


A GPU-based algorithm for fast node label learning in large and unbalanced biomolecular networks
05-07 July 2017

Marco Frasca email@example.com
AnacletoLab -Department of Computer Science
Università degli Studi di Milano
Via Comelico 3920135MilanoItaly

Giuliano Grossi
AnacletoLab -Department of Computer Science
Università degli Studi di Milano
Via Comelico 3920135MilanoItaly

Jessica Gliozzo
Marco Mesiti
AnacletoLab -Department of Computer Science
Università degli Studi di Milano
Via Comelico 3920135MilanoItaly

Marco Notaro
AnacletoLab -Department of Computer Science
Università degli Studi di Milano
Via Comelico 3920135MilanoItaly

Paolo Perlasca
AnacletoLab -Department of Computer Science
Università degli Studi di Milano
Via Comelico 3920135MilanoItaly

Alessandro Petrini
AnacletoLab -Department of Computer Science
Università degli Studi di Milano
Via Comelico 3920135MilanoItaly

Giorgio Valentini
AnacletoLab -Department of Computer Science
Università degli Studi di Milano
Via Comelico 3920135MilanoItaly

From Italian Society of Bioinformatics (BITS): Annual Meeting
2017CagliariItaly

A GPU-based algorithm for fast node label learning in large and unbalanced biomolecular networks
05-07 July 201710.1186/s12859-018-2301-4Frasca et al. BMC Bioinformatics 2018, 19(Suppl 10):353 RESEARCH Open Access *Correspondence: Full list of author information is available at the end of the articleGPU-based Hopfield netsLarge-sized networksProtein function predictionBiological networksNode label prediction
Background: Several problems in network biology and medicine can be cast into a framework where entities are represented through partially labeled networks, and the aim is inferring the labels (usually binary) of the unlabeled part. Connections represent functional or genetic similarity between entities, while the labellings often are highly unbalanced, that is one class is largely under-represented: for instance in the automated protein function prediction (AFP) for most Gene Ontology terms only few proteins are annotated, or in the disease-gene prioritization problem only few genes are actually known to be involved in the etiology of a given disease. Imbalance-aware approaches to accurately predict node labels in biological networks are thereby required. Furthermore, such methods must be scalable, since input data can be large-sized as, for instance, in the context of multi-species protein networks. Results: We propose a novel semi-supervised parallel enhancement of COSNET, an imbalance-aware algorithm build on Hopfield neural model recently suggested to solve the AFP problem. By adopting an efficient representation of the graph and assuming a sparse network topology, we empirically show that it can be efficiently applied to networks with millions of nodes. The key strategy to speed up the computations is to partition nodes into independent sets so as to process each set in parallel by exploiting the power of GPU accelerators. This parallel technique ensures the convergence to asymptotically stable attractors, while preserving the asynchronous dynamics of the original model. Detailed experiments on real data and artificial big instances of the problem highlight scalability and efficiency of the proposed method. Conclusions: By parallelizing COSNET we achieved on average a speed-up of 180x in solving the AFP problem in the S. cerevisiae, Mus musculus and Homo sapiens organisms, while lowering memory requirements. In addition, to show the potential applicability of the method to huge biomolecular networks, we predicted node labels in artificially generated sparse networks involving hundreds of thousands to millions of nodes.

Background

The Automated Function Prediction of proteins (AFP) conveys the need of annotating the huge amount of protein sequences with their biomolecular functions. High-throughput sequencing technologies are rapidly increasing the gap between known protein sequences and proteins with experimentally annotated functions; indeed, more than 60 millions of protein sequences are available at the UniProt repository [1], and for instance less than 1% of these sequences have manually curated annotations in SwissProt [2]. Accordingly, the computational assignment of the biological functions to the proteins of an organism can greatly help in reducing this gap [3]. From this point of view, AFP can be modeled as a set of binary classification problems on graphs (one for every function), where nodes represent proteins, edges encode their functional similarity, and nodes are labeled according to the current function (see Fig. 1). Two main characteristics of AFP are the large imbalance between positive (proteins annotated with the function under study) and negative nodes (the remaining proteins), and the large size of the input graph, since networks can contain millions of proteins (see for instance multi-species protein networks [4]).

Numerous graph/network-based approaches have been proposed by the scientific community to deal with the AFP problem, ranging from methods relying on the guilt-by-association principle [5], assuming proteins topologically close in the graph are likely to share their functions, to label propagation algorithms based on Markov [6] and Gaussian Random Fields [7][8][9][10]. Other studies based their evaluation on global [11] and local convergence properties, the latter one exploiting Hopfield networks (HNs) [12], and parametric variants of this model [13][14][15], including an extension of the Hopfield model Fig. 1 AFP problem as a set of binary classification problems. The aim is determining the color/label of unlabeled nodes/proteins, given the graph topology and the labels of the known part of the graph to a multi-category context [16,17], where nodes are inherently partitionable into separated categories (e.g. in the multi-species protein networks). In addition, since most AFP approaches exploit a protein neighborhood to infer the functions of that protein, some works introduced a generalization of the notion of pairwise-similarity among nodes by taking into account the contribution of neighbors shared by nodes [18,19]. Finally, other relevant studies adopted techniques based on random walks [20][21][22], kernel matrices [23,24], communities [25] and co-citations [26].

Despite their large diversity and their effectiveness in solving the AFP problem, most of the above mentioned methods neglect the class-imbalance problem, leading to classifiers tending to learn mainly the negative class, thus often obtaining a sensible deterioration of their performance [27]. Moreover, they do not suitably scale with the input size, in terms of both memory usage and execution time, since they usually adopt matrix representations to embed the graph (without exploiting the graph sparsity), and basically utilize sequential programming in their model design. Indeed, recent and interesting works proposed to use secondary memory-based technologies to exploit the large disks available in standard computers to apply to big data standard graph-based semi-supervised learning algorithms; nevertheless, this can be done at the expense of the efficiency, since the swapping between secondary and primary memories data increases the computational burden [28].

In this study we propose PARCOSNET (Parallel COS-NET), a methodology for solving AFP problem specifically designed to cope with the label imbalance problem and the big size of input data. It extends COSNET (Cost-Sensitive Neural Network) [14], a state-of-the-art semisupervised method for AFP based on HNs. COSNET introduces a parametric HN to effectively handle the label imbalance, but its available implementation [29] still adopts a matrix representation of input data, allowing its application (on ordinary off-the-shelf computer) only to networks with few tens of thousands of nodes.

As first contribution, PARCOSNET reduces the memory requirements of COSNET by adopting a sparse representation of both network connections and node labeling, leveraging the sparsity of input graphs and the scarcity of positive proteins characterizing data in the AFP context. On the other side, the overall execution time is remarkably reduced 1) by splitting the HN dynamics over independent sets of neurons, where nodes in the same independent set are updated in parallel, and 2) by exploiting Graphics Processing Unit (GPUs) devices under the CUDA (Compute Unified Device Architecture) parallel programming model [30] to use one or multiple GPUs in parallel along with the CPU. Specifically, multiple GPU cores are assigned to a single independent set, thus updating in parallel the neurons within each independent set, while independent sets are in turn sequentially updated to preserve the HN convergence properties. Considering that usually multiple GO terms should be predicted for each protein, the proposed implementation adds another level of parallelism through the multithreading execution, where each CPU thread is given an instance of the AFP problem (a function to be predicted), and multiple threads are run in parallel (each using multiple cores in parallel). This thereby results in a noticeable speed-up with regard to the original COSNET implementation.

We evaluated the PARCOSNET gain in terms of both memory requirements reduction and execution speed-up by testing COSNET and PARCOSNET in predicting the Gene Ontology functional terms [31] for three eukaryotic organisms. Furthermore, synthetic graphs of different sizes, from hundreds of thousands to millions of nodes, and of different densities have been generated to empirically show the applicability of PARCOSNET on large networks.

PARCOSNET source code has been publicly released for evaluation and testing purposes, and is available on the official AnacletoLab GitHub repository, at [32].

Methods

Experimental data

This section is devoted to the description of both real and artificial networks used in throughout the paper.

Real data. Three organisms have been considered for the AFP problem, namely Homo sapiens (human) and two model organisms S.cerevisiae (yeast) and Mus musculus (mouse). The input networks have been retrieved from the STRING database, version 10.0 [4]: the STRING networks are highly informative networks merging several sources of information about proteins, coming from databases collecting experimental data like BIND, DIP, GRID, HPRD, IntAct, MINT or from databases collecting curated data such as Biocarta, BioCyc, KEGG, Reactome. The total number of proteins is 6391, 21151 and 19576 for yeast, mouse and human organisms, respectively.

All networks have one large connected component, with human and mouse networks with one or more smaller connected components. Furthermore, the human network is the most compact, having the smallest ratio between the number of nodes and the network diameter (see Table 1 for the network topological characteristics).

In the STRING database each protein-protein connection is associated with a confidence score: in principle, discarding edges with confidence score lower than a fixed threshold would allow to select more reliable connections; on the other side, it would generate isolated proteins (proteins with no connections), which accordingly should be discarded from the analysis. Since the aim of this study is supplying a methodolgy able to work on large networks, no edge threshold has been applied, thus including all available proteins.

Protein networks have been normalized as follows: denoted byŴ the matrix obtained from the STRING connections, the final network W is obtained by applying the normalization
W = D −1/2Ŵ D −1/2 ,
where D is a diagonal matrix with non-null elements d ii = jŴ ij . Note that W is still symmetric. Protein functional annotations have been retrieved from the Gene Ontology (GO) database, using the UniProt GOA releases 69 (9 May 2017), 155 (6 June 2017) and 168 (9 May 2017) respectively for yeast, mouse and human organisms. The GO terms have been selected from all the three branches Biological Process (BP), Molecular Function (MF), and Cellular Component (CC), by considering terms with at least 50 annotated proteins with experimental evidence, in order to obtain a minimal amount of information for the prediction of GO terms. The number of the resulting GO terms is summarized in Table 2. The mapping from UniProt to STRING protein identifiers was carried out according to the mapping files provided at UniProt repository.

Artificial data. In order to assess the performances in terms of computational time and memory consumption over larger datasets, the parallel implementation has also been tested on several artificial datasets which have been randomly generated. For random graphs we use the Erdős model in which the graph size n and the probability p to have an edge between a pair of nodes are fixed. In particular, the range chosen for n goes from 5 · 10 5 to 1.5 · 10 6 , while for p we chose values so as to reproduce a graph density σ = np close to that of real biological network.  Edge weights were uniformly generated in [ 0, 1], while the corresponding set of labels has been generated so as to respect the unbalancing of realistic cases. We run PARCOSNET, with the resulting 9 artificial datasets and recorded the computation time and memory consumption.

Automated protein function prediction

In the context of the automated protein function prediction (AFP), proteins are represented by a set of nodes V = {1, 2, . . . , n}, and relationships between proteins are encoded through a symmetric n × n real weight matrix W, whose elements w ij represent functional similarities between pairs (i, j) of proteins.

For a given functional class, the nodes V are labeled with {+, −}, leading to the subsets P and N of positive and negative nodes. For most existing taxonomies for AFP usually the functional labeling is known only for a subset S ⊂ V , while is unknown for U = V \ S. Moreover, let be S + = S ∩ P and S − = S ∩ N.

The Automated protein function prediction problem consists in finding a bipartition (U + , U − ) of U, where U + and U − are the subsets of unlabeled proteins considered as candidate for the classes U ∩ P and U ∩ N, respectively. From this standpoint, AFP is set as a semi-supervised learning problem on graphs, since protein functions can be predicted by exploiting both labeled and unlabeled nodes/proteins and the weighted connections between them [33].

COSNET

COSNET (COst Sensitive neural Network) [13,14] is a neural algorithm recently proposed to face with the AFP problem. More specifically, this technique relies on a parametric family of the Hopfield model [34], where the network parameters are learned to cope with the label imbalance and the network equilibrium point is interpreted to classify the unlabeled nodes.

Formally, for a given a set of nodes V = {1, . . . , n}, COSNET is a triple H = W , λ, ρ , where:

-W ∈ R n×n is a symmetric weight matrix whose elements w ij ∈ [ 0, 1] represent the connection strength between the neurons (nodes) i and j (naturally w ii = 0), -λ = {λ 1 , . . . , λ n } ∈ R n denotes the neuron activation thresholds,
-ρ ∈ [0, π 2 )
is a parameter which determines the two neuron activation (state) values {sin ρ, − cos ρ}.

The rationale of the parameter ρ is to conceptually separate node labels and neuron activation values, since for classical HNs activation values are in the set {−1(0), 1}, that means node labels and neuron activation values coincide. Thus, appropriately learning the parameter ρ allows the algorithm to counterbalance the large imbalance towards negatives (see [14]).

A relevant issue for the correct design of this kind of recurrent neural networks is the synchronization of its computing nodes. The Hopfield model is a discrete-time dynamical system which admits synchronous or asynchronous updating or even both if an hybrid setting is admitted. In case of asynchronous (sequential) updating, each unit is updated independently from the others at any time t. Thus, by denoting with π = π(1), · · · , π(n) an arbitrary permutation on nodes V and with x π(i) (t) the state of neuron π(i) at time t, the dynamics assumes the form:
x π(i) (t + 1) = sin ρ, ifh π(i) (t + 1) ≥ 0 − cos ρ, otherwise(1)
where
h π(i) (t + 1) = π(i−1) j=π(1) w π(i)j x j (t + 1) + π(n) j=π(i+1) w π(i)j x j (t) − λ π(i) .
The convergence properties depend on the weight matrix structure W and the rule by which the nodes are updated. In particular, if the matrix is symmetric, it has been proved that the network converges to a stable state when operating in asynchronous mode, while it converges to a cycle of length at most 2 when operating in a synchronous (fully-parallel) mode. The proof of these properties is grounded on the so-called energy function, which is non decreasing when the state of the network x = (x 1 , x 2 , . . . , x n ) changes as a result of a computation (1). Since the energy function is upper-bounded, it follows that the system will converge to some state. In the classic discrete Hopfield model the energy function has the following quadratic form:
E(x) = − 1 2 x T W x + x T λ.( 2 )
As a major result, it has been shown that (2) is a Lyapunov function for the Hopfield dynamical systems with asynchronous dynamics, i.e., for each t > 0, E(x(t + 1)) ≤ E(x(t)) and exists a timet such that E(x(t)) = E(x(t)), for all t ≥t. Moreover, the reached fixed pointx = x(t) is a local minimum of (2).

The overall scheme of the COSNET algorithm adopting dynamics (1) can be sketched as follows:
INPUT. A symmetric weight matrix W ∈ [0, 1] n×n , a
labeling function y : V → {+, −}; the subsets S + , S − and U of positive, negative and unlabeled instances, respectively; an initial value x(0) ∈ {sin ρ, − cos ρ, 0} n ; a permutation π on the set U. Step 1. Learn the parameters ρ =ρ and λ i =λ ∈ R on the sub-network restricted to labeled nodes S such that the state represented by known labels is "as close as possible" to a minimum of the network energy.

Step 2. Regularize the network dynamics in order to prevent the network stucking into trivial energy minima by suitably changing the thresholds and the connection weights (see [14] for more details). Hereafter, abusing notation, we assume this regularization is embedded in the connections w ij and thresholds λ i , fer each i, j ∈ V .

Step 3. Run the sub-network restricted to unlabeled nodes embedding the learned parameterρ andλ until an equilibrium stateû is reached. On the base of stateû compute the bipartition (U + , U − ) of U into the positive and negative neurons.
OUTPUT. A bipartition (U + , U − ).
Step 1 and 2 allow the method to deal with labeling imbalance, since the first step counterbalances the predominance of negatives in the node neighborhood, whereas the second step ensures to avoid trivial states composed of all negative predictions.

Parallel COSNET

The goal of this work is to speed-up the COSNET algorithm by introducing a partial synchronous updating of the computational units in order to parallelize the network evolution stage while preserving the asynchronous dynamics. Since in AFP context the weight matrix W is usually sparse, several nodes are independent from each others.

The basic idea is thereby to partition the nodes belonging to the undirected graph G = V , E W (with edge set E W induced by W ) into a small number of independent subsets, which can be done recasting the problem as a vertex coloring problem of G.

Let a vertex coloring be a map σ : V → C, where C is a set of distinct colors. We say that a coloring is proper if adjacent vertices of G receive distinct colors of C, which in turn means that if (i, j) ∈ E W , then σ (i) = σ (j). Clearly, in any proper vertex coloring of G the vertices that receive the same color are independent. A k-coloring of a graph G is a vertex coloring of G that uses at most k colors and G is said to be k-colorable if it admits a proper vertex coloring using at most k colors. Hereafter, we denote a proper k-
coloring by a partition P = {V 1 , . . . , V k } of the vertex set V into k independent subsets.
Under this setting it is easy to show that by simultaneously updating the nodes of each element V i of P, one at a time, whatever the permutation of {1, . . . , k} is, the Hopfield network asynchronous dynamics is preserved. This implies that, given an initial state, the network is guaranteed to converge to a unique fixed point which is a local minimum of (2). To see that this property holds, we can start by observing that each partition P induces a permutation π P (that for sake of notation we simply denote by π) on the set V such that, for all i = 1, . . . , k, the subse-
quence π i = π i (1), . . . , π i (n i ) collects the nodes within V i = {π i (1), . . . , π i (n i )}, being n i = |V i |.
The permutation π, for a fixed P, is then written as the juxtaposition of all subsequences π = π 1 , · · · , π k .

It is easy to see now that, synchronously updating all nodes in the subsequence π i , but asynchronously accordingly to an arbitrary permutation ω = ω(1), . . . , ω(k) on all subsequence indexes {1, . . . , k}, the constraints prescribed by the asynchronous updating rule are respected. Indeed, given the permutation π(ω) = π ω(1) , · · · , π ω(k) induced by a k-coloring P and the permutation ω of the partition indexes, by applying the following partiallyparallel update
∀u ∈ π ω(i) , x u (t + 1) = sin ρ, ifh u (t + 1) ≥ 0 − cos ρ, otherwise (3) where h u (t + 1) = v∈π ω(1) ···π ω(i−1) w uv x v (t + 1) (4) + n v∈π ω(i+1) ···π ω(k) w uv x v (t) − λ u .
done in parallel on u ∈ π ω(i) and sequentially over π ω (1) , · · · , π ω(k) , we obtain the same attractor as in (1) with sequential update dictated exactly by π = π(ω). Therefore, the following PARCOSNET algorithmwhich stands for "Parallel COSNet" -encompassing a graph coloring strategy and adopting the Hopfield dynamics (3), can be sketched as follows:

INPUT. Idem as in COSNET; a k-coloring P = {U 1 , . . . , U k } of U inducing the permutation π; a permutation ω on the first k integers.

Step 1. Idem as in COSNET.

Step 2. Idem as in COSNET.

Step 3. Run the sub-network restricted to unlabeled nodes embedding the learned parametersρ,λ and following updating (3) in parallel for nodes in U i and sequentially across sets U i by following the update permutation π(ω) = π ω (1) , · · · , π ω(k) until an equilibrium stateû is reached. On the basis of statê u compute the bipartition (U + , U − ) of U into the positive and negative neurons.
OUTPUT. A bipartition (U + , U − ).
Despite the additional computational cost for finding a suitable k-coloring, PARCOSNET shows a significant performance speed-up compared to original algorithm, mainly due to the structure of partition P and the number of processors N at hand. Although in many applications it is required to find a partition P having minimum cardinality (also called the chromatic number of G), our parallelization task is instead aimed at making the evolution of the Hopfield neural system in COSNET as fast as possible; therefore, we rather seek a k-coloring able optimize the computational efficiency of the processors. In the following we discuss some implementation issues regarding a specific parallel architecture provided by GPUs rather than to carry out cost analysis for abstract theoretical models.

GPU Implementation

Originally designed for graphics applications, GPUs have gradually acquired increasing importance for scientific computing and computer simulations and their processing power is one order of magnitude higher than current generation CPUs. This considerable computational power, due to specific hardware design, has paved the way for big data applications. For instance, in many domains it is frequent to address problems on very large graphs, often involving millions of vertices and graphics accelerator hardware has become a cost-effective parallel platform to solve them [35].

For this reasons, an implementation of PARCOSNET specifically targeted to NVIDIA GPUs [32] has been developed for assessing the effectiveness and performances of the algorithm. This implementation has been developed under the CUDA (Compute Unified Device Architecture) programming model [30]. Also under the same programming paradigm, a parallel implementation of a Greedy Graph Coloring (GGC) algorithm has been developed for solving the graph coloring problem, as required by the PARCOSNET algorithm.

A GPU processor is composed by an array of Streaming Multiprocessors (SM), each composed by a variable number of CUDA cores, depending on the generation of the GPU. Therefore, a single GPU processor can be seen as an array of thousands of simplified processing cores capable of scheduling and concurrently running a very large number of threads. Threads are executed according to the SIMT (Single Instruction Multiple Thread) architecture, a variation of the SIMD execution model (Single Instruction, Multiple Data) in which every thread in a block is executed independently.

Both in PARCOSNET and GGC this execution model feature has been exploited to achieve a high level of fine grained parallelism: in the GGC algorithm, each node of the graph is assigned to a thread, while in the Hopfield dynamics, an entire block of threads, ranging from 32 to 512, is assigned to a neuron for updating its state. Launching a very large number of threads has a useful consequence in the CUDA programming model, since it helps to hide latencies between the processing core and the on-board video RAM, where data are stored.

One of the major difficulty in GPU programming lies in managing its complicate memory model, since data can be stored in different address spaces, each one having its own trade-offs in terms of accessing speed and size; on top of that, CUDA kernels -i.e. programs that run on the GPU -are able to access only data residing on the on-board RAM, therefore the host system must copy the relevant part of data on the GPU before the actual computation starts, and copy back the results after the computation ends. Repeatedly copying data back and forth the GPU causes latencies that slow down the computation. In PARCOSNET a good trade-off has been achieved through several strategies. First of all, to cope with the limited amount of video ram, the graph is stored in the host RAM by means of a compressed representation, such that the entire net takes roughly (2 × n + 2 × m) doubles, being n and m respectively the number of nodes and the number of edges of the net; this is possible by exploiting the sparsity of the net. Then, only the unlabeled portion of the graph is copied into GPU memory and processed, further reducing the transmitted data volume.

Another strategy to minimize latencies occurring in the GPU programming model is to reduce the number of synchronization points between the host system and the GPU. In an ideal heterogeneous system, the host and the hardware accelerator should work independently for maximizing concurrency, but this does not hold true if the algorithm requires several synchronization points; when such events happen, one device may be put to halt, waiting for the other to reach the synchronization point, thus slowing down the computation. This was the case of the Greedy Coloring, where each iteration on the GPU required a validation on the host system. We solved this issue by writing a coloring kernel which is rather independent from the system host, exploiting CUDA Dynamic Parallelism, where each kernel can spawn a set of subkernels (this is opposed to the traditional kernel launching model, where only the host system is allowed to launch computational tasks on the GPU). In this way, the entire coloring task does not require any synchronization by the host, which is therefore free to accomplish other tasks and concurrency between GPU and host is maximized.

In the following subsections, the parallelization schemes and some implementation issues about the GGC and PARCOSNET algorithms are presented. Also, CPU multithreading and memory consumption are briefly discussed.

Parallel Greedy Graph Coloring

As highlighted in previous section, graph coloring is a key strategy to make an efficient use of parallel (SIMT) architecture because it allows to split complex tasks into small independent subtasks that can be carried out concurrently. In our setting, each subtask can be identified by the so called Maximal Independent Set (MIS) of a graph, i.e., a maximal collection of vertices I ∈ V subject to the restriction that no pair of vertices in I are adjacent. This subgraph structure is strictly connected to coloring, because it represents a common parallel approach for graph coloring, leveraging the parallel MIS algorithm as a subroutine (see the schema in Algorithm 1). In this approach, a partition P = {V 1 , . . . , V k } is conceived as a collection of MISs yielded by subsequent call of the Luby parallel algorithm [36] (called LUBYMIS in the pseudocode) which works in a greedy fashion. The idea is that in every round i it finds a subsetĪ ⊂ V which is an independent set. Then it addsĪ to the current independent set I, and removesĪ and its neighbors N (Ī) from the current graph V (see pseudocode). IfĪ ∪ N (Ī) is a constant fraction of |V |, then it will only needs O(log |V |) rounds to determine I. It will instead ensure that by removing such subset from the graph, it removes a constant fraction of the edges. The final subset I at round i is then considered as set V i in the partition P.

To chooseĪ in parallel, each vertex v independently adds itself toĪ with a well chosen probability p(v). Since we want to avoid adding adjacent vertices toĪ, we will prefer to add low degree vertices. But, if for some edge (u, v), both endpoints were added toĪ, then we keep the higher degree vertex. The above strategy is concisely summed up in the statement called choice in the GPULUBYMIS pseudocode.

From an implementation standpoint, the latter procedure has been implemented in SIMT fashion through the curand library made available within CUDA toolkit. Each thread handles a node of the whole graph, draws a random number between 0 and 1 uniformly distributed and establishes whether it belongs or not to the MIS under construction.

Parallel COSNET

In this section, the implementation of the parallel Hopfield algorithm using CUDA programming model, sketched in the pseudocode of Algorithm 2, is presented and analyzed.

To face with the parallelization of the asynchronous dynamics (broadly described in previous sections), it should be noted that at the base of this neural architecture there is an intrinsic parallelism in the computation

Algorithm 1 Parallel greedy coloring
Input: graph G = V , E Output: coloring (partition) P = {V 1 , . . . , V k } P ← ∅ i ← 1 first color while G = ∅ do V i ← GPULUBYMIS(G) P ← P ∪ {V i } Z ← V i ∪ N (V i ) union of V i with its neighbors G ← sub V − Z, E induced subgraph excluding Z i ← i + 1 set new color end while procedure GPULUBYMIS(G) I ← ∅ G = V , E ← G = V , E while G = ∅ do choiceĪ ⊆ V select an independent set of G I ← I ∪Ī Z ←Ī ∪ N (Ī) union ofĪ with its neighbors G ← sub V − Z, E
induced subgraph excluding Z end while return I MIS of G end procedure of the activation function (4): each single neuron implements this simple thresholding function, whose state is either "active" or "not active". This state is determined by calculating the weighted sum of the states of its connected neurons. If the sum exceeds the threshold, the state will change to active, otherwise, the neuron will be non-active. W U is the matrix of connections among nodes in U.

All these observations are captured in what we call GPUHOPFIELDNET procedure within the pseudocode. This algorithm naturally exposes two different levels of parallelism which can be exploited for realizing an effective scheme well suited for GPU execution. These levels correspond to two nested and different tasks: the first consists in concurrently compute the update of all nodes sharing the same color (i.e., belonging to i-th cluster V i ), while the second consists in executing in parallel the addition and thresholding required for the updating of the state of each neuron. Also, the tasks are characterized by different granularity, the latter being more fine-grained than the former.

The first and more coarse-grained task, i.e. concurrently updating all the nodes having the same color, has been tackled by sequentially launching k different instances of the GPUHopfieldNet kernel during each iteration, being k the number of colors returned by the coloring algorithm.

Algorithm 2 PARCOSNET: Hopfield net (Step 3)
Input: net H = W U ,λ,ρ , k-coloring P = {U 1 , . . . , U k } of vertices in U Output: final stateû u ← 0 net state nonstop ← true stop flag while nonstop do u ← u for i ← 1, k do u ← GPUHOPFIELDNET(H, U i , u ) end for if u = u then u ← u else nonstop ← false end if end while procedure GPUHOPFIELDNET(H, U, u) for all v ∈ U do in parallel use SIMT to compute h v compute (4) u v ← sin ρ, ifh v ≥ 0 − cos ρ, ifh v < 0 compute (3)
end for u ← u returnû partially update fixed point end procedure

Note that this approach respects the sequential requirements of the Hopfield dynamics. Each kernel is launched with a different configuration reflecting the number of nodes belonging to each cluster V i . In particular, we assign |V i | CUDA thread blocks to the i-th kernel, i.e. one CUDA thread block per node.

To perform the second task, i.e. to update the state of each neuron, a fixed number n T of threads (ranging from 32 to 1024) is in turn assigned to each thread block. To process the weigthed contributions of each neuron neightborhood, we proceed in SIMT fashion using shared memory among all threads in the same block and applying the known primitive called parallel reduction: this is a tree-based approach used within each CUDA thread block frequently applied to process very large arrays of N elements. It can be shown that, having at disposal P CUDA threads physically in parallel (P processors), the time complexity of the parallel reduction is O(N/P + log N). By varying n T , the algorithm can be adapted to the density of the graph: very dense graphs will benefit from an increase of n T , since more threads will speed-up the evaluation of the neighborhood and the computation of the parallel reduction.

CPU multithreading and data representation

On top of the parallelization of the Hopfield dynamics and graph coloring, PARCOSNET exploits the independence of protein functions (target classes) to further accelerate the computation, since as stated in the introduction multiple AFP problems can be solved concurrently. As shown in Fig. 2, in PARCOSNET we put this natural additional level of parallelism by means of CPU multi-threading, where each target class is assigned to a different CPU thread. In this implementation, each thread reads a different class labeling, trains the net and then executes the coloring and Hopfield dynamics on the GPU. This dramatically improves the performances of the overall process since each AFP instance is independent, therefore multiple instances can be run concurrently. On the other side, CPU multi-threading implies to share the resources of a single GPU among multiple CPU threads, leading to serialization latencies. To deal with this problem, we exploited a CUDA compiler option that allows to assign at runtime each CPU thread to a different CUDA stream, where a CUDA stream is a queue of commands or operations that are executed in a specific order; while operations on a stream are executed sequentially, operations in different streams may be executed concurrently or out of order with respect to one another. In the specific case of PARCOSNET, kernel executions belonging to different AFP problems are interleaved and executed concurrently.

Also, the algorithm in principle could benefit of additional GPU devices to split the workload, for instance, in a system with 2 GPUs, all the odd-numbered CPU threads might offload the parallel computation to the first GPU, while the even-numbered to the second GPU.

An issue that arises when working with big datasets is memory consumption. We chose to adopt a compressed format for storing the net and the labeling y. Indeed, by exploiting the sparsity of W, for each node solely its neighbors are kept in memory at run time; moreover, leveraging the scarcity of positives, only the nodes belonging to the minority class are maintained, thus saving huge amount of memory when the input graph contains millions of nodes.

Results

Real data

Although COSNET has already been validated in [14,15,37] to solve AFP, for sake of completeness we report in Table 2 its performances in predicting the GO terms. The generalization abilities of COSNET have been assessed through a 5-fold cross validation (CV), and evaluated in terms of Precision (the proportion of positives correctly predicted) and Recall (the proportion of real positive discovered) combined in the F measure, which is the harmonic mean of precision and recall. In this context, where positives are rare, these measures are more informative than the error rate. Moreover, to Multiple CPU threads are launched in parallel each one solving the AFP problem for a given class/protein function. The GPU thread blocks, each composed of several CUDA threads, first solve the coloring problem for the graph and then concurrently process all neurons of a given color, for all colors in sequence. A further fine-grained level of parallelism is finally introduced by assigning to each neuron a thread block to perform the neuron level local computations evaluate the ability of COSNET as ranker, we also report the Area Under the Precision Recall Curve (AUPRC), measure adopted in the recent CAFA2 international challenge to evaluate protein ranking [38]. To provide a protein ranking for COSNET related to the current GO term, we use the neuron state at equilibrium, as done in [15,39]. Table 3 contains the corresponding results averaged across terms in each GO branch.

The COSNET implementation is publicly available as an R package [29], where the time consuming procedures (e.g. parameter learning and Hopfield dynamics) for efficiency reasons are implemented in C language. Moreover, this package adopts a matrix representation for the input network, since the R language is optimized for matrixbased computations. Then we tested PARCOSNET on the same data sets, pointing out that its classification performances are the same as COSNET. The aim is assessing the computational speed-up achieved by PARCOSNET with respect to the already available COSNET implementation. To this end we tested both versions under the same setting, i.e. 5-fold cross validation, and the same learning parameters.

PARCOSNET has been executed using 1, 4, 8 and 12 CPU threads; to assess the scalability of the multithread approach, even the overall CPU occupancy of each execution has been computed. Also the maximum memory footprint and the execution time have been collected; the latter is used to evaluate the speed-up defined as T s /T p , where T s and T p are the execution times of sequential and parallel implementation, respectively.

All tests have been performed on the same host system, a workstation having 2 Intel Xeon E5 − 2620v3 CPUs clocked at 2.40 GHz, 64 GB of RAM memory, 2 TB disk and Linux Ubuntu 16.04 as operating system. The workstation is equipped with an NVidia GeForce GTX980 GPU card, featuring 2048 CUDA cores, 4 GB of dedicated onboard video memory and having Compute Capability 5.2. The C++ portion of the code has been compiled wit GCC 5.4.0, while the rest (that is, the CUDA kernels) with NVCC 8.0.44. As for COSNET, it has been executed under R version 3.4.2. Table 4 reports the average execution time in seconds of COSNET, and PARCOSNET, for computing an entire CV cycle for a single GO term.

To better evaluate the contribution of multithreading, Table 6 shows the average CPU occupancy of each execution of PARCOSNET. Data collected in this table is useful to assess the scalability of the parallelization over the number of CPU threads. Optimal values for this tables should be ideally near 100% × n, with n number of threads assigned to the task. As an example, optimal scalability for PARCOSNET executed on 12 CPU threads is achieved when occupancy reaches 1200%.

Finally, to evaluate also the memory usage, Table 7 reports the maximum memory footprint of COSNET and PARCOSNET when predicting a single GO term. For COS-NET, memory usage excludes the memory required by the R interpreter itself, thus counting just the space of objects created by R and C procedures.

Artificial data

Tests performed on the artificial dataset are aimed to evaluate the potential application of PARCOSNET on big data. We tested the method on single labeling, and accordingly the method has been executed in a single CPU thread mode. Table 8 shows the average execution time in seconds and the maximum memory occupancy in GB of PARCOSNET. Three dataset sizes have been considered, (each corresponding to a column in the table) and, for each size, datasets having different density (average degree per node) have been generated.

Discussion

Real data

The time reduction obtained by PARCOSNET is impressive (Table 4): for instance on mouse data the execution time is reduced from around 107s to 1.71s, when using a single thread. Multithreading further accelerates the execution, passing to 0.54s, 0.33s and 0.28s respectively when using 4, 8, and 12 CPU threads. To better understand these results, in Table 5 the speed-up gained by PARCOS-NET when compared with COSNET is also reported. Even when using a single thread implementation, PARCOSNET achieves a speed-up of range 31.64× (yeast) and 62.90× (mouse), whereas it gains up to two order of magnitude with respect to COSNET implementation when using in multithreaded version, up to 383.18× on mouse data.

Moreover, PARCOSNET occupancy is not far from the optimal value when using 4 threads, while slightly decreasing (in proportion) when the number of threads  increases (see Table 6). This is due to the fact that all the CPU threads concurrently access the same GPU, creating a minor bottleneck in computation. Indeed, we are fairly sure that adding others GPU to the host system might significantly improve the occupancy in multithread execution, since the workload can be equally divided between the devices. PARCOSNET has been executed with 1, 4, 8 and 12 CPU threads, but only the execution of 1 thread is reported, since the allocation footprints for the 4, 8 and 12 thread execution are pretty similar: as a matter of fact, maximum memory consumption is reached before the actual computation starts, i.e. while importing and compressing the network file (Table 7). Results show that, thanks to the compressed representation of both the net and labeling, memory usage is remarkably decreased, ranging from almost the half memory used by PARCOSNET on yeast, to less than one third on human and mouse.

Artificial data

Interestingly, PARCOSNET is able to predict node labels on graphs with 1.5 millions of nodes and average degree 300 in around 10 minutes, and using around 32 GB of RAM, nowadays available on the majority of ordinary offthe-shelf computers (Table 8). Furthermore, PARCOSNET shows a good scalability in terms of both computational time and memory consumption. The execution time increases less than linearly with the density, and little more than linearly with the number of nodes. The memory usage grows less than linearly with the number of nodes and little more than linearly with the density. This is likely due to the fact that most of the memory  consumption occurs with the import and compression of the net. Nevertheless this limitation can be addressed by off-line preprocessing the data and then importing the resulting file in compressed format for PARCOSNET processing. This strategy would allow PARCOSNET to process even larger datasets: as an example, the actual memory footprint recorded during the computation of the dataset composed by 1.5 million nodes with the average density of 300 edges per nodes, is around 4 GB of RAM memory and 1 GB of GPU memory, which is a tiny fraction compared to the maximum quantity of RAM used for compressing the graph.

Conclusions

PARCOSNET is a method for the automated function prediction (AFP), well-suited to process large protein networks with strongly unbalanced labels. PARCOSNET introduces a parallel and sparse implementation of COS-NET, a state-of-art imbalance-aware method for predicting protein functions, which allows both to remarkably speeding up the computation and reducing the memory requirements. In particular, the dynamics of the Hopfield network on which COSNET builds upon is parallelized by solving a vertex coloring problem on the graph/network, partitioning nodes into sets of independent nodes which are updated in parallel by using Graphics Processing Unit (GPUs) devices and CUDA programming. By leveraging the sparsity of biological networks and of the available annotated proteins characterizing the AFP context, PAR-COSNET adopts a sparse representation for both network connections and protein functions/labels. This, together with the parallel design and the usage of GPU devices, allows to significantly speed-up the computation with ordinary single-species biological networks, and opens the avenue to efficiently predict protein functions in large multi-species networks on ordinary computers, as shown in the experiments performed with synthetic networks having millions of nodes and hundreds of millions of edges.

Abbreviations

Availability of data and materials

The code presented in this study is available at https://github.com/ AnacletoLAB/ParCOSNet.

About this supplement

Fig. 2
2CPU/GPU schema of the PARCOSNET parallelization.

AFP: Automated function prediction; CUDA: Compute unified device architecture; CV: Cross validation; GCC: GNU compiler collection; GO: Gene Ontology; GPU: Graphics processing unit; HN: Hopfield network; SIMT: Single instruction multiple thread; SIMD: Single instruction, multiple data Funding This study was partially funded by Regione Lombardia, LISA project titled "HyperGeV : Detection of Deleterious Genetic Variation through Hyper-ensemble Methods ", and by Università degli Studi di Milano, project number 15983, titled "Discovering Patterns in Multi-Dimensional Data", which also funded the publication costs.

Table 1
1Characteristics of protein networksColumn Components denotes the number of connected components in the network, whereas Largest component size is the number of nodes in the largest connected component. Diameter is the number of edges on the longest path between two nodes, without considering edge weightsOrganism
Nodes
Average degree
Components
Largest component size
Diameter
Weighted diameter

Yeast
6391
314.0563
1
6391
6
1.0925

Mouse
21151
596.3804
21
21105
9
1.8362

Human
19576
579.9477
2
19574
6
1.0302

Table 2
2Number of GO terms with at least 5 annotationsOrganism
CC
MF
BP

Yeast
50
74
191

Mouse
85
112
733

Human
115
193
580

Table 3
3Average COSNET performance in predicting GO protein functionsOrganism
Precision
Recall
F
AUPRC

Yeast -CC
0.3467
0.5139
0.4023
0.363

Yeast -MF
0.3317
0.4609
0.3815
0.3237

Yeast -BP
0.4042
0.5231
0.4486
0.4079

Mouse -CC
0.2068
0.2383
0.2198
0.1517

Mouse -MF
0.1861
0.2377
0.2068
0.1411

Mouse -BP
0.1473
0.1725
0.1582
0.0935

Human -CC
0.2238
0.2765
0.2448
0.1709

Human -MF
0.1847
0.2302
0.2032
0.1356

Human -BP
0.1485
0.1793
0.1611
0.0972

Table 4
4Average CPU time in seconds for COSNET and PARCOSNET to perform a CV cycle on one GO termMethod
Yeast
Mouse
Human

COSNET
8.86
107.57
84.03

PARCOSNET
0.28
1.71
1.49

PARCOSNET 4
0.09
0.54
0.48

PARCOSNET 8
0.07
0.33
0.31

PARCOSNET 12
0.06
0.28
0.26

Table 5
5Average speed-up T s /T p , where T s and T p are the average execution time of COSNET and PARCOSNET to perform an entire cross-validation on one GO termMethod
Yeast
Mouse
Human

PARCOSNET
31.64x
62.90x
56.39x

PARCOSNET 4
98.44x
199.20x
175.06x

PARCOSNET 8
126.57x
325.96x
271.06x

PARCOSNET 12
147.66x
384.18x
323.19x

Table 6
6Average CPU occupancy in percentage for PARCOSNET to perform a CV cycle on every GO term. Optimal scalability is achieved when the occupancy reaches 100% × n, with n the number of CPU threadsMethod
Yeast
Mouse
Human

PARCOSNET
99%
100%
100%

PARCOSNET 4
345%
364%
368%

PARCOSNET 8
613%
659%
692%

PARCOSNET 12
815%
941%
975%

Table 7
7Average memory usage in GigaBytes (GBs) for COSNET and PARCOSNET when running cross-validation to predict GO termsMethod
Yeast
Mouse
Human

COSNET
0.40
3.73
3.26

PARCOSNET
0.27
1.13
0.94

Table 8
8Average CPU time in seconds and maximum memory consumption in GB for PARCOSNET to perform a single CV cycle on one class over the synthetic datasetsNumber of nodes

Density (σ )
500k
1000k
1500k

50
Time
45.3
137
330

Memory
1.94
3.86
5.81

100
Time
54.3
166
360

Memory
3.69
7.37
11.1

300
Time
92.9
248
609

Memory
10.7
21.4
32.1

This article has been published as part of BMC Bioinformatics Volume 19 Supplement 10, 2018: Italian Society of Bioinformatics (BITS): Annual Meeting 2017. The full contents of the supplement are available online at https:// bmcbioinformatics.biomedcentral.com/articles/supplements/volume-19supplement-10.
Authors' contributionsThe authors equally contributed to the development of the method, its implementation and in conducting the experiments. All the authors supervised the study and contributed to write the article, read and approved the final manuscript.Ethics approval and consent to participateNot applicable.Consent for publicationNot applicable.Competing interestsThe authors declare that they have no competing interests.Publisher's NoteSpringer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.Author details
UniProt: a hub for protein information. 10.1093/nar/gku989The UniProt Consortium. 43The UniProt Consortium. UniProt: a hub for protein information. Nucleic Acids Res. 2015;43(D1):204-212. https://doi.org/10.1093/nar/gku989.

. E Boutet, D Lieberherr, M Tognolli, M Schneider, P Bansal, A J Bridge, S Poux, L Bougueleret, I Xenarios, Edwards DSpringerNew YorkBoutet E, Lieberherr D, Tognolli M, Schneider M, Bansal P, Bridge AJ, Poux S, Bougueleret L, Xenarios I. In: Edwards D, editor. New York: Springer;

. 10.1007/978-1-4939-3167-5$_$22016, pp. 23-54. https://doi.org/10.1007/978-1-4939-3167-5$_$2.

Automated protein function prediction-the genomic challenge. I Friedberg, Brief Bioinform. 7Friedberg I. Automated protein function prediction-the genomic challenge. Brief Bioinform. 2006;7:225-42.

STRING v10: protein-protein interaction networks, integrated over the tree of life. D Szklarczyk, A Franceschini, S Wyder, K Forslund, D Heller, J Huerta-Cepas, 10.1093/nar/gku1003Nucleic Acids Res. 43D1Szklarczyk D, Franceschini A, Wyder S, Forslund K, Heller D, Huerta-Cepas J, et al. STRING v10: protein-protein interaction networks, integrated over the tree of life. Nucleic Acids Res. 2015;43(D1):447-52. https://doi.org/10.1093/nar/gku1003.

Guilt-by-association goes global. S Oliver, Nature. 403Oliver S. Guilt-by-association goes global. Nature. 2000;403:601-3.

A few good predictions: Selective node labeling in a social network. G Chaudhari, V Avadhanula, S Sarawagi, 10.1145/2556195.2556241Proceedings of the 7th ACM International Conference on Web Search and Data Mining. WSDM '14. the 7th ACM International Conference on Web Search and Data Mining. WSDM '14New YorkACMChaudhari G, Avadhanula V, Sarawagi S. A few good predictions: Selective node labeling in a social network. In: Proceedings of the 7th ACM International Conference on Web Search and Data Mining. WSDM '14. New York: ACM; 2014. p. 353-62. https://doi.org/10.1145/2556195. 2556241.

Semi-supervised learning using gaussian fields and harmonic functions. X Zhu, Z Ghahramani, J Lafferty, ICML. AAAI PressZhu X, Ghahramani Z, Lafferty J. Semi-supervised learning using gaussian fields and harmonic functions. In: ICML. AAAI Press; 2003. p. 912-19.

Fast protein classification with multiple networks. K Tsuda, H Shin, B Scholkopf, Bioinformatics. 212SupplTsuda K, Shin H, Scholkopf B. Fast protein classification with multiple networks. Bioinformatics. 2005;21(Suppl 2):59-65.

Learning with local and global consistency. D Zhou, O Bousquet, Navin Lal, T Weston, J Scholkopf, B , Advances in Neural Information Processing Systems. CambridgeMIT PressZhou D, Bousquet O, Navin Lal T, Weston J, Scholkopf B. Learning with local and global consistency. In: Advances in Neural Information Processing Systems. Cambridge: MIT Press; 2004. p. 321-28.

GeneMANIA: a real-time multiple association network integration algorithm for predicting gene function. S Mostafavi, D Ray, D W Farley, C Grouios, Q Morris, Genome Biol. 914SupplMostafavi S, Ray D, Farley DW, Grouios C, Morris Q. GeneMANIA: a real-time multiple association network integration algorithm for predicting gene function. Genome Biol. 2008;9(Suppl 1):4.

Global protein function prediction from protein-protein interaction networks. A Vazquez, A Flammini, A Maritan, A Vespignani, Nat Biotechnol. 21Vazquez A, Flammini A, Maritan A, Vespignani A. Global protein function prediction from protein-protein interaction networks. Nat Biotechnol. 2003;21:697-700.

Whole-genome annotation by using evidence integration in functional-linkage networks. U Karaoz, T M Murali, S Letovsky, Y Zheng, C Ding, C R Cantor, S Kasif, Proc Natl Acad Sci U S A. 101Karaoz U, Murali TM, Letovsky S, Zheng Y, Ding C, Cantor CR, Kasif S. Whole-genome annotation by using evidence integration in functional-linkage networks. Proc Natl Acad Sci U S A. 2004;101:2888-93.

Cosnet: A cost sensitive neural network for semi-supervised learning in graphs. A Bertoni, M Frasca, G Valentini, Proceedings of the. theBertoni A, Frasca M, Valentini G. Cosnet: A cost sensitive neural network for semi-supervised learning in graphs. In: Proceedings of the 2011

10.1007/978-3-642-23780-5_24European Conference on Machine Learning and Knowledge Discovery in Databases -Volume Part I. ECML PKDD'11. BerlinSpringerEuropean Conference on Machine Learning and Knowledge Discovery in Databases -Volume Part I. ECML PKDD'11. Berlin: Springer; 2011. p. 219-34. https://doi.org/10.1007/978-3-642-23780-5_24.

A neural network algorithm for semi-supervised node label learning from unbalanced data. M Frasca, A Bertoni, M Re, G Valentini, 10.1016/j.neunet.2013.01.021Neural Netw. 43Frasca M, Bertoni A, Re M, Valentini G. A neural network algorithm for semi-supervised node label learning from unbalanced data. Neural Netw. 2013;43:84-98. https://doi.org/10.1016/j.neunet.2013.01.021.

Automated gene function prediction through gene multifunctionality in biological networks. M Frasca, 10.1016/j.neucom.2015.04.007Neurocomputing. 162Frasca M. Automated gene function prediction through gene multifunctionality in biological networks. Neurocomputing. 2015;162: 48-56. https://doi.org/10.1016/j.neucom.2015.04.007.

Learning node labels with multi-category hopfield networks. M Frasca, S Bassis, G Valentini, 10.1007/s00521-015-1965-1Neural Comput & Applic. 276Frasca M, Bassis S, Valentini G. Learning node labels with multi-category hopfield networks. Neural Comput & Applic. 2016;27(6):1677-92. https:// doi.org/10.1007/s00521-015-1965-1.

A Neural Procedure for Gene Function Prediction. Smart Innovation, Systems and Technologies. M Frasca, A Bertoni, A Sion, 10.1007/978-3-642-35467-0$_$19Neural Nets and Surroundings: 22nd Italian Workshop on Neural Nets, WIRN 2012. Vietri sul Mare, Salerno, Italy; BerlinSpringerFrasca M, Bertoni A, Sion A. Neural Nets and Surroundings: 22nd Italian Workshop on Neural Nets, WIRN 2012, May 17-19, Vietri sul Mare, Salerno, Italy. A Neural Procedure for Gene Function Prediction. Smart Innovation, Systems and Technologies. Berlin: Springer; 2013, pp. 179-88. https://doi. org/10.1007/978-3-642-35467-0$_$19.

Exploiting indirect neighbours and topological weight to predict protein function from protein-protein interactions. H N Chua, W-K Sung, L Wong, 10.1093/bioinformatics/btl145Bioinformatics. 22Chua HN, Sung W-K, Wong L. Exploiting indirect neighbours and topological weight to predict protein function from protein-protein interactions. Bioinformatics. 2006;22:1623-30. https://doi.org/10.1093/ bioinformatics/btl145.

Molecular function prediction using neighborhood features. P Bogdanov, A K Singh, IEEE/ACM Trans Comput Biol Bioinforma. 7Bogdanov P, Singh AK. Molecular function prediction using neighborhood features. IEEE/ACM Trans Comput Biol Bioinforma. 2010;7: 208-17.

Partially labeled classification with Markov random walks. M Szummer, T Jaakkola, Advances in Neural Information Processing Systems (NIPS). CambridgeMIT Press14Szummer M, Jaakkola T. Partially labeled classification with Markov random walks. In: Advances in Neural Information Processing Systems (NIPS), vol. 14. Cambridge: MIT Press; 2001. p. 945-52.

The rendezvous algorithm: Multi-class semi-supervised learning with Markov random walks. A Azran, 10.1145/1273496.1273503Proceedings of the 24th International Conference on Machine Learning (ICML). the 24th International Conference on Machine Learning (ICML)New YorkACMAzran A. The rendezvous algorithm: Multi-class semi-supervised learning with Markov random walks. In: Proceedings of the 24th International Conference on Machine Learning (ICML). New York: ACM; 2007. p. 49-56. https://doi.org/10.1145/1273496.1273503.

Walking the interactome for prioritization of candidate disease genes. S Kohler, S Bauer, D Horn, P N Robinson, Am J Hum Genet. 824Kohler S, Bauer S, Horn D, Robinson PN. Walking the interactome for prioritization of candidate disease genes. Am J Hum Genet. 2008;82(4): 948-58.

RANKS: a flexible tool for node label ranking and classification in biological networks. G Valentini, G Armano, M Frasca, J Lin, M Mesiti, M Re, 10.1093/bioinformatics/btw23Bioinformatics. 32Valentini G, Armano G, Frasca M, Lin J, Mesiti M, Re M. RANKS: a flexible tool for node label ranking and classification in biological networks. Bioinformatics. 2016;32:2872-4. https://doi.org/10.1093/bioinformatics/ btw23.

Multitask protein function prediction through task dissimilarity. M Frasca, N Cesa-Bianchi, 10.1109/TCBB.2017.2684127IEEE/ACM Trans Comput Biol Bioinforma. 99In pressFrasca M, Cesa-Bianchi N. Multitask protein function prediction through task dissimilarity. IEEE/ACM Trans Comput Biol Bioinforma. 2017;99. https://doi.org/10.1109/TCBB.2017.2684127. In press.

You are who you know: Inferring user profiles in online social networks. A Mislove, B Viswanath, K P Gummadi, P Druschel, 10.1145/1718487.1718519Proceedings of the Third ACM International Conference on Web Search and Data Mining. WSDM '10. the Third ACM International Conference on Web Search and Data Mining. WSDM '10New YorkACMMislove A, Viswanath B, Gummadi KP, Druschel P. You are who you know: Inferring user profiles in online social networks. In: Proceedings of the Third ACM International Conference on Web Search and Data Mining. WSDM '10. New York: ACM; 2010. p. 251-60. https://doi.org/10.1145/ 1718487.1718519.

Node classification in social networks. S Bhagat, G Cormode, S Muthukrishnan, Social Network Data Analytics. BostonSpringer USBhagat S, Cormode G, Muthukrishnan S. Node classification in social networks. In: Social Network Data Analytics. Boston: Springer US; 2011. p. 115-148.

The class imbalance problem: A systematic study. N Japkowicz, S Stephen, Intell Data Anal. 65Japkowicz N, Stephen S. The class imbalance problem: A systematic study. Intell Data Anal. 2002;6(5):429-49.

Think globally and solve locally: secondary memory-based network learning for automated multi-species function prediction. M Mesiti, M Re, G Valentini, 10.1186/2047-217X-3-5GigaScience. 31Mesiti M, Re M, Valentini G. Think globally and solve locally: secondary memory-based network learning for automated multi-species function prediction. GigaScience. 2014;3(1):1-14. https://doi.org/10.1186/2047- 217X-3-5.

COSNet: An R package for label prediction in unbalanced biological networks. M Frasca, G Valentini, 10.1016/j.neucom.2015.11.096Neurocomputing. 237Frasca M, Valentini G. COSNet: An R package for label prediction in unbalanced biological networks. Neurocomputing. 2017;237:397-400. https://doi.org/10.1016/j.neucom.2015.11.096.

. Nvidia, Cuda, Guide, NVIDIA. CUDA programming guide. 2017. http://docs.nvidia.com/cuda. Accessed 8 July 2018.

Gene ontology: tool for the unification of biology. M Ashburner, C A Ball, J A Blake, D Botstein, H Butler, J M Cherry, Nat Genet. 251Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, et al. Gene ontology: tool for the unification of biology. Nat Genet. 2000;25(1):25-9.

ParCOSNet official repository. Anacletolab, AnacletoLab. ParCOSNet official repository. 2017. https://github.com/ AnacletoLAB/ParCOSNet. Accessed 30 Nov 2017.

Label Propagation and Quadratic Criterion. Y Bengio, O Delalleau, Le Roux, N , Semi-Supervised Learning. Chapelle O, Scholkopf B, Zien AMIT Press Scholarship OnlineBengio Y, Delalleau O, Le Roux N. Label Propagation and Quadratic Criterion. In: Chapelle O, Scholkopf B, Zien A, editors. Semi-Supervised Learning. MIT Press Scholarship Online; 2006. p. 193-216.

Neural networks and physical systems with emergent collective compatational abilities. J J Hopfield, Proc Natl Acad Sci. 798Hopfield JJ. Neural networks and physical systems with emergent collective compatational abilities. Proc Natl Acad Sci. 1982;79(8):2554-8.

Accelerating Large Graph Algorithms on the GPU Using CUDA. P Harish, P J Narayanan, S Aluru, M Parashar, R Badrinath, V K Prasanna, SpringerBerlin, HeidelbergHarish P, Narayanan PJ. In: Aluru S, Parashar M, Badrinath R, Prasanna VK, editors. Accelerating Large Graph Algorithms on the GPU Using CUDA. Berlin, Heidelberg: Springer; 2007, pp. 197-208.

A simple parallel algorithm for the maximal independent set problem. M Luby, Proceedings of the Seventeenth Annual ACM Symposium on Theory of Computing. STOC '85. the Seventeenth Annual ACM Symposium on Theory of Computing. STOC '85New YorkACMLuby M. A simple parallel algorithm for the maximal independent set problem. In: Proceedings of the Seventeenth Annual ACM Symposium on Theory of Computing. STOC '85. New York: ACM; 1985. p. 1-10.

UNIPred: unbalance-aware Network Integration and Prediction of protein functions. M Frasca, A Bertoni, G Valentini, 10.1089/cmb.2014.0110J Comput Biol. 2212Frasca M, Bertoni A, Valentini G. UNIPred: unbalance-aware Network Integration and Prediction of protein functions. J Comput Biol. 2015;22(12):1057-74. https://doi.org/10.1089/cmb.2014.0110.

An expanded evaluation of protein function prediction methods shows an improvement in accuracy. Y Jiang, T R Oron, W T Clark, A R Bankapur, D 'andrea, D Lepore, R , 10.1186/s13059-016-1037-6Genome Biol. 17184Jiang Y, Oron TR, Clark WT, Bankapur AR, D'Andrea D, Lepore R, et al. An expanded evaluation of protein function prediction methods shows an improvement in accuracy. Genome Biol. 2016;17(184) .https://doi.org/10. 1186/s13059-016-1037-6.

A neural network based algorithm for gene expression prediction from chromatin structure. M Frasca, G Pavesi, 10.1109/IJCNN.2013.6706954IJCNN. IEEEFrasca M, Pavesi G. A neural network based algorithm for gene expression prediction from chromatin structure. In: IJCNN. IEEE; 2013. p. 1-8. https://doi.org/10.1109/IJCNN.2013.6706954.