1College of Psychology and Child, Hanshin University, Osan, Republic of Korea.
2Department of Psychiatry, Chung-Ang University Hospital, Seoul, Republic of Korea.
3Department of Psychology, Korea University, Seoul, Republic of Korea.
Received 2016 February 20; Revised 2016 April 15; Accepted 2016 April 23.
This study aimed to develop and validate a Structured Clinical Interview for Internet Gaming Disorder (SCI-IGD) in adolescents.
First, we generated preliminary items of the SCI-IGD based on the information from the DSM-5 literature reviews and expert consultations. Next, a total of 236 adolescents, from both community and clinical settings, were recruited to evaluate the psychometric properties of the SCI-IGD.
First, the SCI-IGD was found to be consistent over the time period of about one month. Second, diagnostic concordances between the SCI-IGD and clinician's diagnostic impression were good to excellent. The Likelihood Ratio Positive and the Likelihood Ratio Negative estimates for the diagnosis of SCI-IGD were 10.93 and 0.35, respectively, indicating that SCI-IGD was ‘very useful test’ for identifying the presence of IGD and ‘useful test’ for identifying the absence of IGD. Third, SCI-IGD could identify disordered gamers from non-disordered gamers.
The implications and limitations of the study are also discussed.
Over the past decade, an increasing amount of research has been published regarding Internet Gaming Disorder (IGD). While preliminary in nature, it has been suggested that individuals suspected of IGD commonly display features of compulsive use, withdrawal, tolerance, and negative repercussions that characterize substance use disorders.1 Recent studies have also reported individuals demonstrating similar neurobio-psychosocial characteristics when screening for IGD and substance use disorders.2 However, there is considerable debate on the legitimacy of IGD as an independent clinical disorder because of the conceptual confusion and frequent appearance of IGD in the context of comorbid conditions. To establish its legitimacy, it is essential to develop an agreed-upon definition and accumulate data regarding its presentation across different ages and cultures, temporal stability, and mechanisms underlying its psychopathology.
Recently Petry et al.3 presented an international consensus related to diagnostic criteria for IGD in the Diagnostic and Statistical Manual for Mental Disorder, fifth edition (DSM-5), as a condition worthy of future study.4 The critical first step of specifying the consensus-based diagnostic criteria was taken in the gaming addiction field where the progress had been hindered by lack of a standard set of diagnostic criteria and no standardized assessment tool to measure IGD. Although Petry et al.3 paved the way for assessing IGD in some consistent manner, the appropriateness of the DSM-5 criteria, the best wordings to measure them, and the threshold for diagnosis are remained to be addressed. In order for IGD to be included as a separate mental disorder, robust empirical evidence needs to be accumulated to elucidate the conceptualization of IGD either as an addiction or not.
The clinical diagnosis of IGD comprises a cognitive and behavioral pattern encompassing persistent and recurrent use of Internet games, leading to significant impairment or distress over a period of 12 months as indicated by endorsing five or more out of nine criteria. The nine criteria for IGD include: 1) preoccupation with Internet games; 2) withdrawal symptoms when Internet gaming is taken away; 3) tolerance, resulting in the need to spend increasing amounts of time engaged in Internet games; 4) unsuccessful attempts to control participation in Internet games; 5) loss of interest in previous hobbies and entertainment as a result of, and with the exception of, Internet games; 6) continued excessive use of Internet games despite knowledge of psychosocial problems; 7) deceiving family members, therapists, or others regarding the amount of time spent participating in Internet gaming; 8) use of Internet games to escape or relieve negative moods; and 9) jeopardizing or losing a significant relationship, job, or education or career opportunity because of participation in Internet games. The IGD diagnostic criteria in the DSM-5, which are based upon an international consensus, have mostly been borrowed from substance use disorder or gambling disorder. Although these criteria are provisionally agreed characteristics for an IGD diagnosis among researchers,5 it is necessary to determine the diagnostic validity of each individual criterion by systematic investigation.
A recent review of instruments assessing game addiction reported that 18 different instruments had been developed and used in 63 studies.6 Despite excellent internal consistency and convergent validity, reviewed instrumentation showed a lack of consistent core addiction indicators, inconsistent cut-off points relating to clinical status, poor inter-rater reliability and predictability. Griffiths et al.7 strongly argued for a unified approach to assessment of IGD, which would enable comparisons across different demographic groups and different cultures. Since the introduction of IGD in the DSM-5 researchers have enthusiastically developed new diagnostic instruments, such as the Internet Gaming Disorder Scale5 or have modified preexisting instruments that were thought to reflect the nine criteria of IGD, such as the Video Game Dependency Scale8 and the Internet Gaming Disorder Test.9 These instruments are self-report measures that were designed to screen and classify possible cases of disordered gamers vs. non-disordered gamers.
Self-report questionnaires have some strength in that they are cost-efficient and easy to administer. However, they have some limitations. First, children and adolescents may find it difficult to concentrate on the long questions printed on the papers. Secondly, they may lack the awareness necessary to judge their own behavior in an accurate way. Thirdly, they may have difficulty in placing their own behavior in an appropriate time/duration context. For these reasons, a structured diagnostic interview has been strongly recommended for diagnosing psychiatric disorders of children and adolescents.1011 The same argument is very relevant in assessing and diagnosing IGD of children and adolescents especially because they tend to deny their problematic gaming or lack the awareness to judge their own behaviors. Therefore, it is of great demand to develop a structured diagnostic interview schedule for assessing IGD of adolescents.
Structured interview schedules have some advantages over open clinical interviews. Even with the DSM-5 diagnostic system, there can be substantial disagreement among raters when diagnosis is based on an open clinical interview. Clinicians often make an intuitive diagnosis without checking all of the diagnostic criteria. When they use the DSM-5 criteria, the order used to explore different criteria varies among clinicians and their interpretation of the criteria depends on their own clinical experience. Unlike open clinical interviews, structured diagnostic interviews are carefully linked to diagnostic criteria and the wordings and order of questions are predetermined. As a result, the inter-rater reliability is higher when using structured interview schedules because they are less susceptible to interviewer biases. Thus, development of a structured clinical interview has been greatly needed in this new field of IGD to assure that the criteria of the DSM-5 can be reliably assessed. The main aim of this study was to develop a structured clinical interview for adolescents to measure the nine IGD criteria from the DSM-5, and test the reliability and validity of the Structured Clinical Interview for Internet Gaming Disorder in the DSM-5 (SCI-IGD).
Another aim was to evaluate the diagnostic validity of nine individual criteria of IGD in the DSM-5. Although most of the proposed DSM-5 criteria of IGD were considered to adequately capture the phenomenon, some of criteria have become a focus of debate among researchers in the field.121314 So far, there have been some attempts to use a semi-structured interview to make a diagnosis of IGD in the DSM-5. Ko et al.15 recently evaluated the diagnostic validity of individual criteria of IGD in the DSM-5 using a diagnostic interview. It was reported that all criteria of IGD had diagnostic accuracy ranging from 77.3% to 94.7% except for the “deceiving” and “escape” criteria to differentiate university students with IGD from remitted students. van Rooij et al.16 also expanded the preexisting clinician-administered assessment tool (Clinical video game addiction test, C-VAT) to examine the sensitivity of nine DSM-5 criteria in a clinical youth sample and demonstrated that the C-VAT 2.0 correctly identified 91% of the sample using the proposed DSM-5 cut-off score. However, the specificity of the C-VAT 2.0 could not be examined because they did not include healthy gamers. Although these two studies provided some valuable information on the validity of the DSM-5 criteria, IGD diagnostic criteria in the DSM-5 need to be subjected to extensive psychometric testing using both community samples and clinical samples in order to establish good reliability and validity.
SCI-IGD was developed through three stages. The first stage of the study consisted of item generation. Authors defined IGD tentatively as a specific kind of behavioral addiction which not only shares similarities in presentation with substance use disorder and gambling disorder (e.g., loss of control, negative consequences) but also has features unique to IGD (e.g., irritability, health-related problems). Literature review and consultation with 8 experts who have substantial IGD-related clinical experience were made to establish a set of components for IGD workgroup. As a result, a total of 7 components such as preoccupation, salience, loss of control, tolerance, withdrawal, mood modification, and negative consequences, were selected. To develop items, items tapping the 7 components were oversampled from existing, psychometrically established instruments as well as suggested wordings from the DSM workgroup.31718192021 On examination the initial pool of items, items that were overlapping or had ambiguous meanings were deleted. To finalize items and phrasing of questions, discussion among authors and a consultation meeting with experts were made, which resulted in the preliminary SCI-IGD of 16 items assessing 6 components: preoccupation (included salience), withdrawal, tolerance, loss of control (DSM-5 criteria; ‘unsuccessful attempt to control’ and ‘continue despite problems’), mood modification (DSM-5 criteria; ‘escape’), negative consequences (DSM-5 criteria; ‘loss of interest’, ‘deceive’, ‘jeopardize’). In the second stage, the preliminary SCI-IGD was administered to a community sample of 28 middle school students with problems with gaming (19 males and 9 females) who agreed to participate in the interview. In order to examine the face validity of interview items, any discrepancy between responses to the interview items and the general impression was closely monitored. In this process, it was found that extra care should be taken when interviewees did not acknowledge the presence of problematic gaming. Because of ambiguous meanings, 4 items were excluded from the final version. Based on the preliminary testing of the SCI-IGD, a total of 12 items were selected as the final version of SCI-IGD.
The SCI-IGD allows for the assessment of DSM-5 Internet Gaming Disorder for the occurrence in the past 6 months.
The SCI-IGD is a comprehensive, fully standardized diagnostic interview primarily for use in epidemiological surveys and mental health research. The final version of the SCI-IGD was composed of two parts. The first part of the SCI-IGD was pre-diagnosis section composed of questions including demographical information and game use patterns. The second part of SCI-IGD was diagnostic interview section.
The SCI-IGD requires at least one of the one, two or three diagnostic questions to be acknowledged.
The final version of the SCI-IGD was administered to a total of 236 middle school students [mean age: 13.61 years (SD=0.87)] in Seoul, Korea [69 girls (29.3%), 167 boys (70.7%)]; 192 participants were recruited from five middle schools in Seoul and Gyeonggi province in Korea (in some schools, school administrators encouraged students with heavy game to participate in the study for the purpose promoting awareness, and 39 were sampled from Internet cafés where adolescents with severe Internet related problems usually spend the majority of their leisure time, and 5 in-patients who sought treatment for game related problems from ‘A’ University Hospital in Seoul. Participants were selected based on the following criteria: 1) they could attend a 20-min interview and 2) they could provide coherent responses to questions. Among 236 participants, 111 [mean age: 13.53 (SD=0.73); 27 girls (24.3%), 84 boys (75.7%); 93 from middle schools, 18 from Internet cafés] underwent two interviews to examine the diagnostic agreement; once by an interviewer using a SCI-IGD and once by a psychiatrist conducting an open clinical interview.
The Institutional Review Board (IRB) of ‘B’ University approved all procedures. In addition, all assessment sessions were conducted in private and by individuals blind to the findings of other interviews. The order of administration was counter-balanced. The mean duration of each interview ranged between 15 and 20 minutes. Informed consent was received from all participants and their parents prior to the interview; after which participants additionally completed self-report questionnaires. Each youth received a $10 gift certificate to buy books for their participation. For test-retest reliability, 16 participants, after having had their first SCI-IGD interview, were invited to a second independent identical SCI-IGD interview by a different interviewer, who was unaware of any findings from the first interview. They also were informed that they should not assume that symptoms indicated in the test interview would not need to be reported again in the retest interview. The mean time interval between each investigation in this study was approximately four weeks.
The two participating psychiatrists had extensive experience in the assessment and treatment of IGD at the Internet Game Addiction Counseling Center, which was affiliated with the department of psychiatry in ‘A’ University Hospital. To assess the reliability of the psychiatrist's diagnoses, kappa was calculated at the criteria and diagnostic level. The agreement between the two psychiatrists ranged from good to excellent, all of which ranged above 0.89.
Four doctoral level clinical psychologists with at least five years of trained clinical experience, and six graduate students supervised by doctoral level clinical psychologists administered each SCI-IGD. Prior to meeting with the participants, all interviewers were instructed in a 60 minute SCI-IGD education training. The agreement between the interviewers ranged from good to excellent with most above 0.89.
A K-scale was administered for the purpose of checking the concurrent validity of the SCI-IGD. K-scale consists of 40 items, each item is scored using a 4-point scale ranging from 1 (not at all) to 4 (always). Originally, there were three contributing factor subscales, such as subscales of disturbance of reality testing, automatic addictive thoughts, and virtual interpersonal relationships, as well as four symptom-related factor subscales such as subscales of daily life disturbance, deviant behavior, tolerance, and withdrawal. Koo et al.23 recently examined the diagnostic validity of the K-symptom scale, composing of 24 items from four symptom-related subscales and calculated the new diagnostic cut-off points. Cronbach's alpha of the K-scale was 0.96 in this study.
The Korean version of the BSI26 was administered to assess the depression and anxiety levels of the subjects. Subjects endorsed the relevance of each item to their experience in the past 7 days on a 5-point scale, from 0 (not at all) to 4 (extremely). The Cronbach's alpha for depression and anxiety subscale was 0.85 and 0.81 in the original validation study24 and 0.89 and 0.91 in the current study.
The Korean version of the SDQ27 was used to assess conduct problems, attention problems, and peer problems. It is composed of 25-items with 5-items in each of its five subscales, scored by using a 4-point scale from 0 (not at all) to 3 (extremely). The Cronbach's Alpha for conduct, attention, and peer problem subscales of the SDQ were from 0.50 to 0.80 in the Korean sample27 and from 0.70 to 0.87 in the current study.
The Korean version of DERQ29 was used to assess the emotion regulation ability. It has 36 items and is evaluated using a 5-point scale from 1 (almost never) to 6 (almost always). The Cronbach's Alpha for the DERQ was 0.93 in the Korean sample29 and 0.90 in the current study.
Table 1 summarizes all relevant socio-demographic information of the current sample. Twenty-three (11.0%, n=26) participants had indicated that their longest time spent on game play in a 24-hour period has been for more than 12 hours. Seventy four (31.4%) responded that they played games every day. Furthermore, most gamers reported they first began playing games at a very early age, typically before the age of 6 (15.3%, n=36), and between the age of 7–12 (69.9%, n=165).
Table 2 presents the sensitivity (Sen), the specificity (Spe), the positive likelihood ratio (LRP), and the negative likelihood ratio (LRN) estimates for the SCI-IGD at the criteria and diagnostic level for DSM-5. Among 111 participants, twelve (10.8%) were diagnosed with IGD according to the SCI-IGD [n=7 among 93 (7.5%) from schools; n=5 among 18 (27.8%) from internet cafes]. Among 12 diagnosed by SCI-IGD, eight (66.7%) were also diagnosed as IGD by the psychiatrist's clinical interview based on the DSM-5 of IGD. The LRP and LRN estimates for the final diagnosis of SCI-IGD were 10.93 and 0.35, respectively, indicating that the SCI-IGD was ‘very useful test’ for identifying the presence of IGD and ‘useful test’ for identifying the absence of IGD. In specific, most LRP of the SCI-IGD items were shown to be greater than 2, suggesting that they are useful for identifying the presence of diagnostic symptoms of IGD. Although the LRN of ‘withdrawal’ and ‘unsuccessful attempt to control’ items slightly exceeded 0.5, most LRN of the SCI-IGD items were below 0.5, demonstrating that the SCI-IGD items were useful for identifying the absence of the diagnostic symptoms of IGD. By contrast, the LRP and LRN of the 8th criterion (‘escape’) were below 2 and above 0.5, respectively, suggesting that the ‘escape’ item was proven to be ‘rarely useful’ for identifying the absence of ‘escape’ diagnostic symptom. It may have been resulted from the difficulty of the assessing the symptom because there were no participants who responded positive to the ‘escape’ criterion during the clinician's open interview, it warrants extra caution in interpreting this result.
The results showed that all diagnostic criteria had ‘moderate’ to ‘almost perfect’ agreement, with PABAK coefficients ranging between 0.41 and 0.91, An ‘almost perfect’ PABAK coefficient of 0.91 was obtained on the withdrawal and deceiving criteria, indicating that they might be quite consistent over the time period of approximately one month. On the other hand, ‘moderate’ PABAK coefficients of 0.44 were found for ‘unsuccessful attempts to control’ and ‘escaping a negative mood’ criteria, suggesting that these criteria might be relatively more sensitive to temporal or situational change than the other criteria.
All participants (n=236) were further divided into an IGD group (n=27) and non-IGD group (n=209) according to the SCI-IGD. Table 3 demonstrated that there were significant differences on the K-scale (F=45.34, p<0.001) and the K-symptom scale (F=44.37, p<0.001) between the IGD and non-IGD group. It is noteworthy that the mean on the K-symptom scale of the IGD group was found to be approximately equal to the diagnostic cutoff score (60.5) suggested by Koo and her colleagues (2015). Also, the IGD group had higher scores on depression (F=15.03, p<0.001), anxiety (F=12.80, p<0.001), conduct problems (F=16.75, p<0.001), attention problems (F=3.86, p<0.001), and difficulties in emotional regulation (F=3.93, p<0.05) than the non-disordered group assigned by SCI-IGD, except for peer relational problem (F=1.18, n. s.).
This study aimed to develop the SCI-IGD and examined its psychometric properties in adolescents using a community sample. It was demonstrated that SCI-IGD was found to be a quite valid and reliable tool to diagnose IGD in adolescents.
First, test-retest reliability as examined within a 4-week time interval showed significant estimates from moderate level to almost perfect level. This indicates that the SCI-IGD was found to be quite consistent over a long period of time, lasting at least one month. However, some estimates of PABAK coefficients between the two assessments were relatively low. For example, a relatively low PABAK coefficient of 0.44, albeit at moderate levels, was found for ‘unsuccessful attempts to control’ and ‘escaping a negative mood’ items. It might be attributable to the fact that this study used a considerably longer time interval of a month between assessments than other studies. It is also possible that some diagnostic items might be more sensitive to temporal or situational changes than other items. However, caution should be paid in interpreting these findings because of the small sample size.
Next, we examined the diagnostic accuracy of the SCI-IGD using the Likelihood Ratio because it is less affected by the prevalence rate. The SCI-IGD was proven to be a useful tool for identifying the presence and absence of IGD diagnosis assessed by the psychiatrist's clinical interview. At the diagnostic item level, the SCI-IGD showed an overall good ability for identifying the presence of diagnostic criteria of IGD. However, the LRN of the ‘withdrawal’ and ‘unsuccessful attempt to control’ slightly exceeded 0.5, which means that the diagnostic ability of these items is not quite useful for identifying the absence of these criteria. In other words, the items of SCI-IGD may have slightly high ‘miss' rates. This may have resulted from difficulties in drawing accurate reports from adolescents who have lack of awareness to recognize emotional or internal states of ‘withdrawal’ and ‘loss of control’ symptoms. There is also a possibility that most adolescents never attempted to reduce or stop gaming and therefore found it hard to answer questions to assess ‘withdrawal’ and ‘loss of control’ symptoms. Given the complex clinical nature of these criteria, it is also probable that more clarifying questions may be needed to assure a valid judgment. Future validation research should make more effort to reach and study clinical samples. Given the complex clinical nature of these criteria, it is also probable that more clarifying questions may be needed to assure a valid judgment. However, overall Likelihood Ratio estimates obtained from the other criteria were good, suggesting that the SCI-IGD interviewers are able to distinguish between the ‘normal’ and the ‘clinically significant experiences’. One strategy to improve the validity of this interview tool would be to provide interviewers with further training to promote understanding of the nature of the criteria and to address clarifying questions when needed. More generally, however, the tendency for structured diagnostic interviews to under- or over-diagnose compared to clinicians, has been well documented in the literature.35 This is due to the fact that clinicians are able to draw upon multiple sources of information and their own clinical experience in determining diagnoses.
In addition, the diagnostic ability of ‘escape’ symptoms criterion was demonstrated to be problematic, due to the fact that there was an extremely low base rate of it. There are several possibilities that could explain for the extremely low base rate for ‘escape’ diagnostic criterion. One possibility is related to external validity of the DSM-5 ‘escape’ diagnostic criterion. External validity of diagnostic criteria refers to their usefulness for distinguishing between patients on the basis of ‘gold standard’.36 However, until now, there have been very few empirical studies to evaluate the validity of the individual IGD diagnostic criteria of DSM-5. Ko and his colleagues15 examined the validity of IGD criteria for young adults and reported acceptable sensitivity, but relatively low diagnostic accuracy of the ‘deceiving’ and ‘escape’ criteria. It is possible that adolescents may have less awareness of their motivation of escape, compared to young adults. Another possibility is that the ‘escape’ criterion might be rarely endorsed in the community sample, while it might be easily identified in a clinical sample. This finding might also reflect that the ‘escape’ diagnostic criterion could not be one of the essential symptoms that identify Internet game addicts and further distinguish them from normal users, as other researchers also asserted.153738 It deserves further research to examine the validity of individual IGD criteria of the DSM-5.
The results also showed that those who are diagnosed as disordered adolescent gamers, according to the SCI-IGD, showed significantly higher scores on the K-scale, one of the most commonly used instruments in Korea to screen IGD in adolescents, indicating that SCI-IGD can validly differentiate disordered adolescent gamers from non-disordered adolescent gamers. It was also demonstrated that the disordered group assessed by the SCI-IGD were significantly different than the non-disordered group on several psychosocial variables, such as depression, anxiety, conduct and attention problems, and emotional dysregulation, which have all been known to be associated with IGD. By contrast, there was no significant difference on peer problems between the disordered group assessed by the SCI-IGD and the non-disordered group. It is consistent with the previous findings39 that peer problems are less associated with IGD than other factors.
Lastly, this study showed the relatively high prevalence (10.8%) of IGD prevalence compared to those being reported in previous studies.5 This relatively high prevalence can be attributable to the sampling process. As reported above in the ‘participant’ section, students in some middle schools participated in this study as part of the prevention and education process for their heavy game users, and some students were sampled from Internet cafés where adolescents with severe Internet related problems usually spend the majority of their time. Additional analysis showed that the prevalence rate varied according to the sampling sites ranging from 3.3% to 33.3%.
The limitations of this study were as follows. First, some analyses suffered from a relatively low base rate of IGD due to a relatively small community sample. Second, as excessive use of internet games among adolescents is of significant public health importance, this study aimed to validate the SCI-IGD for adolescents of age through 18. However, a fairly young sample of middle school students was recruited because we wanted to develop interview questions easy to understand for young adolescents and examine the reliability and diagnostic accuracy. As the pattern of adolescents' game usage was demonstrated to be similar across ages (Gentile 2009), it was assumed that the current findings on the reliability and validity of the SCI-IGD could be generalized to the older adolescents. However, in future studies, the current findings should be replicated using a larger sample with older participants.
In spite of these limitations, it is the first attempt to develop a diagnostic structured interview measure of well documented reliability and validity that offers 1) items that correspond closely to DSM-5 criteria; 2) binary statements on the presence/absence of disorder and each of its symptom criteria; and 3) sufficient simplicity to permit administration by a trained lay-interviewer. This newly developed structured clinical interview of IGD can fill the need for a psychometrically sound interview tool to assess IGD with more precision than the brief screening questionnaires. It will contribute to improve the accuracy of clinical diagnosis of IGD and enhance agreement among clinicians. It could also promote research to evaluate the prevalence, course, prognosis, and risk factors of IGD. Overall, the current study's findings lend empirical support for the concept of IGD suggested by the DSM-5 (APA, 2013). Although the crucial first step of reaching general consensus on the concept and diagnosis of IGD was taken, questions still remain to be addressed in future research about the nature and presentations of IGD at different stages or ages.
The National Information Society Agency (NIA), Korea, provided funding of this study. NIA had no role in the study design, collection, analysis or interpretation of the data, writing the manuscript, or the decision to submit the paper for publication.
1. Block JJ. Issues for DSM-V: internet addiction. Am J Psychiatry 2008;165:306–307. 18316427.
2. Kuss DJ, van Rooij AJ, Shorter GW, Griffiths MD, van de Mheen D. Internet addiction in adolescents: prevalence and risk factors. Comput Human Behav 2013;29:1987–1996.
3. Petry NM, Rehbein F, Gentile DA, Lemmens JS, Rumpf HJ, Mößle T, et al. An international consensus for assessing internet gaming disorder using the new DSM-5 approach. Addiction 2014;109:1399–1406. 24456155.
4. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders 5th Edth ed. Washington DC: Am Psychiatr Assoc; 2013.
5. Lemmens JS, Valkenburg PM, Gentile DA. The internet gaming disorder scale. Psychol Assess 2015;27:567–582. 25558970.
6. King DL, Haagsma MC, Delfabbro PH, Gradisar M, Griffiths MD. Toward a consensus definition of pathological video-gaming: a systematic review of psychometric assessment tools. Clin Psychol Rev 2013;33:331–342. 23396015.
7. Griffiths MD, King DL, Demetrovics Z. DSM-5 internet gaming disorder needs a unified approach to assessment. Neuropsychiatry 2014;4:1–4.
8. Rehbein F, Kliem S, Baier D, Mößle T, Petry NM. Prevalence of internet gaming disorder in German adolescents: diagnostic contribution of the nine DSM-5 criteria in a state-wide representative sample. Addiction 2015;110:842–851. 25598040.
9. Pontes HM, Király O, Demetrovics Z, Griffiths MD. The conceptualisation and measurement of DSM-5 internet gaming disorder: the development of the IGD-20 Test. PloS One 2014;9:e110137. 25313515.
10. Cohen P, Cohen J, Kasen S, Velez CN, Hartmark C, Johnson J, et al. An epidemiological study of disorders in late childhood and adolescence-I. Age-and gender-specific prevalence. J Child Psychol Psychiatry 1993;34:851–867. 8408371.
11. Flament MF, Whitaker A, Rapoport JL, Davies M, Berg CZ, Kalikow K, et al. Obsessive compulsive disorder in adolescence: an epidemiological study. J Am Acad Child Adolesc Psychiatry 1988;27:764–771. 3264280.
12. Griffiths MD, van Rooij AJ, Kardefelt-Winther D, Starcevic V, Király O, Pallesen S, et al. Working towards an international consensus on criteria for assessing internet gaming disorder: a critical commentary on Petry et al. (2014). Addiction 2016;111:167–175. 26669530.
13. Kardefelt-Winther D. A critical account of DSM-5 criteria for internet gaming disorder. Addict Res Theory 2015;23:93–98.
14. van Rooij A, Prause N. A critical review of “internet addiction” criteria with suggestions for the future. J Behav Addict 2014;3:203–213. 25592305.
15. Ko CH, Yen JY, Chen SH, Wang PW, Chen CS, Yen CF. Evaluation of the diagnostic criteria of internet gaming disorder in the DSM-5 among young adults in Taiwan. J Psychiatr Res 2014;53:103–110. 24581573.
16. van Rooij AJ, Schoenmakers TM, van de Mheen D. Assessment van gameverslaving in de klinischepraktijk met de C-VAT 2.0. Verslaving 2015;11:184–197.
17. Kim EJ, Lee SY, Oh SK. The validation of Korean Adolescent Internet Addiction Scale (K-AIAS). Korean J Clin Psychol 2003;22:125–139.
18. Ko CH, Yen JY, Chen CC, Chen SH, Yen CF. Proposed diagnostic criteria of internet addiction for adolescents. J Nerv Ment Dis 2005;193:728–733. 16260926.
19. Lee H, Ahn C. Development of the internet game addiction diagnostic scale. Korean J Health Psychol 2002;7:211–239.
20. Rehbein F, Kleimann M, Mediasci G. Prevalence and risk factors of video game dependency in adolescence: results of a German nationwide survey. Cyberpsychol Behav Soc Netw 2010;13:269–277. 20557246.
21. Tao R, Huang X, Wang J, Zhang H, Zhang Y, Li M. Proposed diagnostic criteria for internet addiction. Addiction 2010;105:556–564. 20403001.
22. National Information Society Agency. Third Standardization of Korean Internet Addiction Scale Seoul, Korea: National Information Society Agency; 2014.
23. Koo HJ, Cho SH, Kwon JH. A study for examining diagnostic ability of the K-Scale as a diagnostic tool for DSM-5 internet gaming disorder. Korean J Clin Psychol 2015;34:335–352.
24. Derogatis LR, Melisaratos N. The brief symptom inventory: an introductory report. Psychol Med 1983;13:595–605. 6622612.
25. Park KP, Woo SW, Chang MS. Validational study of brief symptoms inventory-18 in college students. Korean J Clin Psychol 2012;31:507–521.
26. Goodman R. The Strengths and Difficulties Questionnaire: a research note. J Child Psychol Psychiatry 1997;38:581–586. 9255702.
27. Ahn JS, Jun SK, Han JK, Noh KS, Goodman R. The development of a Korean version of the Strengths and Difficulties Questionnaire. J Korean Neuropsychiatr Assoc 2003;42:141–147.
28. Gratz KL, Roemer L. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J Psychopathol Behav Assess 2004;26:41–54.
29. Cho Y. Assessing emotion dysregulation: psychometric properties of the Korean version of the difficulties in emotion regulation scale. Korean J Clin Psychol 2007;26:1015–1038.
30. Attia J. Moving beyond sensitivity and specificity: using likelihood ratios to help interpret diagnostic tests. Aust Prescr 2003;26:111–113.
31. Manuel Porcel J, Vives M, Esquerda A, Ruiz A. Usefulness of the British Thoracic Society and the American College of Chest Physicians guidelines in predicting pleural drainage of non-purulent parapneumonic effusions. Respir Med 2006;100:933–937. 16253494.
32. Tacconelli E. Systematic reviews: CRD’s guidance for undertaking reviews in health care. Lancet Infect Dis 2010;10:226.
33. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159–174. 843571.
34. Hallgren KA. Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 2012;8:23–34. 22833776.
35. Wittchen HU, Semler G, von Zerssen D. A comparison of two diagnostic methods: clinical ICD diagnoses vs. DSM-III and research diagnostic criteria using the Diagnostic Interview Schedule (version 2). Arch Gen Psychiatry 1985;42:677–684. 4015309.
36. Merikangas KR, Dartigues JF, Whitaker A, Angst J. Diagnostic criteria for migraine. A validity study. Neurology 1994;44(6 Suppl 4):S11–S16. 8008221.
37. Charlton JP, Danforth ID. Validating the distinction between computer addiction and engagement: online game playing and personality. Behav Inf Technol 2010;29:601–613.
38. Gentile D. Pathological video-game use among youth ages 8 to 18: a national study. Psychol Sci 2009;20:594–602. 19476590.
39. Koo HJ, Kwon JH. Risk and protective factors of Internet addiction: a meta-analysis of empirical studies in Korea. Yonsei Med J 2014;55:1691–1711. 25323910.