Autism spectrum disorder (hereafter autism) is a life-long condition characterised by persistent differences in social communication and interaction alongside restricted and repetitive patterns of behaviour, interests or activities ([@bibr2-1362361319898331]). Studies in Asia, Europe, and North America have identified individuals with autism with an average prevalence of between 1% and 2%, diagnosed in three to four times as many males as females ([@bibr4-1362361319898331]). The impact on the economy from intervention costs, lost earnings, and care and support for children and adults with autism is estimated at £32 billion per year in the United Kingdom and \$175 billion per year in the United States ([@bibr10-1362361319898331]). Supports developed for autism have the potential to help members of the autistic community achieve better life quality, greater autonomy and inclusion ([@bibr9-1362361319898331]). This is best achieved through participatory research approaches which critically reflect on the current status of support with the autistic and broader autism communities, in order to identify how the field can develop more inclusively ([@bibr19-1362361319898331]; [@bibr40-1362361319898331]).

Reviews of the academic literature have shown that there is a growing number of digital supports for autistic individuals ([@bibr12-1362361319898331]; [@bibr22-1362361319898331]; [@bibr37-1362361319898331]; [@bibr41-1362361319898331]; [@bibr42-1362361319898331]; [@bibr43-1362361319898331]; [@bibr51-1362361319898331]; [@bibr52-1362361319898331]; [@bibr53-1362361319898331]; [@bibr54-1362361319898331]). Digital supports are defined as 'any electronic item/equipment/application/or virtual network that is used intentionally to increase/maintain, and/or improve daily living, work/productivity, and recreation/leisure capabilities' ([@bibr37-1362361319898331], p. 3806). In the past decade touchscreen, tangible and immersive digital technologies have become increasingly popular and accessible, with many autistic people and their parents/carers reporting high levels of digital technology use for supporting both leisure and academic pursuits ([@bibr30-1362361319898331]; [@bibr31-1362361319898331]; [@bibr32-1362361319898331]; [@bibr41-1362361319898331]; [@bibr49-1362361319898331]). Digital technologies have been developed to support autism in areas such as social skills and social interaction (for reviews see [@bibr11-1362361319898331]; [@bibr21-1362361319898331]; [@bibr43-1362361319898331]; [@bibr48-1362361319898331]) and emotion recognition (for review see [@bibr6-1362361319898331]). Digital technology, both in school and home settings, is being used in a variety of supportive ways such as increasing autonomy, reducing anxiety and increasing social opportunities for autistic people ([@bibr25-1362361319898331]), encapsulated by the term 'digital support'. To get an idea of how profuse development in this area is, one curated database of autism apps (<http://www.appyautism.com/en/>) lists over 400 apps for the iOS format alone.

Digital supports aim to facilitate a wide range of outcomes for autistic people, across a wide variety of ages ([@bibr53-1362361319898331]). Despite the extensive use of digital supports for autism, most digital supports available to the autistic community have little or no evidence to support their effectiveness ([@bibr13-1362361319898331]; [@bibr29-1362361319898331]). Studies reporting on the effects of digital supports are often in-depth case-study reports (e.g. [@bibr14-1362361319898331]; [@bibr23-1362361319898331]; [@bibr26-1362361319898331]; [@bibr33-1362361319898331]; [@bibr34-1362361319898331]; [@bibr39-1362361319898331]). Beginning with 29,105 potential articles, [@bibr53-1362361319898331] identified 27 focused intervention practices for autism that met their inclusion criteria. An intervention practice met the level of research evidence necessary to be included if it was supported by (1) two high-quality experimental or quasi-experimental design studies conducted by two different research groups, or (2) five high-quality single case design studies conducted by three different research groups and involving a total of 20 participants across studies or (3) a combination of research designs that must include at least one high-quality experimental/quasi-experimental design, three high-quality single case designs and was conducted by more than one researcher or research group ([@bibr53-1362361319898331]). Using similar inclusion criteria for research evidence, [@bibr30-1362361319898331] identified 29 studies that met these inclusion criteria; however, of these studies only three single-subject studies and no group studies met the criteria for quality of research evidence. [@bibr22-1362361319898331] conducted a meta-analysis of digital technology supports for autism and identified only 22 out of 379 (6%) using pre-post group design studies. Studies were included based on criteria for quality of evidence that took into account participants' diagnoses, outcome measures and interaction with digital technology. Of the 22 pre-post group studies, only 10 followed a randomised controlled design (2.6% of the initial sample). The analysis of efficacy on these 10 studies provided evidence for a beneficial effect of technology-based training for autistic children overall, irrespective of age and intelligence quotient (IQ). The effect size was in the small-to-medium range, with a significant heterogeneity among studies. Together, these reviews indicate that digital supports can be effective for autism, but only a very small proportion of the research evidence from group or single case designs is of sufficient quality to permit an informed decision on whether to use the digital support.

Under these circumstances, using evidence to make an informed choice about supports for members of the autistic community and broader autism community (including professionals and researchers) is challenging. The digital supports that do have published evidence of effectiveness are frequently developed in research projects, and are rarely made available to the autistic consumer ([@bibr13-1362361319898331]). [@bibr17-1362361319898331] argue that research findings and related expert opinions represent only one source of potential information influencing support-related decisions in health professions and sciences in general. Other sources of potential evidence include (1) personal (e.g. own experience and expertise as well as recommendations from peers), (2) professional practice guidelines (e.g. clinical recommendations) and (3) personal values and preferences (alongside societal values and norms). In addition, online information, from a product website to digital store reviews, may be a potential source of evidence for digital supports, especially when these are commercially available. Such sources of evidence have varying degrees of perceived independence, and relevance to the consumer's priorities. The abundance of digital supports for autism and the lack of research evidence makes it difficult for the autistic and broader autism communities to select the most appropriate digital support for their needs. A framework to evaluate the effectiveness of digital supports from multiple sources of potential evidence is urgently required to support evidence-based practice (EBP) and is the aim of this study.

EBP describes the integration of the best available research, clinical expertise, patient values and circumstances and healthcare system policies ([@bibr16-1362361319898331]; [@bibr47-1362361319898331]). EBP has its origins within medicine, and clinical expertise refers to both the clinician's individual knowledge acquired through clinical experience and practice, as well as external clinical evidence based on relevant scientific research using the best available methodologies, with meta-analyses and randomised controlled trials (RCTs) being considered 'gold standard' methods ([@bibr47-1362361319898331]). This is integrated with the values and circumstances within the context of broader healthcare policy to ensure that practice is evidence-based. If a practice is not evidence-based, it risks being inapplicable to a specific patient, out of date or even potentially harmful to the patient.

Efforts have been made to extend EBP to primary care and specialised clinics for autism ([@bibr3-1362361319898331]), but defining EBP is not straightforward in this context ([@bibr35-1362361319898331]). [@bibr17-1362361319898331] outline issues faced by professionals when trying to apply evidence-based practice. For instance, their routine clinical practice is often remote from the controlled circumstances in which an RCT is conducted. Specifically, patients often have specific comorbidities (e.g. attention deficit hyperactivity disorder (ADHD), intellectual disabilities) that do not match the inclusion criteria for an RCT. In autism, other factors include unexpected life events occurring during the evaluation period that affect the outcome of the support and, specifically for autism, the heterogeneity of the condition ([@bibr35-1362361319898331]). Criteria for EBP in autism support have been proposed by [@bibr45-1362361319898331] which take into account both group designs and single case designs. Strong, adequate or weak judgements can be made concerning the rigour of the underlying research based on common primary quality indicators, including clear and reproducible accounts of participant characteristics, dependent measures and independent variables (in addition to secondary quality indicators including inter-observer agreement, blind raters, procedural fidelity, generalisation and maintenance and social validity). Depending on the quality and quantity of the research (e.g. see the inclusion criteria described by [@bibr53-1362361319898331], above), EBP for a support for autism can then be classified as established or promising (see [@bibr44-1362361319898331]; [@bibr45-1362361319898331]). Such criteria are particularly valuable in the context of digital supports as they are more able to take account of the range of possible uses of technology, the multitude of ways technology can be personalised and the variety of potential outcome measures for digital supports. Evaluating rapidly developing technology-based supports in RCTs is difficult, given the mismatch between the timelines of commercial and academic progress ([@bibr18-1362361319898331]).

EBP is the integration of best available research and clinical expertise with patient values. Integrating the values and opinions of the autism community within the consideration of the research evidence is therefore essential to EBP (see also [@bibr19-1362361319898331]; [@bibr40-1362361319898331]). This study aimed to co-develop a framework for evaluating the evidence base for digital supports for autism, through better understanding of what constitutes evidence for the autistic and broader autism communities and what sources are being used to obtain that knowledge when considering digital supports for autism. We used an online, four-round Delphi study methodology, ideal for integrating the perspectives of multiple stakeholders ([@bibr24-1362361319898331]), with feedback managed by a moderator at all stages ([@bibr50-1362361319898331]). The Delphi study methodology was selected as it has been proposed to be more effective for group-based judgement and decision-making than traditional group meetings by both increasing a group's access to multiple interpretations and views and decreasing the negative features of group discussions such as domineering individuals and opinions ([@bibr5-1362361319898331]; [@bibr24-1362361319898331]; [@bibr46-1362361319898331]; see also [@bibr28-1362361319898331]). The Delphi methodology was therefore chosen as an ideal format for systematically capturing and integrating opinion from a diverse group of experts, who were not co-located and remained anonymous from each other ([@bibr20-1362361319898331]; [@bibr27-1362361319898331]). Since the method allows each individual to contribute anonymously and in their own time, the study allowed us to accommodate different communication preferences that do not include face-to-face communication and to avoid direct confrontation between people of differing opinions. Allowing participants to contribute at their own pace without having to manage live group discussions therefore made it easier to include autistic individuals.

Methods {#section2-1362361319898331}
=======

Panel members {#section3-1362361319898331}
-------------

Four key groups of stakeholders were identified: (1) autistic people, (2) families of autistic people, (3) professionals who support autistic people and (4) researchers -- all with experience of using or developing digital supports for autism and advising others on the topic (see [Table 1](#table1-1362361319898331){ref-type="table"}). The literature recommends between 15 and 30 panel members ([@bibr24-1362361319898331]; [@bibr38-1362361319898331]), and we aimed for 10 participants from each of our stakeholder subgroups. We contacted members of our networks directly and invited them to take part or to recommend another expert if they were unable to participate. We only contacted people that met our inclusion criteria as an 'expert' and we asked those who were referred by other people to confirm they met these criteria. We defined 'experts' as people with the necessary experience with technology for autism to advise others. All potential participants completed a brief questionnaire detailing their experience with digital interventions for autism. Experts were therefore those who reported that they had experience using and advising others on technologies for autism and could therefore be researchers, practitioners and/or members of the autism community. As the needs of the autistic community and their immediate providers of support were of primary importance to this study, researchers were not included in the first two Delphi study rounds (see [Table 1](#table1-1362361319898331){ref-type="table"}), but joined at the mid-point to help refine statements on evidence. Panel members were recruited on the basis of recommendations from autism networks and associations internationally, especially those relevant to digital technology for supporting autism (e.g. [www.asdtech.ed.ac.uk](http://www.asdtech.ed.ac.uk)). Panel members were recruited through personal invitations to experts from autism-related networks in different countries for relevant experts: Asociación Española de Profesionales del Autismo AETAPI (Spain), Autism Speaks (United States), Research Autism (United Kingdom) and Centres Ressources Autisme (France). The inclusion criteria were that panel members were adults, fluent in English and that autistic panel members had formal evidence of diagnosis. As a screener, we asked all potential panel members to discuss their knowledge and experience with digital technology including (but not limited to) touchscreen tablets, smartphones, gaming devices, computers, robots and argumentative and alternative communication (AAC) devices.

###### 

Number of panel members per round.

![](10.1177_1362361319898331-table1)

  Round   Community members   Researchers   Total per round        
  ------- ------------------- ------------- ----------------- ---- ----
  1       6                   8             13                --   27
  2       6                   7             11                --   24
  3       5                   2             6                 12   25
  4       5                   2             5                 11   23

The age range of the community group was between 22 and 72 years (mean = 38.76 SD = 12.35) including 8 males and 17 females. In the researchers' group there were 9 males and 3 females and no information on age was given. The sample was recruited from the United Kingdom (21), France (6), United States (3), Spain (3), Israel (3) and one each from Germany, Austria and Ireland. Some panel members fulfilled the criteria for multiple groups (e.g. autistic practitioners) but are only listed in one group here -- as selected by themselves.

Procedure {#section4-1362361319898331}
---------

The study was conducted using an online survey software ([www.qualtrics.com](http://www.qualtrics.com)) over four rounds ([Table 2](#table2-1362361319898331){ref-type="table"}). A literature review was conducted on EBP for digital supports for autism ([@bibr54-1362361319898331]) providing information about the goals of existing digital supports. These informed the design of the first round of the Delphi study, providing context for panel members to consider how they may seek sources of potential evidence. Panel members' comments and ratings in each round were collected and analysed by the moderator (first author), and used to create content for the following round.

###### 

Goals and panel members in each round.

![](10.1177_1362361319898331-table2)

  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  Round   Description                                                                                                                             Panel members
  ------- --------------------------------------------------------------------------------------------------------------------------------------- -------------------------------
  1       Brainstorming: open enquiry about the reasons for using digital supports and the kind of information used to select digital supports.   Family members\
                                                                                                                                                  Professionals--practitioners\
                                                                                                                                                  Autistic people

  2       Categorisation of evidence: organise evidence in categories, locate new types of evidence                                               Family members\
                                                                                                                                                  Professionals--practitioners\
                                                                                                                                                  Autistic people

  3       Drafting the framework: ranking and editing lists of statements about evidence                                                          Family members\
                                                                                                                                                  Professionals--practitioners\
                                                                                                                                                  Autistic people\
                                                                                                                                                  Researchers

  4       Finalisation: ranking a selection of the statements and final modifications in wording                                                  Family members\
                                                                                                                                                  Professionals--practitioners\
                                                                                                                                                  Autistic people\
                                                                                                                                                  Researchers
  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

### Round 1 -- Brainstorming {#section5-1362361319898331}

In Round 1 the panel answered open-ended questions (see [Supplemental material, Appendix I](https://journals.sagepub.com/doi/suppl/10.1177/1362361319898331)) about their goals and sources of evidence when selecting a digital support (as defined above). They were asked to think about recent experiences when choosing or recommending a digital support intended for an autistic person (potentially including themselves). A thematic analysis was performed on panel responses, and illustrative quotes were selected ([@bibr8-1362361319898331]). This identified recurrent themes pertaining to the purpose of digital supports and the outcomes which are sought when using digital supports. We also identified potential sources of information that panel members detailed in regard to these purposes and outcomes.

### Round 2 -- categorisation {#section6-1362361319898331}

The panel was asked to rate potential sources of information identified during round 1 using 5-point Likert-type scales for the following dimensions:

-   Relevance: whether information from this source is likely to relate to their situation;

-   Importance: whether information from this source is likely to be of high quality;

-   Usefulness: whether information from this source is likely to make a difference to their decisions/actions;

-   Accessibility: whether information from this source is likely to be easy to find and understand.

We also aimed to refine the list of features and outcomes of digital support the panel may want evidence for and to match sources of evidence to these features/outcomes. The list of features/outcomes was derived from comments and illustrative quotes made during round 1. The panel were presented with features/outcomes beginning with the phrase 'You want to know whether . . .' (see [Table 3](#table3-1362361319898331){ref-type="table"}) and asked to list the sources of information they would use to find out specifically about those features/outcomes.

###### 

Desired features and outcomes of a piece of technology.

![](10.1177_1362361319898331-table3)

  ------------------------------------------------------------------------------------------------------
  • The product has ongoing tech support from the development team
  • Special interests of autistic people are taken into consideration in the product design
  • The product encourages original creations
  • The product is aesthetically pleasant
  • The product is easy to use
  • The product is customisable
  • The product is easy to find and order/buy
  • The product can be used autonomously by the autistic person
  • The product helps the autistic person to be more autonomous in their life
  • The product contributes to a better life quality for the family/carers of the autistic person
  • The product is amusing/entertaining
  • The product helps the autistic person develop new skills/improve existing skills
  • The product encourages social interaction between the autistic user and other people
  • The effects of its usage are long-lasting
  • The autistic user can generalise the skills they acquired via the technology in different contexts
  • The product achieves better results than similar products that are not technology-based
  • The product matches the *abilities* of the autistic user
  • The product matches the needs of the autistic user
  • The product is age-appropriate for the autistic user
  • There are opportunities to try out the product before buying it
  ------------------------------------------------------------------------------------------------------

Third, in an open commentary the panel members were asked to discuss whether their personal experience was similar to specific quotes from the panel's responses in the previous round. Those were selected to match sources of information proposed in the first round ([Table 4](#table4-1362361319898331){ref-type="table"}).

###### 

Quotes from panel members.

![](10.1177_1362361319898331-table4)

  ------------------------------------------------------------------------------------------------------------------------
  'I need a piece of technology to help me keep track of anxiety and offer suggestions and tips based on my experiences'
  'I would appreciate being able to buy harder levels or aspects of a game'
  'I'd look whether this technology is approved by several scientific communities specialised in autism'
  'I'd only rely on my personal judgement resulting from hands on experience'
  'As there were bugs in the app the person got a bit angry with it and stopped using it'
  'The technology we currently have does keep him entertained and occupied'
  'He uses elsewhere the things he has learnt with the app'
  ------------------------------------------------------------------------------------------------------------------------

Mean ratings for relevance, importance, usefulness and accessibility for each source of information were computed. Using thematic analysis, codes representing desired features and outcomes of digital supports were clustered into sub-themes and then top-level themes by two raters independently (first two authors). The themes were reviewed, validated and, if necessary, revised by two other independent raters (last two authors). In the first round, to gather input from the community we made an open-ended enquiry regarding the kind of evidence they seek when considering whether to use, or recommend that someone else use, a new technology. Specific examples of evidence were requested as illustrations of this. Analysis of these responses culminated in three high-level categories of evidence: 'engagement' (how the user experiences the product itself, its ease of use and attractiveness), 'effectiveness' (outcomes reached/directly observed changes) and 'reliability' (the technology is functional). The resulting output was composed of statements on potential sources of evidence grouped into three high-level categories: that is, evidence for reliability, evidence for engagement and evidence for effectiveness. This constituted the basis of what was used in round 3 to create the first draft of the framework.

### Round 3 -- refinement {#section7-1362361319898331}

Round 3 integrated the perspectives of autistic community, families and professionals with researchers. The expanded panel were asked to rank and edit the statements that had emerged from Round 2 regarding what constitutes evidence. They were given the opportunity to remove statements that they thought were inappropriate or irrelevant. They were told that not all statements would make it to the final framework and that they should prioritise statements that they would want to see appear in the final framework. The moderator merged revisions that were similar, yielding a list of ranked statements for each category of evidence. For the framework to be easy to use, the number of statements per category was restricted. Hence, only the five most highly ranked statements in each category were maintained.

### Round 4 -- finalisation {#section8-1362361319898331}

The panel was required to review and, if necessary, revise each statement in a draft framework. They were given three possibilities for each statement: (1) accept it as is, (2) make adjustments and (3) remove it from the framework. They were required to justify their edits when they chose to make adjustments or remove a statement. The panel was also asked to signal any 'words of caution' concerning the finalised framework. They ranked the five top statements from 1 to 5 with 1 being the most important source of evidence for them.

The moderator merged the edits suggested by the panel when they were similar and then classified them and responded as listed below. The classification was reviewed by two independent coders (last two authors). In case of disagreement between them, consensus was achieved through discussion.

1.  Amendment: This is clarification or expansion of the scope of the statement without fundamentally changing it. For each amendment, two independent coders gave a score from 1 to 3:Should be integrated into the statement;Neutral stance regarding integration in the statement;Need not to be integrated.

To integrate an amendment, it had to have a mean score of less than 2.

1.  2\. Words of caution: These are important risks or constraints associated with the statement to an extent that they should be acknowledged in conjunction with the statement. These were adjoined to the statements or category of evidence they were associated with (see [Supplemental material, Appendix II](https://journals.sagepub.com/doi/suppl/10.1177/1362361319898331)).

2.  3\. Rejections: This is when a panel member opposed a statement, or criticised major aspects of it. A threshold of 90% agreement (i.e. fewer than 10% of the panel rejected a specific statement) was set for inclusion of statements in the final list, following the emerging convention in Delphi studies ([@bibr1-1362361319898331]).

3.  4\. Misunderstandings: Comments that appeared to be unrelated to the statement. The statement was double-checked and reworded for clarity if needed.

Results {#section9-1362361319898331}
=======

Round 1 {#section10-1362361319898331}
-------

The desired outcomes of digital support for the autistic and autism communities that emerged from the panel's responses primarily related to autonomy, time awareness and management, enhanced quality of life for family/carers, better communication, social participation, fun/leisure, learning support, creativity and enhanced cognitive skills. The *features of* digital support sought by the autistic and autism communities revolved mainly around the products being reliable (bug-free, tech support and battery life) and how the products look and feel (clear instructions, scaffolding and progress awareness, customisability, attractive design, user control and reward system). [Table 5](#table5-1362361319898331){ref-type="table"} summarises the desirable features and outcomes that were derived from the thematic analysis.

###### 

Desirable features and outcomes for digital technology derived from thematic analysis.

![](10.1177_1362361319898331-table5)

  Features                                                Outcomes
  ------------------------------------------------------- ------------------------------------------
  Technical support                                       Encouraging original creativity
  Bug-free                                                Encouraging social interactions
  Aesthetically pleasing                                  Amusing/entertaining
  Ease of use                                             Autonomy
  Customisation                                           Better quality of life for family/carers
  Accessible and affordable                               Generalisability of learnt skills
  Adapted to autistic user's special interest and needs   Long-term effectiveness
  Age-appropriate                                         
  Progressive levels of difficulty                        

Sources of information with regard to those features and outcomes were reviews and recommendations specifically from the autism community, personal hands-on experience and direct observation, expertise of the design team, involvement of autistic users in the design, scientific evidence and non-specific online reviews.

Round 2 {#section11-1362361319898331}
-------

Six potential sources of information relevant to choosing and using technologies were rated for relevance, importance, usefulness and accessibility (see [Table 6](#table6-1362361319898331){ref-type="table"}).

###### 

Ratings for source of information derived from round 1 according to four parameters, ranked from the highest to the lowest mean score over all dimensions.

![](10.1177_1362361319898331-table6)

  Source of information                                                              Mean position (SD)                               
  ---------------------------------------------------------------------------------- -------------------- ------------- ------------- -------------
  Positive online reviews specifically from the autistic community                   4.4 (0.70)           4.5 (0.76)    4.46 (0.71)   4.29 (1.02)
  Expertise of the product's development/design team                                 3.25 (1.48)          3.75 (1.13)   3.71 (1.31)   3.96 (1.17)
  Observable positive changes in the autistic user's behaviour                       4.25 (1.05)          4.21 (1.08)   4.13 (1.05)   4.25 (0.88)
  The product's development/design team specifically includes autistic people        4.04 (1.27)          4.13 (0.88)   4.08 (0.95)   3.88 (1.17)
  Academic research                                                                  3.88 (0.93)          3.75 (0.92)   3.79 (0.82)   3.50 (1)
  Positive online reviews (e.g. Amazon stars, comments on product's Facebook page)   3.00 (1.15)          3.08 (1.04)   3.04 (0.89)   3.33 (1.11)

Thematic analysis of panel responses produced three high-level categories of for which evidence might be required, defined as follows:

1.  *The product is reliable*: The efficacy of a product at the level of engineering. Is it technically sound/functional? How well does it work? For example, does the face recognition functionality actually work? Does the app crash often?

2.  *The product is engaging*: The user perception of the technology. How usable, agreeable, pleasant and accessible a product is for the specific users? Its ease of use/look and feel.

3.  *The product is effective*: The outcome of using the product. How much impact does it have for the people using it? Does it make an observable difference in the user's life/behaviour?

Round 3 {#section12-1362361319898331}
-------

The sources of evidence used for these three categories are summarised in [Table 7](#table7-1362361319898331){ref-type="table"}, followed by descriptions that were summaries of comments that appeared across panel members and groups.

###### 

Statements ranked in the top five positions per category of evidence.

![](10.1177_1362361319898331-table7)

  Reliability
  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   *Try it out.* You might request a trial version from the developer, or borrow a copy/device from a friend. Take your time to explore all the functionalities. Bear in mind that a trial version might differ from the full version.
   *Get an expert opinion.* Ask people you know who have skills and experience with technology, or read official documentation provided by agencies such as a government council on technology standards.
   *Read online reviews*. Look on app review sites and social media. Include reviews from autistic users and their families and pay attention to long-time users. Keep in mind that reviewers' circumstances (e.g. their needs, goals or budget) may not be the same as yours.
   *Seek academic opinions.* Read an academic article evaluating a product's design. You might also see scientists writing in the mainstream media or find a talk given online. Check the academic's relevant qualifications, affiliations and potential conflicts of interest when you decide how much trust to put in them.
   *Consult the company's website.* While this information does not constitute independent evidence, the technical description of the product and the kind of technical support may be informative. You can also look for tech industry accreditations such as kite marks, badges, ISO norms and so on.
  Engagement
   *Try it out.* You might request a trial version from the developer, or borrow a copy/device from a friend. Take your time to explore all the functionalities. Bear in mind that a trial version might differ from the full version
   *Read online reviews.* Include reviews from autistic users and their families and pay attention to long-time users. Keep in mind that reviewers' circumstances (e.g. their needs, goals or budget) may not be the same as yours.
   *Get an expert opinion.* Ask people you know who have skills and experience with technology. Talk to relevant professionals such as a teacher or speech and language therapist.
   *Consult review websites.* Look for sites that compare different technologies and search for case studies.
   *Seek academic opinions.* Read an academic article evaluating a product's design. You might also see scientists writing in the mainstream media or find a talk given online. Check the academic's relevant qualifications, affiliations and potential conflicts of interest when you decide how much trust to put in them.
  Effectiveness
   *Read an academic paper*. Ideally look for a review that systematically combines the results from multiple independent studies. It may be worth checking the quality of the original studies too and the journals where they were published.
   *Get an expert opinion.* Talk to relevant professionals such as a teacher or speech and language therapist. Ask people you know who have skills and experience with technology, or take the advice of reference centres. Read official documentation provided by agencies such as a government council on technology standards.
   *Read online reviews.* Include reviews from autistic users and their families and pay attention to long-time users. Keep in mind that reviewers' circumstances (e.g. their needs, goals or budget) may not be the same as yours.
   *Try it out.* You might request a trial version from the developer, or borrow a copy/device from a friend. Take your time to explore all the functionalities. Bear in mind that a trial version might differ from the full version.
   *Search online for expert perspectives.* Read a media article on the technology written by a scientist or listen to a talk given by an academic. Check the academic's relevant qualifications, affiliations and potential conflicts of interest when you decide how much trust to put in them. Consult websites that review and compare technologies, and search for relevant case studies.

Round 4 {#section13-1362361319898331}
-------

In this final round the panel had to edit and rank statements in each category of sources of evidence. The statements that had the highest mean ratings as a source of evidence were similar for the three categories ([Table 8](#table8-1362361319898331){ref-type="table"}).

###### 

Ranking of statements for all categories.

![](10.1177_1362361319898331-table8)

  -------------------------------------------------------------------------------------------------------------
  Statements                              Reliability ranking\   Engagement ranking\   Effectiveness ranking\
                                          Mean (SD)              Mean (SD)             Mean (SD)
  --------------------------------------- ---------------------- --------------------- ------------------------
  Try it out                              1.43 (0.82)            1.39 (0.87)           3.39 (1.69)

  Get an expert opinion                   2.57 (0.88)            2.87 (1.08)           2.48 (1.02)

  Read online reviews                     2.87 (1.19)            2.74 (0.85)           3.09 (1.14)

  Seek academic opinions                  3.43 (0.97)            4.13 (1.12)           *Not featured*

  Consult the company's website           4.70 (0.69)            *Not featured*        *Not featured*

  Consult review websites                 *Not featured*         3.87 (1.19)           *Not featured*

  Read an academic paper                  *Not featured*         *Not featured*        2.26 (1.45)

  Search online for expert perspectives   *Not featured*         *Not featured*        3.78 (1.06)
  -------------------------------------------------------------------------------------------------------------

Of the 23 panel members, 3 (more than 10%) rejected the statements shown in [Table 9](#table9-1362361319898331){ref-type="table"}, so they were excluded. The final framework statements reaching inclusion for consensus by the autistic and autism communities as well as researchers are listed in [Table 10](#table10-1362361319898331){ref-type="table"}, with the agreed explanatory text.

###### 

Statements that were removed from the framework.

![](10.1177_1362361319898331-table9)

  Category of desirable evidence   Removed statements
  -------------------------------- ---------------------------------------
  Reliability                      Consult the company's website
  Engagement                       Consult review websites
                                   Seek academic opinions
  Effectiveness                    Search online for expert perspectives

###### 

An evidence-based framework for digital supports for autism.

![](10.1177_1362361319898331-table10)

  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  How to select digital supports for autistic users: an evidence-based framework
  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
   1. Try it out\
  You might request a trial version from the developer, or borrow a copy/device from a friend. Take your time to explore all the functionalities. Ask how the trial version differs from the full version.

   2. Get an expert opinion\
  Talk to relevant professionals (e.g. a specialist teacher, speech and language therapist, specialist psychologist, etc.). Ask (autistic) people, organisations or agencies you know who have specialist skills and relevant experience with technology.

   3. Read online reviews\
  Look on app review websites and social media. Include reviews from autistic users and their families and pay attention to people that have been using the product for a (relatively) long time. Read and compare as many reviews as possible to improve objectivity. Keep in mind that reviewers' circumstances (e.g. their needs, age, goals or budget) may not be the same as yours and individual experiences may not be generalisable.

   4. Seek academic opinions\
  Read an academic article evaluating the product, or find an article/online talk in the mainstream media by a qualified scientist. Check the academic's relevant qualifications, affiliations and potential conflicts of interest when you decide how much trust to put in them.

  Is it engaging?

   1. Try it out\
  You might request a trial version from the developer, or borrow a copy / device from a friend. Explore all the functionalities and see if it might be motivating to keep using it in the medium and long term, as well as the short term. Ask how the trial version differs from the full version.

   2. Read online reviews\
  Include reviews from autistic users and their families and pay attention to people that have been using the product for a (relatively) long time. Keep in mind that reviewers' circumstances (e.g. their needs, age, goals or budget) may not be the same as yours and individual experiences may not be generalisable.

   3. Get an expert opinion\
  Ask people you know who have skills and experience with this technology and/or autistic users. Talk to relevant professionals such as a as a teacher, therapist or support worker. Ideally look for someone who also knows you as your personality has a key role in how engaging you will find it.

  Is it efficient?

   1. Read an academic paper\
  Ideally look for a review that systematically combines the results from multiple independent studies. It may be worth checking the quality and potential affiliations/bias of the original studies too and the journals where they were published.

   2. Get an expert opinion\
  Talk to relevant professionals (e.g. a specialist teacher, speech and language therapist, specialist psychologist, etc.). Ask (autistic) people, organisations or agencies you know who have specialist skills and relevant experience with technology.

   3. Read online reviews\
  Include reviews from autistic users and their families and pay attention to people that have been using the product for a (relatively) long time. Keep in mind that reviewers' circumstances (e.g. their needs, age, goals or budget) may not be the same as yours and individual experiences may not be generalisable.

   4. Try it out\
  You might request a trial version from the developer, or borrow a copy/device from a friend. Take your time to explore all the functionalities. Ask how the trial version differs from the full version.
  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Discussion {#section14-1362361319898331}
==========

There is a plethora of highly accessible digital supports purporting to support the autistic community ([@bibr12-1362361319898331]; [@bibr22-1362361319898331]; [@bibr37-1362361319898331]; [@bibr41-1362361319898331]; [@bibr42-1362361319898331]; [@bibr43-1362361319898331]; [@bibr51-1362361319898331]; [@bibr52-1362361319898331]; [@bibr53-1362361319898331]; [@bibr54-1362361319898331]) but no mechanism by which consumers, practitioners or researchers can gauge the level of evidence supporting their use. This is the first study to generate a consensus from an international group made up from the autistic and broader autism communities as well as researchers as to what constitutes good evidence for digital supports for autism. Through a Delphi study methodology, consensus was achieved on a detailed framework providing the parameters for which evidence is sought and the sources of evidence perceived to be important. This novel framework allows users of digital supports to incorporate evidence into their decision-making regarding the selection and use of digital support, for themselves, or their autistic family members, pupils, clients, participants and so on. The framework can also inform those developing digital supports for the autistic community, highlighting what types of evidence are considered important. For the first time, the autistic and autism communities can incorporate EBP into the development, application and use of digital supports. Importantly, this framework has been co-developed through a participatory research approach which connects researchers with relevant autistic and broader autism communities to achieve shared goals. These methods can deliver results that are relevant to people's lives and thus likely to have a positive impact ([@bibr19-1362361319898331]; [@bibr40-1362361319898331]).

The study revealed that academic evidence obtained with carefully conducted empirical research was just one of the aspects that may inform the autistic and broader autism communities when selecting appropriate digital support. This was clearly expressed by the panel from the very beginning of the study, when the importance of reliability, engagement and effectives emerged. Clinical research methodologies, such as RCTs, need to be augmented with other sources of empirical evidence, as well as hands-on experience or other users' feedback, to identify the extent to which a digital support is reliable, engaging or effective. Reliability and engagement may be particularly pertinent as features of digital support which are not present in the same way for non-technology-based supports ([@bibr35-1362361319898331]). EBP for digital supports therefore departs from non-technology-based EBP for autism, highlighting the need for a specific EBP framework.

There are important similarities and differences between the proposed EBP framework for digital supports for autism and other general models of evidence provision in the field of system and software engineering. For example, the ISO/IEC 25010:2011 standard (International Organisation for Standardisation/International Electrotechnical Commission) defines a 'quality in use' model composed of five characteristics: effectiveness, efficiency, satisfaction, freedom from risk and context coverage ([@bibr7-1362361319898331]). 'Effectiveness' is a common theme between the framework co-developed with the autistic and broader autism communities and this standard, and 'engagement' maps closely to 'satisfaction'. 'Reliability', however, represents a different perspective, potentially related to (but distinct from) 'efficiency', which also takes into account task time, time efficiency, cost-effectiveness, productive time ratio, unnecessary actions and fatigue. 'Reliability' focuses more upon, 'will the app crash?', for example, reflecting the end-users experience of technology which is not captured separately by the ISO/IEC model, but is embedded within 'satisfaction' which incorporates 'proportion of users complaining, proportion of user complaints about a particular feature and user trust'. Thus, there are parallels between the requirements of the autistic and broader autism communities and international standards, but the participatory research approach ensures the relevance of the framework to those who the digital supports are being developed for. In addition, words of caution associated with the framework (see [Supplemental material, Appendix II](https://journals.sagepub.com/doi/suppl/10.1177/1362361319898331)) emphasised potential downsides of technology and thus introduced the notion of freedom from risk. Indeed, risks for health and social status were acknowledged in the word of caution related to 'engagement', which warned about possible over-engagement with technology that would monopolise the child's time, and the framework needs to be interpreted within reference to the words of caution.

There were also similarities and differences in which sources of evidence were perceived to be most salient for reliability, engagement and effectiveness. While trying out the product was identified as the best source of evidence for informing reliability and engagement, academic research was viewed as the best source of evidence for effectiveness. This highlights that EBP, as informed by the broader autism community, requires multiple sources of information. Online reviews and expert opinions were also identified as key sources of evidence in all three domains. Recent accounts of fictitious online reviews ([@bibr36-1362361319898331]) and the independence of the expert are important considerations when evaluating these sources of evidence, and this is highlighted in the 'words of caution' (see [Supplemental material, Appendix II](https://journals.sagepub.com/doi/suppl/10.1177/1362361319898331)). Thus while similar sources of evidence are identified for reliability, engagement and effectiveness, they are weighted differently for each category.

As noted above, EBP is informed by integrating best available evidence along with practitioner expertise and the values of recipients of the practice. Co-developing the proposed EBP framework with researchers, technology developers, practitioners and the autism community helps ensure that it will be useful for these communities (see [@bibr19-1362361319898331]; [@bibr40-1362361319898331]). However, different participant groups may have different levels of access to different kinds of evidence, which may lead to inconsistencies in the type of sources of information which will actually be used by different types of potential users. For example, researchers may have greater access to and expertise in interpreting academic papers, while educators and caregivers may have more experience supporting day-to-day use and evaluating the long-term utility of digital technologies. In addition, while the framework identified commonalities in what constitutes evidence, it is important to note that there may be additional sources of evidence that are also significant for only one of the participant groups. Finally, we found the Delphi study methodology to be an effective method for integrating potentially divergent perspectives into an agreed-upon framework. However, despite our number of participants being consistent with those proposed from previous research ([@bibr24-1362361319898331]; [@bibr38-1362361319898331]), our sample was relatively small given the heterogeneity of autism, and of stakeholder perspectives in the broader autism community. This needs to be borne in mind when considering if the framework is suitable for the entire autism community.

Future work will apply this framework to digital supports for the autistic community, to identify the level of evidence available (complete, adequate, limited, none) from each source for reliability, engagement and effectiveness to highlight if the available evidence is strong, adequate or weak (after [@bibr45-1362361319898331]). An online version of the framework that enables researchers, developers and the autism community to evaluate the evidence base for any digital supports they are interested in is freely available at [beta-project.org](http://beta-project.org). Importantly, this framework identifies the strength (i.e. availability, quality) of the evidence, not the outcome of the evidence. It is possible, for example, that there could be strong evidence that an app is not engaging. For instance, [@bibr15-1362361319898331] conducted an RCT trial on a computerised support for training executive functions that yielded non-significant changes associated with high attrition rate in autistic participants, thus discouraging continuing practice. The framework developed here supports the sourcing and consideration of evidence into best practice, not necessarily what that best practice should be.

Supplemental Material {#section15-1362361319898331}
=====================

###### AUT898331_Supplemental_material_Appendix_I -- Supplemental material for A framework of evidence-based practice for digital support, co-developed with and for the autism community

###### 

Click here for additional data file.

Supplemental material, AUT898331_Supplemental_material_Appendix_I for A framework of evidence-based practice for digital support, co-developed with and for the autism community by Vanessa Zervogianni, Sue Fletcher-Watson, Gerardo Herrera, Matthew Goodwin, Patricia Pérez-Fuster, Mark Brosnan and Ouriel Grynszpan in Autism

###### AUT898331_Supplemental_material_Appendix_II -- Supplemental material for A framework of evidence-based practice for digital support, co-developed with and for the autism community

###### 

Click here for additional data file.

Supplemental material, AUT898331_Supplemental_material_Appendix_II for A framework of evidence-based practice for digital support, co-developed with and for the autism community by Vanessa Zervogianni, Sue Fletcher-Watson, Gerardo Herrera, Matthew Goodwin, Patricia Pérez-Fuster, Mark Brosnan and Ouriel Grynszpan in Autism

The authors thank all the panel members who participated in the Delphi study. Without their input, the project would not have been possible.

**Declaration of conflicting interests:** The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

**Funding:** The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: We are grateful to FIRAH who funded the study (Grant: APa2016_026: <https://www.firah.org/>).

**ORCID iDs:** Vanessa Zervogianni ![](10.1177_1362361319898331-img1.jpg) <https://orcid.org/0000-0003-0638-4087>

Sue Fletcher-Watson ![](10.1177_1362361319898331-img1.jpg) <https://orcid.org/0000-0003-2688-1734>

Mark Brosnan ![](10.1177_1362361319898331-img1.jpg) <https://orcid.org/0000-0002-0683-1492>

**Supplemental material:** Supplemental material for this article is available online.
