Background: My Health Record is Australia’s electronic personal health record system, which was introduced in July 2012. As of August 2017, approximately 21 percent of Australia’s total population was registered to use My Health Record. Internationally, usability issues have been shown to negatively influence the uptake and use of electronic health record systems, and this scenario may particularly affect people who have low e-health literacy. It is likely that usability issues are negatively affecting the uptake and use of My Health Record in Australia.
Objective: To identify potential e-health literacy–related usability issues within My Health Record through a heuristic evaluation method.
Methods: Between September 14 and October 12, 2016, three of the authors conducted a heuristic evaluation of the two consumer-facing components of My Health Record—the information website and the electronic health record itself. These two components were evaluated against two sets of heuristics—the Health Literacy Online checklist and the Monkman Heuristics. The Health Literacy Online checklist and Monkman Heuristics are evidence-based checklists of web design elements with a focus on design for audiences with low health literacy. During this heuristic evaluation, the investigators individually navigated through the consumer-facing components of My Health Record, recording instances where the My Health Record did not conform to the checklist criteria. After the individual evaluations were completed, the investigators conferred and aggregated their results. From this process, a list of usability violations was constructed.
Results: When evaluated against the Health Literacy Online Checklist, the information website demonstrated violations in 12 of 35 criteria, and the electronic health record demonstrated violations in 16 of 35 criteria. When evaluated against the Monkman Heuristics, the information website demonstrated violations in 7 of 11 criteria, and the electronic health record demonstrated violations in 9 of 11 criteria. The identified violations included usability issues with the reading levels used within My Health Record, the graphic design elements, the layout of web pages, and a lack of images and audiovisual tools to support learning. Other important usability issues included a lack of translated resources, difficulty using accessibility tools, and complexity of the registration processes.
Conclusion: My Health Record is an important piece of technology that has the potential to facilitate better communication between consumers and their health providers. However, this heuristic evaluation demonstrated that many usability-related elements of My Health Record cater poorly to users at risk of having low e-health literacy. Usability issues have been identified as an important barrier to use of personal health records internationally, and the findings of this heuristic evaluation demonstrate that usability issues may be substantial barriers to the uptake and use of My Health Record.
Usability issues are recognized as being deterrents to both consumers’ and health professionals’ use of electronic health records.12–14 Usability is the extent to which users in a specific context can achieve effective, efficient, and satisfactory use of a product.15 In a systematic review of 50 usability studies of electronic medical records and electronic health records, Zahabi et al. (2015)16 identified nine major types of usability problems affecting both consumers’ and health professionals’ use of records. These problems were poor ease of use and familiarity (naturalness); poor consistency of interface elements (such as hyperlinks, fonts, buttons); interfaces not designed in ways that prevent or minimize common errors; information overload; navigation problems; lack of feedback; poor use of language (such as use of acronyms or jargon); interface design that creates visual barriers to accessing information; and a lack of customizability to users’ needs.
It is tempting to categorize usability as a purely system-design problem—for example, ensuring that web pages are easy to navigate or that fonts and white space are used properly. If we take this approach, then usability most strongly relates to the “computer literacy” component in Norman and Skinner’s e-health literacy model.17 However, if we accept the definition of usability as the degree to which users can achieve effective, efficient, and satisfactory use of a product,18 then usability is actually a broader concept that can affect each component of e-health literacy. Any feature of an e-health resource or tool that facilitates or creates barriers to use—from design elements such as size of fonts or methods of navigation to pure literacy elements, such as the use of plain English or availability of translated resources—can be viewed as either negatively or positively influencing usability and thus affecting the ability of people with differing levels of e-health literacy to effectively use online health resources.
The aim of this study was to evaluate the usability of both the information website and the health record components of MyHR, taking into account the specific usability needs of people with low e-health literacy, using a heuristic evaluation method.30 A heuristic evaluation is one method of assessing usability characteristics and identifying potential usability issues. In a heuristic evaluation, a small number of users evaluate the design of an interface against a set of predetermined principles.31 Heuristic tools help to identify potential usability issues as part of an iterative design process, and ideally would be paired with user testing to identify all potential and actual usability issues.32 Despite recognition that the quality of resources and tools strongly affects the ability of users to complete health-related tasks,33 measures of health literacy and e-health literacy to date have focused on the ability or characteristics of the user,34 rather than on the features of the tools or resources.35 A heuristic evaluation shifts the focus away from the abilities of the user and toward the features of the tools, and thus may be useful for identifying aspects of website design and content that present potential usage barriers to people with low e-health literacy.
The heuristics used for this analysis were taken from the US Office of Disease Prevention and Health Promotion’s Health Literacy Online Checklist37 (henceforth referred to as “HLO Checklist”) and from Monkman et al. (2015)38 (henceforth referred to as “Monkman Heuristics”). Based on this heuristic evaluation, potential usability issues related to e-health literacy within MyHR were identified. In accordance with the principles of heuristic evaluation,39 the findings from this study could inform future user testing to comprehensively identify improvements that could increase the usability of MyHR.
The usability of MyHR, including the information website and the health record, was heuristically evaluated using Nielsen’s method (1994).40 In this method, a minimum of three heuristic evaluators individually navigate through the interface under examination and note any instances where the interface does not conform to a checklist of predetermined criteria. These instances are considered “violations” regardless of their perceived severity. Only after the individual evaluations are completed are the evaluators allowed to confer and aggregate their results. This process results in the construction of a list of usability violations that can be used to inform design changes to enhance usability. Heuristic evaluation can also include the evaluators’ judgments of the severity of violations; however, judgments of severity have been shown to have very low interrater reliability,41 and therefore severity ratings were not included as part of this study.
Between September 14 and October 12, 2016, three of the authors (L.W., N.A., M.A.) each separately conducted a heuristic evaluation of MyHR, analyzing the information website and the health record against two sets of heuristics—the HLO Checklist (see Table 1)44 and the Monkman Heuristics (see Table 2).45 These heuristic sets were chosen because they were the only existing heuristic sets the authors could find that considered the specific usability needs of web users with low literacy when interacting with health information and resources online.
The HLO Checklist is taken from Health Literacy Online, an evidence-based electronic manual for producing web-based health content for audiences with low literacy.46 The HLO Checklist comprises 35 separate criteria, categorized into four domains: (1) “write actionable content,” (2) “display content clearly on page,” (3) “organize content and simplify navigation,” and (4) “engage users.”47 A fifth domain (“test your site with users with limited literacy skills”), with eight additional criteria, was not included in the checklist used for the heuristic evaluation because it refers to user testing. The full checklist can be seen in Table 1.
The Monkman Heuristics are a novel heuristic set developed by summarizing design guidelines from the Health Literacy Online guide (from which the HLO Checklist comes),48 altering design guidelines to be relevant for use with mobile devices and apps,49 and incorporating further research from e-health/health literacy and usability literature.50 The Monkman Heuristics comprise 11 items (see Table 2).
The authors decided to use both the HLO Checklist and the Monkman Heuristics to enable comparison of the findings from these two different but related heuristic sets, as well as the investigators’ experience of using them. Any differences in the findings and/or the investigators’ experience related to the use of the two different heuristic sets may help to inform future researchers or developers needing to choose a heuristic set for the analysis of online health-related content.
This study was the first heuristic evaluation conducted by the three investigators. However, all three investigators were already users of the MyHR system, with one investigator being a very experienced MyHR user as a person with traumatic brain injury who uses augmentative and alternative communication methods. The investigators’ prior experience with the MyHR system, combined with their knowledge of the broader health system, means that the investigators who conducted the heuristic evaluation can be considered “domain experts” and, according to Nielsen (1994),51 sufficiently equipped to perform the evaluation.
The MyHR system does not allow the creation of mock records for the purpose of research. To conduct the heuristic evaluation, the investigators had to use their own existing records. Because the purpose of the evaluation was to discover usability violations rather than to analyze the content of the records, and because the interactive and usability elements of the records are unable to be tailored (making them standard across all users), the use of the investigators’ actual records was deemed appropriate.
In conducting the evaluation, the three investigators examined all web pages on both sites, conducted trials of all interactive elements, and opened a sample of uploaded documents on the record from government health agencies and healthcare professionals to ensure that the links, formatting, and downloading worked. Whether the sites met or violated the heuristic criteria was recorded on an evaluation table, accompanied by a short commentary explaining how each criterion was violated. After their individual investigations were completed, the three authors compared their analyses, clarified any explanatory notes, and aggregated their individual results into a combined table of results.
The heuristic evaluation of the information website against the HLO Checklist identified violations in 12 of 35 criteria, and the evaluation of the record identified violations in 16 of 35 criteria (see Table 3). Violations were seen across all domains of the HLO Checklist.
The record showed violations in five of the seven criteria for writing actionable content. The “identify user motivation and goals” and “put the most important information first” criteria were violated by a lack of introductory information about the record after the user had logged in. The criteria of “describe the health behavior—just the basics” and “provide specific action steps” were violated because of a lack of consistent instructions throughout all pages of the record. Finally, poor use of the active voice in the text meant that the record violated the “plain language” criterion, potentially affecting readability.
The information website demonstrated violations in 3 of the 13 criteria for displaying content clearly on the page. Information layout was identified as an issue in the “most important content above the fold” criterion. This criterion is violated when users need to scroll down to find key information, which was common throughout the website. The “images to assist learning” criterion was also violated, with the website relying heavily on text and using very few images throughout. Finally, the “disability accessible” criterion was violated overall. Although the website has instructions to improve accessibility,56 none of the evaluators could use the access keys successfully.
The health record had six criteria violations in the “display content clearly on page” domain. As with the information website, the “most important content above the fold,” “images to assist learning,” and “disability accessible” criteria were violated, for the same reasons stated above. Additionally, the fact that hyperlinks were not well differentiated from surrounding text (i.e., they were the same color and same size as the surrounding text, and were not underlined) meant that two criteria were violated: “use links effectively” and “color or underline to identify links.” Finally, the evaluators found that the record had too many headings, with complicated layering of headings and subheadings, which violated the “meaningful headings” criterion.
The information website demonstrated violations in 3 of the 10 criteria in this domain. Two violations (of the “simple and engaging home page” and “make clickable elements recognizable” criteria) were associated with links being difficult to differentiate from surrounding text or other graphic elements throughout the website. One tester also noted that the image of the health professional used on the home page made the website appear to target health professionals rather than consumers, violating the “simple and engaging home page” criterion. Additionally, the search function was considered not to be prominent enough, having only a magnifying glass symbol and the term “search” in light gray in the search box, and no “get started” or “go” button as advised by the HLO Checklist.57 This function therefore violated the “simple search function” criterion.
Three criteria violations were identified in the health record in the “organize content and simplify navigation” domain. The lack of a search function within the record led to the violation of two criteria—“simple search function” and “display search results clearly.” The links within the record were also not clearly differentiated from surrounding text, leading to a violation of the “make clickable elements recognizable” criterion.
The information website demonstrated violations in all five criteria of this domain. The heavy reliance on text-based information, the lack of multimedia, and the lack of quizzes or forms led to violations in the “share information through multimedia,” “intuitive interactive graphics and tools,” and “user-friendly forms and quizzes” criteria. Users cannot enter personal details to assist their information gathering, which violated the “tailored information” criterion. Finally, the website had no instructions for social media sharing or social media sharing buttons to assist with sharing information on social media, which violates the “social media sharing options” criterion.
The health record violated four of the five criteria in the “engage users” domain. The lack of images and multimedia violated the “share information through multimedia” criterion. Forms in the record can be challenging to find, and the investigators judged the forms as not intuitive or easy to use, violating the “user-friendly forms and quizzes” criterion. A number of interactive child health quizzes were found to be available through the “Parents” section, but these quizzes were suitable only for parents of infant children, not for all users of the record. Therefore, investigators considered the “intuitive interactive graphics and tools” domain to be violated. Finally, the record offered no option to share information through social media, violating the “social media sharing option” criterion (see Table 3).
The evaluation of the information website against the Monkman Heuristics identified violations in 7 of the 11 criteria. The evaluation of the health record identified violations in 9 of the 11 criteria (see Table 4).
In the analysis of the information website against the Monkman Heuristics, web design elements were strongly associated with criteria violations. The use of few visuals, and the lack of audio, video, and interactive elements, led to violations in the “use complementary interaction methods,” “leverage interactivity,” and “use visuals to complement text, but avoid tables” criteria. A lack of options for tailoring and no obvious prioritization of the information presented violated the “provide tailored, flexible, layered content” and “provide accurate, colloquial, comprehensive, succinct content” criteria. The need to scroll or click a link to find important information violated the “simplistic, consistent displays” criterion.
In terms of content, the poor use of plain language, including the use of jargon and a mean Flesch-Kincaid grade level greater than 11,58 a lack of comprehensive information in multiple languages,59 and the lack of ability to tailor the information for the user violated the “provide tailored, flexible, layered content” and “provide accurate, colloquial, comprehensive, succinct content” criteria. Risks (e.g., about privacy/security) were not clearly conveyed, violating the “clear and comprehensive communication of risks” criterion. Finally, the image of a health professional that appeared on the home page did not represent the user, and there was a lack of other meaningful images throughout the website, which violated the “use visuals to complement text, but avoid tables” criterion.
In the evaluation of the health record according to the Monkman Heuristic, the lack of visuals, video, and audio; the lack of content in languages other than English; the need for scrolling to find important information; and the poor communication of risks led to violations in the “use complementary interaction methods,” “use visuals to complement text, but avoid tables,” “provide tailored, flexible, layered content,” “clear and comprehensive communication of risks,” and “simplistic, consistent displays” criteria. Interactive elements in the record were limited to users completing forms and entering their health information, which did not provide opportunities for learning or improving performance or knowledge through feedback. Therefore the “leverage interactivity” criterion was considered to be violated. Links within the record were not clear and therefore constituted a violation of the “simplistic, consistent navigation” criteria.
Two important violations within the record were related to the “immediately inform users of purpose and engage users; avoid registration” and “clear depiction of monitoring data and/or test results” criteria. Registration is a multiple-step process, involving registration on and navigation through another government website (myGov).60 Registration and access to myGov is reportedly complicated,61–63 but myGov was not included in this heuristic evaluation, except to acknowledge that the requirement for registration on both MyGov and MyHR violates the “immediately inform users of purpose and engage users; avoid registration” criterion.
Evaluation of the criterion of “clear depiction of monitoring data and/or test results” was complicated because this criterion depends on the quality of information uploaded into the record by health professionals or health service databases. The investigators considered this criterion to be violated because the record itself had no systems to facilitate pattern recognition (e.g., development of graphs to monitor change over time) and had no links to external sources that could aid in the understanding or interpretation of any personal health information uploaded into the system.
Screenshots of the information website and the health record with annotated examples of criteria violations can be seen in Figure 1, Figure 2, Figure 3, and Figure 4. The annotated figures do not show all criteria violations but are intended to provide an illustrative selection of violations for readers who are unfamiliar with MyHR.
This heuristic evaluation of MyHR identified a range of potential usability issues that may be acting as a barrier to widespread use of MyHR by the Australian population. A wide range of usability issues were identified in this study, including little use of plain language and translated resources, problems with website navigation, complex registration processes, and issues with design elements, all of which may account for the fact that only 21 percent of Australians have a MyHR record64 more than four years after the record became available. Negative user experiences,65, 66 usability issues (including complex registration processes),67 and low levels of user digital or health literacy68–70 are recognized barriers to patient recruitment and engagement with PHRs internationally, and research into usability was recently identified, in a systematic review by Roehrs et al. (2017),71 as a priority area for PHR research. Although the findings from this study have particular application in the Australian context, the methods and identified usability violations are likely to be applicable to PHR usability research more broadly.
This study demonstrates how existing heuristic sets can be used to analyze a PHR. It must be acknowledged that both sets of heuristics used are general guidelines for the evaluation of online health information and are not specific to the evaluation of a PHR. Therefore, some criteria might be viewed as less important or not appropriate in the context of PHRs. For example, a lack of social media sharing options was identified as a violation according to the heuristic sets used; however, the investigators acknowledge that, in the context of MyHR, facilitating social media sharing may be appropriate only for the information website, and not for the confidential information contained within the health record. Similarly, the need to ensure the safety and security of personal health data in the record may mean that violations of criteria around registration are unavoidable. However, the investigators hope that the evaluation of usability and identification of criteria violations may help the MyHR website designers either to find innovative solutions to the issues identified or, in the case of unavoidable criteria violations, to simplify processes as much as possible and provide additional support or information for consumers to help overcome any potential barriers to the use of the site.
Ultimately, user testing with a range of users having various skills, abilities, and backgrounds needs to be incorporated alongside heuristic evaluation into the development processes of MyHR and other PHRs to ensure that usability issues are fully identified and corrected. A review of the literature relating to the development of MyHR does not indicate whether or how user testing was conducted. Concerns relating to user testing of the record were raised in the terms of reference (the aspects/issues which were examined as part of the review of the personally controlled EHR completed in 2013) for a review of the record in 2013, but were not addressed in the review report.72 A lack of usability testing by vendors of health records internationally has been identified and proposed as a major factor contributing to the poor usability of electronic health records generally.73 Adopting a usability design framework that includes both usability and user testing to guide future changes to the health record may help address usability issues, which could improve uptake and use of MyHR.
A secondary purpose of this study was to share the findings and experience of researchers using two different, but related, heuristic sets to analyze a PHR. Both checklists were useful tools for evaluation. The HLO Checklist was perceived as simpler to use and returned more specific information, especially regarding design elements. This finding is not surprising, given that the HLO Checklist was designed for anyone involved in the creation of online health content74 whereas the Monkman Heuristics were designed to be used by people experienced in heuristic evaluation.75 One important difference between the evaluations was that the usability issues of a complex registration process and the lack of translated resources about the MyHR were identified only in the evaluation conducted with the Monkman Heuristics.
One potential limitation of this study was the lack of experience of the investigators who performed the heuristic evaluation. This study was the first time any of the investigators had performed a heuristic evaluation, which may have meant that some violations were missed. This limitation was addressed through the use of category-specific heuristics to guide the evaluation, rather than employing broad heuristic sets commonly used by more experienced usability evaluators.76 In addition, all three investigators were already users of the MyHR system, and, as public health researchers, they have extensive knowledge of the broader health system, which means that they can be considered “domain experts” and, according to Nielsen (1994), sufficiently equipped to perform the evaluation.77 For future studies, or if heuristic testing is to be implemented as part of the MyHR development process, investigators with greater experience in heuristic evaluation, or a team of evaluators comprising both heuristic evaluation experts and domain experts, could be considered. Additionally, the use of more than three investigators may also uncover more violations. Nielsen (1994)78 recommends three investigators as a minimum, and up to a maximum of 15.
A second limitation was that the use of the two heuristic sets by the same investigator team may have meant that an accurate comparison of the outcomes and the experience of use between the HLO Checklist and the Monkman Heuristics could not be made. Conducting the evaluation twice with different, but related, heuristics may have resulted in the outcomes of the first evaluation influencing the identification of violations in the second evaluation. This limitation means that the investigator team is unable to make a definitive recommendation of the use of one heuristic set over the other for testing the usability of a PHR. In future comparisons of heuristic sets, the two sets should be tested by different investigator teams for a truer comparison of the outcomes and the experience of use of the heuristics.
Finally, this evaluation was a “snapshot in time” of a system that is evolving through quality improvement and user feedback.79 In a previous examination of the MyHR online consumer-facing information, changes were made to the layout and information on the MyHR information website within the three-month period of the study,80 which potentially influenced usability. Similarly, the health record underwent a redesign process during 2016.81 It is anticipated that redesigns and updates of MyHR will continue to influence usability into the future. Therefore, usability trials need to be responsive to changes made in MyHR while remaining firmly grounded in the research and evolving models of e-health literacy. Reapplying the heuristic evaluation in this study might provide one way to determine comparability across usability trials of MyHR as it evolves over time.
Heuristic evaluation is not a stand-alone technique and is best paired with user testing to fully identify usability issues.82 The heuristic evaluation applied in this study could be used in future user testing of MyHR with different user groups. User testing could be particularly valuable for groups that have been identified as high priority for use of the record (e.g., Aboriginal and Torres Strait Islander people, individuals with chronic and complex conditions, older Australians, mothers with newborn children, people with mental health conditions, people with disabilities, people living in rural and remote communities)83 or groups that are identified as being at risk of or having either communication disabilities84, 85 or low health literacy.86–88 User testing with these different groups may help to enable MyHR to meet the needs of all consumers and guard against exclusion of individuals who could benefit from using MyHR.
Registration and ongoing access to MyHR might also be affected by registration being through the Australian government’s myGov website.89 Problems with usability of the myGov website have been reported since its introduction in 2013.90–92 It is possible that using myGov presents usability issues for users and potential users, which could be further assessed through user testing. Such user testing could also take the results of the “opt out” and “opt in” trial sites into account, in that these trials may result in further changes in the requirements and user interactions with myGov and MyHR sites.
MyHR is an important piece of technology that has the potential to facilitate better communication between consumers and their health providers. Better communication facilitates information sharing, limits adverse events, and consequently improves health care.94 However, this heuristic evaluation demonstrated that many of the usability elements of MyHR cater poorly to users at risk of having low e-health literacy. When tested with the HLO Checklist,95 the information website demonstrated violations in 12 of 35 criteria, and the health record demonstrated violations in 16 of 35 criteria. When tested with the Monkman Heuristics,96 the information website demonstrated violations in 7 of the 11 criteria, and the health record demonstrated violations in 9 of 11 criteria. The identified violations included usability issues with the language MyHR uses, the graphic design elements, the layout of web pages, and a lack of images and audiovisual tools to support learning. Important usability issues with MyHR also included a lack of translated resources, difficulty using accessibility tools (e.g., access keys), and the complexity of the registration process.
Usability issues have been identified as an important barrier to the use of PHRs internationally,97–99 and it is likely that the specific usability issues identified in this study may be significant barriers to uptake and use of MyHR in Australia.
This review was funded in part by a project grant of the National Health and Medical Research Council of Australia.
The authors wish to acknowledge Andrew Ingersoll from the Australian Digital Health Agency for providing feedback regarding the terminology and timelines of the operation of My Health Record.
Louisa Walsh, MStratComm, is a research officer at the La Trobe University Centre for Health Communication and Participation in Victoria, Australia.
Bronwyn Hemsley, PhD, is an Associate Professor at the The University of Newcastle in Australia.
Meredith Allan is a PhD candidate at Deakin University in Australia.
Natalie Adams is a research associate at The University of Newcastle in Australia.
Susan Balandin, PhD, is a Professor at Deakin University in Australia.
Andrew Georgiou, MSc, FCHSM, FACHI, FSc, RCPA, PhD, is a Professor at the Australian Institute of Health Innovation in Australia.
Isabel Higgins, RN, MN, PhD, is a Professor at the University of Newcastle in Australia.
Sophie Hill, MA, PhD, is the Head of the Centre for Health Communication and Participation at La Trobe University in Australia.
Czaja, S. J., C. Zarcadoolas, W. L. Vaughon, C. C. Lee, M. L. Rockoff, and J. Levy. “The Usability of Electronic Personal Health Record Systems for an Underserved Adult Population.” Human Factors 57, no. 3 (2015): 491–506.
Tang, P. C., J. S. Ash, D. W. Bates, J. M. Overhage, and D. Z. Sands. “Personal Health Records: Definitions, Benefits and Strategies for Overcoming Barriers to Adoption.” Journal of the American Medical Informatics Association 13, no. 2 (2006): 121–26.
Sartain, S. A., S. Stressing, and J. Prieto. “Patient Views on the Effectiveness of Patient-held Records: A Systematic Review and Thematic Synthesis of Qualitative Studies.” Health Expectations 18 (2014): 2666–77.
Sorensen, K., S. Van den Broucke, J. Fullam, G. Doyle, J. Pelikan, Z. Slonska, and H. Brand. “Health Literacy and Public Health: A Systematic Review and Integration of Definitions and Models.” BMC Public Health 12 (2012): 80.
Norman, C. “eHealth Literacy 2.0: Problems and Opportunities with an Evolving Concept.” Journal of Medical Internet Research 13, no. 4 (2011): e125.
Norman, C. D., and H. A. Skinner. “eHealth Literacy: Essential Skills for Consumer Health in a Networked World.” Journal of Medical Internet Research 8, no. 2 (2006): e9.
Otte-Trojel, T., A. de Bont, T. G. Rundall, and J. van de Klundert. “What Do We Know about Developing Patient Portals? A Systematic Literature Review.” Journal of the American Medical Informatics Association 23 (2016): e162–e168.
Ammenwerth, E., P. Schnell-Inderst, and A. P. Hoerbst. “The Impact of Electronic Patient Portals on Patient Care: A Systematic Review of Controlled Trials.” Journal of Medical Internet Research 14, no. 6 (2012): 1–13.
Goldzweig, C. L., G. Orshansky, N. M. Paige, A. A. Towfigh, D. A. Haggstrom, I. Miake-Lye, J. M. Beroes, and P. G. Shekelle. “Electronic Patient Portals: Evidence on Health Outcomes, Satisfaction, Efficiency, and Attitudes.” Annals of Internal Medicine 159, no. 10 (2013): 677–86.
Bush, R. A., C. D. Connelly, M. Fuller, and A. Pérez. “Implementation of the Integrated Electronic Patient Portal in the Pediatric Population: A Systematic Review.” Telemedicine Journal and e-Health 22, no. 2 (2015): 144–52.
Zahabi, M., D. B. Kaber, and M. Swangnetr. “Usability and Safety in Electronic Medical Records Interface Design: A Review of Recent Literature and Guideline Formulation.” Human Factors: The Journal of the Human Factors and Ergonomics Society 57, no. 5 (2015): 805–34.
International Organization for Standardization. Ergonomic requirements for office work with visual display terminals (VDTs)—part 11: Guidance on usability. ISO 9241-11.
International Organization for Standardization. Ergonomic requirements for office work with visual display terminals (VDTs)—part 11: Guidance on usability.
National E-Health Transition Authority. Evolution of eHealth in Australia: Achievements, Lessons, and Opportunities. Sydney, Australia: National E-Health Transition Authority, 2016.
National eHealth Transition Authority. “My Health Record News Issue 1 February 2016.” 2016. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/news-hcp-007.
Australian Digital Health Agency. “My Health Record Statistics—at 13 August 2017.” 2017. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/news-002.
Walsh, L., S. Hill, M. Allan, S. Balandin, A. Georgiou, I. Higgins, B. Kraal, S. McCarthy, and B. Hemsley. “A Content Analysis of the Consumer-facing Online Information about My Health Record: Implications for Increasing Knowledge and Awareness to Facilitate Uptake and Use.” HIM Journal (2017). Available at http://journals.sagepub.com/doi/10.1177/1833358317712200.
Australian Digital Health Agency. “Frequently Asked Questions: Managing Your My Health Record.” 2016. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/find-out-more?OpenDocument&cat=Managing%20your%20My%20Health%20Record.
Australian Digital Health Agency. “Getting a My Health Record.” 2016. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/before_you_register_yourself_or_your_child.
26. Australian Digital Health Agency. “My Health Record for Northern Queensland and Nepean Blue Mountains Areas.” 2016. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/trials.
Australian Government. “About myGov.” 2016. Available at https://my.gov.au/mygov/content/html/about.html.
Nielsen, J. “Heuristic Evaluation.” In J. Nielsen and R. Mack (Editors), Usability Inspection Methods. New York: Wiley, 1994, 25–62.
US Department of Health and Human Services, Office of Disease Prevention and Health Promotion. “Health Literacy Online: A Guide to Simplifying the User Experience.” 2nd ed. 2015. Available at http://health.gov/healthliteracyonline/.
Nguyen, T. H., H. Park, H. R. Han, K. S. Chan, M. K. Paasche-Orlow, J. Haun, and M. T. Kim. “State of the Science of Health Literacy Measures: Validity Implications for Minority Populations.” Patient Education and Counseling 98, no. 12 (2015): 1492–1512.
Monkman, H., and A. Kushniruk. “Applying Usability Methods to Identify Health Literacy Issues: An Example Using a Personal Health Record.” Studies in Health Technology & Informatics 183 (2013): 179–85.
Hemsley, B., A. Georgiou, R. Carter, S. Hill, I. Higgins, P. van Vliet, and S. Balandin. “Use of the My Health Record by People with Communication Disability in Australia: A Review to Inform the Design and Direction of Future Research.” HIM Journal 45, no. 3 (2016): 107–15.
Monkman, H., J. Griffith, and A. W. Kushniruk. “Evidence-based Heuristics for Evaluating Demands on eHealth Literacy and Usability in a Mobile Consumer Health Application.” Studies in Health Technology and Informatics 216 (2015): 358–62.
Australian Digital Health Agency. “Managing Access, Privacy and Security.” 2016. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/privacy.
Australian Digital Health Agency. “Accessibility.” 2016. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/accessibility.
Cowan, P. “DHS’ Plan to Stop People Complaining about MyGov.” IT News. April 12, 2016. Available at http://www.itnews.com.au/news/dhs-plan-to-stop-people-complaining-about-mygov-417939.
Francis, H. “Taxpayers, IT Expert Slam ATO’s ‘Appalling’ myGov Integration.” Sydney Morning Herald, July 7, 2015. Available at http://www.smh.com.au/technology/technology-news/taxpayers-it-expert-slam-atos-appalling-mygov-integration-20150706-gi6bru.html.
Towell, N. “Troubled myGov Website to be Taken from Human Services and Given to Digital Transformation Office for Streamlining.” Sydney Morning Herald, January 28, 2016. Available at http://www.smh.com.au/it-pro/government-it/troubled-mygov-website-to-be-taken-from-human-services-and-given-to-digital-transformation-office-for-streamlining-20160127-gmfoe3.html.
O’Connor, S., P. Hanlon, C. A. O’Donnell, S. Garcia, J. Glanville, and F. S. Mair. “Understanding Factors Affecting Patient and Public Engagement and Recruitment to Digital Health Interventions: A Systematic Review of Qualitative Studies.” BMC Medical Informatics and Decision Making 16, no. 1 (2016): 120.
Roehrs, A., C. A. da Costa, R. da Rosa Righi, and K. S. F. de Oliveira. “Personal Health Records: A Systematic Literature Review.” Journal of Medical Internet Research 19, no. 1 (2017): e13.
Australian Department of Health. Review of the Personally Controlled Electronic Health Record. Canberra, Australia: Department of Health, 2013.
Ratwani, R. M., N. C. Benda, A. Hettinger, and R. J. Fairbanks. “Electronic Health Record Vendor Adherence to Usability Certification Requirements and Testing Standards.” JAMA 314, no. 10 (2015): 1070–71.
Hemsley, B., S. McCarthy, N. Adams, A. Georgiou, S. Hill, and S. Balandin. “Legal, Ethical, and Rights Issues in the Adoption and Use of the Personally Controlled Electronic Health Record ‘My Health Record’ by People with Communication Disabilities in Australia.” Journal of Intellectual & Developmental Disability (2016). Available at http://www.tandfonline.com/doi/abs/10.3109/13668250.2017.1294249.
Australian Digital Health Agency. “Welcome to My Health Record.” 2017. Available at https://myhealthrecord.gov.au/internet/mhr/publishing.nsf/Content/home.
Australian Department of Health and Ageing. Concept of Operations Relating to the Introduction of a Personally Controlled Electronic Health Record System. Canberra, Australia: Australian Department of Health and Ageing, 2011.
Nutbeam, D. “The Evolving Concept of Health Literacy.” Social Science & Medicine 67, no. 12 (2008): 2072–78.
Rowlands, G., J. Protheroe, J. Winkley, M. Richardson, P. T. Seed, and R. Rudd. “A Mismatch between Population Health Literacy and the Complexity of Health Information: An Observational Study.” British Journal of General Practice 65, no. 635 (2015): e379–e386.
Berkman, N. D., S. L. Sheridan, K. E. Donahue, D. J. Halpern, and K. Crotty. “Low Health Literacy and Health Outcomes: An Updated Systematic Review.” Annals of Internal Medicine 155, no. 2 (2011): 97–107.
Australian Commission on Safety and Quality in Health Care (ACSQHC). Health Literacy: Taking Action to Improve Safety and Quality. Sydney, Australia: ACSQHC, 2014.
Louisa Walsh, MStratComm; Bronwyn Hemsley, PhD; Meredith Allan; Natalie Adams; Susan Balandin, PhD; Andrew Georgiou, MSc, FCHSM, FACHI, FSc, RCPA, PhD; Isabel Higgins, RN, MN, PhD; Shaun McCarthy, LLB; and Sophie Hill, MA, PhD. “The E-health Literacy Demands of Australia’s My Health Record: A Heuristic Evaluation of Usability.” Perspectives in Health Information Management (Fall 2017): 1-28.