--- abstract: 'While the public claim concern for their privacy, they frequently appear to overlook it. This disparity between concern and behaviour is known as the Privacy Paradox. Such issues are particularly prevalent on wearable devices. These products can store personal data, such as text messages and contact details. However, owners rarely use protective features. Educational games can be effective in encouraging changes in behaviour. Therefore, we developed the first privacy game for (Android) Wear OS watches. 10 participants used smartwatches for two months, allowing their high-level settings to be monitored. Five individuals were randomly assigned to our treatment group, and they played a dynamically-customised privacy-themed game. To minimise confounding variables, the other five received the same app but lacking the privacy topic. The treatment group improved their protection, with their usage of screen locks significantly increasing (*p* = 0.043). In contrast, 80% of the control group continued to never restrict their settings. After the posttest phase, we evaluated behavioural rationale through semi-structured interviews. Privacy concerns became more nuanced in the treatment group, with opinions aligning with behaviour. Actions appeared influenced primarily by three factors: convenience, privacy salience and data sensitivity. This is the first smartwatch game to encourage privacy-protective behaviour.' author: - | \ \ bibliography: - 'bib.bib' title: 'Smartwatch Games: Encouraging Privacy-Protective Behaviour in a Longitudinal Study' --- Privacy, game, smartwatch, behavior, wearable, education Introduction {#sec:one} ============ Study Motivation ---------------- The public claim to be concerned about privacy, as suggested by a range of polls and surveys @MorarConsulting2016 [@Pike2017]. However, we frequently exhibit behaviour which places our data at risk @Beresford2012 [@Felt2012]. This disparity between claimed concern and empirical action is known as the Privacy Paradox @Norberg2007. The situation often arises through a lack of awareness @Deuker2009. This poses a particular risk to wearables, which are both novel and unfamiliar @Williams2017. Smartwatches offer exciting functionality, providing interactive apps and online connectivity. They can also store a variety of personal data, from text messages to contact details @Do2017. Despite this, users rarely use available settings to protect their privacy @Udoh2016. This has led to the Privacy Paradox being prevalent in this environment @Williams2017. Previous work has suggested that this issue can be mitigated by increasing awareness @Deuker2009. Therefore, many studies have sought to educate users on privacy matters @Kelley2009 [@Helou2012]. Unfortunately, highlighting a problem is often not sufficient to change behaviour @Bada2015. Since privacy is rarely a primary goal @Hughes-Roberts2015, individuals might lack the motivation to protect their data. If we hope to incentivise protection, privacy should be aligned with user wants @Dolan2010. Rather than mandating compliance, we can then highlight the empowering aspects of protection. Serious games embed incentives within interactivity, using positive reinforcement to instil knowledge @Kumar2013. Such apps have succeeded in phishing training @Sheng2007 and network defence @Irvine2005. However, privacy games have never been developed for smartwatches. In previous work (*under review*), we constructed and evaluated an online prototype. Through a 504-person survey, we found that protective actions could be encouraged. Therefore, to empirically assess behaviour, we develop and evaluate the first smartwatch privacy game. Background and Related Work --------------------------- **Privacy and awareness.** Privacy is a nebulous topic, encompassing confidentiality, anonymity and autonomy @Solove2008. However, since we target technological behaviour, we scope our focus to these interactions. Therefore, we consider informational privacy: "*the interest an individual has in controlling, or at least significantly influencing, the handling of data about themselves*" @Clarke1999. Polls repeatedly suggest that the public care about their privacy @Pike2017 [@MorarConsulting2016]. A 2017 survey found 84% of US consumers were worried about their data, with 70% stating their concerns have increased @Pike2017. Despite these assertions, we rarely act to protect our information. We ignore permissions @Felt2012, skim policies @Glanville2018 and settle for lax default settings @Bonneau2010. This attitude-behaviour gap has been labelled the Privacy Paradox @Norberg2007. We define the Privacy Paradox as the "*discrepancy between the expressed concern and the actual behavior of users*" @Barth2017. Due to its popularity, it has been deconstructed in many previous studies. Veltri and Ivchenko [-@Veltri2017] explored the influence of cognitive scarcity. Through an experiment with 969 users, they discovered that fatigue encouraged disclosure. They used this factor to partially justify the Privacy Paradox. Hallam and Zanella [-@Hallam2017] adopted the lens of Construal Level Theory, which studies whether concepts are considered abstract or concrete @Trope2010. If issues are hypothetical or temporally distant, as often the case with privacy, they are frequently deemed to be abstract. The authors describe how this 'psychological distance' tends to lead to the topic lacking salience. As a result, they concluded that concerns have little influence on protective behaviour. Our work also considers the influence of privacy salience. While the above studies discuss the topic, we actively seek to mitigate the Privacy Paradox. Research suggests that increasing awareness should address the Privacy Paradox. Deuker [-@Deuker2009] found a concern-behaviour disparity existed due to bounded rationality, incomplete information and psychological variables. When describing bounded rationality, he highlighted that "*users' capabilities in processing information and drawing the right conclusions are restricted by nature*". Since individuals fail to process the technical details, they tend to underestimate the risks of privacy invasion. Through building awareness, he believed that both bounded rationality and incomplete information could be addressed. P<span>ö</span>tzsch [-@Potzsch2009] saw two solutions to the disparity: align concern to behaviour or behaviour to concern. By highlighting privacy risk, individuals should be more likely to act. Indeed, Bartsch and Dienlin [-@Bartsch2016] found that knowledge can increase the chance of protective action. However, informed individuals must also have the motivation to put that knowledge into practice. While these researchers theorised wise solutions, we evaluate the success of a privacy intervention. Jackson and Wang [-@Jackson2018] successfully mitigated the Privacy Paradox on mobile phones. They used customised notifications, with charts highlighting the discrepancy between a user's attitude and their app permissions. Attitudes were evaluated through a concern questionnaire at the start of the study. Based on their selected permissions, the system then predicted their degree of privacy risk. Through an online simulation, the authors found that the disparity decreased after these notifications were viewed. This is encouraging, and we adopt personalised challenges within our games. However, the Privacy Paradox was studied on mobile phones at a single point in time. In contrast, we evaluate smartwatch interactions over a two-month period. **Privacy behaviour change.** Awareness can highlight the existence of a particular risk. However, this is often insufficient to change privacy behaviour @Bada2015. Sasse et al. [-@Sasse2007] recommended a three-stage approach: raise awareness, give education and provide training. In this manner, individuals have an opportunity to practice and refine their behaviour. Finally, even if users possess the knowledge, they must be incentivised to act @Bada2015. Our game, introduced in Section \[sec:four\], seeks to implement all these approaches. We explore privacy-protective behaviour through the lens of Protection Motivation Theory (PMT) @Rogers1983. This model "*postulates the three crucial components of a fear appeal to be (a) the magnitude of noxiousness of a depicted event; (b) the probability of that event's occurrence; and (c) the efficacy of a protective response*" @Rogers1975. It seeks to deconstruct why individuals do (or do not) use protection @Rogers1983. Therefore, it appeared relevant to our efforts at behaviour change. It is comprised of two primary components: threat appraisal and coping appraisal. The former is informed by the severity and vulnerability of a risk. The rewards of functionality are also taken into account. For the latter, self-efficacy and response efficacy is considered. This is balanced against the costs of protection. Our games sought to influence these components to encourage privacy. We also considered the Theory of Reasoned Action[^1] @Fishbein1979, but this model does not recognise constraints on action @Briggs2017. It is not deemed appropriate for skilled tasks @Liska1984, and privacy protection appears to require skill. While we investigated the Theory of Planned Behaviour[^2] @Ajzen1991, this fails to account for susceptibility or response efficacy @Norman1996. In contrast, PMT aligns well with privacy and has been recommended for security behaviour change @Briggs2017. In non-wearable environments, education has prompted protection. Albayram et al. [-@Albayram2017] encouraged screen lock usage on smartphones. In their 228-person study, they divided their participants between a treatment group and a control group. The former watched an educational video, whereas the latter did not. Both groups reported their smartphone actions in pretest and posttest. The treatment group reported improved behaviour, suggesting the video was persuasive. Since we evaluate the efficacy of an intervention, we also adopt a pretest-posttest two-group design. However, rather than using smartphone self-reports, we study smartwatches empirically. Albayram et al. [-@Albayram2017a] later explored whether videos can encourage the use of Two-Factor Authentication (2FA). Through a 2x2x2 design, they generated and evaluated eight videos. Their content varied on whether risk, self-efficacy and contingency were included. When the first two components were highlighted in the videos, participants were found to adopt 2FA. Both risk and self-efficacy are considered within PMT, and we also use the theory to encourage alterations. However, while Albayram et al. [-@Albayram2017a] used Amazon Mechanical Turk, we analyse participants through a field study. Our in-person approach delivers several advantages. Since our behaviour is empirical rather than self-reported, it should be less prone to falsehood @Fielding2006. With participants using a real smartwatch in a native environment, our findings should also have external validity. Finally, although our in-person approach limited our sample size, it supported rationale extraction through rich interviews. 'Nudging' has become a popular approach to encourage protection @Wisniewski2016. Wang et al. [-@Wang2014] augmented Facebook to highlight the audience of a person's posts. Through their six-week trial, they found unintended disclosures were decreased. Although temporarily influential, behaviour can revert when nudges are removed @Bruyneel2016. This approach differs from techniques within serious games. Nudging modifies the choice architecture to encourage certain decisions. In contrast, serious games seek to instil lessons through education and positive reinforcement @Connolly2012. Since intrinsic motivation can be highly persuasive @Ryan2000, the latter approach might prove more persistent. **Behaviour change games.** Serious games can be defined as "*any form of interactive computer-based game software...that has been developed with the intention to be more than entertainment*" @Ritterfeld2009. Such tools have been highly successful, often considered more persuasive than direct training @Wouters2013. In non-smartwatch environments, security has been frequently addressed. Anti-Phishing Phil @Sheng2007 challenged users to identify fraudulent URLs. After playing an aquatic game, players were better able to avoid phishing campaigns. We differ by targeting smartwatches, but adopt similar Learning Science principles. For example, we implement reflection @Donovan1999, where players contemplate their learning experience. We also include story-based agents by using Non-Player Characters to guide the user through our narrative @Moreno2001. Finally, we use the conceptual-procedural principle by augmenting high-level information with specific instructions @Rittle-Johnson2002. These techniques sought to encourage protective behaviour. Immaculacy @Suknot2014 is a proposed privacy game, in which the user faces dystopian scenarios. Characters progress through challenges by undertaking privacy-protective actions. This encourages reflection on behaviour, and we adopt a similar approach. Vaidya et al. [-@Vaidya2014] considered interactive techniques to teach privacy. Since privacy is inherently complex, they recommended that scenarios be used. We implement scenario-based challenges, developing the first smartwatch privacy game. **Smartwatch behaviour.** Although wearables have existed for decades, smartwatches have gained recent popularity. They can be defined as "*an electronic wristwatch that is able to perform many of the functions of a smartphone*" @CollinsEnglishDictionary2017. The environment differs greatly from other contexts, particularly when concerning the topic of privacy. Internet-of-Things (IoT) products (defined as belonging to a "*global network interconnecting smart objects by means of extended Internet technologies*" @Miorandi2012) have been criticised for lacking usability @Williams2016x. Smartwatches have small screens and few buttons, with this constraining the use of protective settings @Horcher2015 [@Benbunan-Fich2017]. The devices are also unfamiliar and therefore their navigation is less likely to be understood @Williams2017. Furthermore, they can possess highly-sensitive data @Al-Sharrah2018, while often having great vulnerability @HewlettPackardEnterprise2014. Due to the novelty and idiosyncrasy of this environment, behavioural studies might uncover new insights. Pizza et al. [-@Pizza2016] evaluated behaviour for 34 days, with their participants possessing wearable cameras. They found smartwatches were most often used as timepieces, though they also provided notifications. Jeong et al. [-@Jeong2017] undertook a 203-day longitudinal study, collecting data on 50 participants. They analysed wear, but never considered security or privacy. Indeed, there have been no empirical studies on smartwatch privacy. Our analysis offers a rare glimpse into how these settings are used. Smartwatches are a challenging interface for games, since they possess small screens and few buttons. Casano et al. [-@Casano2016] evaluated an app entitled 'Estimate It!', which sought to teach measurement and geometry. The game was ported to a Tizen OS watch, and users were engaged in gameplay. Design requirements and usability guidelines have also been created for this environment @Li2017a [@JimenezVargas2016]. However, educational games remain greatly underexplored. Before undertaking this research, we developed an online prototype of our smartwatch game (*under review*). Through this app, we evaluated behaviour change and qualitative rationale. To achieve this, we recruited 504 smartwatch users through a crowdsourcing platform. The treatment half played the prototype, which included privacy questions and challenges. The control participants did not interact with an application. In pretest and posttest, we solicited concern and behaviour through an online survey. Whereas the Privacy Paradox was mitigated in the treatment group, control actions failed to change. While this study was encouraging, the game was emulated and behaviour was self-reported. To empirically evaluate the matter, we now construct and evaluate the first smartwatch privacy game. Method {#sec:three} ====== Recruitment ----------- **Sampling process.** 10 Huawei Watch 2 devices were purchased for this study. Since we monitored all participants over the same two-month period (to minimise extraneous variables), the size of our sample was practically limited. Individuals were loaned an expensive device, and as such there was a security risk. Therefore, in compliance with our university's ethical requirements, we recruited from the institution's students. This demographic offers decent external validity, since smartwatch owners tend to be young and educated @Desarnauts2016. **Recruitment.** To participate, individuals were required to fulfil three criteria. They had to be full-time university students, and therefore accountable for their device. They also needed to be 18 years or older, so they could provide informed consent. Finally, they had to possess a modern Android smartphone, to allow their watch to be configured. Of those eligible applicants, we sought to prioritise diversity. Rather than compiling a white British sample, we included a range of nationalities. Privacy is inherently cultural @Alashoor2015a, with research suggesting Asian societies do less to protect personal data @Huang2016a. Therefore, we explored whether European students would use greater protection than those from Asian nations. We also selected individuals from a variety of degree specialisms. We felt this would be more-representative of the public than choosing technologists. To recruit, flyers were affixed to notice boards across the halls of the university. Emails were also sent to mailing list curators, who could forward the messages if they wished. Participants were fully informed of, and consented to, the monitoring of their watch settings. In addition to the privacy-relevant data, we also received approval to track font size, screen brightness and battery level. This disguised the purpose of our study, while also ensuring high ethical standards. Experimental Structure ---------------------- **Overview.** Our longitudinal study was divided into three distinct phases: pretest, gameplay and posttest. The experimental structure is shown in Fig \[fig:structure\]. In a 18-day pretest phase, we monitored the baseline concerns and behaviour of our 10 participants. During a 16-day gameplay phase, these individuals were randomly divided into a treatment group and a control group. Group allocation was truly random, with the process undertaken before study commencement. We considered matching, but thought pure randomisation would reflect the external environment. Fortunately, our groups still appeared well-matched on demographics. ![Experimental structure<span data-label="fig:structure"></span>](structure3.png){width="48.00000%"} The treatment group (*n* = 5) received a customised privacy game, including challenges to refine behaviour. To minimise confounding variables, the control group (*n* = 5) received the same app, but without the privacy theme. Their game concerned general smartwatch use, such as using gestures and adjusting screen brightness. We originally considered having the control group play no game. However, we were concerned that treatment participants might adjust behaviour purely due to study interaction. Therefore, to reduce bias from the Hawthorne Effect @Adair1984, both groups received a game. All participants played their game over a five-day and four-day period, with a one-week gap in the middle. Such two-stage approaches can help to construct mental models @Mayer2002. After these periods were completed, we ended with an 18-day posttest phase. This allowed us to explore whether actions had changed. We decided against including control variables in our study. We could have considered users' smartwatch familiarity, but all participants lacked prior experience. By collecting non-watch privacy opinions, we might have identified baseline concerns. However, since we sought to disguise the topic (to avoid priming), we decided against this approach. **Pretest.** In total, we monitored concerns and behaviour over a 52-day period. We were limited to this span due to the term lengths of our student participants. On day one, 10 individuals were given a Wear OS smartwatch. Once configuration was complete, the monitoring app was installed on each device. This (ethically-approved) service logged settings every five minutes, with details outlined in Subsection \[features\]. At the end of the phase, we distributed a concern questionnaire to each participant. Its queries can be found in the appendices as Table \[tbl:concerns\]. We considered soliciting these opinions at the start of the study. However, users might be unfamiliar with smartwatches and therefore unable to provide informed responses. As none of the participants had used such a device before (as revealed in our posttest interviews), our notion was validated. We also sought to assess concern directly before gameplay, since we wished to explore our games' influence. At the end of the pretest phase, individuals had used their smartwatches for 18 days. Therefore, they should have now been able to provide informed opinions. **Gameplay.** While users completed the questionnaire, we installed one of two games on their watches. The treatment group received a privacy-themed game, which sought to encourage protection. The control group were given an app with identical gameplay, but concerning a different theme. Rather than the challenges (highlighted below) targeting privacy, they related to general smartwatch usage (e.g., adjusting screen brightness). Since both groups received interactive games, we restricted the influence of extraneous variables. Users were instructed to play the games three times per day for 10 minutes each time. At the end of this 16-day phase, participants completed an evaluation questionnaire. These questions can be found in Table \[tbl:evalquestionnaire\] of the appendices. This sought to inform future refinements to the games. The questions did not concern privacy, since we did not wish to prime the topic before the posttest phase. **Posttest.** To prevent further gameplay influencing behaviour, the apps were uninstalled from the watches. For the final 18 days, we continued to monitor privacy behaviour. At the end, we distributed identical concern questionnaires to pretest. This enabled analysis of whether opinions changed as the study progressed. Since these forms were completed 18 days after gameplay, we doubt concerns were unfairly primed. Finally, we conducted semi-structured interviews to explore behavioural rationale. The questions were all open-ended and can be found in Table \[tbl:interview\] of the appendices. Smartwatches were then reset while users received their compensation. They were each compensated with a £40 voucher and entry into a £70 draw. They were also debriefed on our privacy focus, as this was disguised in forms and recruitment. With concerns and behaviour collected, we could now analyse the Privacy Paradox. Threat Model ------------ As will be described, we evaluated privacy concerns through hypothetical scenarios. For these concerns to be assessed fairly, we must define a reasonable threat model. All individuals made use of a Wear OS smartwatch. This watch contains a number of apps, with some developed by companies other than Google/Huawei. Apps are constrained by permissions, allowing data access to be restricted. Data which is read is often shared (potentially anonymised or aggregated) with external parties @Schneier2015. The watches can access GPS, offering location-customised functionality. While this provides navigational benefits, the device's current location is accessed @Ashbrook2003a. These watches are also small, expensive, and consumer-oriented. Like Android smartphones, this places them at a reasonable risk of loss or theft @Matthews2016. Indeed, their "*size and portability makes them easy to steal*" @Baggili2015. Therefore, if a threat is encountered, it would likely come from app companies or petty criminals. Protective Features and Concern Scenarios {#features} ----------------------------------------- **Selection.** To gauge privacy concern, we were required to solicit personal opinions. However, privacy is highly contextual @Nissenbaum2009, and this can challenge a simple rating @Paine2007. For example, 'very concerned' means little when it is divorced from the particular situation. Context is also important when comparing concerns against behaviour. Trepte et al. [-@Trepte2014] were critical of works that juxtaposed abstract opinions against concrete actions. To compare these factors, it is wise to situate them within the same context. We adapted the design of Lee et al.'s influential work [-@Lee2016b], by requesting responses to hypothetical scenarios. This supports an analysis grounded within the smartwatch environment. To enable a fair evaluation, scenarios were selected through three criteria: 1. The issue must be feasible and part of our threat model. 2. The situation should be comprehensible to our sample. 3. Most importantly, there should be a direct correspondence between scenario and privacy-protective tools. Responses were made on a five-point Likert Scale, also adapted from Lee et al. [-@Lee2016b], which ranged from *Indifferent* to *Very Concerned*. As highlighted earlier, these questions can be found in Table \[tbl:concerns\] of the appendices. Individuals then provided a qualitative justification for each answer. Since concerns can be inflated when insufficiently considered @Baek2014, these queries provided a pause for reflection. When analysing behaviour, it was important to consider the available protective tools. In Wear OS environments, three features appeared particularly relevant. These comprised of: *app permissions*, *GPS disabling* and *screen locks*. The settings are outlined below, alongside their respective concern scenario. **Permissions.** Apps[^3] provide useful functions to the smartwatch owner. To provide these services, they often access personal data. While this access is legitimate, details are commonly traded with third parties @Schneier2015. Fortunately, as on smartphones, privacy permissions can restrict access. When applications cannot read details, they cannot share them with partners. To gauge concern, we asked users how they would feel if their data was accessed. We also asked how they would react if data was shared with others. If a person is opposed, they can reduce their risk through permissions. **GPS.** GPS can support great functionality, such as navigation and fitness tracking. To provide these features, a satellite geolocates the smartwatch. By its very nature, this allows the position of a device to be monitored @Ashbrook2003a. If an individual wants to limit this, they can easily disable their GPS. Then, when functionality is required, it can be briefly re-enabled. To evaluate concerns, we asked users how they would feel if their position was monitored. We also asked how they would react if this data was shared with others. If a person fears this, disabling GPS can reduce the risk. **Screen locks.** Passcodes are well-known barriers, and have been suggested to deter smartphone theft @ConsumerReports2014a. Since the Watch OS interface is similar to Android, this deterrent could apply to watches. Smartwatches are small, expensive and popular. As a result, they have been deemed a feasible target for theft @Baggili2015. Through using a screen lock, personal data is better-protected. To gauge concerns, we asked users how they would feel if their missing device was accessed. We also solicited reactions to their apps being used by a stranger. If users are concerned about physical access, a screen lock is a simple solution. Research Questions {#sec:hypo} ------------------ We explore whether the Privacy Paradox can be mitigated through an educational game. To achieve this, we must compare concerns and behaviour in pretest and posttest. Furthermore, we must judge our treatment group results against those of our control group. Therefore, it is crucial that we first define our study metrics. **Metrics.** Concerns are evaluated based on reactions to the above scenarios. Since our Likert data is ordinal, it is conventional to avoid means. However, if questions consider the same topic, it is deemed acceptable to aggregate the scores @Norman2010 [@Carifio2008]. By taking means of these responses, we receive *location scores*, *stranger scores* and *app scores*. The Cronbach alpha values for these question pairs were 0.837, 0.204 and 0.631, respectively @Cronbach1951. Since the second alpha was particularly low, we report responses to the two 'stranger' scenarios (stranger app access and stranger app use) separately. To evaluate behaviour, we developed metrics to summarise participant activity. For the *GPS score*, we calculated the percentage of recordings (taken every five minutes) in which the feature was enabled. Similarly, for the *lock score*, we analysed the percentage of logs in which a lock was present. When assessing permissions, we chose to consider the context of the application. Some permissions were deemed to be innocuous, such as waking the screen or increasing the volume. We made this judgement by considering the personal data that might be accessed. Two permissions concerned particularly private details: precise location and text message contents. We analysed these elements since such details could support privacy invasions @Creese2012. The *permission score* comprised the average acceptance percentage of these two permissions. **Research questions.** We both monitor 52 days of empirical behaviour and conduct 10 in-depth interviews. Through this quantity of data, we seek to address the following questions: 1. Do smartwatch users take action to protect their data? If not, this has implications for smartwatch risk and interface design. 2. In smartwatch environments, does a Privacy Paradox appear to be prevalent? If so, users might place their personal holdenkimberly@example.com. 3. Can the smartwatch game encourage privacy-protective behaviour? If so, such apps could offer an interactive and low-cost complement to awareness campaigns. 4. What factors influence smartwatch behaviour? If we can understand behavioural rationale, we might be better-placed to design interventions. Game Design and Rationale {#sec:four} ========================= We now move forward to discuss the design of our two smartwatch games. Most attention will be given to the privacy version, since this sought to encourage protective behaviour. A YouTube video of the game can be found at: <https://goo.gl/K7DVfL>. Game Narrative and Mechanics ---------------------------- **Overview.** Both games challenged users to navigate across a maze-like map. The privacy version can be found below in Figure \[fig:priv\]. The player starts at their house and then must traverse four levels to reach the shops. En route, they collect coins to increase their score. When the game ends (or is completed), this score is ranked on a competitive leaderboard. During their journey, users encounter two types of Non-Player Character (NPC): 'villagers' and 'thieves'. Villagers ask functionality questions and reward correct answers with points. For example, a player might be asked, "*How can I prevent apps accessing contact details?*", and select the "*Revoke contacts permission*" response. Thieves block the user's path and trigger functionality challenges. In these challenges, characters must configure a settings menu before their health expires. For example, a player might be tasked to enable a screen lock. Success is rewarded by additional coins, while failure ends the game. If the final level is completed on normal difficulty, extra modes are unlocked. All aforementioned components are identical on both games. The only differences were that the privacy app's challenges/questions related to protective features. ![Smartwatch privacy game. Left: SAM playing the game in 'Morning' mode. Right: BOB facing a customised challenge in 'Night' mode.<span data-label="fig:priv"></span>](combined2.jpg){width="48.00000%"} **Challenges.** The general version concerned generic non-privacy smartwatch features. These included adjusting font size, changing screen brightness and configuring alarm volume. In our concern questionnaires, we included decoy questions relating to these settings. Therefore, the purpose of our study was further disguised. The privacy game focused on the three protective approaches: restricting permissions, disabling GPS and enabling a screen lock. These were divided into 14 challenges, shown below in Table \[tbl:challenges\]. <span>cc</span>\ *Disable GPS* & Enable a screen lock pattern\ Check app permissions & Revoke contacts permissions\ \ Enable a screen lock pattern & Check app permissions\ Revoke contacts permissions & *Revoke audio permissions*\ \ Enable a screen lock PIN & Revoke location permissions\ Check system app permissions & *Disable GPS & location perms*\ \ *Revoke SMS permissions* & *Enable a screen lock password*\ *Revoke sensors permissions* & *Lock pattern, GPS, location perms*\ \ \[tbl:challenges\] In seeking to highlight each participant's risk exposure, these challenges were dynamically customised around user behaviour. This was achieved through reading the recent log files of the monitoring app. Based on GPS, screen lock, apps installed and app permissions, we contextualised the tasks. For example, a participant might grant ACCESS\_FINE\_LOCATION permissions to their Uber app. If this had occurred, location challenges (in the game) would be customised with these details. This design followed the influential work of Harbach et al. [-@Harbach2014], who used a similar approach on smartphones. Since our contextualisation was dynamic, the game adjusted to reflect recent user behaviour. This provided an educational feedback loop to encourage protection @Kiili2005. Behaviour Change Principles --------------------------- The games were designed with educational techniques from psychology @Garg2012, learning science @Quinn2005 and HCI @Richards2014. They could be defined as 'operative games', since they "*leverage knowledge gained from the study of games or play to exert control upon the world such as encouraging exercise or learning*" @Carter2014. **Personalisation.** When participants first open the app, they assign themselves a three-digit name. Since customisation contributes to immersion @Annetta2006, individuals should continue in a more-retentive manner. They then personalise their character, toggling gender, hair colour and skin colour. These avatars tend to further increase immersion @Annetta2006a, and might lead to the receptive 'flow' state @Kiili2005. **Practice.** Game challenges required the configuration of a settings menu. Through completing tasks, participants learned directly-applicable skills. Rather than adjusting the real menu, we implemented a simulated interface (found in Figure \[fig:challenge\]). This allowed us to provide on-screen hints, found to offer further education @Woolf2010. It also enabled users to experiment in a safe environment, without being forced to change their own settings. Challenges were time-pressured, encouraging players to remember the menu layout. This sought to trigger 'pleasurable frustration', where users enjoy a fun but challenging task @Gee2004. ![Smartwatch privacy challenges. Left: DISABLE\_GPS challenge in progress. Right: DISABLE\_GPS failure screen, describing the consequences.<span data-label="fig:challenge"></span>](challenges2.jpg){width="48.00000%"} **Education.** Since we wished to enhance privacy knowledge, we included education within the application. When a new game is started, individuals can watch a brief slideshow. The presentation highlights protective approaches and the potential consequences of inaction. In the control group game, the slides concerned general elements of functionality. Again, by including similar features in both apps, we sought to minimise the influence of extraneous variables. **Contextualisation.** We selected an accessible real-world narrative: that of going to the shops. Although we considered more exciting scenarios, we took guidance from the literature @Maldonado2005 and selected a relatable situation. Since understanding is enhanced by aligning physical and virtual risks @Garg2012, we also matched challenges to possible real-world situations. For example, the character faces a challenge when they are near their (gameplay) house. This task requires GPS disabling, since their home location is being 'tracked'. In a later level, a thief and villager are adjacent to each other. Since questions could be overheard, the challenge concerns microphone eavesdropping. By relating risks to real situations, participants might consider threats in the future. **Principles.** To encourage protective behaviour, we implemented the four learning science principles (goal-oriented, challenging, contextual and interactive) @Quinn2005. We achieved this through (1) privacy challenges, (2) difficulty modes, (3) dynamic customisation and (4) rich interactivity. The six principles of educational game design were also implemented @Annetta2010. This was done through avatars (unique identity), rich narrative (immersion), high responsiveness (interactivity), difficulties (increased complexity), challenges (informed teaching), and feedback (instructional). **Behaviour change.** The above paragraphs outline our design techniques but they do not specify our behaviour change mechanism. Primarily, we sought to a) appeal to the availability heuristic @Tversky1973 and b) increase user self-efficacy @Bandura1977. This heuristic describes how "*a person evaluates the ... probability of events by availability, i.e., by the ease with which relevant instances come to mind*" @Tversky1973. By increasing the salience of privacy, we wished to enhance the perceived risk of data infractions. Through self-efficacy, participants should gain confidence to put new skills into action. In concert, this seeks to increase behavioural control. In an attempt to increase salience, we use gameplay feedback, informative questions and interactive challenges. Even when individuals recognise the risks, they need knowledge to protect themselves. Through our educational game, we seek to deliver information and provide an opportunity for practice. This should increase individuals' self-efficacy: the confidence that they have in their own expertise. Aligning with Protection Motivation Theory, if users possess self-efficacy and appreciate the risks, they should be more likely to take action @Rogers1983. We hope this encourages our participants to change their behaviour to a protective state. Findings and Discussion {#sec:five} ======================= Participants and Techniques {#techs} --------------------------- **Participants.** 10 participants used smartwatches for two months. Although four came from the UK, we also had individuals from Ireland, Italy, Russia, Mexico, Singapore and the US. While we suspected that concerns and behaviour might differ by culture, this was not found. A larger sample is required to evaluate the influence of this factor. Eight of the users were male, while two were female. Smartwatch users have been disproportionately male and young @NPDConnectedIntelligence2014, and this trend appears to continue[^4]. Since many also tend to be well-educated @Desarnauts2016, our sample has some validity. None of our participants had ever used or owned a smartwatch before. This inexperience should limit the influence from prior familiarity. As mature smartwatches are relatively recent[^5], an inexperienced sample should be externally valid. **Quantitative techniques.** Since our sample size was small, we used non-parametric measures for our behavioural comparisons @Siegal1956. To significance test independent groups, we selected the Mann-Whitney U Test [-@Mann1947]. If the two samples were related, we chose the Wilcoxon Signed-Rank Test [-@Wilcoxon1945]. We required *p* &lt;0.05 for significance, though its likelihood is limited by our small sample. We used Cohen's *d* for effect sizes, with 0.2, 0.5 and 0.8 representing small, medium and large, respectively [-@Cohen1977]. This metric is less affected by sample size @Cohen1977, and gives an indication of the game's influence. We use $\bar{x}$ for means, as is standard notation. We did not undertake significance testing when comparing concerns or opinions. Since these scores only ranged from 1/5 to 5/5, significance was unlikely in a 10-person sample. We also chose not to apply significance testing to our rationale proportions. As these metrics were based on thematic coding, we preferred to use qualitative analyses. **Qualitative techniques.** Through our questionnaires and interviews, we collected a large quantity of qualitative data. This enabled a rare exploration of the privacy rationale of smartwatch behaviour. To ensure a robust evaluation, we undertook best practice through inductive thematic analysis @Ritchie2003. First, all data was formatted in a consistent manner. For our interviews, the researcher undertook verbatim transcription. This approach provides the most detailed account of a discussion @MacLean2004, further enhancing our validity. We moved on to label recurring topics and concepts. This was undertaken iteratively, seeking to establish consistency between similar replies. Once labelling began to converge, we divided our topics into subtopics. Through this process, we developed conceptual frameworks. These indices then served as our coding frames. Once coding was completed, we selected vivid examples @Braun2006 of rich participant quotes. These are excerpts which we deemed to exemplify a qualitative theme. To select examples, we reviewed those quotes categorised within each topic. If an excerpt was deemed to explain a matter with clarity, it was presented as a vivid example. Through this approach, we aimed to include qualitative description alongside our quantitative findings. These examples are included throughout this section to illustrate user opinions. **Validation.** To maximise our validity, we followed four best-practice procedures. Firstly, since we explored rationale through both questionnaires and interviews, we triangulated our findings @Flick2004. Secondly, our interviews were analysed through multiple coding @Patton1999. A second researcher, not familiar with with the authors' topic, also coded the transcripts. We analysed consistency by comparing the theme distributions through 'proportion agreement' @Morrissey1974. We selected this method over Cohen's kappa @Cohen1960 for two reasons. Firstly, there were a large number of themes, reducing the risk that matching is due to chance. Secondly, since responses often mentioned multiple themes, kappa is not appropriate @Cohen1960. The matching accuracy was 83.4%, suggesting that raters frequently agreed on the categorisation. Thirdly, we did not seek to hide deviant cases. Where opinions could not be conveniently grouped, distinct themes were retained. Finally, we used respondent validation to verify our understanding @Brink1991. Each participant was sent their interview transcript and their assigned codes. They were asked to evaluate the accuracy and to suggest refinements. Fortunately, 100% of the sample agreed with our decisions. Therefore, we believe our findings adequately encapsulate our participants' rationale. Pretest Findings ---------------- Our pretest concern questionnaires were completed 18 days into the study. This ensured that the participants were familiar with their device, but not in possession of training. **Opinions.** Before addressing concerns, the questionnaire assessed general opinions. However, we reserve discussion of these to the posttest section. Importantly, we included an instructional manipulation check within the questions @Oppenheimer2009. In Question 9, participants were asked to indicate their attentiveness by replying 'Strongly Disagree'. This query did not serve other purposes and was not related to privacy. Since all individuals answered correctly, the reliability of our responses was deemed enhanced @Oppenheimer2009. We then assessed privacy concerns, analysing reactions to violation scenarios. In addition to our described incidents, we included six decoy questions. Through this technique, privacy priming should have been mitigated. The proportion of those at least 'concerned' is illustrated below in Table \[tbl:pretest\], while the questions can be found in Table \[tbl:concerns\] of the appendices. **Q** **Concerned (%)** ------- ---------------------------------- ------------------- 16 GPS: Location tracking 50 25 GPS: Location sharing 80 19 Screen locks: Stranger access 80 21 Screen locks: Stranger app usage 70 18 Permissions: App data collection 90 23 Permissions: App data sharing 80 : Pretest privacy concerns<span data-label="tbl:pretest"></span> **GPS.** Our participants expressed some opposition to location tracking, with 50% indicating their concern. However, opinions varied, with another 30% being quite indifferent. When assessing these scenarios, we also considered the participants' rationale. 16 justifications were given, and while 37.5% expressed concerns, 37.5% were dependent on the situation. For example, 18.8% claimed their reaction depended on whether the tracking was optional. If the tracking could be disabled, as GPS can be, they would be less worried. However, for location data to be protected, intentions must turn into action. To directly illustrate participant opinions, we display vivid examples below @Braun2006. We also report the participant ID of the cited individual. Users *A*-*E* were in the treatment group, while *F*-*J* were control participants. At this pretest stage, group membership was irrelevant. "*I want to be able to decide when I can be tracked*" (\#C). The second concern scenario, considering location sharing, faced strong opposition. For this incident, 80% were concerned and nobody expressed indifference. We then considered qualitative justifications, with 21 comments provided. The vast majority expressed concern (85.7%), with most individuals objecting to the principle (23.8%). Many also thought this was illegal (14.3%), but it might be consented through privacy policies. If individuals are truly opposed to this sharing, access can be limited by disabling GPS. "*I would very much feel as though my privacy was invaded*" (\#A). Whereas concerns were reported, behaviour was collected in our smartwatch logs. At this pretest stage, all 10 users had their GPS enabled. Indeed, not a single person had adjusted this setting. This implies that their location was accessible for the first 18 days. When considering **RQ1**, it appears that protective actions are rarely taken. Since the public are unlikely to be receive training, this presents a worrying baseline. We hope that through our game, the issue will gain salience. **Screen locks.** When considering unauthorised access, participants were worried. 50% indicated their strong opposition, while not a single respondent was indifferent. This suggests that users generally reject this intrusion. 15 justifications were given, with 60% of these fearing great damage. Concerns were primarily driven by the security risk (20%) and the importance of personal data (20%). This suggests that our participants place value on their smartwatch. Two individuals were less concerned, with one believing that that their password would protect them. If protective features are enabled, the privacy risk might be mitigated. "*I don't want a stranger to know my whereabouts*" (\#G). Participants expressed similar concerns over their apps being used. 70% were in opposition, with only one person expressing indifference. This suggests that application access is strongly rejected. On this occasion, 14 comments were provided for justification. 64.3% of these feared great damage, with data access being the most common concern (28.6%). This is understandable, since apps can contain personal details. Only one participant was unconcerned, and they believed their apps were not sensitive (7.1%). However, if app usage is feared, a screen lock might be appropriate. "*They could cause issues through contact and they could gain my details*" (\#E). Fortunately, four participants had enabled a screen lock. Since games had not yet been played, this suggests the feature is well-known. This might be due to the prevalence of smartphone PINs and patterns. The other six participants had never used a lock. Despite their inaction, they still claimed concern over the scenarios. In these situations, it is likely that the setting was never noticed. Considering **RQ1**, this implies that protective action is far from constant. **Permissions.** We next gauged concern towards an app accessing data. Respondents were strongly in opposition, with 90% disliking this situation. When considering the justifications, 16 different comments were given. While 25% depended on particular details, 62.5% expressed strong concern. Many objected to the access on principle (25%), whether or not it posed a risk. Individuals might not act to prevent an issue, but oppose it at an ideological level. In such cases, a disparity is often found between concern and behaviour. "*I would want them to respect my privacy*" (\#E). When considering data sharing, 80% opposed the incident. Our users appear to reject these practices, despite them being commonly found @Schneier2015. We then considered the qualitative justifications, with 17 comments provided. 58.8% of these expressed concern, compared to only 17.6% with little worries. As before, the most popular objection was purely on principle (23.5%). These individuals found this data sharing to be invasive. If they wish to limit the content, they could choose to change their permissions. "*I value my online privacy*" (\#F). To assess empirical behaviour, we analysed the pretest logs. Throughout this 18-day period, not a single participant had restricted their permissions. In fact, these settings had been loosened by 9/10 users. Furthermore, two installed additional apps with sensitive permissions. Since the public are unlikely to receive education, this presents a worrying baseline. **RQ1.** In our first research question, we explored whether smartwatch users protect their data. Although screen locks were used by some, 60% neglected the feature. GPS was used constantly by all individuals, enabling locations to be identified. In the case of permissions, settings were loosened rather than tightened. Based on these results, smartwatch users rarely behave in a protective manner. **RQ2.** Our second research question explored whether the Privacy Paradox was common in this environment. To illustrate the degree of concern, the mean scores are displayed below in Figure \[fig:pretest\]. ![Pretest mean privacy concerns.<span data-label="fig:pretest"></span>](pretest3.png){width="48.00000%"} Although location tracking was not strongly opposed, sharing provoked negative responses. This did not encourage any participants to disable their GPS. Our respondents also rejected both unauthorised access and app usage. While some mitigated the risk through passwords, others expressed similar concern. Finally, users appeared to strongly disagree with apps accessing or sharing data. However, they chose to loosen their permissions. Based on these results, concerns and behaviour appear misaligned. Our smartwatch game seeks to mitigate this issue. Gameplay Opinions ----------------- **Evaluations.** At the end of the gameplay phase, participants completed an evaluation questionnaire. This form can be found in Table \[tbl:evalquestionnaire\] within the appendices. It did not concern the topic of privacy, since we did not wish to prime posttest behaviour. We first solicited agreement with statements through Likert Scale questions. Users were asked whether they assessed the games as enjoyable, usable, educational and challenging, respectively. Across our sample, 60% expressed that they enjoyed the games. While it is encouraging that most participants were pleased, we would hope to increase this percentage in future iterations. The agreement level was equal in each group (60%), suggesting the privacy theme did not detract from enjoyment. 70% deemed the apps to be usable; promising since usability can encourage retention @Annetta2010. In this case, 80% agreed in the control group, compared to 60% in the treatment group. As will be outlined in Section IV-F, some of the privacy menus were challenging to navigate. This might have contributed to the decreased percentage. Since we wished to inform participants, we hoped the apps were considered educational. Fortunately, all respondents agreed with the statement. However, differences existed between our groups. Whereas 80% of control participants were in strong agreement, this was matched by 20% of the treatment group. Surprisingly, this suggests that the generic game was considered more educational. This opinion might have emerged for two reasons. Firstly, since the privacy app only concentrated on the three protective features, its content was narrower. Secondly, as evidenced in our final Likert-Scale question, privacy tasks were found more challenging. Whereas generic tasks were deemed simple (100%), nobody thought the same of treatment challenges (0%). This suggests that privacy is a more-complex topic, and might explain why protective settings are frequently overlooked. **Opinions.** We then proceeded to extract opinions through qualitative questions. Their open-ended responses were coded through the thematic analysis highlighted in Subsection \[techs\]. Firstly, we asked individuals what they most liked about the game. Usability was most praised, with this contributing to 41.2% of responses. The ease of interaction was particularly appreciated (23.5%), suggesting our game was simple to play. This is encouraging, since usability has been found to encourage retention @Annetta2010. When discussing dislikes, participants mentioned 14 factors. The most-frequent complaint was that the games were repetitive (28.6%). This was partially intentional, since repetition is a standard approach to ingrain knowledge @Franzwa2013. In future implementations, greater randomness might make the issue less apparent. When we asked for suggested improvements, 19 comments were submitted. 57.9% were in favour of extending the game, with 26.3% requesting additional 'challenges'. This might suggest that users saw feasibility in our approach. While our challenges currently concern installed programs, we could add tasks based on app installation. Through these participant responses, we can refine our games for future interaction. Posttest Concerns ----------------- Users returned their smartwatches at the end of the posttest phase. They then completed final surveys, identical in design to the pretest forms. This allowed fair examination of whether opinions had changed. Since the games had not been played for almost three weeks, they should not prime privacy. Furthermore, our purpose should be disguised by the decoy questions. Due to the small sample sizes, we do not include *p*-values in our below discussion. For ordinal comparisons in a 10-person sample, significance is highly unlikely. As an overview, the pretest-posttest concern proportions (the percentage of those at least responding 'concerned') are presented below in Table \[tbl:prepost\]. The final columns highlight the pretest-posttest change. --------------------- --------- --------- --------- --------- --------- --------- **Tmt** **Cnt** **Tmt** **Cnt** **Tmt** **Cnt** Location tracking 20 80 60 60 +40 -20 Location sharing 60 100 40 100 -20 0 Stranger access 60 100 100 60 +40 -40 Stranger app usage 100 40 80 80 -20 +40 App data collection 100 100 60 100 -40 0 App data sharing 80 80 60 80 -20 0 --------------------- --------- --------- --------- --------- --------- --------- : Pretest-posttest privacy concerns\ Tmt: Treatment Group, Cnt: Control Group<span data-label="tbl:prepost"></span> **Opinions.** In this section, we explored our participants' privacy perceptions. To assess awareness, we asked whether personal data could be read by apps. While agreement decreased in the control group (from $\bar{x}$ = 4.8 to 4.6), it increased in the treatment group (from $\bar{x}$ = 4.4 to 4.8). We then solicited their confidence in their own understanding. As expected, treatment users appeared to have greater self-efficacy than the other group ($\bar{x}$ = 4.6 vs 3.8). In our third question, participants were asked whether an app might threaten smartwatch data. Whereas the treatment group perceived a threat ($\bar{x}$ = 4.8), controls appeared to lack this knowledge ($\bar{x}$ = 3.4). Although our sample size impedes significance, the privacy game might have enhanced understanding. All users again succeeded in the instructional manipulation check. This implies that responses were made in an engaged manner @Oppenheimer2009. **Location.** When considering location tracking, both groups became more worried. Concerns were now slightly less contingent, and instead focused on the principle of violation (3/10 participants). Users also began to consider targeted advertising
{'source': 'arXiv_out/1905'}