<reponame>marcusnygren/YoungDriveMasterThesis<filename>discussion/4_how_design_affects_usability_learning.tex
%\section{Learning development}

\section{How Does Design Affect Usability and Learning done via the App?}

  The quiz design is made with <NAME>'s model of deliberate practice in mind. \cite{sierra} Depending on the combination of correct and sure, the coach knowledge for each question is put into three different buckets: "Can't do", "Can do with effort" och "Can do effortlessly". By retaking the "Can't do" questions with Try again, reflecting on the "Can do with effort" questions (for example: correct but unsure), and waiting to test the "Can do effortlessly" questions until the certification, all questions are eventually put into "Can do effortlessly".

  The coach can choose to leave the training without doing the certification quiz, later repeating the test. This is to make the learning self-directed and just-in-time, and to allow the coach to do it's own scaffolding. If the coach uses the app before a youth session, and has a low result, the coach will get feedback as such (for example: "Nice effort, but you still need to practice and prepare yourself even more! How can you do that?"). The goal is to allow the coach to move knowledge from "Can't do" to "Can do effortlessly" to "Certified" in a pace that suits the coach. This might be neccessary, instead of a one-fits-all solution, as the coaches' preferences are so different.

  The learning goes faster in iteration 4 than in iteration 3, which is most notably shown by the fact that the speed from try 1 into getting 100\% in 1 try has increased. This is largely because usability issues has been addressed, and because design choices has been made that stimulates and makes learning more efficient. Most notably, the score board has been improved to show which questions the coach is correct and sure of ("Can do effortlessly"), unsure but correct on ("Can do with effort"), and were wrong on ("Can't do"). One mistake made, was no not think about reducing cognitive load \cite{sierra} until iteration 4, when usability issues were identified as serious problems in iteration 3.

  The YoungDrive app is the first known application which uses a confidence metric (in this case "Are you sure?") for the student's own sake, and not only assessment, like in the case study detailed by \cite{nicol}. The effect is reaching meta-cognitive on Bloom's.

  %\subsubsection{Benefits of "Are you sure?"}
  %"Det här kan motverka %traditioner och ”så här vi alltid gjort det” genom att %tvinga en att reflektera över varför inte ens rätta svar %korrelerar med hur empowered du känner dig. Bryter normer, %sätter sig emot lathet och agerar proaktivt för en skarpare %utveckling tillsammans."



  %Lena frågades också om vilken skala jag ska ha på 5, 4 eller %vad jag inte tänkt på, 2-gradig skala. 5 eller 4 är vilket %som enligt literatur, det finns två olika skolor. 2-gradiga %skalan bedömde jag vara bäst, p.g.a. användarvänlighet, tydligt för coacherna

  \subsection{Benefits with Confidence Level and Correctness in Combination}
  If a coach is wrong and sure about a lot of questions, it might be the indication that the coach is teaching the wrong information to the youth, which might potentially hurt hundreds of their youth's businesses. If the coach is correct but not confident, it could be considered a guess. In Sierra's framework of building expertise, it would be called "Can do with effort". In the app during iteration 3, dampened for iteration 4, there were troubles when coaches passing the training with too many correct guesses, knowledge that were not yet "Can do effortlessly". This meant that they would fail the Certification, because they could not answer reliably, providing the wrong alternative. This in turn meant, that they were put back into training instead of getting a medal, which was not motivating. The conclusion was that the coach should not start the Certification mode before being truly ready.
  %Fanns extremt många fördelar med denna, och kom bara fram %till ännu fler efter diskussioner med människor och Lena %Tibell, framför allt hur denna kan förbättra utbildningen %och 1-on-1 coachning, och bli väldigt bra självreflektion %för coachen.

  One solution could have been that "Improve" would not only include repeating wrong answers, but also answers where the coach had been correct but said "No" on "Are you sure?". However, coaches seldom believed they would be wrong, or at least did not determine it worthwhile to be honest. Some preventative measures were taken to try to make the coach more honest. Unfortunately, for iteration 3, only some coaches took notice of number of tries as an indication that they should pay more attention. For iteration 4, the design had improved by giving coaches minus points for being sure and wrong, but here as well, not everyone paid attention. This goes in-line with research both from deliberate practice being a desirable difficulty (learning being hard might make the coach more likely to want to "cheat", or take an easier route like guessing instead of putting more effort).

  The bigger problem was that the knowledge were not yet reliant, not that coaches were not honest. The solution for iteration 4 was to improve learning in the app, partly by a more personal score board with feedback, which could show the coach which kind of questions she needed to repeat ("Can't do" or "Can do with effort"), but instead of forcing the coach to redo correct guesses ("Can do with effort"), she could reinforce the correct answers by personal feedback. Sierra \cite{sierra} calls this "high pay-off tips", which can be very effective.

  \subsection{Deciding on Learning Methods}
  There was a lot of work behind choosing the learning design methods in Iteration \#3. The way to progress, was to brainstorm various solution, discuss them with experts, and then create trigger material and test some of these approaches.

  Retaking questions that were wrong ("Try again", called "Improve" in iteration 3) was inspired by deliberate practice \cite{sierra}, and is already common in e-learning driver license software to learn traffic signs or how to act in various situations.

  Showing the coach how many quiz tries they have done, was inspired by Linköping University's work with the e-learning tool NTA Digital, where they reward students with badges for getting 100\% in few tries. Their goal with this kind of "gamification", is to reward students for studying before taking the test. Similarly, for the training mode, the coach seeing number of tries was a method of studying the correct answers more thoroughly. For the certification, where the coach was supposed to have trained before taking the test, badges worked with the same purpose as NTA Digital, to reward students that had studied properly.

  %In their application, this is so subtle to the user that only performance-driven students might take notice, whereas students that are not motivated by this are not discouraged.

  "Are you sure?" was inspired by a Swedish teacher, and has been used before by others \citep{nicol}. It has then been used to determine if a right answer should award a point or not. \cite{nicol} and the pedagogical expert for this thesis, <NAME>, suggested that the teacher had overlooked a learning benefit of this approach: the student reflecting on their own knowledge, which is proved great for learning. This was extended in this thesis where personalised feedback has the goal of the coach getting both confident and correct. In a school situation, this might not be necessary, but in the YoungDrive context, the purpose of the app was to build both correctness and confidence with the material. To pass the certification after training, getting 100\% without faults, made the coach feel that confidence, at the same time reassuring the teacher that the coach had learned the material.

  After iteration 4 a bonus test was made with Plan Tororo staff, which showed the relevancy of the certification mode: one group that were 100\% correct on the first try, did get 100\% correct on their second try, meaning guesses had been present. On the other hand, a person have 1 wrong answer, passing the training on the second try, did then pass the certification on the first try, with confidence. It can therefore be determined, that when all of the answers are answered correctly, after having gotten all answers correct once, that the coach has both correct information and confident - this is a good example of deliberate practice: the information has gotten reliable.

  Other learning design methods were considered, as previously discussed. It was discussed if multiple-choice answers should be completely abandoned, replaced with flashcards. Challenges were that the coaches had no previous knowledge of typing on a keyboard, and analysing recorded answers would be too technically demanding. The integration and benefits of flashcards was however discussed in various ways, see Future work.
