Learning Launch Reflection 

 Challenge 

 One of the responsibilities of my role as online chat channel leader was to increase customer   satisfaction ( C - SAT ) scores .   Customer satisfaction is measured through surveys given to customers   immediately after the chat is over .   Agents can see the results of these surveys , but since the   information is available only after the chat is over , there is obviously no way for agents to go back and   improve upon that poor interaction .   They can only try to do better with the next customer .   Moreover ,   supervisors can only see their agents ’ scores as interactions are completed throughout the day .    Therefore , they can only coach to deficiencies after - the - fact . 

 While searching for tools and practices that might help increase customer satisfaction , I learned that   data scientists at my company had developed a Natural Language Processing ( NLP ) model that used   account notes to predict if a customer would file a regulatory complaint . I asked , “ What if we used this   same to approach to analyze historical chat transcripts and uncover the correlation between certain   words / phrases and C - SAT scores ? ”   The data scientists quickly proved that such a model could be built   and was indeed highly predictive .   I then asked , “ What if we predicted the C - SAT score in real - time so   that agents can better understand the customer ’s emotional state and adjust their responses and so   that supervisors can intervene on the chats that are most at risk of ending with dissatisfaction ? ” 

 Selection 

 These “ what - if ” questions were intriguing – particularly because 1 ) our chat operation was staffed with   offshore , non - native English - speakers , 2 ) the agents handled up to three chats simultaneously , and 3 )   the emotive content of written channels like chat or correspondence is naturally lower than in - person or   phone channels .   We hypothesized that our agents ( or at least some subgroup ) could improve their C- SAT scores if they had a visual , quantitative display of customer sentiment throughout the conversation .    At the same time , we worried the tool , if also given to supervisors , would put too much pressure on the   agents and cause them to take too long to complete normal requests .   Since such a tool had never been   used at the company , we had no prior data to help us assess the impact on C - SAT scores or on handle   time .   Therefore , it was fairly clear that we would need to run a pilot / learning launch .   The alternative   ( which had been done far too often in the past ) would have been to assume the idea was a good one   and to immediate start full - scale development . 

 A key assumption was that C - SAT scores are materially impacted by the degree of empathy shown by   the agent , as opposed to say , the policies of the company or the capability of the agents .   We knew from   experience all of these play a role , but it is nearly impossible to quantify the relative degree .   Another   assumption was that one or more of the factors above impeded the agents from empathizing with   customers .   There was a hard push from our analysts to set up a statistically significant A - B test to   measure impact to C - SAT scores and handle time , but this would have required significantly more work ,   a longer time to market , and greater cost .   As James Angelo pointed out in the “ Learning Launch ”   video , it was critical to get learnings quickly and cheaply .   Therefore , we decided that the goal of the first 

 learning launch would be to gather qualitative feedback , and if it was favorable , we would then make   the additional investment in a true A - B test . 

 Application 

 We assembled a team that had visual design , software engineering , data science , and operations   expertise .   Our first step was to develop paper prototypes showing potential features and different   visual displays of what agents and supervisors would see when using the tool .   We got feedback from   several focus groups and used their input to make decisions on specific design elements .   Our second   step was to develop a working prototype so we could do a show - and - tell , rather than a PowerPoint   presentation , to get leadership and governance approvals .   The last step was to develop a working   solution that could be used by up to 10 % of our agents simultaneously .   Because of our decision to not   pursue the more difficult A - B test , we completed the entire development cycle , from concept to learning   launch , in only 8 weeks – this was virtually unheard of and was held up as a beacon achievement . 

 Insights 

 Through our interviews with agents , we learned that the visualization tool was only marginally helpful .    It turns out that humans are pretty good at inferring customer emotion even if they are non - native   speakers , working in a written channel , and handling 2 - 3 conversations at once .   This was disconfirming   evidence to our main hypothesis .   However , the tool was extremely helpful for supervisors whose job is   to coach .   It allowed supervisors to monitor all interactions in real - time and intervene on at - risk   interactions while they could still make a difference .   And rather than agents feeling like their   supervisors were looking over their shoulders , agents appreciated supervisors ’ pro - active intervention .    This learning made us change from thinking as agents as the primary users to thinking of supervisors as   the primary users . 

 Approach 

 I believe there were several key things we did correctly in this learning launch .   First , the development of   a paper prototype was instrumental in getting early user feedback and for keeping the software   developers focused only on the essential features – not whistles - and - bells .   Second , we accepted a   lower fidelity prototype that what would have been needed for an A - B test , and thereby obtained   course - altering learnings significantly quicker and at lower cost . 

 I believe the project could have gone even more smoothly if , at the beginning , we trained the project   team and key stakeholders on the design thinking process ( e.g.   What is ? What if ? What wows ? What   works ? ) .   I found that it was very difficult for people to stay in the uncertainty of “ what if ” and “ what   wows ” .   They always wanted to jump to what works .    I believe deeper knowledge of the process may   have enabled people be more patient with the process .   Also , I believe this project was more of the   traditional R&D - to - application approach .   We discovered a technology ( NLP / predictive modelling ) and   applied it to customer service operations .   I think it would have been interesting to do ethnographic   interviews with agents before having any solution at all in mind , because I think we might have   discovered an unstated need that would have taken us in a totally different direction . 


EMAIL: 
ID_NUM: 
NAME_STUDENT: 
PHONE_NUM: 
STREET_ADDRESS: 
URL_PERSONAL: 
USERNAME: 

### Task: You are a PII (Personal Identity Information) Detection. Given a essays written by students, you have to identify ONLY the following:
NAME_STUDENT - The full or partial name of a student that is not necessarily the author of the essay. This excludes instructors, authors, and other person names.
EMAIL - A student’s email address.
USERNAME - A student's username on any platform.
ID_NUM - A number or sequence of characters that could be used to identify a student, such as a student ID or a social security number.
PHONE_NUM - A phone number associated with a student.
URL_PERSONAL - A URL that might be used to identify a student.
STREET_ADDRESS - A full or partial street address that is associated with the student, such as their home address.
### Input: Learning Launch Reflection 

 Challenge 

 One of the responsibilities of my role as online chat channel leader was to increase customer   satisfaction ( C - SAT ) scores .   Customer satisfaction is measured through surveys given to customers   immediately after the chat is over .   Agents can see the results of these surveys , but since the   information is available only after the chat is over , there is obviously no way for agents to go back and   improve upon that poor interaction .   They can only try to do better with the next customer .   Moreover ,   supervisors can only see their agents ’ scores as interactions are completed throughout the day .    Therefore , they can only coach to deficiencies after - the - fact . 

 While searching for tools and practices that might help increase customer satisfaction , I learned that   data scientists at my company had developed a Natural Language Processing ( NLP ) model that used   account notes to predict if a customer would file a regulatory complaint . I asked , “ What if we used this   same to approach to analyze historical chat transcripts and uncover the correlation between certain   words / phrases and C - SAT scores ? ”   The data scientists quickly proved that such a model could be built   and was indeed highly predictive .   I then asked , “ What if we predicted the C - SAT score in real - time so   that agents can better understand the customer ’s emotional state and adjust their responses and so   that supervisors can intervene on the chats that are most at risk of ending with dissatisfaction ? ” 

 Selection 

 These “ what - if ” questions were intriguing – particularly because 1 ) our chat operation was staffed with   offshore , non - native English - speakers , 2 ) the agents handled up to three chats simultaneously , and 3 )   the emotive content of written channels like chat or correspondence is naturally lower than in - person or   phone channels .   We hypothesized that our agents ( or at least some subgroup ) could improve their C- SAT scores if they had a visual , quantitative display of customer sentiment throughout the conversation .    At the same time , we worried the tool , if also given to supervisors , would put too much pressure on the   agents and cause them to take too long to complete normal requests .   Since such a tool had never been   used at the company , we had no prior data to help us assess the impact on C - SAT scores or on handle   time .   Therefore , it was fairly clear that we would need to run a pilot / learning launch .   The alternative   ( which had been done far too often in the past ) would have been to assume the idea was a good one   and to immediate start full - scale development . 

 A key assumption was that C - SAT scores are materially impacted by the degree of empathy shown by   the agent , as opposed to say , the policies of the company or the capability of the agents .   We knew from   experience all of these play a role , but it is nearly impossible to quantify the relative degree .   Another   assumption was that one or more of the factors above impeded the agents from empathizing with   customers .   There was a hard push from our analysts to set up a statistically significant A - B test to   measure impact to C - SAT scores and handle time , but this would have required significantly more work ,   a longer time to market , and greater cost .   As James Angelo pointed out in the “ Learning Launch ”   video , it was critical to get learnings quickly and cheaply .   Therefore , we decided that the goal of the first 

 learning launch would be to gather qualitative feedback , and if it was favorable , we would then make   the additional investment in a true A - B test . 

 Application 

 We assembled a team that had visual design , software engineering , data science , and operations   expertise .   Our first step was to develop paper prototypes showing potential features and different   visual displays of what agents and supervisors would see when using the tool .   We got feedback from   several focus groups and used their input to make decisions on specific design elements .   Our second   step was to develop a working prototype so we could do a show - and - tell , rather than a PowerPoint   presentation , to get leadership and governance approvals .   The last step was to develop a working   solution that could be used by up to 10 % of our agents simultaneously .   Because of our decision to not   pursue the more difficult A - B test , we completed the entire development cycle , from concept to learning   launch , in only 8 weeks – this was virtually unheard of and was held up as a beacon achievement . 

 Insights 

 Through our interviews with agents , we learned that the visualization tool was only marginally helpful .    It turns out that humans are pretty good at inferring customer emotion even if they are non - native   speakers , working in a written channel , and handling 2 - 3 conversations at once .   This was disconfirming   evidence to our main hypothesis .   However , the tool was extremely helpful for supervisors whose job is   to coach .   It allowed supervisors to monitor all interactions in real - time and intervene on at - risk   interactions while they could still make a difference .   And rather than agents feeling like their   supervisors were looking over their shoulders , agents appreciated supervisors ’ pro - active intervention .    This learning made us change from thinking as agents as the primary users to thinking of supervisors as   the primary users . 

 Approach 

 I believe there were several key things we did correctly in this learning launch .   First , the development of   a paper prototype was instrumental in getting early user feedback and for keeping the software   developers focused only on the essential features – not whistles - and - bells .   Second , we accepted a   lower fidelity prototype that what would have been needed for an A - B test , and thereby obtained   course - altering learnings significantly quicker and at lower cost . 

 I believe the project could have gone even more smoothly if , at the beginning , we trained the project   team and key stakeholders on the design thinking process ( e.g.   What is ? What if ? What wows ? What   works ? ) .   I found that it was very difficult for people to stay in the uncertainty of “ what if ” and “ what   wows ” .   They always wanted to jump to what works .    I believe deeper knowledge of the process may   have enabled people be more patient with the process .   Also , I believe this project was more of the   traditional R&D - to - application approach .   We discovered a technology ( NLP / predictive modelling ) and   applied it to customer service operations .   I think it would have been interesting to do ethnographic   interviews with agents before having any solution at all in mind , because I think we might have   discovered an unstated need that would have taken us in a totally different direction .
### Output: 