Domain-Specific Improvement on Psychotherapy Chatbot Using Assistant
Cheng Kang Daniel Novak Katerina Urbanova Yuqing Cheng Yong Hu



Interpretable Machine Learning Models for Predicting the Next Targets of Activist Funds
    [
    April 29, 2024
=======================================================================================



Large language models (LLMs) have demonstrated impressive generalization capabilities on specific tasks with human-written instruction data. However, the limited quantity, diversity, and professional expertise of such instruction data raise concerns about the performance of LLMs in psychotherapy tasks when provided with domain-specific instructions. To address this, we firstly propose Domain-Specific Assistant Instructions based on AlexanderStreet therapy, and secondly we use an adaption fine-tuning method and retrieval augmented generation method to improve pre-trained LLMs. Through quantitative evaluation of linguistic quality using automatic and human evaluation, we observe that pre-trained LLMs on Psychotherapy Assistant Instructions outperform state-of-the-art LLMs response baselines. Our Assistant-Instruction approach offers a half-annotation method to align pre-trained LLMs with instructions, and provide pre-trained LLMs more psychotherapy knowledge. 



Assistant-Instruction, Psychotherapy Chatbot, Large Language Model, Adaption Fine-tuning, Knowledge Retrieval, Parameter Efficient Fine-Tuning  






§ INTRODUCTION



Large Language Models (LLMs) have demonstrated impressive generalization capabilities, such as in-context learning <cit.>, chain-of-thoughts reasoning <cit.>, and biomedical diagnosing <cit.>. Instruction-tuning of LLMs has enabled them to follow natural language instructions and perform real-world tasks <cit.>. Two main methods have been developed for instruction-tuning LLMs: (1) fine-tuning the model on a wide range of tasks using human-annotated prompts and feedback <cit.>, and (2) supervised fine-tuning using public benchmarks and datasets augmented with manually or automatically generated instructions <cit.>. Reinforcement Learning on Human Feedback (RLHF) has proven to be an effective way to improve LLMs in various domains, such as medicine <cit.>, knowledge graphs <cit.>, multimodal data fusion <cit.> and biomedical applications <cit.>, but it comes with a high cost. Self-Instruct tuning <cit.> and Guess-Instruction tuning methods have shown better performance in aligning LLMs with human intent by learning from instruction-following data generated by state-of-the-art instruction-tuned teacher LLMs (e.g., GPT-3, GPT-3.5, and even GPT-4). These lines of instruction-tuning research have proven effective in improving the zero and few-shot generalization abilities of LLMs. To improve the professional knowledge of LLMs on psychotherapy domains, our paper presents the psychotherapy Assistant-Instruction approach, which aims to (1) achieve generalization over different psychological consulting tasks and (2) incorporate psychological knowledge into natural common LLMs. Figure <ref> provides an overview of our proposed approach, in which a single model can perform various NLP tasks in specific psychotherapy domains.









To achieve human-level professional responses in instruction tuning for psychotherapy, we propose a novel approach using GPT-4 as an assistant for Assistant-Instruct tuning (a half self-instruct tuning method) on psychotherapy consulting tasks (Seen in Figure <ref>). Our method makes the following contributions: (a) it covers a wide range of psychological topics and incorporating feedback knowledge generated by GPT-4. (b) it absorbs psychotherapy knowledge from professional data and enables them to generate content close to GPT-4. (c) it demonstrates the effectiveness of using assistant LLMs-revised instruction data to tune LLMs in psychotherapy domains, providing practical insights to build a general-purpose LLM-following agent powered by assistant LLMs (e.g., GPT-4).






§ METHOD






 §.§ Assistant on Annotation and Task Identification



To arrange psychotherapy data to correct tasks, such as (1) concept explanation, (2) question answering, (3) mental status assessment, (4) psychological counseling and (5) information extraction, (6) dialogue generation, (7) sentiment analysis, (8) event ordering, we use an assistant LLM – GPT-4 to identify which task the human-constructed instruction should be. We directly prompt the LLM in a few-shot way to determine this, using 8 classification instructions from the seed tasks. The prompting template is shown in Table 1.










 §.§ Assistant on Generation, and Evaluation






Our approach involves two main steps. Firstly, we optimize formulations that retain the content of the original instructions. We prompt a language model to reformulate the tasks in the core data for each generated task. In some instruction formulations, we embed the input into or add it behind the “INPUT” template – "We are talking about [***]." – to emphasize the topic. This manually constructed “INPUT” also captures the content discussed by members of the audience in Alexander Street Video, merging the discussed topic with the point of interest for the audience or visitors. Secondly, following <cit.>, we use GPT-4 as an assistant to evaluate the retrieved passage's relevance.The prompting template is shown in Table 2.











§ EXPERIMENTS






 §.§ Data Collection


Alexander Street Press is a website known for its vast collection of video transcripts and recordings from therapy and counseling sessions, covering topics such as depression, abuse, trauma, and mental disorders. The video transcript dataset was specifically collected from the Counseling and Therapy channel on the website. We curated the dataset to include only English-language sessions recorded between 1980 and 2023, resulting in a set of 1,333 videos and accompanying transcripts. After filtering out short-length and non-informative videos, the final dataset comprises 1,179 video transcripts, containing a total of 188,421 dialogue turns. To ensure data quality, we performed a cleaning process to remove Unicode characters, pauses, and other unnecessary elements, resulting in a dataset with 3,141,520 words and a vocabulary size of 30,438.[<https://alexanderstreet.com/>]


On the Alexander Street Press website, most video transcripts and recordings consist of knowledge presentations and counseling talks. For knowledge presentations, there are no instruction questions or instance inputs, and the output is the content presented by the speaker. In the first step, we manually set instructions and instance inputs based on the discussed topics (e.g., Depressive disorders, Addiction, etc.). In the second step, we used the GPT-4 API to revise and generate instructions and instance inputs based on the contents.




 §.§ Models



We conducted an evaluation of the language models mentioned above for the task of response generation in the psychotherapy domain, specifically focusing on therapeutic counseling. For generating the assistant instructions based on new psychotherapy data, we utilized the GPT-4 API as the Assistant-LLM. To fine-tune the generated instruction data effectively, we employed the inhibition adaption fine-tuning method <cit.> and self-RAG <cit.> on Llama2-7B <cit.>, as well as ChatGLM2-6B <cit.>. The fine-tuned LLMs were then evaluated by two psychologists on psychotherapy data. The fine-tuning process required two weeks for ChatGLM2-6B/Llama2-7B when using eight NVIDIA Tesla A100 GPUs with 40GB graphic memory cards.[<https://github.com/ChengKang520/psychotherapy-assistant_instruction>]








 §.§ Metrics


As shown in Table 5, our fine-tuned LLMs are evaluated under two main methods: Automatic evaluation and Human evaluation. For the automatic evaluation, we submit 200 generated answers and use Rouge-L <cit.> and Fluency <cit.> to compare the generated output with a reference text and calculate a score for the similarity evaluation. For the human evaluation, we also submit 60 generated outputs (20 examples for each tasks, as we have 3 main tasks) without model and domain information to two professional psychologists, and based on their professional psychotherapy knowledge, they will measure the quality of the generated content according to the questions and answers. There are six grades that they will use to mark the quality of generated content: Extremely  Bad (1), Bad (2), Neutral (3), Acceptable (4), Good (5) and Very Good (6). They should consider the readability, professional and match score (that measures the match with the domain knowledge).





 §.§ Analytic Experiments










  §.§.§ Performance on Revision


Due to numerous vocabulary and grammar errors in the original psychotherapy data, manual processing and revision of these errors were not feasible. In Table 1, we presented the one command that requires generated instructions from GPT-4. The right panel of Table 3 contains additional information contributed by GPT-4, such as common knowledge about depressive disorders, including psychotherapy, cognitive-behavioral therapy, medication, and a combination of treatment approaches. These recommendations from GPT-4 are considered common-sense by psychologists, but the original psychotherapy data lacked sufficient common knowledge (because it always merely has professional knowledge). The instructions and instance inputs were also refined by GPT-4 based on the given command. Perplexity is a metric that gauges the language model's ability to predict a sequence of words. The perplexity of the right panel is lower than that of the left panel. This revision process makes the content more LLM-friendly, transforming "professional knowledge format" into a more accessible "common knowledge format" (or open-domain knowledge format).







  §.§.§ Generation on Psychotherapy Domain


When comparing to the content generated by pre-trained ChatGLM2-6B, we observed that the RAG method contains additional professional knowledge (can be seen from Table 4). While this extra knowledge provides more professional information, pre-trained LLMs could not offer further professional insights into autism. When comparing the generated content of ChatGLM2-6B fine-tuned on inhibited LoRA to that of ChatGLM2-6B, we found that the former provides more refinedly revised information (However, this difference is not obvious). For example, it offers insights on professional explanation, such as " It is a spectrum disorder, which means that symptoms and severity vary widely among individuals.".






















































  §.§.§ Evaluation


We present a performance summary of different instruction-tuning methods applied to two pre-trained LLMs in Table 5. While the Rouge and Fluency evaluation results show improvement with the use of Assistant-Instruction. To validate the performance, we use a selected portion of psychotherapy data as a validation set. Through content revising and leveraging additional common knowledge from GPT-4, both of these two LLMs show significant enhancement in matching the revised answers. Pre-trained LLMs can provide clients with comments to address psychological problems, but the quality of generated content may not always be fully accepted by psychologists. From Table 5, we observe that psychologists tend to prefer models that have been fine-tuned on psychotherapy data. As most LLMs lack specialization in a specific domain, they often require more domain-specific knowledge to improve their performance in professional domains. Because LLMs have been pre-trained on a vast corpus, giving them an inherent advantage in readability, and the size of tokens used does not seem to affect their performance significantly. Regarding the professionalism of the generated content, the psychologists gave higher scores to models that had been fine-tuned on psychotherapy instruction data compared to the corresponding original LLMs. 



  §.§.§ Human Evaluation Agreement
 
To assess the reliability of our human evaluation, we conducted an inner-rater agreement analysis <cit.> between our two evaluators. We used Cohen's κ to measure inter-rater agreement for categorical items. The 6-level rating scale (ranging from 0 to 5) was treated as a categorical variable for each aspect under consideration. The resulting κ value was 0.63, indicating a moderate level of agreement according to common practice. Furthermore, we computed the Spearman correlation coefficient ρ between the ratings of our two evaluators, treating the ratings as ordinal variables (ranging from 0 to 5). The obtained coefficient was ρ = 0.81, demonstrating a high correlation between the two evaluators. These results indicate a reasonably reliable human evaluation process for our study.
























§ CONCLUSION



We propose a novel method called ASSISTANT-INSTRUCT for fine-tuning or retrieving information from language models (LMs) to improve their instruction-following ability. This method combines both common knowledge and psychotherapy professional knowledge to generate instruction data with the help of experts. It retains the general knowledge already present in pre-trained LMs and incorporates psychotherapy-specific knowledge from expert-presented instructions. To enhance fine-tuning, as well as retrieval knowledge, we format the psychotherapy data, such as presentations, talks, and conversations, to make it more compatible with LMs. Human evaluation of this method demonstrates significant improvement compared to existing instruction methods. ASSISTANT-INSTRUCT can serve as an initial step to align pre-trained LMs with LM-revised instructions, and further research can build upon this method to enhance instruction-following models.











IEEEbib


