Augmenting the Author: Exploring the Potential of AI Collaboration in Academic Writing
Joseph Tu Hilda Hadan Derrick M. Wang Sabrina A Sgandurra Reza Hadi Mogavi Lennart E. Nacke



Augmenting the Author]Augmenting the Author: Exploring the Potential of AI Collaboration in Academic Writing





    Authors are also affiliated with the Department of Systems Design Engineering, University of Waterloo
    joseph.tu@uwaterloo.ca
    0000-0002-7703-6234
    
    Stratford School of Interaction Design and Business, University of Waterloo
    200 University Ave W
    Canada



[1]
hhadan@uwaterloo.ca
0000-0002-5911-1405

    Stratford School of Interaction Design and Business, University of Waterloo
    Waterloo
    Canada



[1]
dwmaru@uwaterloo.ca
0000-0003-3564-2532

    Stratford School of Interaction Design and Business, University of Waterloo
    Waterloo
    Canada



sasgandu@uwaterloo.ca
0000-0003-3187-263X

    Stratford School of Interaction Design and Business, University of Waterloo
    Waterloo
    Canada



rhadimog@uwaterloo.ca
0000-0002-4690-2769

  Stratford School of Interaction Design and Business, University of Waterloo
  Waterloo
  Ontario
  Canada
    


[1]
lennart.nacke@acm.org
https://orcid.org/0000-0003-4290-8829

    Stratford School of Interaction Design and Business, University of Waterloo
    Waterloo
    Canada






This workshop paper presents a critical examination of the integration of Generative AI (Gen AI) into the academic writing process, focusing on the use of AI as a collaborative tool. It contrasts the performance and interaction of two AI models, Gemini and ChatGPT, through a collaborative inquiry approach where researchers engage in facilitated sessions to design prompts that elicit specific AI responses for crafting research outlines. This case study highlights the importance of prompt design, output analysis, and recognizing the AI's limitations to ensure responsible and effective AI integration in scholarly work. Preliminary findings suggest that prompt variation significantly affects output quality and reveals distinct capabilities and constraints of each model. The paper contributes to the field of Human-Computer Interaction by exploring effective prompt strategies and providing a comparative analysis of Gen AI models, ultimately aiming to enhance AI-assisted academic writing and prompt a deeper dialogue within the HCI community.






<ccs2012>
   <concept>
       <concept_id>10010147.10010178</concept_id>
       <concept_desc>Computing methodologies Artificial intelligence</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10003120.10003121</concept_id>
       <concept_desc>Human-centered computing Human computer interaction (HCI)</concept_desc>
       <concept_significance>300</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Artificial intelligence
[300]Human-centered computing Human computer interaction (HCI)





26 February 2024
[accepted]13 March 2024


[
    Lennart E. Nacke
    April 29, 2024
====================





§ INTRODUCTION

In the realm of scholarly investigation, traversing the preliminary stages of drafting a research manuscript can pose challenges <cit.>. Some of the most common struggles when writing research papers are the tight schedule, lack of inspiration (for writing), and the complex process of organizing ideas and creating an outline <cit.>. This issue is particularly salient for early career academics and individuals whose first language is not English <cit.>. We ask ourselves, could Artificial Intelligence (AI) help?

The purpose of our workshop paper is to explore the potential of using AI to address challenges faced by the academic research community. Through an exploratory research approach, we aim to gather intellectual feedback and initiate a dialogue within the HCI community. Instead of presenting a traditional or formal research paper, our goal is to open a discussion on this topic. We will be sharing our combined reflections on the use of AI, focusing on two popular Generative AI models - ChatGPT[ChatGPT. <https://chat.openai.com/share/bf65d447-70c8-4117-87ef-f258265e3e43>] and Gemini[Gemini. <https://gemini.google.com/share/259d4d8f69f8>].

However, the application of AI as a writing assistant is often impeded by a dearth of transparency in researchers' access and utilization  <cit.>. This lack of transparency raises concerns about the reliability and credibility of AI-generated text <cit.>. Hence, comparative analysis between ChatGPT and Gemini as writing assistants for research draft outlines becomes crucial in understanding their capabilities and potential limitations.

At its core, our research emphasizes the importance of continual reflection and validation throughout the entire process of interacting with Gen AI. This includes thoughtful, prompt design, comprehensive output analysis, and an awareness of potential limitations. We contend that prioritizing this process-oriented approach is crucial for responsibly integrating Gen AI into academic research, leading to more accurate and reliable outcomes for researchers.

For this workshop paper, our discussion will revolve around two key aspects:

  
  * Exploring Effective Prompt Strategies: In this section, our attention is directed towards the realm of prompts that researchers themselves generate (the authors), as well as the pivotal inquiries that guide the outputs of Gen AI models, including ChatGPT and Gemini. Drawing upon the valuable insights acquired from our weekly deliberations, we aim to distinguish and articulate efficient strategies for crafting prompts.

  
  * A Comparative Perspective on Results: We compare two Gen AI models by giving them the same prompts and asking them to generate outlines for a specific research topic. Our collaborative inquiry approach will reveal the nuanced differences in their responses.


As there is growing research in prompt engineering, we aim to expand upon existing contributions, including recent works from <cit.>, by undertaking a thorough examination of the merits and drawbacks of different models in their handling of various prompt methods. Furthermore, we present a detailed account of the collaborative process with AI, providing the researcher's perspective and promoting transparency and accountability in assessing the efficacy and limitations of ChatGPT and Gemini as writing aids. We focus not on formal and statistical comparisons, as we acknowledge potential subjective biases and confounding factors. Instead, we aim to offer a firsthand experience that can serve as a starting point for meaningful discussions within the HCI community, potentially leading to further and more comprehensive investigations.


This position paper aims to make two significant contributions to the field of HCI. Firstly, it seeks to delineate effective prompt engineering strategies that enhance the collaborative dynamic between human researchers and AI, thereby optimizing the integration of AI in the research outline development process. Secondly, the paper aims to provide a nuanced comparative analysis of the capabilities and limitations of two prevalent generative AI models, ChatGPT and Gemini, grounded in real-world application scenarios that could serve as a foundational framework for future research and development in AI-assisted academic writing.



§ RESEARCH APPROACH

We used a collaborative inquiry approach <cit.>, focusing on facilitating structured and inclusive discussions and deliberations among ourselves as researchers; and culminating in us collectively reaching thoughtful judgments aligned with a specific goal or purpose, such as formulating recommendations or crafting a research outline. This case study aimed to evaluate the strengths and limitations of large language models (LLM), particularly Gemini and ChatGPT. Researchers (4—6) participated in weekly sessions, utilizing Otter.Ai for transcription. Discussions focused on the strengths and limitations of the models, guided by carefully crafted prompts. The strategic use of AI models in research outline development was evaluated based on predefined criteria. AI-generated prompts further facilitated critical thinking and reflection (condensed for a workshop see <ref>).









§ PRELIMINARY ANALYSIS OF CHATGPT AND GEMINI: OUR COLLECTIVE REFLECTION.


Through a series of comparisons and observations, we identified key differences in their approach, information handling, and potential applications within the research process. Exploring the capabilities of ChatGPT and Gemini in various tasks prompts a critical examination of their strengths and limitations within the context of research outline development. While ChatGPT consistently fulfills tasks based on explicit instructions, concerns arise due to its tendency to fabricate responses, raising questions about the reliability of generated content <cit.>. We often noted that the model's word-by-word rephrasing approach, coupled with a reluctance to restructure sentences autonomously, emphasizes its dependence on precise directives. Both models occasionally struggle with LaTeX formatting, which introduces complexities that may impact their usability in crafting research outlines (at the time of this submission).  
On the other hand, Gemini proves to be a dynamic counterpart, not only completing tasks but also providing valuable suggestions and rationales. Its ability to guide users through reference searches enhances the reliability of information retrieval. Gemini's capacity to offer multiple word choice suggestions, each accompanied by detailed explanations, adds sophistication to its linguistic capabilities. In tasks requiring paragraph improvement based on references, both models demonstrate a dependence on explicit prompts as outlined by <cit.>, highlighting the importance of precise instruction in research outline development.




In terms of articulating the potential of generative AI for research outlines, Gemini distinguishes itself with comprehensive summaries that include self-promotion. In contrast, ChatGPT tends to provide more straightforward summaries without articulating into additional insights. We also identified ChatGPT's potential limitations in discerning the best method or meaning, and found its potential deficiency in common sense. This suggest that enhancing ChatGPT's performance could be achieved through greater customization using its customization panel, potentially infusing a form of common sense into its responses, particularly when generating content for research outlines. 

In addition, Gemini impresses with its proactive and initiative-taking nature. It demonstrates a keen understanding of the broader context and consistently suggests new phrasings or alternative options without explicit requests, showcasing its potential to aid research outline development without waiting for specific directives. A notable distinction lies in how these models handle mistakes. While ChatGPT at times resorts to fabricating information, raising concerns about reliability and inadvertent alterations of intended meaning, Gemini prioritizes being helpful over maintaining an appearance of infallibility. Gemini transparently addresses limitations, enhancing its credibility and transparency compared to ChatGPT, particularly in the context of research outline development.

Researcher's Antidote on Certain Concerns. 
Participant 1 (P1) quotes: “This worries me as a reviewer, am I a human validating the author's work or AI's work?” One notable concern is the potential shift in the role of reviewers, who may unintentionally become the primary validators of AI-generated content rather than the authors themselves. The human touch, creativity, and critical thinking are essential components of writing. If AI takes a prominent role in generating content, there is a risk of diminishing these human qualities. We suggest authors utilizing AI in their writing should prioritize transparency by clearly disclosing the involvement of AI tools in the creation process. 

P6 quotes: “After using these tools for a while, I have a bias towards a certain writing style. I can tell that AI writes this or that, but what happens if this is just their writing style? Personally, English as a second language, I was taught to write in a formal construct.” 
As we explore the potential of AI writing tools, we acknowledge concerns surrounding the negative perception of their generated styles, particularly within academia. We understand the apprehension that might stem from unfamiliarity with this format, potentially leading to bias against its lack of established conventions. Additionally, we recognize the fears surrounding the potential for both quality concerns and ethical misuse of these tools. These concerns could translate into various challenges. We might encounter resistance towards adopting AI writing tools, creating an uneven playing field for those with access. This further highlights the need for clear guidelines to ensure responsible and ethical usage within academic settings <cit.>. While addressing these concerns is crucial for successful integration, we believe that continuous development and increasing familiarity with AI writing might eventually alleviate the initial negative connotations.





This workshop was supported by the SSHRC INSIGHT Grant (grant number:435-2022-0476), NSERC Discovery Grant (grant number: RGPIN-2023-03705) and CFI John R. Evans Leaders Fund or CFI JELF (grant number: 41844). Special thanks to the members of the Games Institute at the University of Waterloo for all their support. 




ACM-Reference-Format





