<!DOCTYPE html><html lang="en">
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title>[2310.16164] Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities</title><meta property="og:description" content="Large Language Models (LLMs) are being increasingly employed in data science for tasks like data preprocessing and analytics. However, data scientists encounter substantial obstacles when conversing with LLM-powered ch…">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="twitter:card" content="summary">
<meta name="twitter:title" content="Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities">
<meta name="twitter:image:src" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta name="twitter:image:alt" content="ar5iv logo">
<meta property="og:title" content="Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities">
<meta property="og:site_name" content="ar5iv">
<meta property="og:image" content="https://ar5iv.labs.arxiv.org/assets/ar5iv_card.png">
<meta property="og:type" content="article">
<meta property="og:url" content="https://ar5iv.labs.arxiv.org/html/2310.16164">

<!--Generated on Tue Feb 27 23:22:56 2024 by LaTeXML (version 0.8.8) http://dlmf.nist.gov/LaTeXML/.-->
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<meta name="keywords" lang="en" content="Data Science,  Generative Models,  Large Language Models,  ChatGPT,  Computational Notebooks">

<script>
  function detectColorScheme(){
    var theme="light";
    var current_theme = localStorage.getItem("ar5iv_theme");
    if(current_theme){
      if(current_theme == "dark"){
        theme = "dark";
      } }
    else if(!window.matchMedia) { return false; }
    else if(window.matchMedia("(prefers-color-scheme: dark)").matches) {
      theme = "dark"; }
    if (theme=="dark") {
      document.documentElement.setAttribute("data-theme", "dark");
    } else {
      document.documentElement.setAttribute("data-theme", "light"); } }

  detectColorScheme();

  function toggleColorScheme(){
    var current_theme = localStorage.getItem("ar5iv_theme");
    if (current_theme) {
      if (current_theme == "light") {
        localStorage.setItem("ar5iv_theme", "dark"); }
      else {
        localStorage.setItem("ar5iv_theme", "light"); } }
    else {
        localStorage.setItem("ar5iv_theme", "dark"); }
    detectColorScheme(); }
</script>
<link media="all" rel="stylesheet" href="/assets/ar5iv-fonts.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv.0.7.9.min.css"><link media="all" rel="stylesheet" href="/assets/ar5iv-site.0.2.2.css">
</head>
<body>
<div class="ltx_page_main">
<div class="ltx_page_content">
<article class="ltx_document ltx_authors_1line ltx_leqno">
<h1 class="ltx_title ltx_title_document">Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities</h1>
<div class="ltx_authors">
<span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Bhavya Chopra
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id1.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft</span><span id="id2.2.id2" class="ltx_text ltx_affiliation_country">India</span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:t-bhchopra@microsoft.com">t-bhchopra@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ananya Singha
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id3.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft</span><span id="id4.2.id2" class="ltx_text ltx_affiliation_country">India</span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:t-asingha@microsoft.com">t-asingha@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Anna Fariha
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id5.1.id1" class="ltx_text ltx_affiliation_institution">University of Utah</span><span id="id6.2.id2" class="ltx_text ltx_affiliation_country">United States</span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:afariha@cs.utah.edu">afariha@cs.utah.edu</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Sumit Gulwani
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id7.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft</span><span id="id8.2.id2" class="ltx_text ltx_affiliation_country">United States</span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:sumitg@microsoft.com">sumitg@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Chris Parnin
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id9.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft</span><span id="id10.2.id2" class="ltx_text ltx_affiliation_country">United States</span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:chrisparnin@microsoft.com">chrisparnin@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before">, </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Ashish Tiwari
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id11.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft</span><span id="id12.2.id2" class="ltx_text ltx_affiliation_country">United States</span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:astiwar@microsoft.com">astiwar@microsoft.com</a>
</span></span></span>
<span class="ltx_author_before"> and </span><span class="ltx_creator ltx_role_author">
<span class="ltx_personname">Austin Z. Henley
</span><span class="ltx_author_notes">
<span class="ltx_contact ltx_role_affiliation"><span id="id13.1.id1" class="ltx_text ltx_affiliation_institution">Microsoft</span><span id="id14.2.id2" class="ltx_text ltx_affiliation_country">United States</span>
</span>
<span class="ltx_contact ltx_role_email"><a href="mailto:austinhenley@microsoft.com">austinhenley@microsoft.com</a>
</span></span></span>
</div>

<div class="ltx_abstract">
<h6 class="ltx_title ltx_title_abstract">Abstract.</h6>
<p id="id15.id1" class="ltx_p">Large Language Models (LLMs) are being increasingly employed in data science for tasks like data preprocessing and analytics. However, data scientists encounter substantial obstacles when conversing with LLM-powered chatbots and acting on their suggestions and answers. We conducted a mixed-methods study, including contextual observations, semi-structured interviews (n=14), and a survey (n=114), to identify these challenges. Our findings highlight key issues faced by data scientists, including contextual data retrieval, formulating prompts for complex tasks, adapting generated code to local environments, and refining prompts iteratively. Based on these insights, we propose actionable design recommendations, such as data brushing to support context selection, and inquisitive feedback loops to improve communications with AI-based assistants in data-science tools.</p>
</div>
<div class="ltx_keywords">Data Science, Generative Models, Large Language Models, ChatGPT, Computational Notebooks
</div>
<span id="id1" class="ltx_note ltx_note_frontmatter ltx_role_ccs"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">ccs: </span>Human-centered computing HCI theory, concepts and models</span></span></span><span id="id2" class="ltx_note ltx_note_frontmatter ltx_role_ccs"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">ccs: </span>Human-centered computing Empirical studies in HCI</span></span></span><span id="id3" class="ltx_note ltx_note_frontmatter ltx_role_ccs"><sup class="ltx_note_mark">†</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">†</sup><span class="ltx_note_type">ccs: </span>Software and its engineering Requirements analysis</span></span></span>
<section id="S1" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">1. </span>Introduction</h2>

<div id="S1.p1" class="ltx_para">
<p id="S1.p1.1" class="ltx_p">Data scientists have traditionally relied on resources like documentation, tutorials, online courses, colleagues, Q&amp;A forums, and online communities to develop skills and solve tasks <cite class="ltx_cite ltx_citemacro_citep">(Kim et al<span class="ltx_text">.</span>, <a href="#bib.bib26" title="" class="ltx_ref">2018</a>; Kross and Guo, <a href="#bib.bib27" title="" class="ltx_ref">2019</a>; Shrestha et al<span class="ltx_text">.</span>, <a href="#bib.bib39" title="" class="ltx_ref">2021</a>; Vasilescu et al<span class="ltx_text">.</span>, <a href="#bib.bib45" title="" class="ltx_ref">2014</a>)</cite>.
These resources have been instrumental in helping them navigate the challenges of data acquisition, cleaning, wrangling, visualization, and presentation, especially for those with a non-programming background <cite class="ltx_cite ltx_citemacro_citep">(Kandel et al<span class="ltx_text">.</span>, <a href="#bib.bib24" title="" class="ltx_ref">2012</a>)</cite>.
Although useful, it can be time consuming, tedious, and error-prone to rely on these resources for solving a data-science problem.</p>
</div>
<div id="S1.p2" class="ltx_para">
<p id="S1.p2.1" class="ltx_p">With the emergence of AI-powered chat assistants, data scientists now have access to potentially faster and more accessible resources through a chat interface.
These AI-powered chatbots, like ChatGPT<span id="footnote1" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">1</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">1</sup><span class="ltx_tag ltx_tag_note">1</span>https://chat.openai.com/</span></span></span>, enable users to ask questions in natural language, and get useful responses with little to no latency.
For example, when prompted with “split my date column that is in MM/DD/YYYY format into three columns”, ChatGPT provided usable Python code:
<br class="ltx_break"><span id="S1.p2.1.1" class="ltx_text ltx_phantom"><span style="visibility:hidden">addingsomecenterit</span></span>df[[’Month’, ’Day’, ’Year’]] = df[’Date’].str.split(’/’, expand=True)
<br class="ltx_break">ChatGPT can also explain how the code works, and supports follow-up questions or changes.
Evaluations of these AI tools have demonstrated over 70% accuracy on programming benchmarks <cite class="ltx_cite ltx_citemacro_citep">(Chen et al<span class="ltx_text">.</span>, <a href="#bib.bib8" title="" class="ltx_ref">2021</a>)</cite> and pass numerous tests designed for humans (e.g., the GRE, a college entrance exam, and the LSAT, a law school admission test) <cite class="ltx_cite ltx_citemacro_citep">(OpenAI, <a href="#bib.bib35" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S1.p3" class="ltx_para">
<p id="S1.p3.1" class="ltx_p">However, the effectiveness of these tools is dependent on data scientists successfully communicating their questions, context (overall problem, task at hand, datasets, etc.), assumptions, and domain knowledge to the AI assistant, and doing so through a back-and-forth conversation.
Grice’s maxims of conversation<span id="footnote2" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">2</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">2</sup><span class="ltx_tag ltx_tag_note">2</span>https://effectiviology.com/principles-of-effective-communication/</span></span></span> posits that a successful conversation involves content that has right amount of information, that is truthful and supported by evidence, that is relevant to the specific context, and that is presented clearly <cite class="ltx_cite ltx_citemacro_citep">(Grice, <a href="#bib.bib15" title="" class="ltx_ref">1991</a>)</cite>.
But conversations go awry—either side of the conversation may be making false assumptions, there may be ambiguities, and conversations may require numerous clarifications.
All of these problems are possible with AI assistants; most notably, they are known to <em id="S1.p3.1.1" class="ltx_emph ltx_font_italic">hallucinate</em>, meaning they confidently respond with false statements. Furthermore, studies have found that users find conversations with AI assistants, such as ChatGPT, to have problematic communication styles <cite class="ltx_cite ltx_citemacro_citep">(Skjuve et al<span class="ltx_text">.</span>, <a href="#bib.bib40" title="" class="ltx_ref">2023</a>)</cite> and responses provided to be repetitive and filled with outdated or irrelevant information <cite class="ltx_cite ltx_citemacro_citep">(Skjuve et al<span class="ltx_text">.</span>, <a href="#bib.bib40" title="" class="ltx_ref">2023</a>; Kaddour et al<span class="ltx_text">.</span>, <a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>—a far cry from Grice’s maxims.
As such, for humans to communicate effectively with AI agents, humans must follow an iterative sense-making process that involves describing intent, building hypotheses, gathering evidence, and evaluating responses <cite class="ltx_cite ltx_citemacro_citep">(Cabrera et al<span class="ltx_text">.</span>, <a href="#bib.bib6" title="" class="ltx_ref">2023</a>)</cite>.</p>
</div>
<div id="S1.p4" class="ltx_para">
<p id="S1.p4.1" class="ltx_p">The barriers to express intents are exacerbated in the context of data-science tasks.
First, data scientists work with a variety of artifacts, including raw datasets, code, computational notebooks, visualizations, documentation, and machine-learning pipelines.
Second, datasets are often large and normalized (spilt across multiple tables), which may not be feasible to share (e.g., due to token limits of AI tools or due to data being spread across heterogeneous sources) or to summarize the relevant portions succinctly (e.g., specifying regions of the data).
Third, real-world data is messy, and suffer from quality issues; they may not strictly adhere to a schema or homogeneous format.
Fourth, data-science tasks often require domain expertise and numerous assumptions (e.g., negative values in a column represent an error, 0 means missing value) and it may be laborious to ensure that the AI assistant is aware of such domain knowledge and assumptions.
Furthermore, even when a data scientist is able to articulate their question to the AI assistant, these same problems can cause difficulties to understand and validate the AI assistant’s response.</p>
</div>
<div id="S1.p5" class="ltx_para">
<p id="S1.p5.1" class="ltx_p">In this work, we aim to understand the fundamental problems of a [human] data scientist communicating to an AI chat agent. In particular, we address the following research questions:</p>
<ul id="S1.I1" class="ltx_itemize">
<li id="S1.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i1.p1" class="ltx_para">
<p id="S1.I1.i1.p1.1" class="ltx_p"><em id="S1.I1.i1.p1.1.1" class="ltx_emph ltx_font_italic">RQ1:</em> How do data scientists interact with ChatGPT to complete data-science tasks?</p>
</div>
</li>
<li id="S1.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i2.p1" class="ltx_para">
<p id="S1.I1.i2.p1.1" class="ltx_p"><em id="S1.I1.i2.p1.1.1" class="ltx_emph ltx_font_italic">RQ2:</em> What challenges and unmet needs do data scientists face when interacting with ChatGPT?</p>
</div>
</li>
<li id="S1.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S1.I1.i3.p1" class="ltx_para">
<p id="S1.I1.i3.p1.1" class="ltx_p"><em id="S1.I1.i3.p1.1.1" class="ltx_emph ltx_font_italic">RQ3:</em> How well do these challenges and needs generalize to broader community of data scientists?</p>
</div>
</li>
</ul>
</div>
<div id="S1.p6" class="ltx_para">
<p id="S1.p6.1" class="ltx_p">To answer these questions, we conducted two mixed-method need-finding studies (Section <a href="#S3" title="3. Methodology ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a>). For the first study, we observe 14 professional data scientists as they perform four diverse tasks that are common in data-science pipelines. The participants were presented with the browser-based chat assistant, ChatGPT, to help them solve the tasks whenever needed. We performed inductive thematic analysis to identify needs and challenges by triangulating findings from open coding of semi-structured interview transcripts, observations from field notes, and video annotations. For the second study, we conducted a confirmation survey with 114 professional data scientists to validate and generalize findings from the first study.</p>
</div>
<div id="S1.p7" class="ltx_para">
<p id="S1.p7.1" class="ltx_p">Our results (Sections <a href="#S4" title="4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> and <a href="#S5" title="5. Survey Results ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>) show that data scientists usually follow a typical pattern of articulating their question, collecting context, validating the response made by the AI assistant, and then applying the response to their task.
Throughout this process, they face copious barriers while communicating with the AI assistant, such as spending considerable time writing their question and collecting relevant context.
Then they face a variety of additional barriers while trying to make use of the AI assistant’s response, such as understanding the assumptions being made by the AI and modifying the provided code to work for their scenario.
This arduous workflow resulted in participants applying strategies to overcome the barriers and still make use of the AI assistant.
Based on our findings, we provide practical design recommendations that can improve user experiences while using LLM-powered assistants for their data-science workflows (Section <a href="#S6" title="6. Design Recommendations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>).</p>
</div>
</section>
<section id="S2" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">2. </span>Background</h2>

<section id="S2.SS0.SSSx1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">How data scientists work</h4>

<div id="S2.SS0.SSSx1.p1" class="ltx_para">
<p id="S2.SS0.SSSx1.p1.1" class="ltx_p">Data science is the broad discipline of using data to understand the underlying nature of a domain and provide insights <cite class="ltx_cite ltx_citemacro_citep">(Muller et al<span class="ltx_text">.</span>, <a href="#bib.bib32" title="" class="ltx_ref">2019</a>)</cite>.
Activities include data acquisition, cleaning, wrangling, visualization, sharing, and acting on insights <cite class="ltx_cite ltx_citemacro_citep">(Fisher et al<span class="ltx_text">.</span>, <a href="#bib.bib12" title="" class="ltx_ref">2012</a>; Kim et al<span class="ltx_text">.</span>, <a href="#bib.bib26" title="" class="ltx_ref">2018</a>; Muller et al<span class="ltx_text">.</span>, <a href="#bib.bib32" title="" class="ltx_ref">2019</a>)</cite>.
The majority of time spent on data-science tasks often involves data cleaning and wrangling <cite class="ltx_cite ltx_citemacro_citep">(Guo et al<span class="ltx_text">.</span>, <a href="#bib.bib16" title="" class="ltx_ref">2011a</a>; Kandel et al<span class="ltx_text">.</span>, <a href="#bib.bib22" title="" class="ltx_ref">2011a</a>)</cite>. Sutton et al. stated that handling expansive datasets often involves navigating a myriad of issues, culminating in a situation they term “death by a thousand wranglings” <cite class="ltx_cite ltx_citemacro_citep">(Sutton et al<span class="ltx_text">.</span>, <a href="#bib.bib42" title="" class="ltx_ref">2018</a>)</cite>.</p>
</div>
<div id="S2.SS0.SSSx1.p2" class="ltx_para">
<p id="S2.SS0.SSSx1.p2.1" class="ltx_p">To support exploratory wrangling tasks, data scientists often employ <em id="S2.SS0.SSSx1.p2.1.1" class="ltx_emph ltx_font_italic">computational notebooks</em>, such as Jupyter notebooks<span id="footnote3" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">3</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">3</sup><span class="ltx_tag ltx_tag_note">3</span>https://jupyter.org/</span></span></span>.
These notebooks provide an interactive environment that allows for the seamless integration of text, code, and output, facilitating both the development and documentation of computational workflows in a single document.
Unlike traditional code editors, notebooks enable a more narrative style of coding, where code cells and rich-text commentary can coexist and outputs can be immediately rendered inline.
However, notebooks come with their own pain points: they take considerable effort to keep organized, hidden state information can lead to misconceptions, they can be difficult to deploy, and long-running tasks can be problematic <cite class="ltx_cite ltx_citemacro_citep">(Kery et al<span class="ltx_text">.</span>, <a href="#bib.bib25" title="" class="ltx_ref">2018</a>; Rule et al<span class="ltx_text">.</span>, <a href="#bib.bib36" title="" class="ltx_ref">2018</a>; Chattopadhyay et al<span class="ltx_text">.</span>, <a href="#bib.bib7" title="" class="ltx_ref">2020</a>)</cite>.</p>
</div>
</section>
<section id="S2.SS0.SSSx2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Large language models and AI-powered chat assistants</h4>

<div id="S2.SS0.SSSx2.p1" class="ltx_para">
<p id="S2.SS0.SSSx2.p1.1" class="ltx_p">AI-powered chat assistants are built on top of large language models (LLMs).
LLMs are generative machine-learning models with billion parameters, and are trained on vast amount of data (text and images) to generate human-like responses and perform various language tasks.
These models interact with users through a <em id="S2.SS0.SSSx2.p1.1.1" class="ltx_emph ltx_font_italic">prompt</em>, a natural-language query to which the LLM can respond to, also in natural language.
There are restrictions on the <em id="S2.SS0.SSSx2.p1.1.2" class="ltx_emph ltx_font_italic">token limit</em>, or the amount of information that can be sent or generated from the LLM.
Tools like ChatGPT add a user interface to enable a conversation, such that the model has
<em id="S2.SS0.SSSx2.p1.1.3" class="ltx_emph ltx_font_italic">context</em> of the conversation history.
However, these models do not have direct access to data sources, and only respond based on the data they have seen during training.</p>
</div>
</section>
</section>
<section id="S3" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">3. </span>Methodology</h2>

<div id="S3.p1" class="ltx_para">
<p id="S3.p1.1" class="ltx_p">We performed two studies to answer our research questions. First, we conducted task-based semi-structured interviews and observed 14 professional data scientists engaged in data-science tasks. Then, we distributed a general survey of our initial findings with 114 data scientists.</p>
</div>
<section id="S3.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.1. </span>Study 1: Using ChatGPT for Data Science Tasks</h3>

<div id="S3.SS1.p1" class="ltx_para">
<p id="S3.SS1.p1.1" class="ltx_p">We used ChatGPT, a browser-based LLM-powered chat-based AI assistant, for the study. We avoided using any AI assistants integrated within computational notebooks to better understand fundamental challenges and patterns of usage. This deliberate choice to keep the AI assistant separate from the data science environment(s) of the participants allows us to give them complete control and transparency over the prompt-writing and code adaptation processes—while giving us an opportunity to understand the underlying process of assessing <span id="S3.SS1.p1.1.1" class="ltx_text ltx_font_italic">which</span> context, and <span id="S3.SS1.p1.1.2" class="ltx_text ltx_font_italic">how much</span> context is relevant for prompting. This choice also echoes one of the design recommendations proposed by <cite class="ltx_cite ltx_citemacro_citet">Barke et al<span class="ltx_text">.</span> (<a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> that developers would prefer control over the shared prompt context.</p>
</div>
<section id="S3.SS1.SSSx1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Participants</h4>

<div id="S3.SS1.SSSx1.p1" class="ltx_para">
<p id="S3.SS1.SSSx1.p1.2" class="ltx_p">We recruited participants from a large technology company by randomly selecting employees with the job title “Data Scientist” mentioned in the company address book. We further screened participants based on having at least 1 year of experience with data science and prior experience with Python, computational notebooks, and ChatGPT (or any other LLM-based chat/code assistants) as a pre-requisite to their participation. Table <a href="#S3.T1" title="Table 1 ‣ Participants ‣ 3.1. Study 1: Using ChatGPT for Data Science Tasks ‣ 3. Methodology ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> indicates the demographics collected for all participants, who reported a median of 5 years of experience in the domain of data science (<math id="S3.SS1.SSSx1.p1.1.m1.1" class="ltx_Math" alttext="\mu=5.4" display="inline"><semantics id="S3.SS1.SSSx1.p1.1.m1.1a"><mrow id="S3.SS1.SSSx1.p1.1.m1.1.1" xref="S3.SS1.SSSx1.p1.1.m1.1.1.cmml"><mi id="S3.SS1.SSSx1.p1.1.m1.1.1.2" xref="S3.SS1.SSSx1.p1.1.m1.1.1.2.cmml">μ</mi><mo id="S3.SS1.SSSx1.p1.1.m1.1.1.1" xref="S3.SS1.SSSx1.p1.1.m1.1.1.1.cmml">=</mo><mn id="S3.SS1.SSSx1.p1.1.m1.1.1.3" xref="S3.SS1.SSSx1.p1.1.m1.1.1.3.cmml">5.4</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSSx1.p1.1.m1.1b"><apply id="S3.SS1.SSSx1.p1.1.m1.1.1.cmml" xref="S3.SS1.SSSx1.p1.1.m1.1.1"><eq id="S3.SS1.SSSx1.p1.1.m1.1.1.1.cmml" xref="S3.SS1.SSSx1.p1.1.m1.1.1.1"></eq><ci id="S3.SS1.SSSx1.p1.1.m1.1.1.2.cmml" xref="S3.SS1.SSSx1.p1.1.m1.1.1.2">𝜇</ci><cn type="float" id="S3.SS1.SSSx1.p1.1.m1.1.1.3.cmml" xref="S3.SS1.SSSx1.p1.1.m1.1.1.3">5.4</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSSx1.p1.1.m1.1c">\mu=5.4</annotation></semantics></math>, <math id="S3.SS1.SSSx1.p1.2.m2.1" class="ltx_Math" alttext="sd=2.8" display="inline"><semantics id="S3.SS1.SSSx1.p1.2.m2.1a"><mrow id="S3.SS1.SSSx1.p1.2.m2.1.1" xref="S3.SS1.SSSx1.p1.2.m2.1.1.cmml"><mrow id="S3.SS1.SSSx1.p1.2.m2.1.1.2" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2.cmml"><mi id="S3.SS1.SSSx1.p1.2.m2.1.1.2.2" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2.2.cmml">s</mi><mo lspace="0em" rspace="0em" id="S3.SS1.SSSx1.p1.2.m2.1.1.2.1" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2.1.cmml">​</mo><mi id="S3.SS1.SSSx1.p1.2.m2.1.1.2.3" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2.3.cmml">d</mi></mrow><mo id="S3.SS1.SSSx1.p1.2.m2.1.1.1" xref="S3.SS1.SSSx1.p1.2.m2.1.1.1.cmml">=</mo><mn id="S3.SS1.SSSx1.p1.2.m2.1.1.3" xref="S3.SS1.SSSx1.p1.2.m2.1.1.3.cmml">2.8</mn></mrow><annotation-xml encoding="MathML-Content" id="S3.SS1.SSSx1.p1.2.m2.1b"><apply id="S3.SS1.SSSx1.p1.2.m2.1.1.cmml" xref="S3.SS1.SSSx1.p1.2.m2.1.1"><eq id="S3.SS1.SSSx1.p1.2.m2.1.1.1.cmml" xref="S3.SS1.SSSx1.p1.2.m2.1.1.1"></eq><apply id="S3.SS1.SSSx1.p1.2.m2.1.1.2.cmml" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2"><times id="S3.SS1.SSSx1.p1.2.m2.1.1.2.1.cmml" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2.1"></times><ci id="S3.SS1.SSSx1.p1.2.m2.1.1.2.2.cmml" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2.2">𝑠</ci><ci id="S3.SS1.SSSx1.p1.2.m2.1.1.2.3.cmml" xref="S3.SS1.SSSx1.p1.2.m2.1.1.2.3">𝑑</ci></apply><cn type="float" id="S3.SS1.SSSx1.p1.2.m2.1.1.3.cmml" xref="S3.SS1.SSSx1.p1.2.m2.1.1.3">2.8</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS1.SSSx1.p1.2.m2.1c">sd=2.8</annotation></semantics></math>). All participants have a background and educational degree in CS and Data Science related fields.</p>
</div>
<div id="S3.SS1.SSSx1.p2" class="ltx_para">
<p id="S3.SS1.SSSx1.p2.1" class="ltx_p">All sessions were audio and video recorded for transcription of responses and analysis of on-screen interactions. All participants signed a consent form prior to the study. Participation in the study was voluntary and participants were compensated with gift cards worth $25 USD each.</p>
</div>
<figure id="S3.T1" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.T1.2.1.1" class="ltx_text" style="font-size:90%;">Table 1</span>. </span><span id="S3.T1.3.2" class="ltx_text" style="font-size:90%;">Overview of the participants, including the domain that they work in, their years of experience as a data scientist, and the programming languages they often use.</span></figcaption>
<table id="S3.T1.4" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T1.4.1.1" class="ltx_tr">
<th id="S3.T1.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">ID</span></th>
<th id="S3.T1.4.1.1.2" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.4.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Domain</span></th>
<th id="S3.T1.4.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.4.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Exp. (yrs)</span></th>
<th id="S3.T1.4.1.1.4" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S3.T1.4.1.1.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Languages used</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T1.4.2.1" class="ltx_tr">
<td id="S3.T1.4.2.1.1" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T1.4.2.1.1.1" class="ltx_text" style="font-size:90%;">P1</span></td>
<td id="S3.T1.4.2.1.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T1.4.2.1.2.1" class="ltx_text" style="font-size:90%;">Anomaly Detection</span></td>
<td id="S3.T1.4.2.1.3" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T1.4.2.1.3.1" class="ltx_text" style="font-size:90%;">5</span></td>
<td id="S3.T1.4.2.1.4" class="ltx_td ltx_align_left ltx_border_t"><span id="S3.T1.4.2.1.4.1" class="ltx_text" style="font-size:90%;">Python, SQL</span></td>
</tr>
<tr id="S3.T1.4.3.2" class="ltx_tr">
<td id="S3.T1.4.3.2.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.3.2.1.1" class="ltx_text" style="font-size:90%;">P2</span></td>
<td id="S3.T1.4.3.2.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.3.2.2.1" class="ltx_text" style="font-size:90%;">Big Data Processing</span></td>
<td id="S3.T1.4.3.2.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.3.2.3.1" class="ltx_text" style="font-size:90%;">10</span></td>
<td id="S3.T1.4.3.2.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.3.2.4.1" class="ltx_text" style="font-size:90%;">Python, SQL</span></td>
</tr>
<tr id="S3.T1.4.4.3" class="ltx_tr">
<td id="S3.T1.4.4.3.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.4.3.1.1" class="ltx_text" style="font-size:90%;">P3</span></td>
<td id="S3.T1.4.4.3.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.4.3.2.1" class="ltx_text" style="font-size:90%;">User Segmentation</span></td>
<td id="S3.T1.4.4.3.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.4.3.3.1" class="ltx_text" style="font-size:90%;">4</span></td>
<td id="S3.T1.4.4.3.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.4.3.4.1" class="ltx_text" style="font-size:90%;">Python</span></td>
</tr>
<tr id="S3.T1.4.5.4" class="ltx_tr">
<td id="S3.T1.4.5.4.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.5.4.1.1" class="ltx_text" style="font-size:90%;">P4</span></td>
<td id="S3.T1.4.5.4.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.5.4.2.1" class="ltx_text" style="font-size:90%;">NL Processing</span></td>
<td id="S3.T1.4.5.4.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.5.4.3.1" class="ltx_text" style="font-size:90%;">1</span></td>
<td id="S3.T1.4.5.4.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.5.4.4.1" class="ltx_text" style="font-size:90%;">Python</span></td>
</tr>
<tr id="S3.T1.4.6.5" class="ltx_tr">
<td id="S3.T1.4.6.5.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.6.5.1.1" class="ltx_text" style="font-size:90%;">P5</span></td>
<td id="S3.T1.4.6.5.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.6.5.2.1" class="ltx_text" style="font-size:90%;">Feature Engineering</span></td>
<td id="S3.T1.4.6.5.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.6.5.3.1" class="ltx_text" style="font-size:90%;">4</span></td>
<td id="S3.T1.4.6.5.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.6.5.4.1" class="ltx_text" style="font-size:90%;">SQL</span></td>
</tr>
<tr id="S3.T1.4.7.6" class="ltx_tr">
<td id="S3.T1.4.7.6.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.7.6.1.1" class="ltx_text" style="font-size:90%;">P6</span></td>
<td id="S3.T1.4.7.6.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.7.6.2.1" class="ltx_text" style="font-size:90%;">Statistical Analysis</span></td>
<td id="S3.T1.4.7.6.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.7.6.3.1" class="ltx_text" style="font-size:90%;">2</span></td>
<td id="S3.T1.4.7.6.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.7.6.4.1" class="ltx_text" style="font-size:90%;">Python</span></td>
</tr>
<tr id="S3.T1.4.8.7" class="ltx_tr">
<td id="S3.T1.4.8.7.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.8.7.1.1" class="ltx_text" style="font-size:90%;">P7</span></td>
<td id="S3.T1.4.8.7.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.8.7.2.1" class="ltx_text" style="font-size:90%;">Defect Detection</span></td>
<td id="S3.T1.4.8.7.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.8.7.3.1" class="ltx_text" style="font-size:90%;">5</span></td>
<td id="S3.T1.4.8.7.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.8.7.4.1" class="ltx_text" style="font-size:90%;">Python</span></td>
</tr>
<tr id="S3.T1.4.9.8" class="ltx_tr">
<td id="S3.T1.4.9.8.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.9.8.1.1" class="ltx_text" style="font-size:90%;">P8</span></td>
<td id="S3.T1.4.9.8.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.9.8.2.1" class="ltx_text" style="font-size:90%;">Finance &amp; Forecasting</span></td>
<td id="S3.T1.4.9.8.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.9.8.3.1" class="ltx_text" style="font-size:90%;">6</span></td>
<td id="S3.T1.4.9.8.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.9.8.4.1" class="ltx_text" style="font-size:90%;">Python, SQL, Spark</span></td>
</tr>
<tr id="S3.T1.4.10.9" class="ltx_tr">
<td id="S3.T1.4.10.9.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.10.9.1.1" class="ltx_text" style="font-size:90%;">P9</span></td>
<td id="S3.T1.4.10.9.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.10.9.2.1" class="ltx_text" style="font-size:90%;">Business Analytics</span></td>
<td id="S3.T1.4.10.9.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.10.9.3.1" class="ltx_text" style="font-size:90%;">2</span></td>
<td id="S3.T1.4.10.9.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.10.9.4.1" class="ltx_text" style="font-size:90%;">Python</span></td>
</tr>
<tr id="S3.T1.4.11.10" class="ltx_tr">
<td id="S3.T1.4.11.10.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.11.10.1.1" class="ltx_text" style="font-size:90%;">P10</span></td>
<td id="S3.T1.4.11.10.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.11.10.2.1" class="ltx_text" style="font-size:90%;">Fraud Detection</span></td>
<td id="S3.T1.4.11.10.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.11.10.3.1" class="ltx_text" style="font-size:90%;">5</span></td>
<td id="S3.T1.4.11.10.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.11.10.4.1" class="ltx_text" style="font-size:90%;">Python, SQL</span></td>
</tr>
<tr id="S3.T1.4.12.11" class="ltx_tr">
<td id="S3.T1.4.12.11.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.12.11.1.1" class="ltx_text" style="font-size:90%;">P11</span></td>
<td id="S3.T1.4.12.11.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.12.11.2.1" class="ltx_text" style="font-size:90%;">User Segmentation</span></td>
<td id="S3.T1.4.12.11.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.12.11.3.1" class="ltx_text" style="font-size:90%;">10</span></td>
<td id="S3.T1.4.12.11.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.12.11.4.1" class="ltx_text" style="font-size:90%;">Python, SQL, Spark</span></td>
</tr>
<tr id="S3.T1.4.13.12" class="ltx_tr">
<td id="S3.T1.4.13.12.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.13.12.1.1" class="ltx_text" style="font-size:90%;">P12</span></td>
<td id="S3.T1.4.13.12.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.13.12.2.1" class="ltx_text" style="font-size:90%;">Cloud Computing</span></td>
<td id="S3.T1.4.13.12.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.13.12.3.1" class="ltx_text" style="font-size:90%;">10</span></td>
<td id="S3.T1.4.13.12.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.13.12.4.1" class="ltx_text" style="font-size:90%;">Python</span></td>
</tr>
<tr id="S3.T1.4.14.13" class="ltx_tr">
<td id="S3.T1.4.14.13.1" class="ltx_td ltx_align_left"><span id="S3.T1.4.14.13.1.1" class="ltx_text" style="font-size:90%;">P13</span></td>
<td id="S3.T1.4.14.13.2" class="ltx_td ltx_align_left"><span id="S3.T1.4.14.13.2.1" class="ltx_text" style="font-size:90%;">Failure Prediction</span></td>
<td id="S3.T1.4.14.13.3" class="ltx_td ltx_align_left"><span id="S3.T1.4.14.13.3.1" class="ltx_text" style="font-size:90%;">7</span></td>
<td id="S3.T1.4.14.13.4" class="ltx_td ltx_align_left"><span id="S3.T1.4.14.13.4.1" class="ltx_text" style="font-size:90%;">Python</span></td>
</tr>
<tr id="S3.T1.4.15.14" class="ltx_tr">
<td id="S3.T1.4.15.14.1" class="ltx_td ltx_align_left ltx_border_bb"><span id="S3.T1.4.15.14.1.1" class="ltx_text" style="font-size:90%;">P14</span></td>
<td id="S3.T1.4.15.14.2" class="ltx_td ltx_align_left ltx_border_bb"><span id="S3.T1.4.15.14.2.1" class="ltx_text" style="font-size:90%;">Fraud Detection</span></td>
<td id="S3.T1.4.15.14.3" class="ltx_td ltx_align_left ltx_border_bb"><span id="S3.T1.4.15.14.3.1" class="ltx_text" style="font-size:90%;">5</span></td>
<td id="S3.T1.4.15.14.4" class="ltx_td ltx_align_left ltx_border_bb"><span id="S3.T1.4.15.14.4.1" class="ltx_text" style="font-size:90%;">Python, SQL</span></td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S3.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">3.1.1. </span>Tasks</h4>

<div id="S3.SS1.SSS1.p1" class="ltx_para">
<p id="S3.SS1.SSS1.p1.1" class="ltx_p">We used four essential tasks performed by data scientists in the lifetime of a typical project <cite class="ltx_cite ltx_citemacro_citep">(Muller et al<span class="ltx_text">.</span>, <a href="#bib.bib32" title="" class="ltx_ref">2019</a>)</cite>: (T1 and T2) data wrangling and pre-processing, (T3) feature extraction and selection for training tasks, (T4) data visualization for insight-finding and reporting. The four tasks are described in Table <a href="#S3.T2" title="Table 2 ‣ 3.1.1. Tasks ‣ 3.1. Study 1: Using ChatGPT for Data Science Tasks ‣ 3. Methodology ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>.</p>
</div>
<div id="S3.SS1.SSS1.p2" class="ltx_para">
<p id="S3.SS1.SSS1.p2.1" class="ltx_p">We selected a sample of the <em id="S3.SS1.SSS1.p2.1.1" class="ltx_emph ltx_font_italic">New York EMS emergency calls dataset</em><span id="footnote4" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">4</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">4</sup><span class="ltx_tag ltx_tag_note">4</span><a target="_blank" href="https://www.kaggle.com/datasets/new-york-city/ny-ems-incident-dispatch-data" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.kaggle.com/datasets/new-york-city/ny-ems-incident-dispatch-data</a></span></span></span>, which contains rows corresponding to emergency incidents, and information about time, location, resources, and the Fire Department’s response to the emergency. The dataset was well-suited for supporting the tasks, as several columns had to be transformed before performing further analysis, and event data was amendable to feature and correlation analysis.</p>
</div>
<figure id="S3.T2" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S3.T2.2.1.1" class="ltx_text" style="font-size:90%;">Table 2</span>. </span><span id="S3.T2.3.2" class="ltx_text" style="font-size:90%;">The four data science tasks used in our study and their descriptions.</span></figcaption>
<table id="S3.T2.4" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S3.T2.4.1.1" class="ltx_tr">
<th id="S3.T2.4.1.1.1" class="ltx_td ltx_align_center ltx_align_top ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">#</span></th>
<th id="S3.T2.4.1.1.2" class="ltx_td ltx_align_center ltx_align_top ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Task</span></th>
<th id="S3.T2.4.1.1.3" class="ltx_td ltx_align_center ltx_align_top ltx_th ltx_th_column ltx_border_tt"><span id="S3.T2.4.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Description</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S3.T2.4.2.1" class="ltx_tr">
<td id="S3.T2.4.2.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S3.T2.4.2.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.2.1.1.1.1" class="ltx_p" style="width:8.5pt;"><span id="S3.T2.4.2.1.1.1.1.1" class="ltx_text" style="font-size:90%;">T1</span></span>
</span>
</td>
<td id="S3.T2.4.2.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S3.T2.4.2.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.2.1.2.1.1" class="ltx_p" style="width:71.1pt;"><span id="S3.T2.4.2.1.2.1.1.1" class="ltx_text" style="font-size:90%;">Datetime typecast</span></span>
</span>
</td>
<td id="S3.T2.4.2.1.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S3.T2.4.2.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.2.1.3.1.1" class="ltx_p" style="width:313.0pt;"><span id="S3.T2.4.2.1.3.1.1.1" class="ltx_text" style="font-size:90%;">Change the datatype of column </span><span id="S3.T2.4.2.1.3.1.1.2" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘INCIDENT_DATETIME’’</span><span id="S3.T2.4.2.1.3.1.1.3" class="ltx_text" style="font-size:90%;"> from string to datetime (e.g., </span><span id="S3.T2.4.2.1.3.1.1.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘2020-07-31T23:59:51.000’’</span><span id="S3.T2.4.2.1.3.1.1.5" class="ltx_text" style="font-size:90%;">).</span></span>
</span>
</td>
</tr>
<tr id="S3.T2.4.3.2" class="ltx_tr">
<td id="S3.T2.4.3.2.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S3.T2.4.3.2.1.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.3.2.1.1.1" class="ltx_p" style="width:8.5pt;"><span id="S3.T2.4.3.2.1.1.1.1" class="ltx_text" style="font-size:90%;">T2</span></span>
</span>
</td>
<td id="S3.T2.4.3.2.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S3.T2.4.3.2.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.3.2.2.1.1" class="ltx_p" style="width:71.1pt;"><span id="S3.T2.4.3.2.2.1.1.1" class="ltx_text" style="font-size:90%;">Split using delimiter</span></span>
</span>
</td>
<td id="S3.T2.4.3.2.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S3.T2.4.3.2.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.3.2.3.1.1" class="ltx_p" style="width:313.0pt;"><span id="S3.T2.4.3.2.3.1.1.1" class="ltx_text" style="font-size:90%;">Split the (</span><span id="S3.T2.4.3.2.3.1.1.2" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘INCIDENT_DESCRIPTION’’</span><span id="S3.T2.4.3.2.3.1.1.3" class="ltx_text" style="font-size:90%;">) column using semicolon (</span><span id="S3.T2.4.3.2.3.1.1.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘;’’</span><span id="S3.T2.4.3.2.3.1.1.5" class="ltx_text" style="font-size:90%;">) as the delimiter. Rows contain information about the incident location — borough, incident dispatch area code, and zipcode. Few rows do not contain the zipcode (e.g. rows such as </span><span id="S3.T2.4.3.2.3.1.1.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘Brooklyn; K7; 11211’’</span><span id="S3.T2.4.3.2.3.1.1.7" class="ltx_text" style="font-size:90%;"> contain all information, whereas rows like “Manhattan; M3” are missing the zipcode).</span></span>
</span>
</td>
</tr>
<tr id="S3.T2.4.4.3" class="ltx_tr">
<td id="S3.T2.4.4.3.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S3.T2.4.4.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.4.3.1.1.1" class="ltx_p" style="width:8.5pt;"><span id="S3.T2.4.4.3.1.1.1.1" class="ltx_text" style="font-size:90%;">T3</span></span>
</span>
</td>
<td id="S3.T2.4.4.3.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S3.T2.4.4.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.4.3.2.1.1" class="ltx_p" style="width:71.1pt;"><span id="S3.T2.4.4.3.2.1.1.1" class="ltx_text" style="font-size:90%;">Feature selection</span></span>
</span>
</td>
<td id="S3.T2.4.4.3.3" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S3.T2.4.4.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.4.3.3.1.1" class="ltx_p" style="width:313.0pt;"><span id="S3.T2.4.4.3.3.1.1.1" class="ltx_text" style="font-size:90%;">Extract a new column titled </span><span id="S3.T2.4.4.3.3.1.1.2" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘RESPONSE_TIME’’</span><span id="S3.T2.4.4.3.3.1.1.3" class="ltx_text" style="font-size:90%;"> using the difference between </span><span id="S3.T2.4.4.3.3.1.1.4" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘FIRST_ON_SCENE_DATETIME’’</span><span id="S3.T2.4.4.3.3.1.1.5" class="ltx_text" style="font-size:90%;"> and </span><span id="S3.T2.4.4.3.3.1.1.6" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘INCIDENT_DATETIME’’</span><span id="S3.T2.4.4.3.3.1.1.7" class="ltx_text" style="font-size:90%;">; subsequently perform feature selection with the aim of predicting the </span><span id="S3.T2.4.4.3.3.1.1.8" class="ltx_text ltx_font_typewriter" style="font-size:90%;">‘‘RESPONSE_TIME’’</span><span id="S3.T2.4.4.3.3.1.1.9" class="ltx_text" style="font-size:90%;">.</span></span>
</span>
</td>
</tr>
<tr id="S3.T2.4.5.4" class="ltx_tr">
<td id="S3.T2.4.5.4.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S3.T2.4.5.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.5.4.1.1.1" class="ltx_p" style="width:8.5pt;"><span id="S3.T2.4.5.4.1.1.1.1" class="ltx_text" style="font-size:90%;">T4</span></span>
</span>
</td>
<td id="S3.T2.4.5.4.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S3.T2.4.5.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.5.4.2.1.1" class="ltx_p" style="width:71.1pt;"><span id="S3.T2.4.5.4.2.1.1.1" class="ltx_text" style="font-size:90%;">Heatmap plotting</span></span>
</span>
</td>
<td id="S3.T2.4.5.4.3" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S3.T2.4.5.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="S3.T2.4.5.4.3.1.1" class="ltx_p" style="width:313.0pt;"><span id="S3.T2.4.5.4.3.1.1.1" class="ltx_text" style="font-size:90%;">Obtain a heatmap plot to analyze correlation between any columns of choice.</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</figure>
</section>
<section id="S3.SS1.SSSx2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Study protocol</h4>

<div id="S3.SS1.SSSx2.p1" class="ltx_para">
<p id="S3.SS1.SSSx2.p1.1" class="ltx_p">We conducted a one-hour long, task-based study with each of the 14 data scientists via video conferencing. The study begins with demographic questions about their role and data-science-related work, languages and tools, educational background, and prior experience with using LLMs. Next, participants were presented with 4 tasks, one at a time, to be solved using Python in any computational notebook of their choice, and the browser-based ChatGPT<span id="footnote5" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">5</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">5</sup><span class="ltx_tag ltx_tag_note">5</span><a target="_blank" href="https://chat.openai.com/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://chat.openai.com/</a> (Versions: March 14, 2023; March 23, 2023)</span></span></span>. They were encouraged to make use of any preferred tools and resources (such as web search, notebooks, Excel, and so on), and think aloud while solving the tasks. The completion of any task was not a necessity, and participants were encouraged to use ChatGPT at least once for every task. Once the participants had explored a task sufficiently, they were asked semi-structured questions to assess if ChatGPT helped them solve the task satisfactorily, and if it could have helped them with the task any better.</p>
</div>
</section>
<section id="S3.SS1.SSSx3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Analysis</h4>

<div id="S3.SS1.SSSx3.p1" class="ltx_para">
<p id="S3.SS1.SSSx3.p1.1" class="ltx_p">We transcribed the audio recordings for each session. Subsequently, we annotated videos from each session recording to timestamp and gathered: (1) prompts authored by participants, (2) corresponding responses provided by ChatGPT, and (3) a log of activities performed by the participants (including prompt writing, adapting generated code, and validating generated code).
The first and second author analyzed the transcripts and annotations from video recordings to perform inductive thematic coding <cite class="ltx_cite ltx_citemacro_citep">(Saldaña, <a href="#bib.bib37" title="" class="ltx_ref">2009</a>)</cite>. We first conducted discussions over a span of 2 weeks while collaboratively annotating data for one participant. Following this, we independently annotated 20% of the data (from three participants) with descriptive open codes. Next, we grouped similar codes and analyzed them to obtain axial codes, identifying usage patterns and pain-points faced by data scientists. We then computed the inter-rater agreement score to be 0.872, indicating ‘almost perfect’ agreement <cite class="ltx_cite ltx_citemacro_citep">(Landis and Koch, <a href="#bib.bib28" title="" class="ltx_ref">1977</a>)</cite>. The first two authors then split the remaining participant interviews and annotated them independently, while meeting frequently to discuss and refine the codes and themes.</p>
</div>
<div id="S3.SS1.SSSx3.p2" class="ltx_para">
<p id="S3.SS1.SSSx3.p2.1" class="ltx_p">We observed theoretical saturation after thematic analysis of data from 9 participants. To further support the validity of our findings, we requested 5 study participants (P1, P3, P4, P8, P12) to respond to a member-check survey. The survey consisted of 18 Likert-scale questions, each question representing an open-code from our findings. The ratings indicated agreement with each finding.</p>
</div>
</section>
</section>
<section id="S3.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">3.2. </span>Study 2: Confirmatory Survey</h3>

<div id="S3.SS2.p1" class="ltx_para">
<p id="S3.SS2.p1.1" class="ltx_p">To validate findings from Study 1, we conducted a survey with a broader population of 114 data scientists at a large technology company.</p>
</div>
<section id="S3.SS2.SSSx1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Survey protocol</h4>

<div id="S3.SS2.SSSx1.p1" class="ltx_para">
<p id="S3.SS2.SSSx1.p1.1" class="ltx_p">The survey consisted of two demographic questions, asking respondents to: (D1) indicate years of experience with data science (a single choice question), and (D2) describe at least one scenario where they used any LLM for a data-science task (an open-field question). To ensure the quality of the survey responses, we used responses for D2 to filter out respondents who do not have any experience with solving data-science tasks using LLMs.</p>
</div>
<div id="S3.SS2.SSSx1.p2" class="ltx_para">
<p id="S3.SS2.SSSx1.p2.1" class="ltx_p">We used findings from the qualitative analysis of Study 1 responses to frame at least one question for each identified obstacle. The survey consisted of 8 ‘Agree’/‘Disagree’ statements, and all questions were optional to answer. Finally, to evaluate if our observations from Study 1 reached theoretical saturation, we asked respondents to indicate if they have faced any other difficulties in using LLMs for data science tasks that were not mentioned in the survey questions.</p>
</div>
</section>
<section id="S3.SS2.SSSx2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Recruitment and informed consent</h4>

<div id="S3.SS2.SSSx2.p1" class="ltx_para">
<p id="S3.SS2.SSSx2.p1.1" class="ltx_p">The respondents were contacted randomly via e-mail invites based on their job title (“Data Scientist”) mentioned in the company address book. In our e-mail invites, we also requested participants to forward the survey to other data scientists who might be interested. We offered a drawing for one of two Amazon gift cards worth $50 USD each for completing the survey.</p>
</div>
</section>
<section id="S3.SS2.SSSx3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">Respondents</h4>

<div id="S3.SS2.SSSx3.p1" class="ltx_para">
<p id="S3.SS2.SSSx3.p1.2" class="ltx_p">114 data scientists from a large technology company responded to the survey. After inspecting whether the respondents have meaningful prior experiences (D2), we removed 16 respondents (<math id="S3.SS2.SSSx3.p1.1.m1.1" class="ltx_Math" alttext="14\%" display="inline"><semantics id="S3.SS2.SSSx3.p1.1.m1.1a"><mrow id="S3.SS2.SSSx3.p1.1.m1.1.1" xref="S3.SS2.SSSx3.p1.1.m1.1.1.cmml"><mn id="S3.SS2.SSSx3.p1.1.m1.1.1.2" xref="S3.SS2.SSSx3.p1.1.m1.1.1.2.cmml">14</mn><mo id="S3.SS2.SSSx3.p1.1.m1.1.1.1" xref="S3.SS2.SSSx3.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSSx3.p1.1.m1.1b"><apply id="S3.SS2.SSSx3.p1.1.m1.1.1.cmml" xref="S3.SS2.SSSx3.p1.1.m1.1.1"><csymbol cd="latexml" id="S3.SS2.SSSx3.p1.1.m1.1.1.1.cmml" xref="S3.SS2.SSSx3.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S3.SS2.SSSx3.p1.1.m1.1.1.2.cmml" xref="S3.SS2.SSSx3.p1.1.m1.1.1.2">14</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSSx3.p1.1.m1.1c">14\%</annotation></semantics></math>) that had no experience with using LLMs for data science tasks. We report all findings using the screened responses. <math id="S3.SS2.SSSx3.p1.2.m2.1" class="ltx_Math" alttext="76\%" display="inline"><semantics id="S3.SS2.SSSx3.p1.2.m2.1a"><mrow id="S3.SS2.SSSx3.p1.2.m2.1.1" xref="S3.SS2.SSSx3.p1.2.m2.1.1.cmml"><mn id="S3.SS2.SSSx3.p1.2.m2.1.1.2" xref="S3.SS2.SSSx3.p1.2.m2.1.1.2.cmml">76</mn><mo id="S3.SS2.SSSx3.p1.2.m2.1.1.1" xref="S3.SS2.SSSx3.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S3.SS2.SSSx3.p1.2.m2.1b"><apply id="S3.SS2.SSSx3.p1.2.m2.1.1.cmml" xref="S3.SS2.SSSx3.p1.2.m2.1.1"><csymbol cd="latexml" id="S3.SS2.SSSx3.p1.2.m2.1.1.1.cmml" xref="S3.SS2.SSSx3.p1.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S3.SS2.SSSx3.p1.2.m2.1.1.2.cmml" xref="S3.SS2.SSSx3.p1.2.m2.1.1.2">76</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S3.SS2.SSSx3.p1.2.m2.1c">76\%</annotation></semantics></math> of the respondents self-reported having more than 5 years of experience in data science related professional roles. We discuss survey results in Section <a href="#S5" title="5. Survey Results ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>, where we plotted the responses and inspected them manually to gather any other difficulties that were not mentioned in the survey questions.</p>
</div>
<figure id="S3.F1" class="ltx_figure"><img src="/html/2310.16164/assets/x1.png" id="S3.F1.g1" class="ltx_graphics ltx_centering ltx_img_square" width="461" height="386" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S3.F1.5.1.1" class="ltx_text" style="font-size:90%;">Figure 1</span>. </span><span id="S3.F1.6.2" class="ltx_text" style="font-size:90%;">A gallery of tasks involved in Study 1. Images are taken from participant screen sharing recordings and are anonymized. Snippets marked (C) denote the local <span id="S3.F1.6.2.1" class="ltx_text ltx_font_bold ltx_font_italic">Context</span> retrieved by participants to use in prompt-writing; (P) denotes the <span id="S3.F1.6.2.2" class="ltx_text ltx_font_bold ltx_font_italic">Prompt</span> sent to ChatGPT; and (A) denotes the <span id="S3.F1.6.2.3" class="ltx_text ltx_font_bold ltx_font_italic">Answer</span> provided by ChatGPT.</span></figcaption>
</figure>
</section>
</section>
</section>
<section id="S4" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">4. </span>Observations</h2>

<div id="S4.p1" class="ltx_para">
<p id="S4.p1.1" class="ltx_p">We provide a summary of our participants interactions with ChatGPT and then report observed obstacles and strategies used by the participants.
Table <a href="#S4.T4" title="Table 4 ‣ 4.2.1. Sharing context is difficult ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a> summarizes the themes that emerged from our analysis of the data scientists’ actions and responses. We also report participants who identified with each obstacle and employed prompting strategies in Table <a href="#S4.T4" title="Table 4 ‣ 4.2.1. Sharing context is difficult ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>.</p>
</div>
<section id="S4.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.1. </span>Participants Interactions with ChatGPT</h3>

<section id="S4.SS1.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.1.1. </span>Typical workflow</h4>

<div id="S4.SS1.SSS1.p1" class="ltx_para">
<p id="S4.SS1.SSS1.p1.1" class="ltx_p">Most participants started with loading the CSV files into a notebook of their choice, and constructing a <span id="S4.SS1.SSS1.p1.1.1" class="ltx_text ltx_font_typewriter">pandas</span> dataframe, while viewing the first few rows of the data. Sometimes they opened the CSV in Excel before creating a notebook, to glance over the data. Some participants used pandas.DataFrame.describe() to obtain a high-level summary of their data.</p>
</div>
<div id="S4.SS1.SSS1.p2" class="ltx_para">
<p id="S4.SS1.SSS1.p2.1" class="ltx_p">Next, they were sequentially presented with each of the tasks. Every participant worked on tasks T1, T2, and T3. However, 4 participants were unable to attempt T4 in the available time. For each task, participants generally began with phrasing their question, assessing and gathering context—such as description of the data, sample values—for prompt construction, understanding and taking action on ChatGPT’s response while writing additional prompts that would help them elicit better responses. Figure <a href="#S3.F1" title="Figure 1 ‣ Respondents ‣ 3.2. Study 2: Confirmatory Survey ‣ 3. Methodology ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">1</span></a> presents a gallery of tasks involved in the study.</p>
</div>
<div id="S4.SS1.SSS1.p3" class="ltx_para">
<p id="S4.SS1.SSS1.p3.1" class="ltx_p">Participants faced several challenges in conversing with ChatGPT and using its responses for task completion. In particular, most data scientists struggled with T3 (feature selection), taking them significantly higher number of prompts and time to get a satisfactory response. Feature selection, being a broad task (as compared to the remaining study tasks), had participants thinking about decomposing it to plan for sub-tasks, and providing ChatGPT sufficient context to obtain concrete and actionable next steps.</p>
</div>
</section>
<section id="S4.SS1.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.1.2. </span>Prompting behaviors</h4>

<div id="S4.SS1.SSS2.p1" class="ltx_para">
<p id="S4.SS1.SSS2.p1.7" class="ltx_p">In total, we logged 111 prompts made by 14 participants. Table <a href="#S4.T3" title="Table 3 ‣ 4.1.2. Prompting behaviors ‣ 4.1. Participants Interactions with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> presents an overview of the prompt distribution for each study task. We observed that participants spent <math id="S4.SS1.SSS2.p1.1.m1.1" class="ltx_Math" alttext="64\%" display="inline"><semantics id="S4.SS1.SSS2.p1.1.m1.1a"><mrow id="S4.SS1.SSS2.p1.1.m1.1.1" xref="S4.SS1.SSS2.p1.1.m1.1.1.cmml"><mn id="S4.SS1.SSS2.p1.1.m1.1.1.2" xref="S4.SS1.SSS2.p1.1.m1.1.1.2.cmml">64</mn><mo id="S4.SS1.SSS2.p1.1.m1.1.1.1" xref="S4.SS1.SSS2.p1.1.m1.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.1.m1.1b"><apply id="S4.SS1.SSS2.p1.1.m1.1.1.cmml" xref="S4.SS1.SSS2.p1.1.m1.1.1"><csymbol cd="latexml" id="S4.SS1.SSS2.p1.1.m1.1.1.1.cmml" xref="S4.SS1.SSS2.p1.1.m1.1.1.1">percent</csymbol><cn type="integer" id="S4.SS1.SSS2.p1.1.m1.1.1.2.cmml" xref="S4.SS1.SSS2.p1.1.m1.1.1.2">64</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.1.m1.1c">64\%</annotation></semantics></math> time on preparing prompts, <math id="S4.SS1.SSS2.p1.2.m2.1" class="ltx_Math" alttext="27\%" display="inline"><semantics id="S4.SS1.SSS2.p1.2.m2.1a"><mrow id="S4.SS1.SSS2.p1.2.m2.1.1" xref="S4.SS1.SSS2.p1.2.m2.1.1.cmml"><mn id="S4.SS1.SSS2.p1.2.m2.1.1.2" xref="S4.SS1.SSS2.p1.2.m2.1.1.2.cmml">27</mn><mo id="S4.SS1.SSS2.p1.2.m2.1.1.1" xref="S4.SS1.SSS2.p1.2.m2.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.2.m2.1b"><apply id="S4.SS1.SSS2.p1.2.m2.1.1.cmml" xref="S4.SS1.SSS2.p1.2.m2.1.1"><csymbol cd="latexml" id="S4.SS1.SSS2.p1.2.m2.1.1.1.cmml" xref="S4.SS1.SSS2.p1.2.m2.1.1.1">percent</csymbol><cn type="integer" id="S4.SS1.SSS2.p1.2.m2.1.1.2.cmml" xref="S4.SS1.SSS2.p1.2.m2.1.1.2">27</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.2.m2.1c">27\%</annotation></semantics></math> time on adapting the code returned by ChatGPT, and the remaining on validating the code, as seen in Figure <a href="#S4.F2" title="Figure 2 ‣ 4.1.2. Prompting behaviors ‣ 4.1. Participants Interactions with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (Left). In raw time spent on these activities, on average, <math id="S4.SS1.SSS2.p1.3.m3.1" class="ltx_Math" alttext="302" display="inline"><semantics id="S4.SS1.SSS2.p1.3.m3.1a"><mn id="S4.SS1.SSS2.p1.3.m3.1.1" xref="S4.SS1.SSS2.p1.3.m3.1.1.cmml">302</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.3.m3.1b"><cn type="integer" id="S4.SS1.SSS2.p1.3.m3.1.1.cmml" xref="S4.SS1.SSS2.p1.3.m3.1.1">302</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.3.m3.1c">302</annotation></semantics></math> seconds were spent on prompt writing, <math id="S4.SS1.SSS2.p1.4.m4.1" class="ltx_Math" alttext="57" display="inline"><semantics id="S4.SS1.SSS2.p1.4.m4.1a"><mn id="S4.SS1.SSS2.p1.4.m4.1.1" xref="S4.SS1.SSS2.p1.4.m4.1.1.cmml">57</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.4.m4.1b"><cn type="integer" id="S4.SS1.SSS2.p1.4.m4.1.1.cmml" xref="S4.SS1.SSS2.p1.4.m4.1.1">57</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.4.m4.1c">57</annotation></semantics></math> seconds on adaptation and <math id="S4.SS1.SSS2.p1.5.m5.1" class="ltx_Math" alttext="24" display="inline"><semantics id="S4.SS1.SSS2.p1.5.m5.1a"><mn id="S4.SS1.SSS2.p1.5.m5.1.1" xref="S4.SS1.SSS2.p1.5.m5.1.1.cmml">24</mn><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.5.m5.1b"><cn type="integer" id="S4.SS1.SSS2.p1.5.m5.1.1.cmml" xref="S4.SS1.SSS2.p1.5.m5.1.1">24</cn></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.5.m5.1c">24</annotation></semantics></math> seconds on validation. We also find that <math id="S4.SS1.SSS2.p1.6.m6.1" class="ltx_Math" alttext="37\%" display="inline"><semantics id="S4.SS1.SSS2.p1.6.m6.1a"><mrow id="S4.SS1.SSS2.p1.6.m6.1.1" xref="S4.SS1.SSS2.p1.6.m6.1.1.cmml"><mn id="S4.SS1.SSS2.p1.6.m6.1.1.2" xref="S4.SS1.SSS2.p1.6.m6.1.1.2.cmml">37</mn><mo id="S4.SS1.SSS2.p1.6.m6.1.1.1" xref="S4.SS1.SSS2.p1.6.m6.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.6.m6.1b"><apply id="S4.SS1.SSS2.p1.6.m6.1.1.cmml" xref="S4.SS1.SSS2.p1.6.m6.1.1"><csymbol cd="latexml" id="S4.SS1.SSS2.p1.6.m6.1.1.1.cmml" xref="S4.SS1.SSS2.p1.6.m6.1.1.1">percent</csymbol><cn type="integer" id="S4.SS1.SSS2.p1.6.m6.1.1.2.cmml" xref="S4.SS1.SSS2.p1.6.m6.1.1.2">37</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.6.m6.1c">37\%</annotation></semantics></math> of the total time was spent on writing the initial prompt, whereas making refinements to prompts and asking follow-up questions takes <math id="S4.SS1.SSS2.p1.7.m7.1" class="ltx_Math" alttext="28\%" display="inline"><semantics id="S4.SS1.SSS2.p1.7.m7.1a"><mrow id="S4.SS1.SSS2.p1.7.m7.1.1" xref="S4.SS1.SSS2.p1.7.m7.1.1.cmml"><mn id="S4.SS1.SSS2.p1.7.m7.1.1.2" xref="S4.SS1.SSS2.p1.7.m7.1.1.2.cmml">28</mn><mo id="S4.SS1.SSS2.p1.7.m7.1.1.1" xref="S4.SS1.SSS2.p1.7.m7.1.1.1.cmml">%</mo></mrow><annotation-xml encoding="MathML-Content" id="S4.SS1.SSS2.p1.7.m7.1b"><apply id="S4.SS1.SSS2.p1.7.m7.1.1.cmml" xref="S4.SS1.SSS2.p1.7.m7.1.1"><csymbol cd="latexml" id="S4.SS1.SSS2.p1.7.m7.1.1.1.cmml" xref="S4.SS1.SSS2.p1.7.m7.1.1.1">percent</csymbol><cn type="integer" id="S4.SS1.SSS2.p1.7.m7.1.1.2.cmml" xref="S4.SS1.SSS2.p1.7.m7.1.1.2">28</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS1.SSS2.p1.7.m7.1c">28\%</annotation></semantics></math> of the total time.
In particular, we observed that participants especially struggled to write an initial prompt for T3 (feature selection), a task with more subjectivity, when compared to relatively more objective tasks (T1, T2 and T4). The prompts also greatly varied in verbosity:</p>
</div>
<div id="S4.SS1.SSS2.p2" class="ltx_para">
<blockquote id="S4.SS1.SSS2.p2.1" class="ltx_quote">
<p id="S4.SS1.SSS2.p2.1.1" class="ltx_p"><span id="S4.SS1.SSS2.p2.1.1.1" class="ltx_text ltx_font_italic">You are a data scientist, how can you do feature selection? (P13)</span></p>
</blockquote>
</div>
<div id="S4.SS1.SSS2.p3" class="ltx_para">
<blockquote id="S4.SS1.SSS2.p3.1" class="ltx_quote">
<p id="S4.SS1.SSS2.p3.1.1" class="ltx_p"><span id="S4.SS1.SSS2.p3.1.1.1" class="ltx_text ltx_font_italic">I have a dataset that has information about medical emergency services. I have a response time column which tells me historically how much time will it take to respond to a medical emergency incident. I have other columns some are categorical and some are continuous which explain more about the incident. My task is to predict the response time for a new incident and I am thinking about building a predictive model for that. What would be good features that can help the model predict response time? (P8)</span></p>
</blockquote>
</div>
<figure id="S4.T3" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T3.2.1.1" class="ltx_text" style="font-size:90%;">Table 3</span>. </span><span id="S4.T3.3.2" class="ltx_text" style="font-size:90%;">Overview of the number of prompts queried per task, and average time spent in prompt-writing.</span></figcaption>
<table id="S4.T3.4" class="ltx_tabular ltx_guessed_headers ltx_align_middle">
<thead class="ltx_thead">
<tr id="S4.T3.4.1.1" class="ltx_tr">
<th id="S4.T3.4.1.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_th_row ltx_border_tt"><span id="S4.T3.4.1.1.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Task</span></th>
<th id="S4.T3.4.1.1.2" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.4.1.1.2.1" class="ltx_text ltx_font_bold" style="font-size:90%;"># prompts</span></th>
<th id="S4.T3.4.1.1.3" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.4.1.1.3.1" class="ltx_text ltx_font_bold" style="font-size:90%;">median</span></th>
<th id="S4.T3.4.1.1.4" class="ltx_td ltx_align_center ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.4.1.1.4.1" class="ltx_text ltx_font_bold" style="font-size:90%;">min</span></th>
<th id="S4.T3.4.1.1.5" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.4.1.1.5.1" class="ltx_text ltx_font_bold" style="font-size:90%;">max</span></th>
<th id="S4.T3.4.1.1.6" class="ltx_td ltx_align_left ltx_th ltx_th_column ltx_border_tt"><span id="S4.T3.4.1.1.6.1" class="ltx_text ltx_font_bold" style="font-size:90%;">avg time spent (s)</span></th>
</tr>
</thead>
<tbody class="ltx_tbody">
<tr id="S4.T3.4.2.1" class="ltx_tr">
<th id="S4.T3.4.2.1.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_t"><span id="S4.T3.4.2.1.1.1" class="ltx_text" style="font-size:90%;">T1 – Datetime typecast</span></th>
<td id="S4.T3.4.2.1.2" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.2.1.2.1" class="ltx_text" style="font-size:90%;">22</span></td>
<td id="S4.T3.4.2.1.3" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.2.1.3.1" class="ltx_text" style="font-size:90%;">1</span></td>
<td id="S4.T3.4.2.1.4" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.2.1.4.1" class="ltx_text" style="font-size:90%;">1</span></td>
<td id="S4.T3.4.2.1.5" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.2.1.5.1" class="ltx_text" style="font-size:90%;">3</span></td>
<td id="S4.T3.4.2.1.6" class="ltx_td ltx_align_left ltx_border_t"><span id="S4.T3.4.2.1.6.1" class="ltx_text" style="font-size:90%;">55</span></td>
</tr>
<tr id="S4.T3.4.3.2" class="ltx_tr">
<th id="S4.T3.4.3.2.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T3.4.3.2.1.1" class="ltx_text" style="font-size:90%;">T2 – Split using delimiter</span></th>
<td id="S4.T3.4.3.2.2" class="ltx_td ltx_align_left"><span id="S4.T3.4.3.2.2.1" class="ltx_text" style="font-size:90%;">19</span></td>
<td id="S4.T3.4.3.2.3" class="ltx_td ltx_align_left"><span id="S4.T3.4.3.2.3.1" class="ltx_text" style="font-size:90%;">1</span></td>
<td id="S4.T3.4.3.2.4" class="ltx_td ltx_align_left"><span id="S4.T3.4.3.2.4.1" class="ltx_text" style="font-size:90%;">1</span></td>
<td id="S4.T3.4.3.2.5" class="ltx_td ltx_align_left"><span id="S4.T3.4.3.2.5.1" class="ltx_text" style="font-size:90%;">2</span></td>
<td id="S4.T3.4.3.2.6" class="ltx_td ltx_align_left"><span id="S4.T3.4.3.2.6.1" class="ltx_text" style="font-size:90%;">59</span></td>
</tr>
<tr id="S4.T3.4.4.3" class="ltx_tr">
<th id="S4.T3.4.4.3.1" class="ltx_td ltx_align_left ltx_th ltx_th_row"><span id="S4.T3.4.4.3.1.1" class="ltx_text" style="font-size:90%;">T3 – Feature selection</span></th>
<td id="S4.T3.4.4.3.2" class="ltx_td ltx_align_left"><span id="S4.T3.4.4.3.2.1" class="ltx_text" style="font-size:90%;">47</span></td>
<td id="S4.T3.4.4.3.3" class="ltx_td ltx_align_left"><span id="S4.T3.4.4.3.3.1" class="ltx_text" style="font-size:90%;">3</span></td>
<td id="S4.T3.4.4.3.4" class="ltx_td ltx_align_left"><span id="S4.T3.4.4.3.4.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="S4.T3.4.4.3.5" class="ltx_td ltx_align_left"><span id="S4.T3.4.4.3.5.1" class="ltx_text" style="font-size:90%;">6</span></td>
<td id="S4.T3.4.4.3.6" class="ltx_td ltx_align_left"><span id="S4.T3.4.4.3.6.1" class="ltx_text" style="font-size:90%;">159</span></td>
</tr>
<tr id="S4.T3.4.5.4" class="ltx_tr">
<th id="S4.T3.4.5.4.1" class="ltx_td ltx_align_left ltx_th ltx_th_row ltx_border_bb"><span id="S4.T3.4.5.4.1.1" class="ltx_text" style="font-size:90%;">T4 – Heatmap plotting</span></th>
<td id="S4.T3.4.5.4.2" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T3.4.5.4.2.1" class="ltx_text" style="font-size:90%;">23</span></td>
<td id="S4.T3.4.5.4.3" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T3.4.5.4.3.1" class="ltx_text" style="font-size:90%;">2</span></td>
<td id="S4.T3.4.5.4.4" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T3.4.5.4.4.1" class="ltx_text" style="font-size:90%;">0</span></td>
<td id="S4.T3.4.5.4.5" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T3.4.5.4.5.1" class="ltx_text" style="font-size:90%;">4</span></td>
<td id="S4.T3.4.5.4.6" class="ltx_td ltx_align_left ltx_border_bb"><span id="S4.T3.4.5.4.6.1" class="ltx_text" style="font-size:90%;">58</span></td>
</tr>
</tbody>
</table>
</figure>
<div id="S4.SS1.SSS2.p4" class="ltx_para">
<p id="S4.SS1.SSS2.p4.1" class="ltx_p">Participants also made iterative refinements to their prompts. They sent across a plenitude of follow-up prompts for performing feature selection (T3)—re-emphasizing task goals, decomposing tasks, providing additional context and examples, and requesting for tweaks to code snippets. The need to send follow-up prompts was also exacerbated for plotting visualizations (T4), where participants would go back-and-forth with ChatGPT for minor tweaks—changes in plot size, color scheme, and font size. Figure <a href="#S4.F2" title="Figure 2 ‣ 4.1.2. Prompting behaviors ‣ 4.1. Participants Interactions with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a> (Right) reflects on the distribution of follow-up and refined prompts for each task.</p>
</div>
<figure id="S4.F2" class="ltx_figure">
<div class="ltx_flex_figure">
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S4.F2.1" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_middle" style="width:195.1pt;">
<img src="/html/2310.16164/assets/time-spent-percentage.png" id="S4.F2.1.g1" class="ltx_graphics ltx_img_landscape" width="598" height="362" alt="Refer to caption">
</div>
</div>
<div class="ltx_flex_cell ltx_flex_size_2">
<div id="S4.F2.2" class="ltx_block ltx_figure_panel ltx_minipage ltx_align_middle" style="width:195.1pt;">
<img src="/html/2310.16164/assets/prompt-type-percentage.png" id="S4.F2.2.g1" class="ltx_graphics ltx_img_landscape" width="598" height="360" alt="Refer to caption">
</div>
</div>
</div>
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_figure"><span id="S4.F2.4.1.1" class="ltx_text" style="font-size:90%;">Figure 2</span>. </span><span id="S4.F2.5.2" class="ltx_text" style="font-size:90%;">Percentage of total time spent in each activity — prompt writing, code adaptation, and validation — for each task (Left).
Stacked percentage distribution of initiating prompts, refinements, prompts for task decompositions, and follow-up prompts (Right).</span></figcaption>
</figure>
<div id="S4.SS1.SSS2.p5" class="ltx_para">
<p id="S4.SS1.SSS2.p5.1" class="ltx_p">Overall, we observed that when participants interacted with ChatGPT as an assistant for providing recommendations for code, it often involved many steps for more involved tasks. Further, participants often struggled with providing information about their data and desired goals to ChatGPT and adapt its responses to solve their tasks.</p>
</div>
</section>
</section>
<section id="S4.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.2. </span>Obstacles when Communicating with ChatGPT</h3>

<section id="S4.SS2.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.1. </span>Sharing context is difficult</h4>

<div id="S4.SS2.SSS1.p1" class="ltx_para">
<p id="S4.SS2.SSS1.p1.1" class="ltx_p"><span id="S4.SS2.SSS1.p1.1.1" class="ltx_text ltx_font_bold ltx_font_italic">What information must be shared with ChatGPT?</span> Since LLMs <em id="S4.SS2.SSS1.p1.1.2" class="ltx_emph ltx_font_italic">“cannot make sense of the raw data”</em> (P2, P3, P7, P8, P10, P12), data scientists begin the process of prompt-writing by brainstorming what pieces of data, (such as filename, column names, column data types, sample rows, etc.) and which descriptive information (such as pattern of strings in a column, number of unique values, missing values, outlier information, minimum/maximum for numeric data, etc.) will help ChatGPT in solving the query satisfactorily. Participants reported that starting with curating a prompt is a “daunting” task (P9) because it requires decision-making on what data context needs to be included. P2 said, <em id="S4.SS2.SSS1.p1.1.3" class="ltx_emph ltx_font_italic">“ I doubt if this will be successful, I feel like we need a little bit more of numbers, but let’s see if only using the column names we can get something.”</em></p>
</div>
<figure id="S4.T4" class="ltx_table">
<figcaption class="ltx_caption"><span class="ltx_tag ltx_tag_table"><span id="S4.T4.2.1.1" class="ltx_text" style="font-size:90%;">Table 4</span>. </span><span id="S4.T4.3.2" class="ltx_text" style="font-size:90%;">Themes—obstacles and strategies when using ChatGPT for data-science tasks.</span></figcaption>
<div id="S4.T4.4" class="ltx_inline-block ltx_transformed_outer" style="width:404.8pt;height:14952.3pt;vertical-align:-0.9pt;"><span class="ltx_transformed_inner" style="transform:translate(-17.6pt,650.1pt) scale(0.92,0.92) ;">
<table id="S4.T4.4.1" class="ltx_tabular ltx_align_middle">
<tbody class="ltx_tbody">
<tr id="S4.T4.4.1.1.1" class="ltx_tr">
<td id="S4.T4.4.1.1.1.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S4.T4.4.1.1.1.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.1.1.1.1.1" class="ltx_p" style="width:85.4pt;"><span id="S4.T4.4.1.1.1.1.1.1.1" class="ltx_text ltx_font_smallcaps" style="font-size:90%;">Theme</span></span>
</span>
</td>
<td id="S4.T4.4.1.1.1.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S4.T4.4.1.1.1.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.1.1.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.1.1.2.1.1.1" class="ltx_text ltx_font_smallcaps" style="font-size:90%;">Description</span></span>
</span>
</td>
<td id="S4.T4.4.1.1.1.3" class="ltx_td ltx_align_justify ltx_border_tt">
<span id="S4.T4.4.1.1.1.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.1.1.3.1.1" class="ltx_p"><span id="S4.T4.4.1.1.1.3.1.1.1" class="ltx_text ltx_font_smallcaps" style="font-size:90%;">Representative Examples</span></span>
</span>
</td>
<td id="S4.T4.4.1.1.1.4" class="ltx_td ltx_align_justify ltx_align_top ltx_border_tt">
<span id="S4.T4.4.1.1.1.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.1.1.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.1.1.4.1.1.1" class="ltx_text ltx_font_smallcaps" style="font-size:90%;">Participants</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.2.2" class="ltx_tr">
<td id="S4.T4.4.1.2.2.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" colspan="3"><span id="S4.T4.4.1.2.2.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Obstacles when Communicating with ChatGPT</span></td>
<td id="S4.T4.4.1.2.2.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t"></td>
</tr>
<tr id="S4.T4.4.1.3.3" class="ltx_tr">
<td id="S4.T4.4.1.3.3.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.3.3.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.3.3.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.3.3.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Sharing context is difficult</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.3.3.1.1.1.2" class="ltx_text" style="font-size:90%;">(Section </span><a href="#S4.SS2.SSS1" title="4.2.1. Sharing context is difficult ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">4.2.1</span></a><span id="S4.T4.4.1.3.3.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.3.3.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.3.3.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.3.3.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.3.3.2.1.1.1" class="ltx_text" style="font-size:90%;">Gathering relevant information to prompt with (e.g., column names, datatypes, example datapoints) takes time.</span></span>
</span>
</td>
<td id="S4.T4.4.1.3.3.3" class="ltx_td ltx_align_justify ltx_border_t">
<span id="S4.T4.4.1.3.3.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.3.3.3.1.1" class="ltx_p"><span id="S4.T4.4.1.3.3.3.1.1.1" class="ltx_text" style="font-size:90%;">“Should I add some rows for it to see? I am not sure if [it] will work…” </span>
<br class="ltx_break"><span id="S4.T4.4.1.3.3.3.1.1.2" class="ltx_text" style="font-size:90%;">“So this one is recommending </span><span id="S4.T4.4.1.3.3.3.1.1.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">scikit-learn</span><span id="S4.T4.4.1.3.3.3.1.1.4" class="ltx_text" style="font-size:90%;"> which is not a package I typically use. I usually work in </span><span id="S4.T4.4.1.3.3.3.1.1.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">pyspark</span><span id="S4.T4.4.1.3.3.3.1.1.6" class="ltx_text" style="font-size:90%;">.” </span>
<br class="ltx_break"><span id="S4.T4.4.1.3.3.3.1.1.7" class="ltx_text" style="font-size:90%;">“It is all about the context which we provide. If we could refine it, it would do better.”</span></span>
</span>
</td>
<td id="S4.T4.4.1.3.3.4" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.3.3.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.3.3.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.3.3.4.1.1.1" class="ltx_text" style="font-size:90%;">P1—P6, P8, P10, P12, P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.3.3.4.1.1.2" class="ltx_text" style="font-size:90%;">(10 </span><span id="S4.T4.4.1.3.3.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.3.3.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.4.4" class="ltx_tr">
<td id="S4.T4.4.1.4.4.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.4.4.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.4.4.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.4.4.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">ChatGPT opaquely makes assumptions</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.4.4.1.1.1.2" class="ltx_text" style="font-size:90%;">(Section </span><a href="#S4.SS2.SSS2" title="4.2.2. ChatGPT opaquely makes assumptions ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">4.2.2</span></a><span id="S4.T4.4.1.4.4.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.4.4.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.4.4.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.4.4.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.4.4.2.1.1.1" class="ltx_text" style="font-size:90%;">Data scientists had to correct or adjust prompts in response to unanticipated assumptions made by ChatGPT about data.</span></span>
</span>
</td>
<td id="S4.T4.4.1.4.4.3" class="ltx_td ltx_align_justify">
<span id="S4.T4.4.1.4.4.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.4.4.3.1.1" class="ltx_p"><span id="S4.T4.4.1.4.4.3.1.1.1" class="ltx_text" style="font-size:90%;">“I will provide this example for the format” </span>
<br class="ltx_break"><span id="S4.T4.4.1.4.4.3.1.1.2" class="ltx_text" style="font-size:90%;">“Oh, it thinks the call type is int, let me say that it is categorical”</span></span>
</span>
</td>
<td id="S4.T4.4.1.4.4.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.4.4.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.4.4.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.4.4.4.1.1.1" class="ltx_text" style="font-size:90%;">P1—P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.4.4.4.1.1.2" class="ltx_text" style="font-size:90%;">(14 </span><span id="S4.T4.4.1.4.4.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.4.4.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.5.5" class="ltx_tr">
<td id="S4.T4.4.1.5.5.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.5.5.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.5.5.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.5.5.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Misaligned expectations</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.5.5.1.1.1.2" class="ltx_text" style="font-size:90%;">(Section </span><a href="#S4.SS2.SSS3" title="4.2.3. Misaligned expectations ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">4.2.3</span></a><span id="S4.T4.4.1.5.5.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.5.5.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.5.5.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.5.5.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.5.5.2.1.1.1" class="ltx_text" style="font-size:90%;">ChatGPT often returned overly verbose answers or inappropriate code responses.</span></span>
</span>
</td>
<td id="S4.T4.4.1.5.5.3" class="ltx_td ltx_align_justify">
<span id="S4.T4.4.1.5.5.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.5.5.3.1.1" class="ltx_p"><span id="S4.T4.4.1.5.5.3.1.1.1" class="ltx_text" style="font-size:90%;">“I will skip reading any of this text, unless my code doesn’t work” </span>
<br class="ltx_break"><span id="S4.T4.4.1.5.5.3.1.1.2" class="ltx_text" style="font-size:90%;">“it didn’t bring up </span><span id="S4.T4.4.1.5.5.3.1.1.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">to_datetime()</span><span id="S4.T4.4.1.5.5.3.1.1.4" class="ltx_text" style="font-size:90%;"> and suggested </span><span id="S4.T4.4.1.5.5.3.1.1.5" class="ltx_text ltx_font_typewriter" style="font-size:90%;">strptime()</span><span id="S4.T4.4.1.5.5.3.1.1.6" class="ltx_text" style="font-size:90%;"> instead!”</span></span>
</span>
</td>
<td id="S4.T4.4.1.5.5.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.5.5.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.5.5.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.5.5.4.1.1.1" class="ltx_text" style="font-size:90%;">P1—P10, P12—P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.5.5.4.1.1.2" class="ltx_text" style="font-size:90%;">(13 </span><span id="S4.T4.4.1.5.5.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.5.5.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.6.6" class="ltx_tr">
<td id="S4.T4.4.1.6.6.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" colspan="3"><span id="S4.T4.4.1.6.6.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Obstacles in Leveraging ChatGPT’s Responses</span></td>
<td id="S4.T4.4.1.6.6.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t"></td>
</tr>
<tr id="S4.T4.4.1.7.7" class="ltx_tr">
<td id="S4.T4.4.1.7.7.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.7.7.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.7.7.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.7.7.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Generation of repeated code</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.7.7.1.1.1.2" class="ltx_text" style="font-size:90%;">(</span><a href="#S4.SS3.SSS1" title="4.3.1. Generation of repeated code ‣ 4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">4.3.1</span></a><span id="S4.T4.4.1.7.7.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.7.7.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.7.7.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.7.7.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.7.7.2.1.1.1" class="ltx_text" style="font-size:90%;">ChatGPT repeatedly generated code for imports, data ingestion and exploration, which had to be removed.</span></span>
</span>
</td>
<td id="S4.T4.4.1.7.7.3" class="ltx_td ltx_align_justify ltx_border_t">
<span id="S4.T4.4.1.7.7.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.7.7.3.1.1" class="ltx_p"><span id="S4.T4.4.1.7.7.3.1.1.1" class="ltx_text" style="font-size:90%;">“It erased my data, I will quickly re-run the cells” </span>
<br class="ltx_break"><span id="S4.T4.4.1.7.7.3.1.1.2" class="ltx_text" style="font-size:90%;">“We already have pandas and the data, I’ll delete these lines […] not required”</span></span>
</span>
</td>
<td id="S4.T4.4.1.7.7.4" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.7.7.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.7.7.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.7.7.4.1.1.1" class="ltx_text" style="font-size:90%;">P1—P10, P12—P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.7.7.4.1.1.2" class="ltx_text" style="font-size:90%;">(13 </span><span id="S4.T4.4.1.7.7.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.7.7.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.8.8" class="ltx_tr">
<td id="S4.T4.4.1.8.8.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.8.8.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.8.8.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.8.8.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Data and notebook management preferences</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.8.8.1.1.1.2" class="ltx_text" style="font-size:90%;">(</span><a href="#S4.SS3.SSS2" title="4.3.2. Data and notebook management preferences ‣ 4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">4.3.2</span></a><span id="S4.T4.4.1.8.8.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.8.8.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.8.8.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.8.8.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.8.8.2.1.1.1" class="ltx_text" style="font-size:90%;">Splitting code into cells and aligning it with data management preferences (such as creating temporary views, dropping parent columns, etc).</span></span>
</span>
</td>
<td id="S4.T4.4.1.8.8.3" class="ltx_td ltx_align_justify">
<span id="S4.T4.4.1.8.8.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.8.8.3.1.1" class="ltx_p"><span id="S4.T4.4.1.8.8.3.1.1.1" class="ltx_text" style="font-size:90%;">“a temp df to see the new split columns” </span>
<br class="ltx_break"><span id="S4.T4.4.1.8.8.3.1.1.2" class="ltx_text" style="font-size:90%;">“I would not fill with ‘NULL’ […] still need to understand these columns more”</span></span>
</span>
</td>
<td id="S4.T4.4.1.8.8.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.8.8.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.8.8.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.8.8.4.1.1.1" class="ltx_text" style="font-size:90%;">P2—P9, P12—P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.8.8.4.1.1.2" class="ltx_text" style="font-size:90%;">(11 </span><span id="S4.T4.4.1.8.8.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.8.8.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.9.9" class="ltx_tr">
<td id="S4.T4.4.1.9.9.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.9.9.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.9.9.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.9.9.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Code validation</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.9.9.1.1.1.2" class="ltx_text" style="font-size:90%;">(</span><a href="#S4.SS3.SSS3" title="4.3.3. Code validation ‣ 4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">4.3.3</span></a><span id="S4.T4.4.1.9.9.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.9.9.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.9.9.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.9.9.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.9.9.2.1.1.1" class="ltx_text" style="font-size:90%;">Data scientists often author additional code snippets to validate generated code.</span></span>
</span>
</td>
<td id="S4.T4.4.1.9.9.3" class="ltx_td ltx_align_justify">
<span id="S4.T4.4.1.9.9.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.9.9.3.1.1" class="ltx_p"><span id="S4.T4.4.1.9.9.3.1.1.1" class="ltx_text" style="font-size:90%;">“ChatGPT’s affirmative language is deceiving […] it doesn’t really know my data” </span>
<br class="ltx_break"><span id="S4.T4.4.1.9.9.3.1.1.2" class="ltx_text" style="font-size:90%;">“let me print </span><span id="S4.T4.4.1.9.9.3.1.1.3" class="ltx_text ltx_font_typewriter" style="font-size:90%;">df.dtypes</span><span id="S4.T4.4.1.9.9.3.1.1.4" class="ltx_text" style="font-size:90%;"> to be sure”</span></span>
</span>
</td>
<td id="S4.T4.4.1.9.9.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.9.9.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.9.9.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.9.9.4.1.1.1" class="ltx_text" style="font-size:90%;">P1—P10, P12—P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.9.9.4.1.1.2" class="ltx_text" style="font-size:90%;">(13 </span><span id="S4.T4.4.1.9.9.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.9.9.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.10.10" class="ltx_tr">
<td id="S4.T4.4.1.10.10.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t" colspan="3"><span id="S4.T4.4.1.10.10.1.1" class="ltx_text ltx_font_bold" style="font-size:90%;">Strategies for Prompting and Alternate Resources</span></td>
<td id="S4.T4.4.1.10.10.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t"></td>
</tr>
<tr id="S4.T4.4.1.11.11" class="ltx_tr">
<td id="S4.T4.4.1.11.11.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.11.11.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.11.11.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.11.11.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Techniques for prompt construction</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.11.11.1.1.1.2" class="ltx_text" style="font-size:90%;">(</span><a href="#S4.SS4.SSS1" title="4.4.1. Techniques for prompt construction ‣ 4.4. Strategies for Prompting and Alternate Resources ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">4.4.1</span></a><span id="S4.T4.4.1.11.11.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.11.11.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.11.11.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.11.11.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.11.11.2.1.1.1" class="ltx_text" style="font-size:90%;">Data scientists use prompting techniques and create placeholders for streamlined context gathering from various sources</span></span>
</span>
</td>
<td id="S4.T4.4.1.11.11.3" class="ltx_td ltx_align_justify ltx_border_t">
<span id="S4.T4.4.1.11.11.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.11.11.3.1.1" class="ltx_p"><span id="S4.T4.4.1.11.11.3.1.1.1" class="ltx_text" style="font-size:90%;">“lets ask why it selected these features, why not the others” </span>
<br class="ltx_break"><span id="S4.T4.4.1.11.11.3.1.1.2" class="ltx_text" style="font-size:90%;">“I will paste column names later, so that I can write the prompt first”</span></span>
</span>
</td>
<td id="S4.T4.4.1.11.11.4" class="ltx_td ltx_align_justify ltx_align_top ltx_border_t">
<span id="S4.T4.4.1.11.11.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.11.11.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.11.11.4.1.1.1" class="ltx_text" style="font-size:90%;">P1—P10, P12, P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.11.11.4.1.1.2" class="ltx_text" style="font-size:90%;">(11 </span><span id="S4.T4.4.1.11.11.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.11.11.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.12.12" class="ltx_tr">
<td id="S4.T4.4.1.12.12.1" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.12.12.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.12.12.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.12.12.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Scaffolding with domain expertise</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.12.12.1.1.1.2" class="ltx_text" style="font-size:90%;">(</span><a href="#S4.SS4.SSS2" title="4.4.2. Scaffolding with domain expertise ‣ 4.4. Strategies for Prompting and Alternate Resources ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">4.4.2</span></a><span id="S4.T4.4.1.12.12.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.12.12.2" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.12.12.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.12.12.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.12.12.2.1.1.1" class="ltx_text" style="font-size:90%;">Data scientists often employ techniques to guide ChatGPT with their domain knowledge to elicit better responses.</span></span>
</span>
</td>
<td id="S4.T4.4.1.12.12.3" class="ltx_td ltx_align_justify">
<span id="S4.T4.4.1.12.12.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.12.12.3.1.1" class="ltx_p"><span id="S4.T4.4.1.12.12.3.1.1.1" class="ltx_text" style="font-size:90%;">“I can be more specific and give it only the float columns”</span>
<br class="ltx_break"><span id="S4.T4.4.1.12.12.3.1.1.2" class="ltx_text" style="font-size:90%;">“I will exclude the time columns to get good features”</span></span>
</span>
</td>
<td id="S4.T4.4.1.12.12.4" class="ltx_td ltx_align_justify ltx_align_top">
<span id="S4.T4.4.1.12.12.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.12.12.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.12.12.4.1.1.1" class="ltx_text" style="font-size:90%;">P5, P10, P14</span>
<br class="ltx_break"><span id="S4.T4.4.1.12.12.4.1.1.2" class="ltx_text" style="font-size:90%;">(3 </span><span id="S4.T4.4.1.12.12.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.12.12.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
<tr id="S4.T4.4.1.13.13" class="ltx_tr">
<td id="S4.T4.4.1.13.13.1" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S4.T4.4.1.13.13.1.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.13.13.1.1.1" class="ltx_p" style="width:85.4pt;"><em id="S4.T4.4.1.13.13.1.1.1.1" class="ltx_emph ltx_font_italic" style="font-size:90%;">Choosing alternate resources over ChatGPT</em>
<br class="ltx_break ltx_align_left"><span id="S4.T4.4.1.13.13.1.1.1.2" class="ltx_text" style="font-size:90%;">(</span><a href="#S4.SS4.SSS3" title="4.4.3. Choosing alternate resources over ChatGPT ‣ 4.4. Strategies for Prompting and Alternate Resources ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref ltx_align_left" style="font-size:90%;"><span class="ltx_text ltx_ref_tag">Section</span> <span class="ltx_text ltx_ref_tag">4.4.3</span></a><span id="S4.T4.4.1.13.13.1.1.1.3" class="ltx_text" style="font-size:90%;">)</span></span>
</span>
</td>
<td id="S4.T4.4.1.13.13.2" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S4.T4.4.1.13.13.2.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.13.13.2.1.1" class="ltx_p" style="width:113.8pt;"><span id="S4.T4.4.1.13.13.2.1.1.1" class="ltx_text" style="font-size:90%;">Making use of web-search; and re-use of previously authored notebooks/pipelines</span></span>
</span>
</td>
<td id="S4.T4.4.1.13.13.3" class="ltx_td ltx_align_justify ltx_border_bb">
<span id="S4.T4.4.1.13.13.3.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.13.13.3.1.1" class="ltx_p"><span id="S4.T4.4.1.13.13.3.1.1.1" class="ltx_text" style="font-size:90%;">“must write longer queries instead of 8-10 key words just because I can”
</span>
<br class="ltx_break"><span id="S4.T4.4.1.13.13.3.1.1.2" class="ltx_text" style="font-size:90%;">“copy paste canned code […] 80% of it would be repeating”</span></span>
</span>
</td>
<td id="S4.T4.4.1.13.13.4" class="ltx_td ltx_align_justify ltx_align_top ltx_border_bb">
<span id="S4.T4.4.1.13.13.4.1" class="ltx_inline-block ltx_align_top">
<span id="S4.T4.4.1.13.13.4.1.1" class="ltx_p" style="width:56.9pt;"><span id="S4.T4.4.1.13.13.4.1.1.1" class="ltx_text" style="font-size:90%;">P3—P5, P7, P11, P12</span>
<br class="ltx_break"><span id="S4.T4.4.1.13.13.4.1.1.2" class="ltx_text" style="font-size:90%;">(6 </span><span id="S4.T4.4.1.13.13.4.1.1.3" class="ltx_text ltx_font_italic" style="font-size:90%;">of</span><span id="S4.T4.4.1.13.13.4.1.1.4" class="ltx_text" style="font-size:90%;"> 14)</span></span>
</span>
</td>
</tr>
</tbody>
</table>
</span></div>
</figure>
<div id="S4.SS2.SSS1.p2" class="ltx_para">
<p id="S4.SS2.SSS1.p2.1" class="ltx_p"><span id="S4.SS2.SSS1.p2.1.1" class="ltx_text ltx_font_bold ltx_font_italic">How much information must be shared with ChatGPT?</span> Participants often questioned <em id="S4.SS2.SSS1.p2.1.2" class="ltx_emph ltx_font_italic">how much</em> information needs to be shared with ChatGPT to get a good response. Some participants made attempts to paste the entire CSV file into the prompt and encountered token limit errors (P6, P7, P13). Others also reflected on the importance of sharing only relevant instructions or snippets of the data as <em id="S4.SS2.SSS1.p2.1.3" class="ltx_emph ltx_font_italic">“additional data seems to confuse ChatGPT into providing not-so-relevant results”</em> (P7, P9, P10). P9 said, <em id="S4.SS2.SSS1.p2.1.4" class="ltx_emph ltx_font_italic">“The main obstacle I ran into here was, knowing how to prompt ChatGPT. Because I think at first I almost gave it too much information on how to perform a correlation analysis by describing it.”</em> After sharing 10 sample rows, P2 tried to estimate ChatGPT’s boundaries in being able to understand raw data beyond its semantics, as it provided a <em id="S4.SS2.SSS1.p2.1.5" class="ltx_emph ltx_font_italic">“very basic response for the most relevant fields, by repeating the entire fields names and telling me their [semantic] meaning.”</em></p>
</div>
<div id="S4.SS2.SSS1.p3" class="ltx_para">
<p id="S4.SS2.SSS1.p3.1" class="ltx_p"><span id="S4.SS2.SSS1.p3.1.1" class="ltx_text ltx_font_bold ltx_font_italic">Challenges in fetching context.</span> To construct the prompt, after assessing <span id="S4.SS2.SSS1.p3.1.2" class="ltx_text ltx_font_italic">what</span> context is relevant, participants had to go back and forth between ChatGPT’s browser window and data sources to fetch the context. They had to frequently write code snippets in their computational notebooks to gather the required context. For instance, consider a <span id="S4.SS2.SSS1.p3.1.3" class="ltx_text ltx_font_typewriter">pandas</span> DataFrame named <span id="S4.SS2.SSS1.p3.1.4" class="ltx_text ltx_font_typewriter">df</span>. Participants must write, execute, and copy the output of commands such as:</p>
<ul id="S4.I1" class="ltx_itemize">
<li id="S4.I1.i1" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i1.p1" class="ltx_para">
<p id="S4.I1.i1.p1.1" class="ltx_p">df.head() to obtain a sample of rows and the data header</p>
</div>
</li>
<li id="S4.I1.i2" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i2.p1" class="ltx_para">
<p id="S4.I1.i2.p1.1" class="ltx_p">df.columns or df.dtypes to obtain column names and their data types</p>
</div>
</li>
<li id="S4.I1.i3" class="ltx_item" style="list-style-type:none;">
<span class="ltx_tag ltx_tag_item">•</span> 
<div id="S4.I1.i3.p1" class="ltx_para">
<p id="S4.I1.i3.p1.1" class="ltx_p">df[’INCIDENT_DATETIME’][0] and df[’INCIDENT_DESCRIPTION’][0] to obtain the first value in the respective columns to provide as few-shot examples, and so on.</p>
</div>
</li>
</ul>
</div>
<div id="S4.SS2.SSS1.p4" class="ltx_para">
<p id="S4.SS2.SSS1.p4.1" class="ltx_p">We observed that P10 and P14 also implemented custom logic to extract information to share beyond what is typically shared in a prompt. For instance, P14 wanted to provide names of specific columns to ChatGPT and hence wrote the following code that computes column names that do not contain the sub-strings <span id="S4.SS2.SSS1.p4.1.1" class="ltx_text ltx_font_typewriter">‘TIME’</span> or <span id="S4.SS2.SSS1.p4.1.2" class="ltx_text ltx_font_typewriter">‘ID’</span>:</p>
</div>
<div id="S4.SS2.SSS1.p5" class="ltx_para">
<div id="S4.SS2.SSS1.p5.1" class="ltx_listing ltx_lstlisting ltx_listing">
<div class="ltx_listing_data"><a href="data:text/plain;base64,Cj4+PiBbaSBmb3IgaSBpbiBkZi5jb2x1bW5zIGlmICdUSU1FJyBub3QgaW4gaS51cHBlcigpIG9yICdJRCcgbm90IGluIGkudXBwZXIoKV0K" download="">⬇</a></div>
<div id="lstnumberx1" class="ltx_listingline">
</div>
<div id="lstnumberx2" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx2.1" class="ltx_text ltx_lst_space"> </span>[<span id="lstnumberx2.2" class="ltx_text ltx_lst_identifier">i</span><span id="lstnumberx2.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.4" class="ltx_text ltx_lst_identifier">for</span><span id="lstnumberx2.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.6" class="ltx_text ltx_lst_identifier">i</span><span id="lstnumberx2.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.8" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx2.9" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.10" class="ltx_text ltx_lst_identifier">df</span>.<span id="lstnumberx2.11" class="ltx_text ltx_lst_identifier">columns</span><span id="lstnumberx2.12" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.13" class="ltx_text ltx_lst_identifier">if</span><span id="lstnumberx2.14" class="ltx_text ltx_lst_space"> </span>’<span id="lstnumberx2.15" class="ltx_text ltx_lst_identifier">TIME</span>’<span id="lstnumberx2.16" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.17" class="ltx_text ltx_lst_identifier">not</span><span id="lstnumberx2.18" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.19" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx2.20" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.21" class="ltx_text ltx_lst_identifier">i</span>.<span id="lstnumberx2.22" class="ltx_text ltx_lst_identifier">upper</span>()<span id="lstnumberx2.23" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.24" class="ltx_text ltx_lst_identifier">or</span><span id="lstnumberx2.25" class="ltx_text ltx_lst_space"> </span>’<span id="lstnumberx2.26" class="ltx_text ltx_lst_identifier">ID</span>’<span id="lstnumberx2.27" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.28" class="ltx_text ltx_lst_identifier">not</span><span id="lstnumberx2.29" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.30" class="ltx_text ltx_lst_identifier">in</span><span id="lstnumberx2.31" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx2.32" class="ltx_text ltx_lst_identifier">i</span>.<span id="lstnumberx2.33" class="ltx_text ltx_lst_identifier">upper</span>()]
</div>
</div>
</div>
<figure id="S4.F3" class="ltx_figure"><img src="/html/2310.16164/assets/x2.png" id="S4.F3.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="106" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F3.2.1.1" class="ltx_text" style="font-size:90%;">Figure 3</span>. </span><span id="S4.F3.3.2" class="ltx_text" style="font-size:90%;">Frequency of data context shared by participants for prompt-construction for each task.</span></figcaption>
</figure>
<div id="S4.SS2.SSS1.p6" class="ltx_para">
<p id="S4.SS2.SSS1.p6.1" class="ltx_p">Further, P5, P7 and P8 had to leave their notebook environment to open the CSV file in Excel. This was because the participants wanted to share values from a column, and the raw output from df.head() could not be selected freely to be copied to the clipboard. Opening the file in Excel enabled them to select any range of cells and copy them with ease without writing additional code.</p>
</div>
<div id="S4.SS2.SSS1.p7" class="ltx_para">
<p id="S4.SS2.SSS1.p7.1" class="ltx_p">In addition to data context, participants also used snippets of their code with error messages to prompt ChatGPT for a fix (P6, P7, P12). Figure <a href="#S4.F3" title="Figure 3 ‣ 4.2.1. Sharing context is difficult ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">3</span></a> shows the frequency of different types of data context shared with ChatGPT corresponding to each task.</p>
</div>
<div id="S4.SS2.SSS1.p8" class="ltx_para">
<p id="S4.SS2.SSS1.p8.1" class="ltx_p">After fetching the relevant data context, participants had to paste it into the prompt. Participants would often manually format the pasted information to make the prompt look <em id="S4.SS2.SSS1.p8.1.1" class="ltx_emph ltx_font_italic">“cleaner,”</em> to ensure that ChatGPT can understand the context (P2, P3, P7). While solving T3 (feature selection), P2 wanted to fetch column names, and organize them such that each column name appears in a new line in the prompt. To this, he said, <em id="S4.SS2.SSS1.p8.1.2" class="ltx_emph ltx_font_italic">“Can you copy and paste it for me please? By any chance would you have like all the columns of the dataset one name per line? In the meantime, I’ll be typing [the prompt]… I am so lazy.”</em></p>
</div>
</section>
<section id="S4.SS2.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.2. </span>ChatGPT opaquely makes assumptions</h4>

<div id="S4.SS2.SSS2.p1" class="ltx_para">
<p id="S4.SS2.SSS2.p1.1" class="ltx_p">Based on the context shared with ChatGPT, it made its own “mental model” of the data. Participants were often enthusiastic about the semantic capabilities of ChatGPT, and how it could infer domain knowledge about the data solely using the column names. P4 leveraged these capabilities and prompted ChatGPT to provide a data dictionary. Several participants (P5, P7, P10, P12—P14) were amazed when ChatGPT could understand the data domain sufficiently to convert the extracted <span id="S4.SS2.SSS2.p1.1.1" class="ltx_text ltx_font_typewriter">‘RESPONSE_TIME’</span> column from <span id="S4.SS2.SSS2.p1.1.2" class="ltx_text ltx_font_typewriter">timedelta</span> format to seconds—which was the most appropriate unit of measurement, since emergency medical services are likely to be dispatched quickly. However, despite the semantic abilities of the LLM, participants struggled with several scenarios, and had to re-assess ChatGPT’s understanding of the task and the data. We observed several instances across every task, where lack of adequate context in the prompt led ChatGPT to make assumptions about the data and its format.</p>
</div>
<div id="S4.SS2.SSS2.p2" class="ltx_para">
<p id="S4.SS2.SSS2.p2.1" class="ltx_p">Several participants (P1, P5, P6) received incorrect code generations for T1 (datetime typecast), where ChatGPT assumed the string to be of the format <span id="S4.SS2.SSS2.p2.1.1" class="ltx_text ltx_font_typewriter">‘%d%m%Y %H:%M:%S’</span> (e.g., 31-07-2020 23:59:51) instead of <span id="S4.SS2.SSS2.p2.1.2" class="ltx_text ltx_font_typewriter">‘%Y-%m-%dT%H:%M:%S.%f’</span> (e.g., 2020-07-31T23:59:51.000). This assumption led to generation of the following code snippet to typecast the string column to datetime format, resulting in a run-time error:</p>
<div id="S4.SS2.SSS2.p2.2" class="ltx_listing ltx_lstlisting ltx_listing">
<div class="ltx_listing_data"><a href="data:text/plain;base64,Cj4+PiBwZC50b19kYXRldGltZShkYXRlX3N0ciwgZm9ybWF0PSdFcnJvcjogdGltZSBkYXRhICcyMDIwLTA3LTMxVDIzOjU5OjUxLjAwMCcgZG9lcyBub3QgbWF0Y2ggdGhlIGZvcm1hdCAn" download="">⬇</a></div>
<div id="lstnumberx3" class="ltx_listingline">
</div>
<div id="lstnumberx4" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx4.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.2" class="ltx_text ltx_lst_identifier">pd</span>.<span id="lstnumberx4.3" class="ltx_text ltx_lst_identifier">to_datetime</span>(<span id="lstnumberx4.4" class="ltx_text ltx_lst_identifier">date_str</span>,<span id="lstnumberx4.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.6" class="ltx_text ltx_lst_identifier">format</span>=’<span id="lstnumberx4.7" class="ltx_text ltx_lst_identifier">Error</span>:<span id="lstnumberx4.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.9" class="ltx_text ltx_lst_identifier">time</span><span id="lstnumberx4.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.11" class="ltx_text ltx_lst_identifier">data</span><span id="lstnumberx4.12" class="ltx_text ltx_lst_space"> </span>’2020-07-31<span id="lstnumberx4.13" class="ltx_text ltx_lst_identifier">T23</span>:59:51.000’<span id="lstnumberx4.14" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.15" class="ltx_text ltx_lst_identifier">does</span><span id="lstnumberx4.16" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.17" class="ltx_text ltx_lst_identifier">not</span><span id="lstnumberx4.18" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.19" class="ltx_text ltx_lst_identifier">match</span><span id="lstnumberx4.20" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.21" class="ltx_text ltx_lst_identifier">the</span><span id="lstnumberx4.22" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx4.23" class="ltx_text ltx_lst_identifier">format</span><span id="lstnumberx4.24" class="ltx_text ltx_lst_space"> </span>’
</div>
</div>
<p id="S4.SS2.SSS2.p2.3" class="ltx_p">Upon providing ChatGPT with the error message, or an example string from the column, it re-generated the snippet without the <span id="S4.SS2.SSS2.p2.3.1" class="ltx_text ltx_font_typewriter">format</span> parameter, allowing successful execution, as <span id="S4.SS2.SSS2.p2.3.2" class="ltx_text ltx_font_typewriter">pandas</span> can now correctly infer the format of the input datetime string itself.</p>
</div>
<div id="S4.SS2.SSS2.p3" class="ltx_para">
<p id="S4.SS2.SSS2.p3.1" class="ltx_p">Next, as part of T2 (split using delimiter), participants had to work with the <span id="S4.SS2.SSS2.p3.1.1" class="ltx_text ltx_font_typewriter">‘INCIDENT_DESCRIPTION’</span> column, containing the borough, incident dispatch area code, and zipcode, separated by semicolons. However, few rows in this column did not contain the zipcode. This led to the column having two formats: (1) <span id="S4.SS2.SSS2.p3.1.2" class="ltx_text ltx_font_typewriter">‘‘&lt;borough&gt;; &lt;incident_dispatch_area_code&gt;; &lt;zipcode&gt;’’</span>, for example: “Brooklyn; K7; 11211”, and (2) <span id="S4.SS2.SSS2.p3.1.3" class="ltx_text ltx_font_typewriter">‘‘&lt;borough&gt;;</span> 
<br class="ltx_break"><span id="S4.SS2.SSS2.p3.1.4" class="ltx_text ltx_font_typewriter">&lt;incident_dispatch_area_code&gt;’’</span>, for example: “Manhattan; M3”. Participants generally did not discover this data-formatting inconsistency by themselves, since encountering such issues requires exploratory data analysis and on-ramping. As a result, ChatGPT often did not get visibility into this data quality issue through participant prompts (P1, P2, P5, P11), and it generated non-executable code for splitting the column under the assumption of having clean and consistent data, which contains all three splits in each row:</p>
</div>
<div id="S4.SS2.SSS2.p4" class="ltx_para">
<div id="S4.SS2.SSS2.p4.1" class="ltx_listing ltx_lstlisting ltx_listing">
<div class="ltx_listing_data"><a href="data:text/plain;base64,CiMgc3BsaXQgdGhlIElOQ0lERU5UX0RFU0NSSVBUSU9OIGNvbHVtbiBieSB0aGUgOyBzZXBhcmF0b3IKPj4+IGRmWydCb3JvdWdoJ10gPSBkZlsnSU5DSURFTlRfREVTQ1JJUFRJT04nXS5zdHIuc3BsaXQoJzsnKS5zdHJbMF0KPj4+IGRmWydQcmVjaW5jdCddID0gZGZbJ0lOQ0lERU5UX0RFU0NSSVBUSU9OJ10uc3RyLnNwbGl0KCc7Jykuc3RyWzFdCj4+PiBkZlsnWmlwQ29kZSddID0gZGZbJ0lOQ0lERU5UX0RFU0NSSVBUSU9OJ10uc3RyLnNwbGl0KCc7Jykuc3RyWzJdCkluZGV4RXJyb3I6IGxpc3QgaW5kZXggb3V0IG9mIHJhbmdlCg==" download="">⬇</a></div>
<div id="lstnumberx5" class="ltx_listingline">
</div>
<div id="lstnumberx6" class="ltx_listingline">#<span id="lstnumberx6.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.2" class="ltx_text ltx_lst_identifier">split</span><span id="lstnumberx6.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.4" class="ltx_text ltx_lst_identifier">the</span><span id="lstnumberx6.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.6" class="ltx_text ltx_lst_identifier">INCIDENT_DESCRIPTION</span><span id="lstnumberx6.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.8" class="ltx_text ltx_lst_identifier">column</span><span id="lstnumberx6.9" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.10" class="ltx_text ltx_lst_identifier">by</span><span id="lstnumberx6.11" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.12" class="ltx_text ltx_lst_identifier">the</span><span id="lstnumberx6.13" class="ltx_text ltx_lst_space"> </span>;<span id="lstnumberx6.14" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx6.15" class="ltx_text ltx_lst_identifier">separator</span>
</div>
<div id="lstnumberx7" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx7.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx7.2" class="ltx_text ltx_lst_identifier">df</span>[’<span id="lstnumberx7.3" class="ltx_text ltx_lst_identifier">Borough</span>’]<span id="lstnumberx7.4" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx7.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx7.6" class="ltx_text ltx_lst_identifier">df</span>[’<span id="lstnumberx7.7" class="ltx_text ltx_lst_identifier">INCIDENT_DESCRIPTION</span>’].<span id="lstnumberx7.8" class="ltx_text ltx_lst_identifier">str</span>.<span id="lstnumberx7.9" class="ltx_text ltx_lst_identifier">split</span>(’;’).<span id="lstnumberx7.10" class="ltx_text ltx_lst_identifier">str</span>[0]
</div>
<div id="lstnumberx8" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx8.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx8.2" class="ltx_text ltx_lst_identifier">df</span>[’<span id="lstnumberx8.3" class="ltx_text ltx_lst_identifier">Precinct</span>’]<span id="lstnumberx8.4" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx8.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx8.6" class="ltx_text ltx_lst_identifier">df</span>[’<span id="lstnumberx8.7" class="ltx_text ltx_lst_identifier">INCIDENT_DESCRIPTION</span>’].<span id="lstnumberx8.8" class="ltx_text ltx_lst_identifier">str</span>.<span id="lstnumberx8.9" class="ltx_text ltx_lst_identifier">split</span>(’;’).<span id="lstnumberx8.10" class="ltx_text ltx_lst_identifier">str</span>[1]
</div>
<div id="lstnumberx9" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx9.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx9.2" class="ltx_text ltx_lst_identifier">df</span>[’<span id="lstnumberx9.3" class="ltx_text ltx_lst_identifier">ZipCode</span>’]<span id="lstnumberx9.4" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx9.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx9.6" class="ltx_text ltx_lst_identifier">df</span>[’<span id="lstnumberx9.7" class="ltx_text ltx_lst_identifier">INCIDENT_DESCRIPTION</span>’].<span id="lstnumberx9.8" class="ltx_text ltx_lst_identifier">str</span>.<span id="lstnumberx9.9" class="ltx_text ltx_lst_identifier">split</span>(’;’).<span id="lstnumberx9.10" class="ltx_text ltx_lst_identifier">str</span>[2]
</div>
<div id="lstnumberx10" class="ltx_listingline">
<span id="lstnumberx10.1" class="ltx_text ltx_lst_identifier">IndexError</span>:<span id="lstnumberx10.2" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.3" class="ltx_text ltx_lst_identifier">list</span><span id="lstnumberx10.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.5" class="ltx_text ltx_lst_identifier">index</span><span id="lstnumberx10.6" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.7" class="ltx_text ltx_lst_identifier">out</span><span id="lstnumberx10.8" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.9" class="ltx_text ltx_lst_identifier">of</span><span id="lstnumberx10.10" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx10.11" class="ltx_text ltx_lst_identifier">range</span>
</div>
</div>
</div>
<div id="S4.SS2.SSS2.p5" class="ltx_para">
<p id="S4.SS2.SSS2.p5.1" class="ltx_p">On the other hand, some participants were able to receive correct code generations from ChatGPT owing to their relatively generic prompts. In such cases, where the generated code ran without errors, participants never discovered the underlying data quality issue—leading them to be unaware about missing values in the newly derived column for zipcode (P1, P3–P4, P7–P14). P4 expected similar behavior for other data smells and quality issues, such as <em id="S4.SS2.SSS2.p5.1.1" class="ltx_emph ltx_font_italic">“typos and capitalization inconsistencies in categorical variable columns,”</em> and expressed the <em id="S4.SS2.SSS2.p5.1.2" class="ltx_emph ltx_font_italic">need</em> to be able to quickly discover and convey such data quality issues to the LLM.</p>
</div>
<div id="S4.SS2.SSS2.p6" class="ltx_para">
<p id="S4.SS2.SSS2.p6.1" class="ltx_p">In tasks T3 (feature selection) and T4 (heatmap plotting), ChatGPT often made assumptions about the data type of columns solely based on the column names. That is, when participants did not share sample rows, or the column data types, ChatGPT used the column names to make a semi-informed guess about the data type. This led to run-time errors in T3 while trying to train a model (P5, P12), and visualizations that were not meaningful for any insight-finding in T4, such as scatter-plots for categorical and boolean variables (seen in Figure <a href="#S4.F4" title="Figure 4 ‣ 4.2.2. ChatGPT opaquely makes assumptions ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>) (P2, P3, P10).</p>
</div>
<figure id="S4.F4" class="ltx_figure"><img src="/html/2310.16164/assets/x3.png" id="S4.F4.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="175" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F4.3.1.1" class="ltx_text" style="font-size:90%;">Figure 4</span>. </span><span id="S4.F4.4.2" class="ltx_text" style="font-size:90%;">ChatGPT suggests a highly skewed boolean variable for a scatter plot. P3 says that, <em id="S4.F4.4.2.1" class="ltx_emph ltx_font_italic">“Should I choose another column? […] This being categorical and skewed isn’t useful for the visualization. So this is where I think [ChatGPT] only looked at the column names, and not the actual values in the data I shared.”</em></span></figcaption>
</figure>
</section>
<section id="S4.SS2.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.2.3. </span>Misaligned expectations</h4>

<div id="S4.SS2.SSS3.p1" class="ltx_para">
<p id="S4.SS2.SSS3.p1.1" class="ltx_p">Responses generated by ChatGPT are highly temperamental to the phrasing of prompts. Participants form a mental model of the nature of responses they would get from ChatGPT, in terms of whether the response will contain code or not, and the libraries, packages and APIs it would use in code generation. However, participants report that the presence or absence of specific words in the prompt can lead to unanticipated responses from ChatGPT.</p>
</div>
<div id="S4.SS2.SSS3.p2" class="ltx_para">
<p id="S4.SS2.SSS3.p2.1" class="ltx_p">For instance, not mentioning “pandas” or “dataframe” in the initial prompt to ChatGPT led to code generations with APIs belonging to standard python libraries, instead of <span id="S4.SS2.SSS3.p2.1.1" class="ltx_text ltx_font_typewriter">pandas</span> APIs (P1, P3, P12). P12 reflected on the issue and realized that they <em id="S4.SS2.SSS3.p2.1.2" class="ltx_emph ltx_font_italic">“didn’t talk about pandas, that’s why it didn’t bring up to_datetime() over here and suggested strptime() instead!”</em></p>
</div>
<div id="S4.SS2.SSS3.p3" class="ltx_para">
<blockquote id="S4.SS2.SSS3.p3.1" class="ltx_quote">
<p id="S4.SS2.SSS3.p3.1.1" class="ltx_p"><span id="S4.SS2.SSS3.p3.1.1.1" class="ltx_text ltx_font_italic">Convert datetime string like “2020-07-31T23:59:51.000” to datetime object in python (P12)</span></p>
</blockquote>
<div id="S4.SS2.SSS3.p3.2" class="ltx_listing ltx_lstlisting ltx_listing">
<div class="ltx_listing_data"><a href="data:text/plain;base64,Cj4+PiBmcm9tIGRhdGV0aW1lIGltcG9ydCBkYXRldGltZQo+Pj4gZGF0ZXRpbWVfc3RyID0gJzIwMjAtMDctMzFUMjM6NTk6NTEuMDAwJwo+Pj4gZGF0ZXRpbWVfb2JqID0gZGF0ZXRpbWUuc3RycHRpbWUoZGF0ZXRpbWVfc3RyLCAn" download="">⬇</a></div>
<div id="lstnumberx11" class="ltx_listingline">
</div>
<div id="lstnumberx12" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx12.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx12.2" class="ltx_text ltx_lst_identifier">from</span><span id="lstnumberx12.3" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx12.4" class="ltx_text ltx_lst_identifier">datetime</span><span id="lstnumberx12.5" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx12.6" class="ltx_text ltx_lst_identifier">import</span><span id="lstnumberx12.7" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx12.8" class="ltx_text ltx_lst_identifier">datetime</span>
</div>
<div id="lstnumberx13" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx13.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx13.2" class="ltx_text ltx_lst_identifier">datetime_str</span><span id="lstnumberx13.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx13.4" class="ltx_text ltx_lst_space"> </span>’2020-07-31<span id="lstnumberx13.5" class="ltx_text ltx_lst_identifier">T23</span>:59:51.000’
</div>
<div id="lstnumberx14" class="ltx_listingline">&gt;&gt;&gt;<span id="lstnumberx14.1" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx14.2" class="ltx_text ltx_lst_identifier">datetime_obj</span><span id="lstnumberx14.3" class="ltx_text ltx_lst_space"> </span>=<span id="lstnumberx14.4" class="ltx_text ltx_lst_space"> </span><span id="lstnumberx14.5" class="ltx_text ltx_lst_identifier">datetime</span>.<span id="lstnumberx14.6" class="ltx_text ltx_lst_identifier">strptime</span>(<span id="lstnumberx14.7" class="ltx_text ltx_lst_identifier">datetime_str</span>,<span id="lstnumberx14.8" class="ltx_text ltx_lst_space"> </span>’
</div>
</div>
</div>
<div id="S4.SS2.SSS3.p4" class="ltx_para">
<p id="S4.SS2.SSS3.p4.1" class="ltx_p">Similar challenges surfaced as participants solved T3 (feature selection). Owing to the subjectivity of T3, participants often mentioned that ChatGPT cannot be completely trusted for suggesting techniques and steps since it lacks domain expertise (P1–P5, P7, P8, P11). Participants observe that ChatGPT’s responses are almost never aligned with their domain knowledge, as it misses necessary data pre-processing steps, or provides technically incorrect responses. In one instance, as P5 went about prompting to train a model that predicts the <span id="S4.SS2.SSS3.p4.1.1" class="ltx_text ltx_font_typewriter">‘RESPONSE_TIME’</span>, ChatGPT generates code to perform training without standardizing column values. P5 emphasized the importance of standardization and suggested careful examination of all generated code: <em id="S4.SS2.SSS3.p4.1.2" class="ltx_emph ltx_font_italic">“One thing it has missed is standardizing data. It is very important for us to scale the data to check for outliers and understand the data distribution.”</em> In other instances, ChatGPT suggested selecting the features: <span id="S4.SS2.SSS3.p4.1.3" class="ltx_text ltx_font_typewriter">‘INCIDENT_DATETIME’</span> and <span id="S4.SS2.SSS3.p4.1.4" class="ltx_text ltx_font_typewriter">‘FIRST_ON_SCENE_DATETIME’</span>, which were used to compute the <span id="S4.SS2.SSS3.p4.1.5" class="ltx_text ltx_font_typewriter">‘RESPONSE_TIME’</span> column. Participants expressed that this is technically incorrect since the target column was arithmetically derived using the features selected by ChatGPT (P2, P3, P5, P13, P14); and that the <span id="S4.SS2.SSS3.p4.1.6" class="ltx_text ltx_font_typewriter">‘FIRST_ON_SCENE_DATETIME’</span> information will not be available in a pragmatic setting—making it impossible to predict the <span id="S4.SS2.SSS3.p4.1.7" class="ltx_text ltx_font_typewriter">‘RESPONSE_TIME’</span> (P8).</p>
</div>
<div id="S4.SS2.SSS3.p5" class="ltx_para">
<p id="S4.SS2.SSS3.p5.1" class="ltx_p">Lastly, participants often found ChatGPT’s responses to be <em id="S4.SS2.SSS3.p5.1.1" class="ltx_emph ltx_font_italic">“excessively verbose and unnecessary”</em>, especially for natural language (NL) code explanations. Participants wanted ChatGPT to omit explanations for APIs that they were already familiar with, and required NL explanations for only those unknown or uncommon APIs and parameters. Several participants held the long NL explanations for code generations as responsible for the prolonged inference time, that is, the time taken by ChatGPT to produce a complete response. P7 said, <em id="S4.SS2.SSS3.p5.1.2" class="ltx_emph ltx_font_italic">“When it was generating the explanation content, I felt like it is time consuming. Just give me the code. I was being impatient.”</em></p>
</div>
<div id="S4.SS2.SSS3.p6" class="ltx_para">
<p id="S4.SS2.SSS3.p6.1" class="ltx_p">We observed that all participants skip reading any NL explanations when the response included a code-snippet, and begin with copying the generated code to adapt it to their notebook. P4 said, <em id="S4.SS2.SSS3.p6.1.1" class="ltx_emph ltx_font_italic">“I guess it takes a bit of time to finish running. It would just be easier for me if it just outputs the code, and then I can ask follow up question on what does <math id="S4.SS2.SSS3.p6.1.1.m1.1" class="ltx_Math" alttext="n=2" display="inline"><semantics id="S4.SS2.SSS3.p6.1.1.m1.1a"><mrow id="S4.SS2.SSS3.p6.1.1.m1.1.1" xref="S4.SS2.SSS3.p6.1.1.m1.1.1.cmml"><mi id="S4.SS2.SSS3.p6.1.1.m1.1.1.2" xref="S4.SS2.SSS3.p6.1.1.m1.1.1.2.cmml">n</mi><mo id="S4.SS2.SSS3.p6.1.1.m1.1.1.1" xref="S4.SS2.SSS3.p6.1.1.m1.1.1.1.cmml">=</mo><mn id="S4.SS2.SSS3.p6.1.1.m1.1.1.3" xref="S4.SS2.SSS3.p6.1.1.m1.1.1.3.cmml">2</mn></mrow><annotation-xml encoding="MathML-Content" id="S4.SS2.SSS3.p6.1.1.m1.1b"><apply id="S4.SS2.SSS3.p6.1.1.m1.1.1.cmml" xref="S4.SS2.SSS3.p6.1.1.m1.1.1"><eq id="S4.SS2.SSS3.p6.1.1.m1.1.1.1.cmml" xref="S4.SS2.SSS3.p6.1.1.m1.1.1.1"></eq><ci id="S4.SS2.SSS3.p6.1.1.m1.1.1.2.cmml" xref="S4.SS2.SSS3.p6.1.1.m1.1.1.2">𝑛</ci><cn type="integer" id="S4.SS2.SSS3.p6.1.1.m1.1.1.3.cmml" xref="S4.SS2.SSS3.p6.1.1.m1.1.1.3">2</cn></apply></annotation-xml><annotation encoding="application/x-tex" id="S4.SS2.SSS3.p6.1.1.m1.1c">n=2</annotation></semantics></math> in the first line of the function do.”</em></p>
</div>
</section>
</section>
<section id="S4.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.3. </span>Obstacles in Leveraging ChatGPT’s Responses</h3>

<div id="S4.SS3.p1" class="ltx_para">
<p id="S4.SS3.p1.1" class="ltx_p">We now present the challenges faced by participant in <span id="S4.SS3.p1.1.1" class="ltx_text ltx_font_italic">using</span> the code generated by ChatGPT. Participants had to edit the generated code in certain ways to successfully adapt it to their notebook contexts. Since the chat instance with ChatGPT was completely detached from the notebook, unless participants explicitly provided variable and attribute names in their prompt, ChatGPT created dummy dataframes and attribute names and included them in the generated code. Thus, the adaptation process frequently involved copying the generated code and replacing the generated variable and attribute names with the actual identifiers. These trivial, yet additional, steps in the adaptation process can be fixed by augmenting the user prompt (i.e., scaffolding the user’s NL prompt with a prefix, and/or suffix to provide relevant context to the model) with the name of the dataframe, attribute names, and any other variables involved. However, we identify several other repeated obstacles across participants, which either require more involved prompt augmentation, post-processing of the generated code, or interface design interventions for easier adaptation. Figure <a href="#S4.F5" title="Figure 5 ‣ 4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a> details the nature of code adaptation and validation activities and their frequencies across tasks, with managing import statements and removing additional code generated for data ingestion being the most frequent adaptations.</p>
</div>
<figure id="S4.F5" class="ltx_figure"><img src="/html/2310.16164/assets/x4.png" id="S4.F5.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="165" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F5.2.1.1" class="ltx_text" style="font-size:90%;">Figure 5</span>. </span><span id="S4.F5.3.2" class="ltx_text" style="font-size:90%;">Frequency of code adaptation and validation activities performed by participants for each task.</span></figcaption>
</figure>
<section id="S4.SS3.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.1. </span>Generation of repeated code</h4>

<div id="S4.SS3.SSS1.p1" class="ltx_para">
<p id="S4.SS3.SSS1.p1.1" class="ltx_p">A common adaptation the participants had to make to the generated code was removing lines that were already present in their notebook. This mostly included (1) repeated library import statements—especially import pandas as pd—which were generated with every code response generated by ChatGPT, (2) repeated data ingestion using the
pandas.read_csv() API,
or by hard-coding a sample of rows as a python dictionary while using the pandas.DataFrame.from_dict() API, and (3) performing exploratory data analysis using pandas.DataFrame.describe() and other APIs. Accidentally executing this code sometimes led the participants’ dataframes to be over-written by the sample values generated by ChatGPT (Figure <a href="#S4.F6" title="Figure 6 ‣ 4.3.1. Generation of repeated code ‣ 4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">6</span></a>), causing them to re-run all notebook cells from the beginning (P1, P9).</p>
</div>
<figure id="S4.F6" class="ltx_figure"><img src="/html/2310.16164/assets/x5.png" id="S4.F6.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="137" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S4.F6.3.1.1" class="ltx_text" style="font-size:90%;">Figure 6</span>. </span><span id="S4.F6.4.2" class="ltx_text" style="font-size:90%;">P1 accidentally executes data ingestion code generated by ChatGPT, causing <span id="S4.F6.4.2.1" class="ltx_text ltx_font_typewriter">df</span> to be over-written by a dummy dataset.</span></figcaption>
</figure>
</section>
<section id="S4.SS3.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.2. </span>Data and notebook management preferences</h4>

<div id="S4.SS3.SSS2.p1" class="ltx_para">
<p id="S4.SS3.SSS2.p1.1" class="ltx_p">We now discuss a few issues related to data and notebook management preferences:</p>
</div>
<div id="S4.SS3.SSS2.p2" class="ltx_para">
<p id="S4.SS3.SSS2.p2.1" class="ltx_p"><span id="S4.SS3.SSS2.p2.1.1" class="ltx_text ltx_font_bold ltx_font_italic">Managing generated code in computational notebooks.</span> We observe that data scientists have implicit preferences for code management, aligned with common practices in the use of computational notebooks, such as having all import statements in a single cell at the beginning of the notebook, splitting functions and data transformations into cells to view intermediate results, interleaving of code with markdown cells to segment the notebook, and so on.</p>
</div>
<div id="S4.SS3.SSS2.p3" class="ltx_para">
<p id="S4.SS3.SSS2.p3.1" class="ltx_p">ChatGPT often generated big chunks of code involving multiple data transformations as a single code-block in its responses. Participants often broke-down such code chunks into meaningful cells (P2, P3, P5, P7, P8, P14). This helped them in isolating lines of code into steps, and execute them sequentially to observe any execution errors. This also enabled them to view the dataframe in between the data-transformation steps, while ensuring that the generated code matches their intent. Further, ChatGPT bundled library import statements with other data transformation/visualization code in the same block. This frequently required participants shifting the new import statements to the cells at the top of their notebook (P3, P5, P7, P12, P13).</p>
</div>
<div id="S4.SS3.SSS2.p4" class="ltx_para">
<p id="S4.SS3.SSS2.p4.1" class="ltx_p">Data scientists have also been studied to commonly author exploratory code; or use the fluent programming pattern, composing multiple transformations into a chain <cite class="ltx_cite ltx_citemacro_citep">(Beth Kery and Myers, <a href="#bib.bib5" title="" class="ltx_ref">2017</a>; Fowler, <a href="#bib.bib13" title="" class="ltx_ref">2005</a>)</cite>. When participants mentioned the phrase “write a function” in their prompts, it led to generation of functions for data transformation tasks, with excessive number of parameters and intermediate steps, contrary to the typical exploratory and fluent patterns. P3 and P7 refactored such code generations to obtain chain of transformations for the target column.</p>
</div>
<div id="S4.SS3.SSS2.p5" class="ltx_para">
<p id="S4.SS3.SSS2.p5.1" class="ltx_p"><span id="S4.SS3.SSS2.p5.1.1" class="ltx_text ltx_font_bold ltx_font_italic">Data transformation and management.</span> Participants often had preferences on how data transformation must be performed, such as—deciding to retain the parent column(s) after having derived newer column(s) (P3, P8, P9, P12), creating temporary dataframes for data transformation tasks (P12, P13), and deciding on appropriate techniques for data imputation (P6). Adaptations had to be made to the generated code to take these transformation preferences into account. For instance, when P6 prompted ChatGPT for T2 (split using delimiter), it generated an additional line of code to impute missing values in zipcode with <span id="S4.SS3.SSS2.p5.1.2" class="ltx_text ltx_font_typewriter">‘‘NULL’’</span>. P6 said, <em id="S4.SS3.SSS2.p5.1.3" class="ltx_emph ltx_font_italic">“No, no, I would not have preferred filling them with ‘NULL’ because right now probably I still need to understand these columns more, so I may just remove null values afterwards,”</em> and subsequently removed the line of code.</p>
</div>
</section>
<section id="S4.SS3.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.3.3. </span>Code validation</h4>

<div id="S4.SS3.SSS3.p1" class="ltx_para">
<p id="S4.SS3.SSS3.p1.1" class="ltx_p">Participants emphasized on the importance of thoroughly validating every operation performed using LLM-generated code. P7 mentioned that validation is even more essential as <em id="S4.SS3.SSS3.p1.1.1" class="ltx_emph ltx_font_italic">“ChatGPT’s confirmatory and affirmative language in responses—like ‘Definitely! Here’s the code you need’—is extremely deceiving because it doesn’t really know about my data.”</em></p>
</div>
<div id="S4.SS3.SSS3.p2" class="ltx_para">
<p id="S4.SS3.SSS3.p2.1" class="ltx_p">Though code validation took the least amount of time when compared to other activities (prompt construction and code adaptation) (Figure <a href="#S4.F2" title="Figure 2 ‣ 4.1.2. Prompting behaviors ‣ 4.1. Participants Interactions with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">2</span></a>), each task required participants to verify the functioning of the generated code in different ways. In some cases, ChatGPT self-generated the code snippet(s) required to validate the data transformation. In other cases, the participants wrote additional code snippets, or manually inspected the data to validate changes. For instance, to ensure that <span id="S4.SS3.SSS3.p2.1.1" class="ltx_text ltx_font_typewriter">‘INCIDENT_DATETIME’</span> has been typecast to <span id="S4.SS3.SSS3.p2.1.2" class="ltx_text ltx_font_typewriter">pd.datetime</span> (T1), participants used df.dtypes and df.columns. To check for newly derived columns after splitting <span id="S4.SS3.SSS3.p2.1.3" class="ltx_text ltx_font_typewriter">‘INCIDENT_DESCRIPTION’</span> (T2), participants printed and manually inspected specific columns. Some participants also checked for missing values after using df.value_counts(). On the other hand, for T4 (heatmap plotting), visual verification of the generated plot would be sufficient to validate the code generated by ChatGPT.</p>
</div>
</section>
</section>
<section id="S4.SS4" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">4.4. </span>Strategies for Prompting and Alternate Resources</h3>

<div id="S4.SS4.p1" class="ltx_para">
<p id="S4.SS4.p1.1" class="ltx_p">We now discuss strategies employed by data scientists to overcome communication and code-adaptation obstacles, along with scenarios where they prefer to leverage other resources (e.g., web-search, documentation, and previously written code).</p>
</div>
<section id="S4.SS4.SSS1" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.1. </span>Techniques for prompt construction</h4>

<div id="S4.SS4.SSS1.p1" class="ltx_para">
<p id="S4.SS4.SSS1.p1.1" class="ltx_p">Alongside data context, participants also brainstormed prompting techniques that are relevant for ChatGPT to solve the problem. We observed frequent use of one-shot and few-shot prompting for T1 (datetime typecast) and T2 (split using delimiter) (P1–P4, P6, P8, P10, P12, P14); chain of thought prompting for T3 (feature selection), to elicit reasons behind <span id="S4.SS4.SSS1.p1.1.1" class="ltx_text ltx_font_italic">why</span>, and <span id="S4.SS4.SSS1.p1.1.2" class="ltx_text ltx_font_italic">on what basis</span> ChatGPT suggests features (P1, P3, P10); and making ChatGPT assume the role of a data scientist to receive expert suggestions, by prefixing each prompt with <em id="S4.SS4.SSS1.p1.1.3" class="ltx_emph ltx_font_italic">“You are a data scientist”</em> (P13).</p>
</div>
<div id="S4.SS4.SSS1.p2" class="ltx_para">
<p id="S4.SS4.SSS1.p2.1" class="ltx_p">To avoid the frequent context-switching between prompt-writing (in the ChatGPT browser-window) and data context gathering (from the notebook/Excel), we observe that participants focused on creating a prompt skeleton while embedding “placeholders” for data context that must be fetched from the notebook (discussed in section <a href="#S4.SS2.SSS1" title="4.2.1. Sharing context is difficult ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.1</span></a>), and pasted into the prompt (P1, P2, P5, P7, P8, P10, P14). For instance, P2 created placeholders (enclosed in square brackets), to replace them afterwards with the required column name and sample value:</p>
</div>
<div id="S4.SS4.SSS1.p3" class="ltx_para">
<blockquote id="S4.SS4.SSS1.p3.1" class="ltx_quote">
<p id="S4.SS4.SSS1.p3.1.1" class="ltx_p"><span id="S4.SS4.SSS1.p3.1.1.1" class="ltx_text ltx_font_italic">I have a dataframe with a column called <span id="S4.SS4.SSS1.p3.1.1.1.1" class="ltx_text ltx_font_bold">[col name]</span>. This column has data separated by ;, here is an example: <span id="S4.SS4.SSS1.p3.1.1.1.2" class="ltx_text ltx_font_bold">[sample value]</span>. Can you split this column? (P2)</span></p>
</blockquote>
</div>
<div id="S4.SS4.SSS1.p4" class="ltx_para">
<p id="S4.SS4.SSS1.p4.1" class="ltx_p">This strategy was commonly used by participants to complete writing the natural-language query in one go, instead of their flow being disrupted by going back and forth between ChatGPT and their notebook to fetch relevant data context. Prompt drafts with placeholders also offered a low cost in switching out pieces of data context while assessing their importance for solving the problem at hand.</p>
</div>
</section>
<section id="S4.SS4.SSS2" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.2. </span>Scaffolding with domain expertise</h4>

<div id="S4.SS4.SSS2.p1" class="ltx_para">
<p id="S4.SS4.SSS2.p1.1" class="ltx_p">Participants often expressed that ChatGPT lacks <em id="S4.SS4.SSS2.p1.1.1" class="ltx_emph ltx_font_italic">“any actual understanding of the data”</em> (discussed in sections <a href="#S4.SS2.SSS2" title="4.2.2. ChatGPT opaquely makes assumptions ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.2</span></a> and <a href="#S4.SS2.SSS3" title="4.2.3. Misaligned expectations ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.3</span></a>). To overcome these obstacles, we observe limited interactions where data scientists successfully guide the LLM with their domain expertise. P10 and P14 narrowed down ChatGPT’s output space by iterative refinement of the data context included in their prompts.</p>
</div>
<div id="S4.SS4.SSS2.p2" class="ltx_para">
<p id="S4.SS4.SSS2.p2.1" class="ltx_p">To obtain meaningful visualizations (T4), P10 refined their prompt to only include names of <span id="S4.SS4.SSS2.p2.1.1" class="ltx_text ltx_font_typewriter">float</span> columns.</p>
</div>
<div id="S4.SS4.SSS2.p3" class="ltx_para">
<blockquote id="S4.SS4.SSS2.p3.1" class="ltx_quote">
<p id="S4.SS4.SSS2.p3.1.1" class="ltx_p"><span id="S4.SS4.SSS2.p3.1.1.1" class="ltx_text ltx_font_italic">I would like to plot out response time vs:</span></p>
<p id="S4.SS4.SSS2.p3.1.2" class="ltx_p"><span id="S4.SS4.SSS2.p3.1.2.1" class="ltx_text ltx_font_italic">[List of ‘float’ attribute names]</span></p>
<p id="S4.SS4.SSS2.p3.1.3" class="ltx_p"><span id="S4.SS4.SSS2.p3.1.3.1" class="ltx_text ltx_font_italic">Can you write python code that does this in seaborn? (P10)</span></p>
</blockquote>
</div>
<div id="S4.SS4.SSS2.p4" class="ltx_para">
<p id="S4.SS4.SSS2.p4.1" class="ltx_p">Similarly, to prevent ChatGPT from suggesting timestamp-based features (T3), P14 provided only names of columns that do not contain the sub-strings ‘TIME’ or ‘ID’ in them.</p>
</div>
<div id="S4.SS4.SSS2.p5" class="ltx_para">
<blockquote id="S4.SS4.SSS2.p5.1" class="ltx_quote">
<p id="S4.SS4.SSS2.p5.1.1" class="ltx_p"><span id="S4.SS4.SSS2.p5.1.1.1" class="ltx_text ltx_font_italic">I want to build a GBM using sklearn. The X variables are [List of attribute names not containing sub-strings ‘TIME’ or ‘ID’], target variable is “RESPONSE_TIME” which is timedelta (P14)</span></p>
</blockquote>
</div>
<div id="S4.SS4.SSS2.p6" class="ltx_para">
<p id="S4.SS4.SSS2.p6.1" class="ltx_p">Surprisingly, we also observe an instance with P5, where ChatGPT asks a “clarifying question” in response to the user provided prompt:</p>
</div>
<div id="S4.SS4.SSS2.p7" class="ltx_para">
<blockquote id="S4.SS4.SSS2.p7.1" class="ltx_quote">
<p id="S4.SS4.SSS2.p7.1.1" class="ltx_p"><span id="S4.SS4.SSS2.p7.1.1.1" class="ltx_text ltx_font_italic">P5: Consider this dataset to be the dataframe which we are going to use for the prediction of the response time: [Top 10 rows with data header]</span></p>
</blockquote>
<blockquote id="S4.SS4.SSS2.p7.2" class="ltx_quote">
<p id="S4.SS4.SSS2.p7.2.1" class="ltx_p"><span id="S4.SS4.SSS2.p7.2.1.1" class="ltx_text ltx_font_italic">ChatGPT: What is response time in this dataset and how is it calculated?</span></p>
</blockquote>
<blockquote id="S4.SS4.SSS2.p7.3" class="ltx_quote">
<p id="S4.SS4.SSS2.p7.3.1" class="ltx_p"><span id="S4.SS4.SSS2.p7.3.1.1" class="ltx_text ltx_font_italic">P5: Response time is calculated as below: FIRST_ON_SCENE_DATETIME – INCIDENT_DATETIME</span></p>
</blockquote>
</div>
<div id="S4.SS4.SSS2.p8" class="ltx_para">
<p id="S4.SS4.SSS2.p8.1" class="ltx_p">This interaction made P5 feel assured that ChatGPT <em id="S4.SS4.SSS2.p8.1.1" class="ltx_emph ltx_font_italic">“is able to understand the prompt”</em>, and <em id="S4.SS4.SSS2.p8.1.2" class="ltx_emph ltx_font_italic">“make sense of the data context”</em>. The clarifying question enabled P5 to provide context that they missed in their initial prompt, and allowed for complete intent expression instead of generating a hallucinated response.</p>
</div>
</section>
<section id="S4.SS4.SSS3" class="ltx_subsubsection">
<h4 class="ltx_title ltx_title_subsubsection">
<span class="ltx_tag ltx_tag_subsubsection">4.4.3. </span>Choosing alternate resources over ChatGPT</h4>

<div id="S4.SS4.SSS3.p1" class="ltx_para">
<p id="S4.SS4.SSS3.p1.1" class="ltx_p">There were several factors that lead to a preference to use alternative resources over ChatGPT:</p>
</div>
<div id="S4.SS4.SSS3.p2" class="ltx_para">
<p id="S4.SS4.SSS3.p2.1" class="ltx_p"><span id="S4.SS4.SSS3.p2.1.1" class="ltx_text ltx_font_bold ltx_font_italic">Data privacy concerns.</span> Participants reflected on the importance of maintaining privacy for sensitive and personally identifiable information (PII). P2 was hesitant about using any confidential business data with ChatGPT and said, <em id="S4.SS4.SSS3.p2.1.2" class="ltx_emph ltx_font_italic">“Do you want me to use normal ChatGPT? And we are not using that for any sensitive data, right, because I don’t want to share any private data with ChatGPT.”</em> Contemporary tools for data might inherently include data context in their prompt, to scaffold user queries. For this reason, participants expressed interest in controlling and reviewing any data context or information being passed on to the LLM. P3 expressed concerns over having their data become part of the model’s training and being presented to a different ChatGPT user as a result to their queries. Participants were comfortable with using enterprise ChatGPT only in accordance with company policy (P2, P7).</p>
</div>
<div id="S4.SS4.SSS3.p3" class="ltx_para">
<p id="S4.SS4.SSS3.p3.1" class="ltx_p"><span id="S4.SS4.SSS3.p3.1.1" class="ltx_text ltx_font_bold ltx_font_italic">Re-using previously written code.</span> Data scientists mentioned that they periodically receive new batches of data, for which they must re-run analyses and visualization reports, and re-train models (P3, P5, P6, P7, P11). In such cases, data scientists are already familiar with the structure of the data, and they reuse previously written code for data pre-processing and cleaning. P11 mentioned, <em id="S4.SS4.SSS3.p3.1.2" class="ltx_emph ltx_font_italic">“We pretty much just copy and paste canned code for every project. A lot of variations can be created on the charts, but 80% of it would be repeating.”</em> The need to author new code for pre-processing or analytics in such scenarios is rare.</p>
</div>
<div id="S4.SS4.SSS3.p4" class="ltx_para">
<p id="S4.SS4.SSS3.p4.1" class="ltx_p"><span id="S4.SS4.SSS3.p4.1.1" class="ltx_text ltx_font_bold ltx_font_italic">Using web search and documentation.</span> Participants described that using LLMs can normally help them save time over making a web search, such as when <em id="S4.SS4.SSS3.p4.1.2" class="ltx_emph ltx_font_italic">“asking for generic directions and steps, saving me the time to go through 4–5 blog posts on the web,”</em> and <em id="S4.SS4.SSS3.p4.1.3" class="ltx_emph ltx_font_italic">“asking for code in my specific context, which needn’t be the exact issue discussed on Stack Overflow”</em> (P4, P11). In these cases, they find the LLM’s responses to be highly contextualized to their data and previously applied steps, and <em id="S4.SS4.SSS3.p4.1.4" class="ltx_emph ltx_font_italic">“less noisy”</em> compared to web-pages. On the other hand, P7 found it easy to make a web search, requiring only <em id="S4.SS4.SSS3.p4.1.5" class="ltx_emph ltx_font_italic">“8–10 key words,”</em> whereas they must write <em id="S4.SS4.SSS3.p4.1.6" class="ltx_emph ltx_font_italic">“longer queries and be more specific with ChatGPT just because it allows me to do that”</em>. P11 also mentioned that the use of LLMs will <em id="S4.SS4.SSS3.p4.1.7" class="ltx_emph ltx_font_italic">“restrict me from learning new techniques or discovering a breadth of alternate approaches. Forums like Stack Overflow support these needs better.”</em></p>
</div>
</section>
</section>
</section>
<section id="S5" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">5. </span>Survey Results</h2>

<div id="S5.p1" class="ltx_para">
<p id="S5.p1.1" class="ltx_p">Our qualitative insights from the task-based study (Section <a href="#S4" title="4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4</span></a>) identify numerous obstacles that data scientists face in conversing with ChatGPT, and subsequently, in acting on the generated responses. The survey results help us understand how well do these obstacles generalize to a diverse set of data science tasks, and to a broader community of data scientists. The survey respondents report experience with using LLMs for varied data science tasks spanning across project timelines, including synthetic data generation, data wrangling, pre-processing, data labelling, exploratory data analysis, insight finding and summarization, hypothesis generation, training various models, outlier detection, and generating plots.</p>
</div>
<div id="S5.p2" class="ltx_para">
<p id="S5.p2.1" class="ltx_p">Figure <a href="#S5.F7" title="Figure 7 ‣ 5. Survey Results ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">7</span></a> shows the distribution of responses for each ‘Agree’/‘Disagree’ question. Results indicate that data scientists find sharing of relevant data context to be important (86%), as well as tedious (59%), echoing findings from Section <a href="#S4.SS2.SSS1" title="4.2.1. Sharing context is difficult ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.1</span></a>. 62% data scientists agreed that prompting ChatGPT for columns with mixed formats is challenging, reflecting the issues discussed in Section <a href="#S4.SS2.SSS2" title="4.2.2. ChatGPT opaquely makes assumptions ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.2</span></a>. These challenges lead to communication breakdowns between data scientists and ChatGPT—causing them to make repeated prompt refinements (92%) (Section <a href="#S4.SS1.SSS2" title="4.1.2. Prompting behaviors ‣ 4.1. Participants Interactions with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1.2</span></a>). Respondents also indicate that LLMs do not have sufficient domain expertise in data science (60%) (Section <a href="#S4.SS2.SSS3" title="4.2.3. Misaligned expectations ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.3</span></a>), and that generated code requires several modifications before it can be used in their notebooks (87%) (Section <a href="#S4.SS3" title="4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>). Lastly, code validation respondents have a split opinion on whether it is easy to verify code generated by ChatGPT (48%) (Section <a href="#S4.SS3.SSS3" title="4.3.3. Code validation ‣ 4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3.3</span></a>).</p>
</div>
<figure id="S5.F7" class="ltx_figure"><img src="/html/2310.16164/assets/x6.png" id="S5.F7.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="461" height="145" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S5.F7.2.1.1" class="ltx_text" style="font-size:90%;">Figure 7</span>. </span><span id="S5.F7.3.2" class="ltx_text" style="font-size:90%;">Distribution of survey responses for ‘Agree’/‘Disagree’ questions. Response Counts: D (Disagree); NR (No Response); A (Agree). </span></figcaption>
</figure>
<div id="S5.p3" class="ltx_para">
<p id="S5.p3.1" class="ltx_p">We manually inspected responses to look for any other challenges encountered by data scientists in conversing with LLMs for data science tasks. Respondents re-emphasized difficulties in <em id="S5.p3.1.1" class="ltx_emph ltx_font_italic">“forming well scoped prompts”</em> and deciding <span id="S5.p3.1.2" class="ltx_text ltx_font_italic">what</span> context to include. They reflected on the need for repeated prompt refinements due to ChatGPT’s lack of domain expertise. A few responses brought up the <em id="S5.p3.1.3" class="ltx_emph ltx_font_italic">“instability”</em> in ChatGPT’s responses, wherein it generates drastically different answers despite sending the same prompt. We did not encounter any new obstacles in response to this question.</p>
</div>
</section>
<section id="S6" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">6. </span>Design Recommendations</h2>

<div id="S6.p1" class="ltx_para">
<p id="S6.p1.1" class="ltx_p">In this section, we discuss the implications of our findings and identify the ways in which they could be adapted to various data-science environments and to broader contexts.</p>
</div>
<section id="S6.SS1" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.1. </span>Recommendation I—Provide preemptive and fluid context when interacting with AI assistants</h3>

<div id="S6.SS1.p1" class="ltx_para">
<p id="S6.SS1.p1.1" class="ltx_p">From our studies, we found that participants spent a significant amount of time constructing prompts (Section  <a href="#S4.SS1.SSS2" title="4.1.2. Prompting behaviors ‣ 4.1. Participants Interactions with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.1.2</span></a>) and gathering and expressing their context (Section <a href="#S4.SS2.SSS1" title="4.2.1. Sharing context is difficult ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.1</span></a>) to ChatGPT. Nearly 45% of the prompts included portions of data that they either manually entered, or wrote additional code snippets to obtain; however, participants did not enjoy being <em id="S6.SS1.p1.1.1" class="ltx_emph ltx_font_italic">“slowed-down”</em> in fetching this data context for every interaction. Thus, data scientists may need help in providing context about their environments, and we should support mechanisms to either preemptively provide implicit context, or provide fluid interactions that allow data scientists to easily refer to fragments of their work in conversations with AI assistants.</p>
</div>
<div id="S6.SS1.p2" class="ltx_para">
<p id="S6.SS1.p2.1" class="ltx_p">In data visualization, classic techniques such as <em id="S6.SS1.p2.1.1" class="ltx_emph ltx_font_italic">brushing</em> <cite class="ltx_cite ltx_citemacro_citep">(Becker and Cleveland, <a href="#bib.bib4" title="" class="ltx_ref">1987</a>)</cite>, enable a user to select and highlight specific data points, allowing the user to use those data points as a filter, or as a point of interest for further analysis. Similarly, providing mechanisms for supporting <em id="S6.SS1.p2.1.2" class="ltx_emph ltx_font_italic">data brushing</em> in data-science environments would allow users share their context with agents more directly. For example, in T2, participants had to split text into three columns. However, some rows had inconsistent formatting making the operation suggested by ChatGPT error-prone. With data brushing (Figure <a href="#S6.F8" title="Figure 8 ‣ 6.1. Recommendation I—Provide preemptive and fluid context when interacting with AI assistants ‣ 6. Design Recommendations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">8</span></a>), participants could mark an example cell with a problematic entry and ask an AI assistant to help split the rows, while dealing with outliers such as the marked cell. Similarly, allowing inline code snippets that are expanded, or referring to entities within the data (<span id="S6.SS1.p2.1.3" class="ltx_text ltx_font_typewriter">@INCIDENT_RESPONSE)</span>, allow less tedious and more fluid expressions of context when prompt writing.</p>
</div>
<figure id="S6.F8" class="ltx_figure"><img src="/html/2310.16164/assets/context-brushes.png" id="S6.F8.g1" class="ltx_graphics ltx_centering ltx_img_landscape" width="598" height="317" alt="Refer to caption">
<figcaption class="ltx_caption ltx_centering"><span class="ltx_tag ltx_tag_figure"><span id="S6.F8.2.1.1" class="ltx_text" style="font-size:90%;">Figure 8</span>. </span><span id="S6.F8.3.2" class="ltx_text" style="font-size:90%;">Design concept—Using data brushing to mark examples (selected in green), and inconsistent entries (selected in red).</span></figcaption>
</figure>
<div id="S6.SS1.p3" class="ltx_para">
<p id="S6.SS1.p3.1" class="ltx_p">Despite these aids, some context, especially providing a summary and schema of the data would be tiresome. Furthermore, users had no clear understanding of what information should be provided, and even for researchers, this question is still being explored <cite class="ltx_cite ltx_citemacro_citep">(Chen, <a href="#bib.bib9" title="" class="ltx_ref">2023</a>)</cite>. However, recall Grice’s maxim of relevance, which states that a successful conversation requires not too little or too much information be provided <cite class="ltx_cite ltx_citemacro_citep">(Grice, <a href="#bib.bib15" title="" class="ltx_ref">1991</a>)</cite>. A simple intermediate measure would be to provide a preemptive summaries of data characteristics. This would help reduce the chances that ChatGPT makes assumptions about data format, which may lead to incorrect code or longer conversations (Section <a href="#S4.SS2.SSS2" title="4.2.2. ChatGPT opaquely makes assumptions ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.2</span></a>).</p>
</div>
</section>
<section id="S6.SS2" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.2. </span>Recommendation II—Provide inquisitive feedback loops and validation-aware operations</h3>

<div id="S6.SS2.p1" class="ltx_para">
<p id="S6.SS2.p1.1" class="ltx_p">For transactional tasks, such as T1 (datetime typecast), ChatGPT often provided satisfactory responses. Unfortunately, open-ended tasks, such as T3 (feature selection), required a more involved conversation with ChatGPT.
Initial responses from ChatGPT were often “<em id="S6.SS2.p1.1.1" class="ltx_emph ltx_font_italic">excessively long</em>” and were not particularly helpful for the task, likely because it violates the fundamental HCI principle of progressive disclosure <cite class="ltx_cite ltx_citemacro_citep">(Nielsen, <a href="#bib.bib33" title="" class="ltx_ref">2006</a>)</cite> and overwhelms the user with details.
As a result, participants were uncertain how much value would be gained by continuing the conversation or what further information would help (Section <a href="#S4.SS2.SSS3" title="4.2.3. Misaligned expectations ‣ 4.2. Obstacles when Communicating with ChatGPT ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.2.3</span></a>). In other cases, participants had to make multiple clarifications and carry out editing operations across several iterations in their data-science environments. This task fragmentation often lead to errors, for example, because ChatGPT always included code to ingest the dataframe—regardless of the stage of conversation—some participants accidentally executed it, leading to their data being over-written (Section <a href="#S4.SS3.SSS1" title="4.3.1. Generation of repeated code ‣ 4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3.1</span></a>).</p>
</div>
<div id="S6.SS2.p2" class="ltx_para">
<p id="S6.SS2.p2.1" class="ltx_p">One mechanism to help participants have more productive conversations with AI assistants is through <em id="S6.SS2.p2.1.1" class="ltx_emph ltx_font_italic">inquisitive feedback loops</em>, where the system guides the user in expressing what they need by proactively asking the user a series of questions, rather than waiting for the user to know when they need help and how to get it <cite class="ltx_cite ltx_citemacro_citep">(Henley et al<span class="ltx_text">.</span>, <a href="#bib.bib18" title="" class="ltx_ref">2021</a>)</cite>.
Moreover, participants in our study could have been aided just by ChatGPT asking clarifying questions to resolve ambiguities instead of making assumptions.
For example, one participant (P9) was able to get more useful advice from ChatGPT, after suggesting that they could reframe the problem of feature selection: “<em id="S6.SS2.p2.1.2" class="ltx_emph ltx_font_italic">I think I am more interested in finding the feature importance so I’ll prompt it for something like a random forest</em>”. In this case, a user could be prompted with questions that helped the user think through which high-level strategy they wanted to use before giving all the details.</p>
</div>
<div id="S6.SS2.p3" class="ltx_para">
<p id="S6.SS2.p3.1" class="ltx_p">Tools for supporting <em id="S6.SS2.p3.1.1" class="ltx_emph ltx_font_italic">refactoring</em>, behavioral-preserving transformations, can be one approach for assisting data scientists who must make repeated edits of similar nature, or adapt generated code snippets to their local notebook contexts (Section <a href="#S4.SS3" title="4.3. Obstacles in Leveraging ChatGPT’s Responses ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.3</span></a>). For instance, <span id="S6.SS2.p3.1.2" class="ltx_text ltx_font_smallcaps">GhostFactor</span> <cite class="ltx_cite ltx_citemacro_citep">(Ge and Murphy-Hill, <a href="#bib.bib14" title="" class="ltx_ref">2014</a>)</cite> is a light-weight program analysis tool with refactoring detection algorithms that can identify incomplete changes to code. Simple approaches can be used for enabling automated post-processing and human-in-the-loop refactoring of generated code snippets based on observed nature of edits—deletion of repeated code,
notebook and data management preferences, and validation.</p>
</div>
</section>
<section id="S6.SS3" class="ltx_subsection">
<h3 class="ltx_title ltx_title_subsection">
<span class="ltx_tag ltx_tag_subsection">6.3. </span>Recommendation III—Provide transparency about shared context and domain expertise slots</h3>

<div id="S6.SS3.p1" class="ltx_para">
<p id="S6.SS3.p1.1" class="ltx_p"><cite class="ltx_cite ltx_citemacro_citet">Barke et al<span class="ltx_text">.</span> (<a href="#bib.bib2" title="" class="ltx_ref">2023</a>)</cite> posit that lack of transparency about the shared context, and lack of control in refining the context selections can leave users in a confused state. Our participants echo a similar consideration, wherein they expressed the urge to <em id="S6.SS3.p1.1.1" class="ltx_emph ltx_font_italic">“get to the first response as fast as possible, but have the ability to look at what context was provided and edit it if I need to refine my prompt”</em> (P3, P4, P7, P12). Furthermore, participants also wanted to be able to provide more context that reflected their personal expertise and experience (Section <a href="#S4.SS4.SSS2" title="4.4.2. Scaffolding with domain expertise ‣ 4.4. Strategies for Prompting and Alternate Resources ‣ 4. Observations ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">4.4.2</span></a>).</p>
</div>
<div id="S6.SS3.p2" class="ltx_para">
<p id="S6.SS3.p2.1" class="ltx_p">Even with automated context management or prompt augmentation, it is important to consider how that context is ultimately shared. For example,
by having sufficient transparency into the shared context can also enable data scientists to ensure that sensitive data does not leak into the prompt.
Data-science environments interacting with AI should consider mechanisms for
ensuring a two-way knowledge transfer between data scientists and AI. For example, an AI assistant chat interface could integrate a <em id="S6.SS3.p2.1.1" class="ltx_emph ltx_font_italic">context panel</em>, a dedicated, editable grid that mirrors the AI assistant’s and user’s assumptions about data and tasks.</p>
</div>
<div id="S6.SS3.p3" class="ltx_para">
<p id="S6.SS3.p3.1" class="ltx_p">Finally, users should be able to inspect and modify the context to provide their domain expertise or reflect their analysis style (e.g., always perform data scaling and impute missing values prior to training a model). One step in this direction is the recent addition to ChatGPT allowing for custom instructions<span id="footnote6" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">6</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">6</sup><span class="ltx_tag ltx_tag_note">6</span><a target="_blank" href="https://openai.com/blog/custom-instructions-for-chatgpt" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://openai.com/blog/custom-instructions-for-chatgpt</a></span></span></span>.</p>
</div>
</section>
</section>
<section id="S7" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">7. </span>Threats to validity</h2>

<div id="S7.p1" class="ltx_para">
<p id="S7.p1.1" class="ltx_p">All study participants had prior experience with using ChatGPT. However, most participants (9 <span id="S7.p1.1.1" class="ltx_text ltx_font_italic">of</span> 14) had not used LLMs for solving data-science tasks. If they had been exposed to tasks of similar nature prior to the study session, their interactions with ChatGPT could have been different. We took this into consideration to ensure the quality of responses in the confirmatory survey, and asked all respondents to share at least one scenario where they used an LLM-powered tool for a data-science task. This enabled us to filter out respondents who did not have any prior experiences to provide informed responses.</p>
</div>
<div id="S7.p2" class="ltx_para">
<p id="S7.p2.1" class="ltx_p">We selected a diverse set of data-science tasks representative of data pre-processing, cleaning, feature selection, and plotting. However, the study duration limited the breadth of tasks that participants could have performed, such as data discovery, capture, and machine learning <cite class="ltx_cite ltx_citemacro_citep">(Muller et al<span class="ltx_text">.</span>, <a href="#bib.bib32" title="" class="ltx_ref">2019</a>)</cite>. Nevertheless, our survey responses (Section <a href="#S5" title="5. Survey Results ‣ Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities" class="ltx_ref"><span class="ltx_text ltx_ref_tag">5</span></a>), from a broader community of data scientists with diverse experiences, confirm the generalizability of our findings. Future work can involve empirical studies that measure the performance of LLMs on assorted data-science tasks with datasets that have not been “seen” by the LLMs during pre-training. We presented the data-science tasks to all participants through verbal descriptions of the objectives. This may have inevitably led to priming, wherein the participants’ formulation of prompts could have been influenced by our description of the task.</p>
</div>
<div id="S7.p3" class="ltx_para">
<p id="S7.p3.1" class="ltx_p">Browser-based version of ChatGPT version was updated as we conducted our task-based studies. P1–P9 used the March 14, 2023 version of ChatGPT, and P10–P14 used the March 23, 2023 version. This might have had a slight impact on study findings, though we did not observe any noticeable shifts in the nature of responses generated by ChatGPT for the study tasks. Lastly, we encouraged participants to think aloud as they solved study tasks, which could have added to the time taken in performing different activities. Our video annotations take this into consideration to only include timestamps where participants are actively working with the data, or with ChatGPT.</p>
</div>
</section>
<section id="S8" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">8. </span>Related Work</h2>

<div id="S8.p1" class="ltx_para">
<p id="S8.p1.1" class="ltx_p">AI-assisted tools for data cleaning suggestions—such as Trifacta<span id="footnote7" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">7</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">7</sup><span class="ltx_tag ltx_tag_note">7</span><a target="_blank" href="https://docs.trifacta.com/display/SS/Overview+of+Predictive+Transformation" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://docs.trifacta.com/display/SS/Overview+of+Predictive+Transformation</a></span></span></span> <cite class="ltx_cite ltx_citemacro_citep">(Guo et al<span class="ltx_text">.</span>, <a href="#bib.bib17" title="" class="ltx_ref">2011b</a>; Kandel et al<span class="ltx_text">.</span>, <a href="#bib.bib23" title="" class="ltx_ref">2011b</a>)</cite>, Data Wrangler<span id="footnote8" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">8</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">8</sup><span class="ltx_tag ltx_tag_note">8</span><a target="_blank" href="https://devblogs.microsoft.com/python/data-wrangler-release/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://devblogs.microsoft.com/python/data-wrangler-release/</a></span></span></span>, CoWrangler <cite class="ltx_cite ltx_citemacro_citep">(Chopra et al<span class="ltx_text">.</span>, <a href="#bib.bib10" title="" class="ltx_ref">2023</a>)</cite>, AWS Glue DataBrew<span id="footnote9" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">9</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">9</sup><span class="ltx_tag ltx_tag_note">9</span><a target="_blank" href="https://docs.trifacta.com/display/SS/Overview+of+Predictive+Transformation" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://docs.trifacta.com/display/SS/Overview+of+Predictive+Transformation</a></span></span></span>, Salesforce Einstein Discovery<span id="footnote10" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">10</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">10</sup><span class="ltx_tag ltx_tag_note">10</span><a target="_blank" href="https://help.salesforce.com/s/articleView?id=sf.bi_edd_prep_terminology.htm" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://help.salesforce.com/s/articleView?id=sf.bi_edd_prep_terminology.htm</a></span></span></span>, AutoPandas <cite class="ltx_cite ltx_citemacro_citep">(Bavishi et al<span class="ltx_text">.</span>, <a href="#bib.bib3" title="" class="ltx_ref">2019</a>)</cite>, Auto-Suggest <cite class="ltx_cite ltx_citemacro_citep">(Yan and He, <a href="#bib.bib46" title="" class="ltx_ref">2020</a>)</cite>—have existed, but LLMs bring forth new opportunities for enhancing these tools further.
More recently, several commercial and open-source tools have emerged—DataChat AI<span id="footnote11" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">11</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">11</sup><span class="ltx_tag ltx_tag_note">11</span><a target="_blank" href="https://datachat.ai/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://datachat.ai/</a></span></span></span>,
Anaconda Assistant<span id="footnote12" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">12</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">12</sup><span class="ltx_tag ltx_tag_note">12</span><a target="_blank" href="https://www.anaconda.com/blog/anaconda-assistant-launches-to-bring-instant-data-analysis-code-generation-and-insights-to-users" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.anaconda.com/blog/anaconda-assistant-launches-to-bring-instant-data-analysis-code-generation-and-insights-to-users</a></span></span></span>,
Databricks Assistant<span id="footnote13" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">13</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">13</sup><span class="ltx_tag ltx_tag_note">13</span><a target="_blank" href="https://www.databricks.com/blog/introducing-databricks-assistant" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.databricks.com/blog/introducing-databricks-assistant</a></span></span></span>,
and Jupyter AI<span id="footnote14" class="ltx_note ltx_role_footnote"><sup class="ltx_note_mark">14</sup><span class="ltx_note_outer"><span class="ltx_note_content"><sup class="ltx_note_mark">14</sup><span class="ltx_tag ltx_tag_note">14</span><a target="_blank" href="https://jupyter-ai.readthedocs.io/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://jupyter-ai.readthedocs.io/</a></span></span></span>—which enable data scientists to access AI-powered chat assistants within their notebook (such as Anaconda, Databricks, and Jupyter notebooks), and other alternate data-science environments (e.g., DataChat AI, SQL and file editors for Databricks, etc.). The semantic abilities of LLMs, coupled with a chat interface, provides users with the agency to converse about their data, ask follow-up questions, and receive highly contextualized responses.</p>
</div>
<div id="S8.p2" class="ltx_para">
<p id="S8.p2.1" class="ltx_p">McNutt et al. <cite class="ltx_cite ltx_citemacro_citep">(Mcnutt et al<span class="ltx_text">.</span>, <a href="#bib.bib31" title="" class="ltx_ref">2023</a>)</cite> studied, through an interview study, the design space of code assistants in computational notebooks, but their focus is on the design space as they do not focus on data-specific issues that data scientists face while communicating their needs to the AI assistants (e.g., communicating the data context or domain knowledge). A number of recent work conducted studies to understand the use of LLMs by software engineers and programmers <cite class="ltx_cite ltx_citemacro_citep">(Liang et al<span class="ltx_text">.</span>, <a href="#bib.bib29" title="" class="ltx_ref">2023</a>; Barke et al<span class="ltx_text">.</span>, <a href="#bib.bib2" title="" class="ltx_ref">2023</a>; Sarkar et al<span class="ltx_text">.</span>, <a href="#bib.bib38" title="" class="ltx_ref">2022</a>; Jayagopal et al<span class="ltx_text">.</span>, <a href="#bib.bib20" title="" class="ltx_ref">2022</a>; Vaithilingam et al<span class="ltx_text">.</span>, <a href="#bib.bib43" title="" class="ltx_ref">2023</a>, <a href="#bib.bib44" title="" class="ltx_ref">2022</a>; Kaddour et al<span class="ltx_text">.</span>, <a href="#bib.bib21" title="" class="ltx_ref">2023</a>)</cite>. However, these mostly focus on the general understanding of AI-powered code assistants and do not particularly focus on data-science tasks, where communicating the data at hand to the assistant is a big challenge.</p>
</div>
<div id="S8.p3" class="ltx_para">
<p id="S8.p3.1" class="ltx_p">There has been a recent line of work that try to make use of LLMs to automate data-science tasks and evaluate performance of LLMs on data-science tasks <cite class="ltx_cite ltx_citemacro_citep">(Noever and McKee, <a href="#bib.bib34" title="" class="ltx_ref">2023</a>; Hollmann et al<span class="ltx_text">.</span>, <a href="#bib.bib19" title="" class="ltx_ref">2023</a>; Yin et al<span class="ltx_text">.</span>, <a href="#bib.bib47" title="" class="ltx_ref">2022</a>)</cite>. This is a promising, but orthogonal, area of research to ours as we focus on human-in-the-loop data science where AI-assistants and human data scientists work as a team.
Similarly, there are work on designing and evaluating tools for end users <cite class="ltx_cite ltx_citemacro_citep">(Liu et al<span class="ltx_text">.</span>, <a href="#bib.bib30" title="" class="ltx_ref">2023</a>; Srinivasa Ragavan et al<span class="ltx_text">.</span>, <a href="#bib.bib41" title="" class="ltx_ref">2022</a>; Ferdowsi et al<span class="ltx_text">.</span>, <a href="#bib.bib11" title="" class="ltx_ref">2023</a>)</cite>, which is complementary to our work.</p>
</div>
<div id="S8.p4" class="ltx_para">
<p id="S8.p4.1" class="ltx_p">Most relevant to our work is the study by Liu et al. <cite class="ltx_cite ltx_citemacro_citep">(Liu et al<span class="ltx_text">.</span>, <a href="#bib.bib30" title="" class="ltx_ref">2023</a>)</cite> on how users of a spreadsheet software articulate their question to an AI-powered tool and how a system they designed can better support this process.
Our work is differentiated by (1) examining the fundamental interactions with a general AI tool like ChatGPT rather than a domain-specific AI tool embedded in a system, (2) we studied professional data scientists rather than non-programmers, and (3) they were specifically investigating the effect of co-editing a prompt by splitting it into natural language steps.</p>
</div>
</section>
<section id="S9" class="ltx_section">
<h2 class="ltx_title ltx_title_section">
<span class="ltx_tag ltx_tag_section">9. </span>Conclusion and Future Work</h2>

<div id="S9.p1" class="ltx_para">
<p id="S9.p1.1" class="ltx_p">The recent rise of AI-powered chat assistants gives hope of greatly increasing the productivity of data scientists as they are engaged in tedious data-wrangling tasks.
To understand the fundamental obstacles, needs, and design opportunities of data scientists using AI assistants, we conducted mixed-method studies involving a task-based study, interviews, and a survey with professional data scientists.
We found that participants faced two key sets of barriers: when communicating with ChatGPT and then when adapting ChatGPT’s response to aid their specific situation.
For example, participants spent considerable time collecting and formatting information to provide to ChatGPT, such as relevant regions of their dataset or descriptive statistics.
Furthermore, ChatGPT often made assumptions unbeknownst to the participant, thus leading to bugs in the code and additional steps to remedy.
When participants did get a useful response from ChatGPT, it required considerable effort to adapt the code to work with the rest of their code and to follow their style.
We provide key design recommendations for tools based on our study, such as the need to provide inquisitive feedback loops and validate-aware operations.
As a next step, we intend to operationalize these design guidelines into a working tool and assess its efficacy in alleviating the workflow challenges that data scientists encountered in our studies.
In conclusion, our work aims to function as a significant catalyst in democratizing data science via AI-powered chat assistants.</p>
</div>
</section>
<section id="bib" class="ltx_bibliography">
<h2 class="ltx_title ltx_title_bibliography">References</h2>

<ul class="ltx_biblist">
<li id="bib.bib1" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">(1)</span>
<span class="ltx_bibblock">        




</span>
</li>
<li id="bib.bib2" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Barke et al<span id="bib.bib2.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023.

</span>
<span class="ltx_bibblock">Grounded Copilot: How Programmers Interact with Code-Generating Models.

</span>
<span class="ltx_bibblock"><em id="bib.bib2.3.1" class="ltx_emph ltx_font_italic">Proc. ACM Program. Lang.</em> 7, OOPSLA1, Article 78 (apr 2023), 27 pages.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1145/3586030" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3586030</a>

</span>
</li>
<li id="bib.bib3" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Bavishi et al<span id="bib.bib3.2.2.1" class="ltx_text">.</span> (2019)</span>
<span class="ltx_bibblock">
Rohan Bavishi, Caroline Lemieux, Roy Fox, Koushik Sen, and Ion Stoica. 2019.

</span>
<span class="ltx_bibblock">AutoPandas: Neural-Backed Generators for Program Synthesis.

</span>
<span class="ltx_bibblock"><em id="bib.bib3.3.1" class="ltx_emph ltx_font_italic">Proc. ACM Program. Lang.</em> 3, OOPSLA, Article 168 (oct 2019), 27 pages.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1145/3360594" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3360594</a>

</span>
</li>
<li id="bib.bib4" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Becker and Cleveland (1987)</span>
<span class="ltx_bibblock">
Richard A. Becker and William S. Cleveland. 1987.

</span>
<span class="ltx_bibblock">Brushing Scatterplots.

</span>
<span class="ltx_bibblock"><em id="bib.bib4.1.1" class="ltx_emph ltx_font_italic">Technometrics</em> 29, 2 (1987), 127–142.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1080/00401706.1987.10488204" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1080/00401706.1987.10488204</a>
arXiv:https://www.tandfonline.com/doi/pdf/10.1080/00401706.1987.10488204

</span>
</li>
<li id="bib.bib5" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Beth Kery and Myers (2017)</span>
<span class="ltx_bibblock">
Mary Beth Kery and Brad A. Myers. 2017.

</span>
<span class="ltx_bibblock">Exploring exploratory programming. In <em id="bib.bib5.1.1" class="ltx_emph ltx_font_italic">2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)</em>. IEEE, New York, NY, USA, 25–29.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/VLHCC.2017.8103446" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/VLHCC.2017.8103446</a>

</span>
</li>
<li id="bib.bib6" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Cabrera et al<span id="bib.bib6.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Ángel Alexander Cabrera, Marco Tulio Ribeiro, Bongshin Lee, Robert Deline, Adam Perer, and Steven M. Drucker. 2023.

</span>
<span class="ltx_bibblock">What Did My AI Learn? How Data Scientists Make Sense of Model Behavior.

</span>
<span class="ltx_bibblock"><em id="bib.bib6.3.1" class="ltx_emph ltx_font_italic">ACM Trans. Comput.-Hum. Interact.</em> 30, 1, Article 1 (mar 2023), 27 pages.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3542921" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3542921</a>

</span>
</li>
<li id="bib.bib7" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chattopadhyay et al<span id="bib.bib7.2.2.1" class="ltx_text">.</span> (2020)</span>
<span class="ltx_bibblock">
Souti Chattopadhyay, Ishita Prasad, Austin Z. Henley, Anita Sarma, and Titus Barik. 2020.

</span>
<span class="ltx_bibblock">What’s Wrong with Computational Notebooks? Pain Points, Needs, and Design Opportunities. In <em id="bib.bib7.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems</em> (Honolulu, HI, USA) <em id="bib.bib7.4.2" class="ltx_emph ltx_font_italic">(CHI ’20)</em>. Association for Computing Machinery, New York, NY, USA, 1–12.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3313831.3376729" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3313831.3376729</a>

</span>
</li>
<li id="bib.bib8" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen et al<span id="bib.bib8.2.2.1" class="ltx_text">.</span> (2021)</span>
<span class="ltx_bibblock">
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis,
Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021.

</span>
<span class="ltx_bibblock">Evaluating Large Language Models Trained on Code.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2107.03374 [cs.LG]

</span>
</li>
<li id="bib.bib9" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chen (2023)</span>
<span class="ltx_bibblock">
Wenhu Chen. 2023.

</span>
<span class="ltx_bibblock">Large Language Models are few(1)-shot Table Reasoners.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2210.06710 [cs.CL]

</span>
</li>
<li id="bib.bib10" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Chopra et al<span id="bib.bib10.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Bhavya Chopra, Anna Fariha, Sumit Gulwani, Austin Z. Henley, Daniel Perelman, Mohammad Raza, Sherry Shi, Danny Simmons, and Ashish Tiwari. 2023.

</span>
<span class="ltx_bibblock">CoWrangler: Recommender System for Data-Wrangling Scripts. In <em id="bib.bib10.3.1" class="ltx_emph ltx_font_italic">Companion of the 2023 International Conference on Management of Data</em> (Seattle, WA, USA) <em id="bib.bib10.4.2" class="ltx_emph ltx_font_italic">(SIGMOD ’23)</em>. Association for Computing Machinery, New York, NY, USA, 147–150.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3555041.3589722" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3555041.3589722</a>

</span>
</li>
<li id="bib.bib11" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ferdowsi et al<span id="bib.bib11.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Kasra Ferdowsi, Jack Williams, Ian Drosos, Andy Gordon, Carina Negreanu, Nadia Polikarpova, Advait Sarkar, and Ben Zorn. 2023.

</span>
<span class="ltx_bibblock">ColDeco: An End User Spreadsheet Inspection Tool for AI-Generated Code. In <em id="bib.bib11.3.1" class="ltx_emph ltx_font_italic">IEEE Symposium on Visual Languages and Human-Centric Computing</em>. IEEE, IEEE, New York, NY, USA.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://www.microsoft.com/en-us/research/publication/coldeco-an-end-user-spreadsheet-inspection-tool-for-ai-generated-code/" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.microsoft.com/en-us/research/publication/coldeco-an-end-user-spreadsheet-inspection-tool-for-ai-generated-code/</a>

</span>
</li>
<li id="bib.bib12" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fisher et al<span id="bib.bib12.2.2.1" class="ltx_text">.</span> (2012)</span>
<span class="ltx_bibblock">
Danyel Fisher, Rob DeLine, Mary Czerwinski, and Steven Drucker. 2012.

</span>
<span class="ltx_bibblock">Interactions with Big Data Analytics.

</span>
<span class="ltx_bibblock"><em id="bib.bib12.3.1" class="ltx_emph ltx_font_italic">Interactions</em> 19, 3 (may 2012), 50–59.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/2168931.2168943" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/2168931.2168943</a>

</span>
</li>
<li id="bib.bib13" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Fowler (2005)</span>
<span class="ltx_bibblock">
Martin Fowler. 2005.

</span>
<span class="ltx_bibblock">Bliki: Fluentinterface.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://www.martinfowler.com/bliki/FluentInterface.html" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://www.martinfowler.com/bliki/FluentInterface.html</a>

</span>
</li>
<li id="bib.bib14" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Ge and Murphy-Hill (2014)</span>
<span class="ltx_bibblock">
Xi Ge and Emerson Murphy-Hill. 2014.

</span>
<span class="ltx_bibblock">Manual Refactoring Changes with Automated Refactoring Validation. In <em id="bib.bib14.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 36th International Conference on Software Engineering</em> (Hyderabad, India) <em id="bib.bib14.2.2" class="ltx_emph ltx_font_italic">(ICSE 2014)</em>. Association for Computing Machinery, New York, NY, USA, 1095–1105.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/2568225.2568280" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/2568225.2568280</a>

</span>
</li>
<li id="bib.bib15" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Grice (1991)</span>
<span class="ltx_bibblock">
Paul Grice. 1991.

</span>
<span class="ltx_bibblock"><em id="bib.bib15.1.1" class="ltx_emph ltx_font_italic">Studies in the Way of Words</em>.

</span>
<span class="ltx_bibblock">Harvard University Press, Cambridge, Mass. [u.a.].

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib16" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al<span id="bib.bib16.2.2.1" class="ltx_text">.</span> (2011a)</span>
<span class="ltx_bibblock">
Philip J. Guo, Sean Kandel, Joseph M. Hellerstein, and Jeffrey Heer. 2011a.

</span>
<span class="ltx_bibblock">Proactive Wrangling: Mixed-Initiative End-User Programming of Data Transformation Scripts. In <em id="bib.bib16.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology</em> (Santa Barbara, California, USA) <em id="bib.bib16.4.2" class="ltx_emph ltx_font_italic">(UIST ’11)</em>. Association for Computing Machinery, New York, NY, USA, 65–74.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/2047196.2047205" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/2047196.2047205</a>

</span>
</li>
<li id="bib.bib17" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Guo et al<span id="bib.bib17.2.2.1" class="ltx_text">.</span> (2011b)</span>
<span class="ltx_bibblock">
Philip J. Guo, Sean Kandel, Joseph M. Hellerstein, and Jeffrey Heer. 2011b.

</span>
<span class="ltx_bibblock">Proactive Wrangling: Mixed-Initiative End-User Programming of Data Transformation Scripts. In <em id="bib.bib17.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology</em> (Santa Barbara, California, USA) <em id="bib.bib17.4.2" class="ltx_emph ltx_font_italic">(UIST ’11)</em>. Association for Computing Machinery, New York, NY, USA, 65–74.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/2047196.2047205" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/2047196.2047205</a>

</span>
</li>
<li id="bib.bib18" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Henley et al<span id="bib.bib18.2.2.1" class="ltx_text">.</span> (2021)</span>
<span class="ltx_bibblock">
Austin Z. Henley, Julian Ball, Benjamin Klein, Aiden Rutter, and Dylan Lee. 2021.

</span>
<span class="ltx_bibblock">An Inquisitive Code Editor for Addressing Novice Programmers’ Misconceptions of Program Behavior. In <em id="bib.bib18.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 43rd International Conference on Software Engineering: Joint Track on Software Engineering Education and Training</em> (Virtual Event, Spain) <em id="bib.bib18.4.2" class="ltx_emph ltx_font_italic">(ICSE-JSEET ’21)</em>. IEEE Press, New York, NY, USA, 165–170.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1109/ICSE-SEET52601.2021.00026" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/ICSE-SEET52601.2021.00026</a>

</span>
</li>
<li id="bib.bib19" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Hollmann et al<span id="bib.bib19.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Noah Hollmann, Samuel Müller, and Frank Hutter. 2023.

</span>
<span class="ltx_bibblock">LLMs for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2305.03403 [cs.AI]

</span>
</li>
<li id="bib.bib20" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Jayagopal et al<span id="bib.bib20.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Dhanya Jayagopal, Justin Lubin, and Sarah E. Chasins. 2022.

</span>
<span class="ltx_bibblock">Exploring the Learnability of Program Synthesizers by Novice Programmers. In <em id="bib.bib20.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology</em> (Bend, OR, USA) <em id="bib.bib20.4.2" class="ltx_emph ltx_font_italic">(UIST ’22)</em>. Association for Computing Machinery, New York, NY, USA, Article 64, 15 pages.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3526113.3545659" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3526113.3545659</a>

</span>
</li>
<li id="bib.bib21" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kaddour et al<span id="bib.bib21.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023.

</span>
<span class="ltx_bibblock">Challenges and Applications of Large Language Models.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2307.10169 [cs.CL]

</span>
</li>
<li id="bib.bib22" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kandel et al<span id="bib.bib22.2.2.1" class="ltx_text">.</span> (2011a)</span>
<span class="ltx_bibblock">
Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011a.

</span>
<span class="ltx_bibblock">Wrangler: Interactive Visual Specification of Data Transformation Scripts. In <em id="bib.bib22.3.1" class="ltx_emph ltx_font_italic">Proceedings of the SIGCHI Conference on Human Factors in Computing Systems</em> (Vancouver, BC, Canada) <em id="bib.bib22.4.2" class="ltx_emph ltx_font_italic">(CHI ’11)</em>. Association for Computing Machinery, New York, NY, USA, 3363–3372.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/1978942.1979444" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/1978942.1979444</a>

</span>
</li>
<li id="bib.bib23" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kandel et al<span id="bib.bib23.2.2.1" class="ltx_text">.</span> (2011b)</span>
<span class="ltx_bibblock">
Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011b.

</span>
<span class="ltx_bibblock">Wrangler: Interactive Visual Specification of Data Transformation Scripts. In <em id="bib.bib23.3.1" class="ltx_emph ltx_font_italic">Proceedings of the SIGCHI Conference on Human Factors in Computing Systems</em> (Vancouver, BC, Canada) <em id="bib.bib23.4.2" class="ltx_emph ltx_font_italic">(CHI ’11)</em>. Association for Computing Machinery, New York, NY, USA, 3363–3372.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/1978942.1979444" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/1978942.1979444</a>

</span>
</li>
<li id="bib.bib24" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kandel et al<span id="bib.bib24.2.2.1" class="ltx_text">.</span> (2012)</span>
<span class="ltx_bibblock">
Sean Kandel, Andreas Paepcke, Joseph M Hellerstein, and Jeffrey Heer. 2012.

</span>
<span class="ltx_bibblock">Enterprise data analysis and visualization: An interview study.

</span>
<span class="ltx_bibblock"><em id="bib.bib24.3.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Visualization and Computer Graphics</em> 18, 12 (2012), 2917–2926.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib25" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kery et al<span id="bib.bib25.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Mary Beth Kery, Marissa Radensky, Mahima Arya, Bonnie E. John, and Brad A. Myers. 2018.

</span>
<span class="ltx_bibblock">The Story in the Notebook: Exploratory Data Science Using a Literate Programming Tool. In <em id="bib.bib25.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems</em> (Montreal QC, Canada) <em id="bib.bib25.4.2" class="ltx_emph ltx_font_italic">(CHI ’18)</em>. Association for Computing Machinery, New York, NY, USA, 1–11.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3173574.3173748" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3173574.3173748</a>

</span>
</li>
<li id="bib.bib26" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kim et al<span id="bib.bib26.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Miryung Kim, Thomas Zimmermann, Robert DeLine, and Andrew Begel. 2018.

</span>
<span class="ltx_bibblock">Data Scientists in Software Teams: State of the Art and Challenges.

</span>
<span class="ltx_bibblock"><em id="bib.bib26.3.1" class="ltx_emph ltx_font_italic">IEEE Transactions on Software Engineering</em> 44, 11 (2018), 1024–1038.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/TSE.2017.2754374" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/TSE.2017.2754374</a>

</span>
</li>
<li id="bib.bib27" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Kross and Guo (2019)</span>
<span class="ltx_bibblock">
Sean Kross and Philip J. Guo. 2019.

</span>
<span class="ltx_bibblock">Practitioners Teaching Data Science in Industry and Academia: Expectations, Workflows, and Challenges. In <em id="bib.bib27.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems</em> (Glasgow, Scotland Uk) <em id="bib.bib27.2.2" class="ltx_emph ltx_font_italic">(CHI ’19)</em>. Association for Computing Machinery, New York, NY, USA, 1–14.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3290605.3300493" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3290605.3300493</a>

</span>
</li>
<li id="bib.bib28" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Landis and Koch (1977)</span>
<span class="ltx_bibblock">
J. Richard Landis and Gary G. Koch. 1977.

</span>
<span class="ltx_bibblock">The measurement of observer agreement for categorical data.

</span>
<span class="ltx_bibblock"><em id="bib.bib28.1.1" class="ltx_emph ltx_font_italic">Biometrics</em> 33, 1 (1977), 159.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.2307/2529310" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.2307/2529310</a>

</span>
</li>
<li id="bib.bib29" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liang et al<span id="bib.bib29.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Jenny T. Liang, Chenyang Yang, and Brad A. Myers. 2023.

</span>
<span class="ltx_bibblock">Understanding the Usability of AI Programming Assistants.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2303.17125 [cs.SE]

</span>
</li>
<li id="bib.bib30" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Liu et al<span id="bib.bib30.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Michael Xieyang Liu, Advait Sarkar, Carina Negreanu, Benjamin Zorn, Jack Williams, Neil Toronto, and Andrew D. Gordon. 2023.

</span>
<span class="ltx_bibblock">“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models. In <em id="bib.bib30.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems</em> (Hamburg, Germany) <em id="bib.bib30.4.2" class="ltx_emph ltx_font_italic">(CHI ’23)</em>. Association for Computing Machinery, New York, NY, USA, Article 598, 31 pages.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3544548.3580817" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3544548.3580817</a>

</span>
</li>
<li id="bib.bib31" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Mcnutt et al<span id="bib.bib31.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Andrew M Mcnutt, Chenglong Wang, Robert A Deline, and Steven M. Drucker. 2023.

</span>
<span class="ltx_bibblock">On the Design of AI-Powered Code Assistants for Notebooks. In <em id="bib.bib31.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems</em> (Hamburg, Germany) <em id="bib.bib31.4.2" class="ltx_emph ltx_font_italic">(CHI ’23)</em>. Association for Computing Machinery, New York, NY, USA, Article 434, 16 pages.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3544548.3580940" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3544548.3580940</a>

</span>
</li>
<li id="bib.bib32" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Muller et al<span id="bib.bib32.2.2.1" class="ltx_text">.</span> (2019)</span>
<span class="ltx_bibblock">
Michael Muller, Ingrid Lange, Dakuo Wang, David Piorkowski, Jason Tsay, Q. Vera Liao, Casey Dugan, and Thomas Erickson. 2019.

</span>
<span class="ltx_bibblock">How Data Science Workers Work with Data: Discovery, Capture, Curation, Design, Creation. In <em id="bib.bib32.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems</em> (Glasgow, Scotland Uk) <em id="bib.bib32.4.2" class="ltx_emph ltx_font_italic">(CHI ’19)</em>. Association for Computing Machinery, New York, NY, USA, 1–15.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3290605.3300356" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3290605.3300356</a>

</span>
</li>
<li id="bib.bib33" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Nielsen (2006)</span>
<span class="ltx_bibblock">
Jakob Nielsen. 2006.

</span>
<span class="ltx_bibblock">Progressive disclosure.

</span>
<span class="ltx_bibblock"><em id="bib.bib33.1.1" class="ltx_emph ltx_font_italic">nngroup.com</em> (2006).

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib34" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Noever and McKee (2023)</span>
<span class="ltx_bibblock">
David Noever and Forrest McKee. 2023.

</span>
<span class="ltx_bibblock">Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Models.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2301.13382 [cs.CL]

</span>
</li>
<li id="bib.bib35" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">OpenAI (2023)</span>
<span class="ltx_bibblock">
OpenAI. 2023.

</span>
<span class="ltx_bibblock">GPT-4 Technical Report.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2303.08774 [cs.CL]

</span>
</li>
<li id="bib.bib36" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Rule et al<span id="bib.bib36.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Adam Rule, Aurélien Tabard, and James D. Hollan. 2018.

</span>
<span class="ltx_bibblock">Exploration and Explanation in Computational Notebooks. In <em id="bib.bib36.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems</em> (Montreal QC, Canada) <em id="bib.bib36.4.2" class="ltx_emph ltx_font_italic">(CHI ’18)</em>. Association for Computing Machinery, New York, NY, USA, 1–12.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3173574.3173606" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3173574.3173606</a>

</span>
</li>
<li id="bib.bib37" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Saldaña (2009)</span>
<span class="ltx_bibblock">
Johnny Saldaña. 2009.

</span>
<span class="ltx_bibblock"><em id="bib.bib37.1.1" class="ltx_emph ltx_font_italic">The Coding Manual for Qualitative Researchers</em>.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="http://ci.nii.ac.jp/ncid/BB20067005" title="" class="ltx_ref ltx_url ltx_font_typewriter">http://ci.nii.ac.jp/ncid/BB20067005</a>

</span>
</li>
<li id="bib.bib38" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sarkar et al<span id="bib.bib38.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022.

</span>
<span class="ltx_bibblock">What is it like to program with artificial intelligence?. In <em id="bib.bib38.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 33rd Annual Conference of the Psychology of Programming Interest Group (PPIG 2022)</em>.

</span>
<span class="ltx_bibblock">
</span>
</li>
<li id="bib.bib39" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Shrestha et al<span id="bib.bib39.2.2.1" class="ltx_text">.</span> (2021)</span>
<span class="ltx_bibblock">
Nischal Shrestha, Titus Barik, and Chris Parnin. 2021.

</span>
<span class="ltx_bibblock">Remote, but Connected: How #TidyTuesday Provides an Online Community of Practice for Data Scientists.

</span>
<span class="ltx_bibblock"><em id="bib.bib39.3.1" class="ltx_emph ltx_font_italic">Proc. ACM Hum.-Comput. Interact.</em> 5, CSCW1, Article 52 (apr 2021), 31 pages.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1145/3449126" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3449126</a>

</span>
</li>
<li id="bib.bib40" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Skjuve et al<span id="bib.bib40.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Marita Skjuve, Asbjørn Følstad, and Petter Bae Brandtzaeg. 2023.

</span>
<span class="ltx_bibblock">The User Experience of ChatGPT: Findings from a Questionnaire Study of Early Users. In <em id="bib.bib40.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 5th International Conference on Conversational User Interfaces</em> (Eindhoven, Netherlands) <em id="bib.bib40.4.2" class="ltx_emph ltx_font_italic">(CUI ’23)</em>. Association for Computing Machinery, New York, NY, USA, Article 2, 10 pages.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3571884.3597144" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3571884.3597144</a>

</span>
</li>
<li id="bib.bib41" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Srinivasa Ragavan et al<span id="bib.bib41.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Sruti Srinivasa Ragavan, Zhitao Hou, Yun Wang, Andrew D Gordon, Haidong Zhang, and Dongmei Zhang. 2022.

</span>
<span class="ltx_bibblock">GridBook: Natural Language Formulas for the Spreadsheet Grid. In <em id="bib.bib41.3.1" class="ltx_emph ltx_font_italic">27th International Conference on Intelligent User Interfaces</em> (Helsinki, Finland) <em id="bib.bib41.4.2" class="ltx_emph ltx_font_italic">(IUI ’22)</em>. Association for Computing Machinery, New York, NY, USA, 345–368.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3490099.3511161" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3490099.3511161</a>

</span>
</li>
<li id="bib.bib42" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Sutton et al<span id="bib.bib42.2.2.1" class="ltx_text">.</span> (2018)</span>
<span class="ltx_bibblock">
Charles Sutton, Timothy Hobson, James Geddes, and Rich Caruana. 2018.

</span>
<span class="ltx_bibblock">Data Diff: Interpretable, Executable Summaries of Changes in Distributions for Data Wrangling. In <em id="bib.bib42.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining</em> (London, United Kingdom) <em id="bib.bib42.4.2" class="ltx_emph ltx_font_italic">(KDD ’18)</em>. Association for Computing Machinery, New York, NY, USA, 2279–2288.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3219819.3220057" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3219819.3220057</a>

</span>
</li>
<li id="bib.bib43" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaithilingam et al<span id="bib.bib43.2.2.1" class="ltx_text">.</span> (2023)</span>
<span class="ltx_bibblock">
Priyan Vaithilingam, Elena L. Glassman, Peter Groenwegen, Sumit Gulwani, Austin Z. Henley, Rohan Malpani, David Pugh, Arjun Radhakrishna, Gustavo Soares, Joey Wang, and Aaron Yim. 2023.

</span>
<span class="ltx_bibblock">Towards More Effective AI-Assisted Programming: A Systematic Design Exploration to Improve Visual Studio IntelliCode’s User Experience.

</span>
<span class="ltx_bibblock">(2023), 185–195.

</span>
<span class="ltx_bibblock">
<a target="_blank" href="https://doi.org/10.1109/ICSE-SEIP58684.2023.00022" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1109/ICSE-SEIP58684.2023.00022</a>

</span>
</li>
<li id="bib.bib44" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vaithilingam et al<span id="bib.bib44.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022.

</span>
<span class="ltx_bibblock">Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In <em id="bib.bib44.3.1" class="ltx_emph ltx_font_italic">Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems</em> (New Orleans, LA, USA) <em id="bib.bib44.4.2" class="ltx_emph ltx_font_italic">(CHI EA ’22)</em>. Association for Computing Machinery, New York, NY, USA, Article 332, 7 pages.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3491101.3519665" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3491101.3519665</a>

</span>
</li>
<li id="bib.bib45" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Vasilescu et al<span id="bib.bib45.2.2.1" class="ltx_text">.</span> (2014)</span>
<span class="ltx_bibblock">
Bogdan Vasilescu, Alexander Serebrenik, Prem Devanbu, and Vladimir Filkov. 2014.

</span>
<span class="ltx_bibblock">How Social Q&amp;A Sites Are Changing Knowledge Sharing in Open Source Software Communities. In <em id="bib.bib45.3.1" class="ltx_emph ltx_font_italic">Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work &amp; Social Computing</em> (Baltimore, Maryland, USA) <em id="bib.bib45.4.2" class="ltx_emph ltx_font_italic">(CSCW ’14)</em>. Association for Computing Machinery, New York, NY, USA, 342–354.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/2531602.2531659" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/2531602.2531659</a>

</span>
</li>
<li id="bib.bib46" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yan and He (2020)</span>
<span class="ltx_bibblock">
Cong Yan and Yeye He. 2020.

</span>
<span class="ltx_bibblock">Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks. In <em id="bib.bib46.1.1" class="ltx_emph ltx_font_italic">Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data</em> (Portland, OR, USA) <em id="bib.bib46.2.2" class="ltx_emph ltx_font_italic">(SIGMOD ’20)</em>. Association for Computing Machinery, New York, NY, USA, 1539–1554.

</span>
<span class="ltx_bibblock">

<a target="_blank" href="https://doi.org/10.1145/3318464.3389738" title="" class="ltx_ref ltx_url ltx_font_typewriter">https://doi.org/10.1145/3318464.3389738</a>

</span>
</li>
<li id="bib.bib47" class="ltx_bibitem">
<span class="ltx_tag ltx_role_refnum ltx_tag_bibitem">Yin et al<span id="bib.bib47.2.2.1" class="ltx_text">.</span> (2022)</span>
<span class="ltx_bibblock">
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Alex Polozov, and Charles Sutton. 2022.

</span>
<span class="ltx_bibblock">Natural Language to Code Generation in Interactive Data Science Notebooks.

</span>
<span class="ltx_bibblock">
</span>
<span class="ltx_bibblock">arXiv:2212.09248 [cs.CL]

</span>
</li>
</ul>
</section>
<div class="ltx_pagination ltx_role_newpage"></div>
</article>
</div>
<div class="ar5iv-footer"><a href="/html/2310.16162" class="ar5iv-nav-button ar5iv-nav-button-prev">◄</a>
    <a class="ar5iv-home-button" href="/"><img height="40" alt="ar5iv homepage" src="/assets/ar5iv.png"></a>
    <a href="/feeling_lucky" class="ar5iv-text-button">Feeling<br>lucky?</a>
    <a href="/log/2310.16164" class="ar5iv-text-button ar5iv-severity-error">Conversion<br>report</a>
    <a class="ar5iv-text-button" target="_blank" href="https://github.com/dginev/ar5iv/issues/new?template=improve-article--arxiv-id-.md&title=Improve+article+2310.16164">Report<br>an issue</a>
    <a href="https://arxiv.org/abs/2310.16164" class="ar5iv-text-button arxiv-ui-theme">View&nbsp;original<br>on&nbsp;arXiv</a><a href="/html/2310.16165" class="ar5iv-nav-button ar5iv-nav-button-next">►</a>
</div><footer class="ltx_page_footer">
<a class="ar5iv-toggle-color-scheme" href="javascript:toggleColorScheme()" title="Toggle ar5iv color scheme"><span class="color-scheme-icon"></span></a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/license" target="_blank">Copyright</a>
<a class="ar5iv-footer-button" href="https://arxiv.org/help/policies/privacy_policy" target="_blank">Privacy Policy</a>

<div class="ltx_page_logo">Generated  on Tue Feb 27 23:22:56 2024 by <a target="_blank" href="http://dlmf.nist.gov/LaTeXML/" class="ltx_LaTeXML_logo"><span style="letter-spacing:-0.2em; margin-right:0.1em;">L<span class="ltx_font_smallcaps" style="position:relative; bottom:2.2pt;">a</span>T<span class="ltx_font_smallcaps" style="font-size:120%;position:relative; bottom:-0.2ex;">e</span></span><span style="font-size:90%; position:relative; bottom:-0.2ex;">XML</span><img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAsAAAAOCAYAAAD5YeaVAAAAAXNSR0IArs4c6QAAAAZiS0dEAP8A/wD/oL2nkwAAAAlwSFlzAAALEwAACxMBAJqcGAAAAAd0SU1FB9wKExQZLWTEaOUAAAAddEVYdENvbW1lbnQAQ3JlYXRlZCB3aXRoIFRoZSBHSU1Q72QlbgAAAdpJREFUKM9tkL+L2nAARz9fPZNCKFapUn8kyI0e4iRHSR1Kb8ng0lJw6FYHFwv2LwhOpcWxTjeUunYqOmqd6hEoRDhtDWdA8ApRYsSUCDHNt5ul13vz4w0vWCgUnnEc975arX6ORqN3VqtVZbfbTQC4uEHANM3jSqXymFI6yWazP2KxWAXAL9zCUa1Wy2tXVxheKA9YNoR8Pt+aTqe4FVVVvz05O6MBhqUIBGk8Hn8HAOVy+T+XLJfLS4ZhTiRJgqIoVBRFIoric47jPnmeB1mW/9rr9ZpSSn3Lsmir1fJZlqWlUonKsvwWwD8ymc/nXwVBeLjf7xEKhdBut9Hr9WgmkyGEkJwsy5eHG5vN5g0AKIoCAEgkEkin0wQAfN9/cXPdheu6P33fBwB4ngcAcByHJpPJl+fn54mD3Gg0NrquXxeLRQAAwzAYj8cwTZPwPH9/sVg8PXweDAauqqr2cDjEer1GJBLBZDJBs9mE4zjwfZ85lAGg2+06hmGgXq+j3+/DsixYlgVN03a9Xu8jgCNCyIegIAgx13Vfd7vdu+FweG8YRkjXdWy329+dTgeSJD3ieZ7RNO0VAXAPwDEAO5VKndi2fWrb9jWl9Esul6PZbDY9Go1OZ7PZ9z/lyuD3OozU2wAAAABJRU5ErkJggg==" alt="Mascot Sammy"></a>
</div></footer>
</div>

    <script>
      var canMathML = typeof(MathMLElement) == "function";
      if (!canMathML) {
        var body = document.querySelector("body");
        body.firstElementChild.setAttribute('style', 'opacity: 0;');
        var loading = document.createElement("div");
        loading.setAttribute("id", "mathjax-loading-spinner");
        var message = document.createElement("div");
        message.setAttribute("id", "mathjax-loading-message");
        message.innerText = "Typesetting Equations...";
        body.prepend(loading);
        body.prepend(message);

        var el = document.createElement("script");
        el.src = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js";
        document.querySelector("head").appendChild(el);

        window.MathJax = {
          startup: {
            pageReady: () => {
              return MathJax.startup.defaultPageReady().then(() => {
                body.removeChild(loading);
                body.removeChild(message);
                body.firstElementChild.removeAttribute('style');
              }); } } };
      }
    </script>
    <script>
    // Auxiliary function, building the preview feature when
    // an inline citation is clicked
    function clicked_cite(e) {
      e.preventDefault();
      let cite = this.closest('.ltx_cite');
      let next = cite.nextSibling;
      if (next && next.nodeType == Node.ELEMENT_NODE && next.getAttribute('class') == "ar5iv-bibitem-preview") {
        next.remove();
        return; }
      // Before adding a preview modal,
      // cleanup older previews, in case they're still open
      document.querySelectorAll('span.ar5iv-bibitem-preview').forEach(function(node) {
        node.remove();
      })

      // Create the preview
      preview = document.createElement('span');
      preview.setAttribute('class','ar5iv-bibitem-preview');
      let target = document.getElementById(this.getAttribute('href').slice(1));
      target.childNodes.forEach(function (child) {
        preview.append(child.cloneNode(true));
      });
      let close_x = document.createElement('button');
      close_x.setAttribute("aria-label","Close modal for bibliography item preview");
      close_x.textContent = "×";
      close_x.setAttribute('class', 'ar5iv-button-close-preview');
      close_x.setAttribute('onclick','this.parentNode.remove()');
      preview.append(close_x);
      preview.querySelectorAll('.ltx_tag_bibitem').forEach(function(node) {
        node.remove();
      });
      cite.parentNode.insertBefore(preview, cite.nextSibling);
      return;
    }
    // Global Document initialization:
    // - assign the preview feature to all inline citation links
    document.querySelectorAll(".ltx_cite .ltx_ref").forEach(function (link) {
      link.addEventListener("click", clicked_cite);
    });
    </script>
    </body>
</html>

2310.16164
Large Language Models (LLMs)  are being increasingly employed in data science for tasks like data preprocessing and analytics. However, data scientists encounter substantial obstacles when conversing with LLM-powered chatbots and acting on their suggestions and answers. We conducted a mixed-methods study, including contextual observations, semi-structured interviews (n=14) , and a survey (n=114) , to identify these challenges. Our findings highlight key issues faced by data scientists, including contextual data retrieval, formulating prompts for complex tasks, adapting generated code to local environments, and refining prompts iteratively. Based on these insights, we propose actionable design recommendations, such as data brushing to support context selection, and inquisitive feedback loops to improve communications with AI-based assistants in data-science tools.
abstract
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
Abstract
2310.16164
1.IntroductionData scientists have traditionally relied on resources like documentation, tutorials, online courses, colleagues, Q&A forums, and online communities to develop skills and solve tasks.
These resources have been instrumental in helping them navigate the challenges of data acquisition, cleaning, wrangling, visualization, and presentation, especially for those with a non-programming background.
Although useful, it can be time consuming, tedious, and error-prone to rely on these resources for solving a data-science problem.With the emergence of AI-powered chat assistants, data scientists now have access to potentially faster and more accessible resources through a chat interface.
These AI-powered chatbots, like ChatGPT111https://chat.openai.com/, enable users to ask questions in natural language, and get useful responses with little to no latency.
For example, when prompted with “split my date column that is in MM/DD/YYYY format into three columns”, ChatGPT provided usable Python code:addingsomecenteritdf[[’Month’, ’Day’, ’Year’]] = df[’Date’].str.split(’/’, expand=True) ChatGPT can also explain how the code works, and supports follow-up questions or changes.
Evaluations of these AI tools have demonstrated over 70% accuracy on programming benchmarksand pass numerous tests designed for humans (e.g., the GRE, a college entrance exam, and the LSAT, a law school admission test) .However, the effectiveness of these tools is dependent on data scientists successfully communicating their questions, context (overall problem, task at hand, datasets, etc.) , assumptions, and domain knowledge to the AI assistant, and doing so through a back-and-forth conversation.
Grice’s maxims of conversation222https://effectiviology.com/principles-of-effective-communication/posits that a successful conversation involves content that has right amount of information, that is truthful and supported by evidence, that is relevant to the specific context, and that is presented clearly.
But conversations go awry—either side of the conversation may be making false assumptions, there may be ambiguities, and conversations may require numerous clarifications.
All of these problems are possible with AI assistants; most notably, they are known tohallucinate, meaning they confidently respond with false statements. Furthermore, studies have found that users find conversations with AI assistants, such as ChatGPT, to have problematic communication stylesand responses provided to be repetitive and filled with outdated or irrelevant information—a far cry from Grice’s maxims.
As such, for humans to communicate effectively with AI agents, humans must follow an iterative sense-making process that involves describing intent, building hypotheses, gathering evidence, and evaluating responses.The barriers to express intents are exacerbated in the context of data-science tasks.
First, data scientists work with a variety of artifacts, including raw datasets, code, computational notebooks, visualizations, documentation, and machine-learning pipelines.
Second, datasets are often large and normalized (spilt across multiple tables) , which may not be feasible to share (e.g., due to token limits of AI tools or due to data being spread across heterogeneous sources)  or to summarize the relevant portions succinctly (e.g., specifying regions of the data) .
Third, real-world data is messy, and suffer from quality issues; they may not strictly adhere to a schema or homogeneous format.
Fourth, data-science tasks often require domain expertise and numerous assumptions (e.g., negative values in a column represent an error, 0 means missing value)  and it may be laborious to ensure that the AI assistant is aware of such domain knowledge and assumptions.
Furthermore, even when a data scientist is able to articulate their question to the AI assistant, these same problems can cause difficulties to understand and validate the AI assistant’s response.In this work, we aim to understand the fundamental problems of a [human] data scientist communicating to an AI chat agent. In particular, we address the following research questions:•RQ1:How do data scientists interact with ChatGPT to complete data-science tasks?•RQ2:What challenges and unmet needs do data scientists face when interacting with ChatGPT?•RQ3:How well do these challenges and needs generalize to broader community of data scientists?To answer these questions, we conducted two mixed-method need-finding studies (Section3) . For the first study, we observe 14 professional data scientists as they perform four diverse tasks that are common in data-science pipelines. The participants were presented with the browser-based chat assistant, ChatGPT, to help them solve the tasks whenever needed. We performed inductive thematic analysis to identify needs and challenges by triangulating findings from open coding of semi-structured interview transcripts, observations from field notes, and video annotations. For the second study, we conducted a confirmation survey with 114 professional data scientists to validate and generalize findings from the first study.Our results (Sections4and5)  show that data scientists usually follow a typical pattern of articulating their question, collecting context, validating the response made by the AI assistant, and then applying the response to their task.
Throughout this process, they face copious barriers while communicating with the AI assistant, such as spending considerable time writing their question and collecting relevant context.
Then they face a variety of additional barriers while trying to make use of the AI assistant’s response, such as understanding the assumptions being made by the AI and modifying the provided code to work for their scenario.
This arduous workflow resulted in participants applying strategies to overcome the barriers and still make use of the AI assistant.
Based on our findings, we provide practical design recommendations that can improve user experiences while using LLM-powered assistants for their data-science workflows (Section6) .
introduction
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
1.Introduction
2310.16164
2.BackgroundHow data scientists workData science is the broad discipline of using data to understand the underlying nature of a domain and provide insights.
Activities include data acquisition, cleaning, wrangling, visualization, sharing, and acting on insights.
The majority of time spent on data-science tasks often involves data cleaning and wrangling. Sutton et al. stated that handling expansive datasets often involves navigating a myriad of issues, culminating in a situation they term “death by a thousand wranglings”.To support exploratory wrangling tasks, data scientists often employcomputational notebooks, such as Jupyter notebooks333https://jupyter.org/.
These notebooks provide an interactive environment that allows for the seamless integration of text, code, and output, facilitating both the development and documentation of computational workflows in a single document.
Unlike traditional code editors, notebooks enable a more narrative style of coding, where code cells and rich-text commentary can coexist and outputs can be immediately rendered inline.
However, notebooks come with their own pain points: they take considerable effort to keep organized, hidden state information can lead to misconceptions, they can be difficult to deploy, and long-running tasks can be problematic.Large language models and AI-powered chat assistantsAI-powered chat assistants are built on top of large language models (LLMs) .
LLMs are generative machine-learning models with billion parameters, and are trained on vast amount of data (text and images)  to generate human-like responses and perform various language tasks.
These models interact with users through aprompt, a natural-language query to which the LLM can respond to, also in natural language.
There are restrictions on thetoken limit, or the amount of information that can be sent or generated from the LLM.
Tools like ChatGPT add a user interface to enable a conversation, such that the model hascontextof the conversation history.
However, these models do not have direct access to data sources, and only respond based on the data they have seen during training.
body
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
2.Background
2310.16164
3.MethodologyWe performed two studies to answer our research questions. First, we conducted task-based semi-structured interviews and observed 14 professional data scientists engaged in data-science tasks. Then, we distributed a general survey of our initial findings with 114 data scientists.3.1.Study 1: Using ChatGPT for Data Science TasksWe used ChatGPT, a browser-based LLM-powered chat-based AI assistant, for the study. We avoided using any AI assistants integrated within computational notebooks to better understand fundamental challenges and patterns of usage. This deliberate choice to keep the AI assistant separate from the data science environment(s)  of the participants allows us to give them complete control and transparency over the prompt-writing and code adaptation processes—while giving us an opportunity to understand the underlying process of assessingwhichcontext, andhow muchcontext is relevant for prompting. This choice also echoes one of the design recommendations proposed bythat developers would prefer control over the shared prompt context.ParticipantsWe recruited participants from a large technology company by randomly selecting employees with the job title “Data Scientist” mentioned in the company address book. We further screened participants based on having at least 1 year of experience with data science and prior experience with Python, computational notebooks, and ChatGPT (or any other LLM-based chat/code assistants)  as a pre-requisite to their participation. Table1indicates the demographics collected for all participants, who reported a median of 5 years of experience in the domain of data science (,) . All participants have a background and educational degree in CS and Data Science related fields.All sessions were audio and video recorded for transcription of responses and analysis of on-screen interactions. All participants signed a consent form prior to the study. Participation in the study was voluntary and participants were compensated with gift cards worth $25 USD each.Table 1.Overview of the participants, including the domain that they work in, their years of experience as a data scientist, and the programming languages they often use.IDDomainExp. (yrs) Languages usedP1Anomaly Detection5Python, SQLP2Big Data Processing10Python, SQLP3User Segmentation4PythonP4NL Processing1PythonP5Feature Engineering4SQLP6Statistical Analysis2PythonP7Defect Detection5PythonP8Finance & Forecasting6Python, SQL, SparkP9Business Analytics2PythonP10Fraud Detection5Python, SQLP11User Segmentation10Python, SQL, SparkP12Cloud Computing10PythonP13Failure Prediction7PythonP14Fraud Detection5Python, SQL3.1.1.TasksWe used four essential tasks performed by data scientists in the lifetime of a typical project: (T1 and T2)  data wrangling and pre-processing, (T3)  feature extraction and selection for training tasks, (T4)  data visualization for insight-finding and reporting. The four tasks are described in Table2.We selected a sample of theNew York EMS emergency calls dataset444https://www.kaggle.com/datasets/new-york-city/ny-ems-incident-dispatch-data, which contains rows corresponding to emergency incidents, and information about time, location, resources, and the Fire Department’s response to the emergency. The dataset was well-suited for supporting the tasks, as several columns had to be transformed before performing further analysis, and event data was amendable to feature and correlation analysis.Table 2.The four data science tasks used in our study and their descriptions.#TaskDescriptionT1Datetime typecastChange the datatype of column‘‘INCIDENT_DATETIME’’from string to datetime (e.g.,‘‘2020-07-31T23:59:51.000’’) .T2Split using delimiterSplit the (‘‘INCIDENT_DESCRIPTION’’)  column using semicolon (‘‘;’’)  as the delimiter. Rows contain information about the incident location — borough, incident dispatch area code, and zipcode. Few rows do not contain the zipcode (e.g. rows such as‘‘Brooklyn; K7; 11211’’contain all information, whereas rows like “Manhattan; M3” are missing the zipcode) .T3Feature selectionExtract a new column titled‘‘RESPONSE_TIME’’using the difference between‘‘FIRST_ON_SCENE_DATETIME’’and‘‘INCIDENT_DATETIME’’; subsequently perform feature selection with the aim of predicting the‘‘RESPONSE_TIME’’.T4Heatmap plottingObtain a heatmap plot to analyze correlation between any columns of choice.Study protocolWe conducted a one-hour long, task-based study with each of the 14 data scientists via video conferencing. The study begins with demographic questions about their role and data-science-related work, languages and tools, educational background, and prior experience with using LLMs. Next, participants were presented with 4 tasks, one at a time, to be solved using Python in any computational notebook of their choice, and the browser-based ChatGPT555https://chat.openai.com/(Versions: March 14, 2023; March 23, 2023) . They were encouraged to make use of any preferred tools and resources (such as web search, notebooks, Excel, and so on) , and think aloud while solving the tasks. The completion of any task was not a necessity, and participants were encouraged to use ChatGPT at least once for every task. Once the participants had explored a task sufficiently, they were asked semi-structured questions to assess if ChatGPT helped them solve the task satisfactorily, and if it could have helped them with the task any better.AnalysisWe transcribed the audio recordings for each session. Subsequently, we annotated videos from each session recording to timestamp and gathered: (1)  prompts authored by participants, (2)  corresponding responses provided by ChatGPT, and (3)  a log of activities performed by the participants (including prompt writing, adapting generated code, and validating generated code) .
The first and second author analyzed the transcripts and annotations from video recordings to perform inductive thematic coding. We first conducted discussions over a span of 2 weeks while collaboratively annotating data for one participant. Following this, we independently annotated 20% of the data (from three participants)  with descriptive open codes. Next, we grouped similar codes and analyzed them to obtain axial codes, identifying usage patterns and pain-points faced by data scientists. We then computed the inter-rater agreement score to be 0.872, indicating ‘almost perfect’ agreement. The first two authors then split the remaining participant interviews and annotated them independently, while meeting frequently to discuss and refine the codes and themes.We observed theoretical saturation after thematic analysis of data from 9 participants. To further support the validity of our findings, we requested 5 study participants (P1, P3, P4, P8, P12)  to respond to a member-check survey. The survey consisted of 18 Likert-scale questions, each question representing an open-code from our findings. The ratings indicated agreement with each finding.3.2.Study 2: Confirmatory SurveyTo validate findings from Study 1, we conducted a survey with a broader population of 114 data scientists at a large technology company.Survey protocolThe survey consisted of two demographic questions, asking respondents to: (D1)  indicate years of experience with data science (a single choice question) , and (D2)  describe at least one scenario where they used any LLM for a data-science task (an open-field question) . To ensure the quality of the survey responses, we used responses for D2 to filter out respondents who do not have any experience with solving data-science tasks using LLMs.We used findings from the qualitative analysis of Study 1 responses to frame at least one question for each identified obstacle. The survey consisted of 8 ‘Agree’/‘Disagree’ statements, and all questions were optional to answer. Finally, to evaluate if our observations from Study 1 reached theoretical saturation, we asked respondents to indicate if they have faced any other difficulties in using LLMs for data science tasks that were not mentioned in the survey questions.Recruitment and informed consentThe respondents were contacted randomly via e-mail invites based on their job title (“Data Scientist”)  mentioned in the company address book. In our e-mail invites, we also requested participants to forward the survey to other data scientists who might be interested. We offered a drawing for one of two Amazon gift cards worth $50 USD each for completing the survey.Respondents114 data scientists from a large technology company responded to the survey. After inspecting whether the respondents have meaningful prior experiences (D2) , we removed 16 respondents ()  that had no experience with using LLMs for data science tasks. We report all findings using the screened responses.of the respondents self-reported having more than 5 years of experience in data science related professional roles. We discuss survey results in Section5, where we plotted the responses and inspected them manually to gather any other difficulties that were not mentioned in the survey questions.Figure 1.A gallery of tasks involved in Study 1. Images are taken from participant screen sharing recordings and are anonymized. Snippets marked (C)  denote the localContextretrieved by participants to use in prompt-writing; (P)  denotes thePromptsent to ChatGPT; and (A)  denotes theAnswerprovided by ChatGPT.
body
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
3.Methodology
2310.16164
4.ObservationsWe provide a summary of our participants interactions with ChatGPT and then report observed obstacles and strategies used by the participants.
Table4summarizes the themes that emerged from our analysis of the data scientists’ actions and responses. We also report participants who identified with each obstacle and employed prompting strategies in Table4.4.1.Participants Interactions with ChatGPT4.1.1.Typical workflowMost participants started with loading the CSV files into a notebook of their choice, and constructing apandasdataframe, while viewing the first few rows of the data. Sometimes they opened the CSV in Excel before creating a notebook, to glance over the data. Some participants used pandas.DataFrame.describe()  to obtain a high-level summary of their data.Next, they were sequentially presented with each of the tasks. Every participant worked on tasks T1, T2, and T3. However, 4 participants were unable to attempt T4 in the available time. For each task, participants generally began with phrasing their question, assessing and gathering context—such as description of the data, sample values—for prompt construction, understanding and taking action on ChatGPT’s response while writing additional prompts that would help them elicit better responses. Figure1presents a gallery of tasks involved in the study.Participants faced several challenges in conversing with ChatGPT and using its responses for task completion. In particular, most data scientists struggled with T3 (feature selection) , taking them significantly higher number of prompts and time to get a satisfactory response. Feature selection, being a broad task (as compared to the remaining study tasks) , had participants thinking about decomposing it to plan for sub-tasks, and providing ChatGPT sufficient context to obtain concrete and actionable next steps.4.1.2.Prompting behaviorsIn total, we logged 111 prompts made by 14 participants. Table3presents an overview of the prompt distribution for each study task. We observed that participants spenttime on preparing prompts,time on adapting the code returned by ChatGPT, and the remaining on validating the code, as seen in Figure2(Left) . In raw time spent on these activities, on average,seconds were spent on prompt writing,seconds on adaptation andseconds on validation. We also find thatof the total time was spent on writing the initial prompt, whereas making refinements to prompts and asking follow-up questions takesof the total time.
In particular, we observed that participants especially struggled to write an initial prompt for T3 (feature selection) , a task with more subjectivity, when compared to relatively more objective tasks (T1, T2 and T4) . The prompts also greatly varied in verbosity:You are a data scientist, how can you do feature selection? (P13) I have a dataset that has information about medical emergency services. I have a response time column which tells me historically how much time will it take to respond to a medical emergency incident. I have other columns some are categorical and some are continuous which explain more about the incident. My task is to predict the response time for a new incident and I am thinking about building a predictive model for that. What would be good features that can help the model predict response time? (P8) Table 3.Overview of the number of prompts queried per task, and average time spent in prompt-writing.Task# promptsmedianminmaxavg time spent (s) T1 – Datetime typecast2211355T2 – Split using delimiter1911259T3 – Feature selection47306159T4 – Heatmap plotting2320458Participants also made iterative refinements to their prompts. They sent across a plenitude of follow-up prompts for performing feature selection (T3) —re-emphasizing task goals, decomposing tasks, providing additional context and examples, and requesting for tweaks to code snippets. The need to send follow-up prompts was also exacerbated for plotting visualizations (T4) , where participants would go back-and-forth with ChatGPT for minor tweaks—changes in plot size, color scheme, and font size. Figure2(Right)  reflects on the distribution of follow-up and refined prompts for each task.Figure 2.Percentage of total time spent in each activity — prompt writing, code adaptation, and validation — for each task (Left) .
Stacked percentage distribution of initiating prompts, refinements, prompts for task decompositions, and follow-up prompts (Right) .Overall, we observed that when participants interacted with ChatGPT as an assistant for providing recommendations for code, it often involved many steps for more involved tasks. Further, participants often struggled with providing information about their data and desired goals to ChatGPT and adapt its responses to solve their tasks.4.2.Obstacles when Communicating with ChatGPT4.2.1.Sharing context is difficultWhat information must be shared with ChatGPT?Since LLMs“cannot make sense of the raw data”(P2, P3, P7, P8, P10, P12) , data scientists begin the process of prompt-writing by brainstorming what pieces of data, (such as filename, column names, column data types, sample rows, etc.)  and which descriptive information (such as pattern of strings in a column, number of unique values, missing values, outlier information, minimum/maximum for numeric data, etc.)  will help ChatGPT in solving the query satisfactorily. Participants reported that starting with curating a prompt is a “daunting” task (P9)  because it requires decision-making on what data context needs to be included. P2 said,“ I doubt if this will be successful, I feel like we need a little bit more of numbers, but let’s see if only using the column names we can get something.”Table 4.Themes—obstacles and strategies when using ChatGPT for data-science tasks.ThemeDescriptionRepresentative ExamplesParticipantsObstacles when Communicating with ChatGPTSharing context is difficult(Section4.2.1) Gathering relevant information to prompt with (e.g., column names, datatypes, example datapoints)  takes time.“Should I add some rows for it to see? I am not sure if [it] will work…”“So this one is recommendingscikit-learnwhich is not a package I typically use. I usually work inpyspark.”“It is all about the context which we provide. If we could refine it, it would do better.”P1—P6, P8, P10, P12, P14(10of14) ChatGPT opaquely makes assumptions(Section4.2.2) Data scientists had to correct or adjust prompts in response to unanticipated assumptions made by ChatGPT about data.“I will provide this example for the format”“Oh, it thinks the call type is int, let me say that it is categorical”P1—P14(14of14) Misaligned expectations(Section4.2.3) ChatGPT often returned overly verbose answers or inappropriate code responses.“I will skip reading any of this text, unless my code doesn’t work”“it didn’t bring upto_datetime() and suggestedstrptime() instead!”P1—P10, P12—P14(13of14) Obstacles in Leveraging ChatGPT’s ResponsesGeneration of repeated code(Section4.3.1) ChatGPT repeatedly generated code for imports, data ingestion and exploration, which had to be removed.“It erased my data, I will quickly re-run the cells”“We already have pandas and the data, I’ll delete these lines […] not required”P1—P10, P12—P14(13of14) Data and notebook management preferences(Section4.3.2) Splitting code into cells and aligning it with data management preferences (such as creating temporary views, dropping parent columns, etc) .“a temp df to see the new split columns”“I would not fill with ‘NULL’ […] still need to understand these columns more”P2—P9, P12—P14(11of14) Code validation(Section4.3.3) Data scientists often author additional code snippets to validate generated code.“ChatGPT’s affirmative language is deceiving […] it doesn’t really know my data”“let me printdf.dtypesto be sure”P1—P10, P12—P14(13of14) Strategies for Prompting and Alternate ResourcesTechniques for prompt construction(Section4.4.1) Data scientists use prompting techniques and create placeholders for streamlined context gathering from various sources“lets ask why it selected these features, why not the others”“I will paste column names later, so that I can write the prompt first”P1—P10, P12, P14(11of14) Scaffolding with domain expertise(Section4.4.2) Data scientists often employ techniques to guide ChatGPT with their domain knowledge to elicit better responses.“I can be more specific and give it only the float columns”“I will exclude the time columns to get good features”P5, P10, P14(3of14) Choosing alternate resources over ChatGPT(Section4.4.3) Making use of web-search; and re-use of previously authored notebooks/pipelines“must write longer queries instead of 8-10 key words just because I can”“copy paste canned code […] 80% of it would be repeating”P3—P5, P7, P11, P12(6of14) How much information must be shared with ChatGPT?Participants often questionedhow muchinformation needs to be shared with ChatGPT to get a good response. Some participants made attempts to paste the entire CSV file into the prompt and encountered token limit errors (P6, P7, P13) . Others also reflected on the importance of sharing only relevant instructions or snippets of the data as“additional data seems to confuse ChatGPT into providing not-so-relevant results”(P7, P9, P10) . P9 said,“The main obstacle I ran into here was, knowing how to prompt ChatGPT. Because I think at first I almost gave it too much information on how to perform a correlation analysis by describing it.”After sharing 10 sample rows, P2 tried to estimate ChatGPT’s boundaries in being able to understand raw data beyond its semantics, as it provided a“very basic response for the most relevant fields, by repeating the entire fields names and telling me their [semantic] meaning.”Challenges in fetching context.To construct the prompt, after assessingwhatcontext is relevant, participants had to go back and forth between ChatGPT’s browser window and data sources to fetch the context. They had to frequently write code snippets in their computational notebooks to gather the required context. For instance, consider apandasDataFrame nameddf. Participants must write, execute, and copy the output of commands such as:•df.head()  to obtain a sample of rows and the data header•df.columns or df.dtypes to obtain column names and their data types•df[’INCIDENT_DATETIME’][0] and df[’INCIDENT_DESCRIPTION’][0] to obtain the first value in the respective columns to provide as few-shot examples, and so on.We observed that P10 and P14 also implemented custom logic to extract information to share beyond what is typically shared in a prompt. For instance, P14 wanted to provide names of specific columns to ChatGPT and hence wrote the following code that computes column names that do not contain the sub-strings‘TIME’or‘ID’:⬇>>>[iforiindf.columnsif’TIME’notini.upper() or’ID’notini.upper() ]Figure 3.Frequency of data context shared by participants for prompt-construction for each task.Further, P5, P7 and P8 had to leave their notebook environment to open the CSV file in Excel. This was because the participants wanted to share values from a column, and the raw output from df.head()  could not be selected freely to be copied to the clipboard. Opening the file in Excel enabled them to select any range of cells and copy them with ease without writing additional code.In addition to data context, participants also used snippets of their code with error messages to prompt ChatGPT for a fix (P6, P7, P12) . Figure3shows the frequency of different types of data context shared with ChatGPT corresponding to each task.After fetching the relevant data context, participants had to paste it into the prompt. Participants would often manually format the pasted information to make the prompt look“cleaner,”to ensure that ChatGPT can understand the context (P2, P3, P7) . While solving T3 (feature selection) , P2 wanted to fetch column names, and organize them such that each column name appears in a new line in the prompt. To this, he said,“Can you copy and paste it for me please? By any chance would you have like all the columns of the dataset one name per line? In the meantime, I’ll be typing [the prompt]… I am so lazy.”4.2.2.ChatGPT opaquely makes assumptionsBased on the context shared with ChatGPT, it made its own “mental model” of the data. Participants were often enthusiastic about the semantic capabilities of ChatGPT, and how it could infer domain knowledge about the data solely using the column names. P4 leveraged these capabilities and prompted ChatGPT to provide a data dictionary. Several participants (P5, P7, P10, P12—P14)  were amazed when ChatGPT could understand the data domain sufficiently to convert the extracted‘RESPONSE_TIME’column fromtimedeltaformat to seconds—which was the most appropriate unit of measurement, since emergency medical services are likely to be dispatched quickly. However, despite the semantic abilities of the LLM, participants struggled with several scenarios, and had to re-assess ChatGPT’s understanding of the task and the data. We observed several instances across every task, where lack of adequate context in the prompt led ChatGPT to make assumptions about the data and its format.Several participants (P1, P5, P6)  received incorrect code generations for T1 (datetime typecast) , where ChatGPT assumed the string to be of the format‘%d%m%Y %H:%M:%S’(e.g., 31-07-2020 23:59:51)  instead of‘%Y-%m-%dT%H:%M:%S.%f’(e.g., 2020-07-31T23:59:51.000) . This assumption led to generation of the following code snippet to typecast the string column to datetime format, resulting in a run-time error:⬇>>>pd.to_datetime(date_str,format=’Error:timedata’2020-07-31T23:59:51.000’doesnotmatchtheformat’Upon providing ChatGPT with the error message, or an example string from the column, it re-generated the snippet without theformatparameter, allowing successful execution, aspandascan now correctly infer the format of the input datetime string itself.Next, as part of T2 (split using delimiter) , participants had to work with the‘INCIDENT_DESCRIPTION’column, containing the borough, incident dispatch area code, and zipcode, separated by semicolons. However, few rows in this column did not contain the zipcode. This led to the column having two formats: (1) ‘‘<borough>; <incident_dispatch_area_code>; <zipcode>’’, for example: “Brooklyn; K7; 11211”, and (2) ‘‘<borough>;<incident_dispatch_area_code>’’, for example: “Manhattan; M3”. Participants generally did not discover this data-formatting inconsistency by themselves, since encountering such issues requires exploratory data analysis and on-ramping. As a result, ChatGPT often did not get visibility into this data quality issue through participant prompts (P1, P2, P5, P11) , and it generated non-executable code for splitting the column under the assumption of having clean and consistent data, which contains all three splits in each row:⬇#splittheINCIDENT_DESCRIPTIONcolumnbythe;separator>>>df[’Borough’]=df[’INCIDENT_DESCRIPTION’].str.split(’;’) .str[0]>>>df[’Precinct’]=df[’INCIDENT_DESCRIPTION’].str.split(’;’) .str[1]>>>df[’ZipCode’]=df[’INCIDENT_DESCRIPTION’].str.split(’;’) .str[2]IndexError:listindexoutofrangeOn the other hand, some participants were able to receive correct code generations from ChatGPT owing to their relatively generic prompts. In such cases, where the generated code ran without errors, participants never discovered the underlying data quality issue—leading them to be unaware about missing values in the newly derived column for zipcode (P1, P3–P4, P7–P14) . P4 expected similar behavior for other data smells and quality issues, such as“typos and capitalization inconsistencies in categorical variable columns,”and expressed theneedto be able to quickly discover and convey such data quality issues to the LLM.In tasks T3 (feature selection)  and T4 (heatmap plotting) , ChatGPT often made assumptions about the data type of columns solely based on the column names. That is, when participants did not share sample rows, or the column data types, ChatGPT used the column names to make a semi-informed guess about the data type. This led to run-time errors in T3 while trying to train a model (P5, P12) , and visualizations that were not meaningful for any insight-finding in T4, such as scatter-plots for categorical and boolean variables (seen in Figure4)  (P2, P3, P10) .Figure 4.ChatGPT suggests a highly skewed boolean variable for a scatter plot. P3 says that,“Should I choose another column? […] This being categorical and skewed isn’t useful for the visualization. So this is where I think [ChatGPT] only looked at the column names, and not the actual values in the data I shared.”4.2.3.Misaligned expectationsResponses generated by ChatGPT are highly temperamental to the phrasing of prompts. Participants form a mental model of the nature of responses they would get from ChatGPT, in terms of whether the response will contain code or not, and the libraries, packages and APIs it would use in code generation. However, participants report that the presence or absence of specific words in the prompt can lead to unanticipated responses from ChatGPT.For instance, not mentioning “pandas” or “dataframe” in the initial prompt to ChatGPT led to code generations with APIs belonging to standard python libraries, instead ofpandasAPIs (P1, P3, P12) . P12 reflected on the issue and realized that they“didn’t talk about pandas, that’s why it didn’t bring up to_datetime()  over here and suggested strptime()  instead!”Convert datetime string like “2020-07-31T23:59:51.000” to datetime object in python (P12) ⬇>>>fromdatetimeimportdatetime>>>datetime_str=’2020-07-31T23:59:51.000’>>>datetime_obj=datetime.strptime(datetime_str,’Similar challenges surfaced as participants solved T3 (feature selection) . Owing to the subjectivity of T3, participants often mentioned that ChatGPT cannot be completely trusted for suggesting techniques and steps since it lacks domain expertise (P1–P5, P7, P8, P11) . Participants observe that ChatGPT’s responses are almost never aligned with their domain knowledge, as it misses necessary data pre-processing steps, or provides technically incorrect responses. In one instance, as P5 went about prompting to train a model that predicts the‘RESPONSE_TIME’, ChatGPT generates code to perform training without standardizing column values. P5 emphasized the importance of standardization and suggested careful examination of all generated code:“One thing it has missed is standardizing data. It is very important for us to scale the data to check for outliers and understand the data distribution.”In other instances, ChatGPT suggested selecting the features:‘INCIDENT_DATETIME’and‘FIRST_ON_SCENE_DATETIME’, which were used to compute the‘RESPONSE_TIME’column. Participants expressed that this is technically incorrect since the target column was arithmetically derived using the features selected by ChatGPT (P2, P3, P5, P13, P14) ; and that the‘FIRST_ON_SCENE_DATETIME’information will not be available in a pragmatic setting—making it impossible to predict the‘RESPONSE_TIME’(P8) .Lastly, participants often found ChatGPT’s responses to be“excessively verbose and unnecessary”, especially for natural language (NL)  code explanations. Participants wanted ChatGPT to omit explanations for APIs that they were already familiar with, and required NL explanations for only those unknown or uncommon APIs and parameters. Several participants held the long NL explanations for code generations as responsible for the prolonged inference time, that is, the time taken by ChatGPT to produce a complete response. P7 said,“When it was generating the explanation content, I felt like it is time consuming. Just give me the code. I was being impatient.”We observed that all participants skip reading any NL explanations when the response included a code-snippet, and begin with copying the generated code to adapt it to their notebook. P4 said,“I guess it takes a bit of time to finish running. It would just be easier for me if it just outputs the code, and then I can ask follow up question on what doesin the first line of the function do.”4.3.Obstacles in Leveraging ChatGPT’s ResponsesWe now present the challenges faced by participant inusingthe code generated by ChatGPT. Participants had to edit the generated code in certain ways to successfully adapt it to their notebook contexts. Since the chat instance with ChatGPT was completely detached from the notebook, unless participants explicitly provided variable and attribute names in their prompt, ChatGPT created dummy dataframes and attribute names and included them in the generated code. Thus, the adaptation process frequently involved copying the generated code and replacing the generated variable and attribute names with the actual identifiers. These trivial, yet additional, steps in the adaptation process can be fixed by augmenting the user prompt (i.e., scaffolding the user’s NL prompt with a prefix, and/or suffix to provide relevant context to the model)  with the name of the dataframe, attribute names, and any other variables involved. However, we identify several other repeated obstacles across participants, which either require more involved prompt augmentation, post-processing of the generated code, or interface design interventions for easier adaptation. Figure5details the nature of code adaptation and validation activities and their frequencies across tasks, with managing import statements and removing additional code generated for data ingestion being the most frequent adaptations.Figure 5.Frequency of code adaptation and validation activities performed by participants for each task.4.3.1.Generation of repeated codeA common adaptation the participants had to make to the generated code was removing lines that were already present in their notebook. This mostly included (1)  repeated library import statements—especially import pandas as pd—which were generated with every code response generated by ChatGPT, (2)  repeated data ingestion using the
pandas.read_csv()  API,
or by hard-coding a sample of rows as a python dictionary while using the pandas.DataFrame.from_dict()  API, and (3)  performing exploratory data analysis using pandas.DataFrame.describe()  and other APIs. Accidentally executing this code sometimes led the participants’ dataframes to be over-written by the sample values generated by ChatGPT (Figure6) , causing them to re-run all notebook cells from the beginning (P1, P9) .Figure 6.P1 accidentally executes data ingestion code generated by ChatGPT, causingdfto be over-written by a dummy dataset.4.3.2.Data and notebook management preferencesWe now discuss a few issues related to data and notebook management preferences:Managing generated code in computational notebooks.We observe that data scientists have implicit preferences for code management, aligned with common practices in the use of computational notebooks, such as having all import statements in a single cell at the beginning of the notebook, splitting functions and data transformations into cells to view intermediate results, interleaving of code with markdown cells to segment the notebook, and so on.ChatGPT often generated big chunks of code involving multiple data transformations as a single code-block in its responses. Participants often broke-down such code chunks into meaningful cells (P2, P3, P5, P7, P8, P14) . This helped them in isolating lines of code into steps, and execute them sequentially to observe any execution errors. This also enabled them to view the dataframe in between the data-transformation steps, while ensuring that the generated code matches their intent. Further, ChatGPT bundled library import statements with other data transformation/visualization code in the same block. This frequently required participants shifting the new import statements to the cells at the top of their notebook (P3, P5, P7, P12, P13) .Data scientists have also been studied to commonly author exploratory code; or use the fluent programming pattern, composing multiple transformations into a chain. When participants mentioned the phrase “write a function” in their prompts, it led to generation of functions for data transformation tasks, with excessive number of parameters and intermediate steps, contrary to the typical exploratory and fluent patterns. P3 and P7 refactored such code generations to obtain chain of transformations for the target column.Data transformation and management.Participants often had preferences on how data transformation must be performed, such as—deciding to retain the parent column(s)  after having derived newer column(s)  (P3, P8, P9, P12) , creating temporary dataframes for data transformation tasks (P12, P13) , and deciding on appropriate techniques for data imputation (P6) . Adaptations had to be made to the generated code to take these transformation preferences into account. For instance, when P6 prompted ChatGPT for T2 (split using delimiter) , it generated an additional line of code to impute missing values in zipcode with‘‘NULL’’. P6 said,“No, no, I would not have preferred filling them with ‘NULL’ because right now probably I still need to understand these columns more, so I may just remove null values afterwards,”and subsequently removed the line of code.4.3.3.Code validationParticipants emphasized on the importance of thoroughly validating every operation performed using LLM-generated code. P7 mentioned that validation is even more essential as“ChatGPT’s confirmatory and affirmative language in responses—like ‘Definitely! Here’s the code you need’—is extremely deceiving because it doesn’t really know about my data.”Though code validation took the least amount of time when compared to other activities (prompt construction and code adaptation)  (Figure2) , each task required participants to verify the functioning of the generated code in different ways. In some cases, ChatGPT self-generated the code snippet(s)  required to validate the data transformation. In other cases, the participants wrote additional code snippets, or manually inspected the data to validate changes. For instance, to ensure that‘INCIDENT_DATETIME’has been typecast topd.datetime(T1) , participants used df.dtypes and df.columns. To check for newly derived columns after splitting‘INCIDENT_DESCRIPTION’(T2) , participants printed and manually inspected specific columns. Some participants also checked for missing values after using df.value_counts() . On the other hand, for T4 (heatmap plotting) , visual verification of the generated plot would be sufficient to validate the code generated by ChatGPT.4.4.Strategies for Prompting and Alternate ResourcesWe now discuss strategies employed by data scientists to overcome communication and code-adaptation obstacles, along with scenarios where they prefer to leverage other resources (e.g., web-search, documentation, and previously written code) .4.4.1.Techniques for prompt constructionAlongside data context, participants also brainstormed prompting techniques that are relevant for ChatGPT to solve the problem. We observed frequent use of one-shot and few-shot prompting for T1 (datetime typecast)  and T2 (split using delimiter)  (P1–P4, P6, P8, P10, P12, P14) ; chain of thought prompting for T3 (feature selection) , to elicit reasons behindwhy, andon what basisChatGPT suggests features (P1, P3, P10) ; and making ChatGPT assume the role of a data scientist to receive expert suggestions, by prefixing each prompt with“You are a data scientist”(P13) .To avoid the frequent context-switching between prompt-writing (in the ChatGPT browser-window)  and data context gathering (from the notebook/Excel) , we observe that participants focused on creating a prompt skeleton while embedding “placeholders” for data context that must be fetched from the notebook (discussed in section4.2.1) , and pasted into the prompt (P1, P2, P5, P7, P8, P10, P14) . For instance, P2 created placeholders (enclosed in square brackets) , to replace them afterwards with the required column name and sample value:I have a dataframe with a column called[col name]. This column has data separated by ;, here is an example:[sample value]. Can you split this column? (P2) This strategy was commonly used by participants to complete writing the natural-language query in one go, instead of their flow being disrupted by going back and forth between ChatGPT and their notebook to fetch relevant data context. Prompt drafts with placeholders also offered a low cost in switching out pieces of data context while assessing their importance for solving the problem at hand.4.4.2.Scaffolding with domain expertiseParticipants often expressed that ChatGPT lacks“any actual understanding of the data”(discussed in sections4.2.2and4.2.3) . To overcome these obstacles, we observe limited interactions where data scientists successfully guide the LLM with their domain expertise. P10 and P14 narrowed down ChatGPT’s output space by iterative refinement of the data context included in their prompts.To obtain meaningful visualizations (T4) , P10 refined their prompt to only include names offloatcolumns.I would like to plot out response time vs:[List of ‘float’ attribute names]Can you write python code that does this in seaborn? (P10) Similarly, to prevent ChatGPT from suggesting timestamp-based features (T3) , P14 provided only names of columns that do not contain the sub-strings ‘TIME’ or ‘ID’ in them.I want to build a GBM using sklearn. The X variables are [List of attribute names not containing sub-strings ‘TIME’ or ‘ID’], target variable is “RESPONSE_TIME” which is timedelta (P14) Surprisingly, we also observe an instance with P5, where ChatGPT asks a “clarifying question” in response to the user provided prompt:P5: Consider this dataset to be the dataframe which we are going to use for the prediction of the response time: [Top 10 rows with data header]ChatGPT: What is response time in this dataset and how is it calculated?P5: Response time is calculated as below: FIRST_ON_SCENE_DATETIME – INCIDENT_DATETIMEThis interaction made P5 feel assured that ChatGPT“is able to understand the prompt”, and“make sense of the data context”. The clarifying question enabled P5 to provide context that they missed in their initial prompt, and allowed for complete intent expression instead of generating a hallucinated response.4.4.3.Choosing alternate resources over ChatGPTThere were several factors that lead to a preference to use alternative resources over ChatGPT:Data privacy concerns.Participants reflected on the importance of maintaining privacy for sensitive and personally identifiable information (PII) . P2 was hesitant about using any confidential business data with ChatGPT and said,“Do you want me to use normal ChatGPT? And we are not using that for any sensitive data, right, because I don’t want to share any private data with ChatGPT.”Contemporary tools for data might inherently include data context in their prompt, to scaffold user queries. For this reason, participants expressed interest in controlling and reviewing any data context or information being passed on to the LLM. P3 expressed concerns over having their data become part of the model’s training and being presented to a different ChatGPT user as a result to their queries. Participants were comfortable with using enterprise ChatGPT only in accordance with company policy (P2, P7) .Re-using previously written code.Data scientists mentioned that they periodically receive new batches of data, for which they must re-run analyses and visualization reports, and re-train models (P3, P5, P6, P7, P11) . In such cases, data scientists are already familiar with the structure of the data, and they reuse previously written code for data pre-processing and cleaning. P11 mentioned,“We pretty much just copy and paste canned code for every project. A lot of variations can be created on the charts, but 80% of it would be repeating.”The need to author new code for pre-processing or analytics in such scenarios is rare.Using web search and documentation.Participants described that using LLMs can normally help them save time over making a web search, such as when“asking for generic directions and steps, saving me the time to go through 4–5 blog posts on the web,”and“asking for code in my specific context, which needn’t be the exact issue discussed on Stack Overflow”(P4, P11) . In these cases, they find the LLM’s responses to be highly contextualized to their data and previously applied steps, and“less noisy”compared to web-pages. On the other hand, P7 found it easy to make a web search, requiring only“8–10 key words,”whereas they must write“longer queries and be more specific with ChatGPT just because it allows me to do that”. P11 also mentioned that the use of LLMs will“restrict me from learning new techniques or discovering a breadth of alternate approaches. Forums like Stack Overflow support these needs better.”
body
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
4.Observations
2310.16164
5.Survey ResultsOur qualitative insights from the task-based study (Section4)  identify numerous obstacles that data scientists face in conversing with ChatGPT, and subsequently, in acting on the generated responses. The survey results help us understand how well do these obstacles generalize to a diverse set of data science tasks, and to a broader community of data scientists. The survey respondents report experience with using LLMs for varied data science tasks spanning across project timelines, including synthetic data generation, data wrangling, pre-processing, data labelling, exploratory data analysis, insight finding and summarization, hypothesis generation, training various models, outlier detection, and generating plots.Figure7shows the distribution of responses for each ‘Agree’/‘Disagree’ question. Results indicate that data scientists find sharing of relevant data context to be important (86%) , as well as tedious (59%) , echoing findings from Section4.2.1. 62% data scientists agreed that prompting ChatGPT for columns with mixed formats is challenging, reflecting the issues discussed in Section4.2.2. These challenges lead to communication breakdowns between data scientists and ChatGPT—causing them to make repeated prompt refinements (92%)  (Section4.1.2) . Respondents also indicate that LLMs do not have sufficient domain expertise in data science (60%)  (Section4.2.3) , and that generated code requires several modifications before it can be used in their notebooks (87%)  (Section4.3) . Lastly, code validation respondents have a split opinion on whether it is easy to verify code generated by ChatGPT (48%)  (Section4.3.3) .Figure 7.Distribution of survey responses for ‘Agree’/‘Disagree’ questions. Response Counts: D (Disagree) ; NR (No Response) ; A (Agree) .We manually inspected responses to look for any other challenges encountered by data scientists in conversing with LLMs for data science tasks. Respondents re-emphasized difficulties in“forming well scoped prompts”and decidingwhatcontext to include. They reflected on the need for repeated prompt refinements due to ChatGPT’s lack of domain expertise. A few responses brought up the“instability”in ChatGPT’s responses, wherein it generates drastically different answers despite sending the same prompt. We did not encounter any new obstacles in response to this question.
body
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
5.Survey Results
2310.16164
6.Design RecommendationsIn this section, we discuss the implications of our findings and identify the ways in which they could be adapted to various data-science environments and to broader contexts.6.1.Recommendation I—Provide preemptive and fluid context when interacting with AI assistantsFrom our studies, we found that participants spent a significant amount of time constructing prompts (Section4.1.2)  and gathering and expressing their context (Section4.2.1)  to ChatGPT. Nearly 45% of the prompts included portions of data that they either manually entered, or wrote additional code snippets to obtain; however, participants did not enjoy being“slowed-down”in fetching this data context for every interaction. Thus, data scientists may need help in providing context about their environments, and we should support mechanisms to either preemptively provide implicit context, or provide fluid interactions that allow data scientists to easily refer to fragments of their work in conversations with AI assistants.In data visualization, classic techniques such asbrushing, enable a user to select and highlight specific data points, allowing the user to use those data points as a filter, or as a point of interest for further analysis. Similarly, providing mechanisms for supportingdata brushingin data-science environments would allow users share their context with agents more directly. For example, in T2, participants had to split text into three columns. However, some rows had inconsistent formatting making the operation suggested by ChatGPT error-prone. With data brushing (Figure8) , participants could mark an example cell with a problematic entry and ask an AI assistant to help split the rows, while dealing with outliers such as the marked cell. Similarly, allowing inline code snippets that are expanded, or referring to entities within the data (@INCIDENT_RESPONSE) , allow less tedious and more fluid expressions of context when prompt writing.Figure 8.Design concept—Using data brushing to mark examples (selected in green) , and inconsistent entries (selected in red) .Despite these aids, some context, especially providing a summary and schema of the data would be tiresome. Furthermore, users had no clear understanding of what information should be provided, and even for researchers, this question is still being explored. However, recall Grice’s maxim of relevance, which states that a successful conversation requires not too little or too much information be provided. A simple intermediate measure would be to provide a preemptive summaries of data characteristics. This would help reduce the chances that ChatGPT makes assumptions about data format, which may lead to incorrect code or longer conversations (Section4.2.2) .6.2.Recommendation II—Provide inquisitive feedback loops and validation-aware operationsFor transactional tasks, such as T1 (datetime typecast) , ChatGPT often provided satisfactory responses. Unfortunately, open-ended tasks, such as T3 (feature selection) , required a more involved conversation with ChatGPT.
Initial responses from ChatGPT were often “excessively long” and were not particularly helpful for the task, likely because it violates the fundamental HCI principle of progressive disclosureand overwhelms the user with details.
As a result, participants were uncertain how much value would be gained by continuing the conversation or what further information would help (Section4.2.3) . In other cases, participants had to make multiple clarifications and carry out editing operations across several iterations in their data-science environments. This task fragmentation often lead to errors, for example, because ChatGPT always included code to ingest the dataframe—regardless of the stage of conversation—some participants accidentally executed it, leading to their data being over-written (Section4.3.1) .One mechanism to help participants have more productive conversations with AI assistants is throughinquisitive feedback loops, where the system guides the user in expressing what they need by proactively asking the user a series of questions, rather than waiting for the user to know when they need help and how to get it.
Moreover, participants in our study could have been aided just by ChatGPT asking clarifying questions to resolve ambiguities instead of making assumptions.
For example, one participant (P9)  was able to get more useful advice from ChatGPT, after suggesting that they could reframe the problem of feature selection: “I think I am more interested in finding the feature importance so I’ll prompt it for something like a random forest”. In this case, a user could be prompted with questions that helped the user think through which high-level strategy they wanted to use before giving all the details.Tools for supportingrefactoring, behavioral-preserving transformations, can be one approach for assisting data scientists who must make repeated edits of similar nature, or adapt generated code snippets to their local notebook contexts (Section4.3) . For instance,GhostFactoris a light-weight program analysis tool with refactoring detection algorithms that can identify incomplete changes to code. Simple approaches can be used for enabling automated post-processing and human-in-the-loop refactoring of generated code snippets based on observed nature of edits—deletion of repeated code,
notebook and data management preferences, and validation.6.3.Recommendation III—Provide transparency about shared context and domain expertise slotsposit that lack of transparency about the shared context, and lack of control in refining the context selections can leave users in a confused state. Our participants echo a similar consideration, wherein they expressed the urge to“get to the first response as fast as possible, but have the ability to look at what context was provided and edit it if I need to refine my prompt”(P3, P4, P7, P12) . Furthermore, participants also wanted to be able to provide more context that reflected their personal expertise and experience (Section4.4.2) .Even with automated context management or prompt augmentation, it is important to consider how that context is ultimately shared. For example,
by having sufficient transparency into the shared context can also enable data scientists to ensure that sensitive data does not leak into the prompt.
Data-science environments interacting with AI should consider mechanisms for
ensuring a two-way knowledge transfer between data scientists and AI. For example, an AI assistant chat interface could integrate acontext panel, a dedicated, editable grid that mirrors the AI assistant’s and user’s assumptions about data and tasks.Finally, users should be able to inspect and modify the context to provide their domain expertise or reflect their analysis style (e.g., always perform data scaling and impute missing values prior to training a model) . One step in this direction is the recent addition to ChatGPT allowing for custom instructions666https://openai.com/blog/custom-instructions-for-chatgpt.
body
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
6.Design Recommendations
2310.16164
7.Threats to validityAll study participants had prior experience with using ChatGPT. However, most participants (9of14)  had not used LLMs for solving data-science tasks. If they had been exposed to tasks of similar nature prior to the study session, their interactions with ChatGPT could have been different. We took this into consideration to ensure the quality of responses in the confirmatory survey, and asked all respondents to share at least one scenario where they used an LLM-powered tool for a data-science task. This enabled us to filter out respondents who did not have any prior experiences to provide informed responses.We selected a diverse set of data-science tasks representative of data pre-processing, cleaning, feature selection, and plotting. However, the study duration limited the breadth of tasks that participants could have performed, such as data discovery, capture, and machine learning. Nevertheless, our survey responses (Section5) , from a broader community of data scientists with diverse experiences, confirm the generalizability of our findings. Future work can involve empirical studies that measure the performance of LLMs on assorted data-science tasks with datasets that have not been “seen” by the LLMs during pre-training. We presented the data-science tasks to all participants through verbal descriptions of the objectives. This may have inevitably led to priming, wherein the participants’ formulation of prompts could have been influenced by our description of the task.Browser-based version of ChatGPT version was updated as we conducted our task-based studies. P1–P9 used the March 14, 2023 version of ChatGPT, and P10–P14 used the March 23, 2023 version. This might have had a slight impact on study findings, though we did not observe any noticeable shifts in the nature of responses generated by ChatGPT for the study tasks. Lastly, we encouraged participants to think aloud as they solved study tasks, which could have added to the time taken in performing different activities. Our video annotations take this into consideration to only include timestamps where participants are actively working with the data, or with ChatGPT.
body
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
7.Threats to validity
2310.16164
8.Related WorkAI-assisted tools for data cleaning suggestions—such as Trifacta777https://docs.trifacta.com/display/SS/Overview+of+Predictive+Transformation, Data Wrangler888https://devblogs.microsoft.com/python/data-wrangler-release/, CoWrangler, AWS Glue DataBrew999https://docs.trifacta.com/display/SS/Overview+of+Predictive+Transformation, Salesforce Einstein Discovery101010https://help.salesforce.com/s/articleView?id=sf.bi_edd_prep_terminology.htm, AutoPandas, Auto-Suggest—have existed, but LLMs bring forth new opportunities for enhancing these tools further.
More recently, several commercial and open-source tools have emerged—DataChat AI111111https://datachat.ai/,
Anaconda Assistant121212https://www.anaconda.com/blog/anaconda-assistant-launches-to-bring-instant-data-analysis-code-generation-and-insights-to-users,
Databricks Assistant131313https://www.databricks.com/blog/introducing-databricks-assistant,
and Jupyter AI141414https://jupyter-ai.readthedocs.io/—which enable data scientists to access AI-powered chat assistants within their notebook (such as Anaconda, Databricks, and Jupyter notebooks) , and other alternate data-science environments (e.g., DataChat AI, SQL and file editors for Databricks, etc.) . The semantic abilities of LLMs, coupled with a chat interface, provides users with the agency to converse about their data, ask follow-up questions, and receive highly contextualized responses.McNutt et al.studied, through an interview study, the design space of code assistants in computational notebooks, but their focus is on the design space as they do not focus on data-specific issues that data scientists face while communicating their needs to the AI assistants (e.g., communicating the data context or domain knowledge) . A number of recent work conducted studies to understand the use of LLMs by software engineers and programmers. However, these mostly focus on the general understanding of AI-powered code assistants and do not particularly focus on data-science tasks, where communicating the data at hand to the assistant is a big challenge.There has been a recent line of work that try to make use of LLMs to automate data-science tasks and evaluate performance of LLMs on data-science tasks. This is a promising, but orthogonal, area of research to ours as we focus on human-in-the-loop data science where AI-assistants and human data scientists work as a team.
Similarly, there are work on designing and evaluating tools for end users, which is complementary to our work.Most relevant to our work is the study by Liu et al.on how users of a spreadsheet software articulate their question to an AI-powered tool and how a system they designed can better support this process.
Our work is differentiated by (1)  examining the fundamental interactions with a general AI tool like ChatGPT rather than a domain-specific AI tool embedded in a system, (2)  we studied professional data scientists rather than non-programmers, and (3)  they were specifically investigating the effect of co-editing a prompt by splitting it into natural language steps.
body
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
8.Related Work
2310.16164
9.Conclusion and Future WorkThe recent rise of AI-powered chat assistants gives hope of greatly increasing the productivity of data scientists as they are engaged in tedious data-wrangling tasks.
To understand the fundamental obstacles, needs, and design opportunities of data scientists using AI assistants, we conducted mixed-method studies involving a task-based study, interviews, and a survey with professional data scientists.
We found that participants faced two key sets of barriers: when communicating with ChatGPT and then when adapting ChatGPT’s response to aid their specific situation.
For example, participants spent considerable time collecting and formatting information to provide to ChatGPT, such as relevant regions of their dataset or descriptive statistics.
Furthermore, ChatGPT often made assumptions unbeknownst to the participant, thus leading to bugs in the code and additional steps to remedy.
When participants did get a useful response from ChatGPT, it required considerable effort to adapt the code to work with the rest of their code and to follow their style.
We provide key design recommendations for tools based on our study, such as the need to provide inquisitive feedback loops and validate-aware operations.
As a next step, we intend to operationalize these design guidelines into a working tool and assess its efficacy in alleviating the workflow challenges that data scientists encountered in our studies.
In conclusion, our work aims to function as a significant catalyst in democratizing data science via AI-powered chat assistants.
conclusion
Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
9.Conclusion and Future Work