Physical models have gender-specific effects on student understanding of protein structure-function relationships
Understanding how basic structural units influence function is identified as a foundational/core concept for undergraduate biological and biochemical literacy. It is essential for students to understand this concept at all size scales, but it is often more difficult for students to understand structure-function relationships at the molecular level, which they cannot as effectively visualize. Students need to develop accurate, 3-dimensional (3D) mental models of biomolecules to understand how biomolecular structure affects cellular functions at the molecular level, yet most traditional curricular tools such as textbooks include only 2-dimensional (2D) representations. We used a controlled, backwards design approach to investigate how hand-held physical molecular model use affected students\xe2\x80\x99 ability to logically predict structure-function relationships. Brief (one class period) physical model use increased quiz score for females, whereas there was no significant increase in score for males using physical models. Females also self-reported higher learning gains in their understanding of context-specific protein function. Gender differences in spatial visualization may explain the gender-specific benefits of physical model use observed.
Introduction<!>Curriculum and Study Sample<!>Conceptual Goals<!>Instructional Activities<!>Treatment groups<!>Performance Assessment<!>Student Assessment of Learning Gains (SALG)<!>Statistical Analysis<!>Assessment of our Initial Model<!>Assessment of our Design<!>Assessment of our Final Model<!>SALG<!>Discussion
<p>Undergraduate biology and biochemistry instructors have the challenge of helping students understand structure-function relationships, which has been identified as foundational/core concept for undergraduate biological and biochemical literacy [1,2]. It is essential for students to understand this concept at all size scales; yet understanding cellular processes has been identified as potentially "troublesome" content knowledge because it requires an expert level of understanding of spatial scale and dynamic interactions [3–5]. An expert level of understanding of these kinds of "threshold" concepts is achieved when learning is "transformative, irreversible, and integrative" [6,7]. For example, biochemists have identified the physical basis of molecular interactions as a key threshold concept because it requires an expert level of understanding of how molecular electrostatic properties ultimately determine interactions between biomolecules [8]. Understanding difficult threshold concepts requires accommodation. This means that the information cannot be simply assimilated into existing knowledge, but instead students must reorganize their knowledge and transform their thinking, allowing them to cross conceptual thresholds [5,9]. Therefore, it is important for instructors to provide students with effective curricular tools and learning environments as they grapple with these difficult concepts.</p><p>Biology and biochemistry instructors use a variety of external representations, such as pictures, animations, and physical biomolecular models of different complexities and size scales, in order to represent biomolecular structures, and the choice of representation used impacts student learning [10,11]. The type of representation chosen by an instructor should depend on desired student learning outcomes. Typical 2D imagery often depicts complex biomolecules as schematic blobs, which may be adequate if the goal is for students to understand a chronological sequence of molecular events, such as cell signaling pathways. However, for many learning outcomes, these schematic blobs may not suffice. These blobs may lead to "deceptive clarity"- images that allow for only a superficial understanding, which can lead students to believe they understand something that they actually do not understand [12]. Schematics often obscure basic principles that experts know and therefore infer from the image, and instructors may incorrectly assume that students are able to infer this information as well [13–15]. Complex biomolecular representations may offer students visual "desirable difficulties", which help overcome "deceptive clarity" by giving students an opportunity to engage with visual representations [12]. For example, more complex representations of receptor-ligand binding events which depict random, non-directional ligand movement instead of directed movement, leads to a deeper understanding of these dynamic molecular events [16]. Another study found that students who studied 2D molecular representations combining schematic blob images with realistic images based on Protein Data Bank (PDB) structure entries more accurately labeled novel PDB images than students who studied only with schematic blob images [17]. Additionally, realistic representations with many details do not negatively affect student performance on other assessments [17,18].</p><p>Scientific experts have high spatial visualization ability, which is the ability to understand the spatial relationship between objects and to mentally manipulate images [19,20]. Translating a 2D external representation into an accurate 3D mental model is identified as a key cognitive skill possessed by biochemistry experts, yet this is typically a difficult skill for undergraduate students to develop [11]. This is particularly true for females, who typically have lower spatial visualization ability than males [21–23]. It is important to support students' development of skills such as visualizing orders of magnitude and relative size and scale [10,11], as these visual literacy skills are fundamental for students to accurately interpret and learn from a variety of external representations. As such, it is important to identify curricular tools and/or activities that help students to think "like an expert" about biomolecules, presumably by helping students hone their spatial visualization skills of the molecular level. Specifically, what curricular tools enhance students' understanding of biomolecules as dynamic structural entities whose interactions determine key cellular functions?</p><p>Hand-held physical models are common in the chemistry classroom and multiple studies have documented their effectiveness, particularly for higher-order chemistry questions [24–26]. "3D-like" computer imagery, which allows students to manipulate and rotate an image on a 2D screen, has also been shown to improve student understanding of chemical representations [27]. The use of physical models and/or 3D-like imagery in biology and biochemistry classrooms is less well studied; however, a combination of physical molecular model use with computer 3D-like molecular imagery over several weeks increased student performance on high-order interview questions about molecular structure-function relationships [28]. Additionally, after a semester-long undergraduate biochemistry class, students rated physical models as the most helpful of seven learning tools for understanding molecular structure-function relationships [29]. However, little is known about the optimal learning environment for using 3D physical models and/or 3D-like imagery in a biology or biochemistry classroom, particularly details such as the impact of each tool alone, the number of exposures students should have to the tools, and the optimal time spent with the tools.</p><p>We are interested in identifying instructional approaches that help students to deeply understand molecular structure-function relationships (i.e., acquire an expert level of understanding). Specifically, we want to use student performance as evidence to identify instructional tools that help students develop a sophisticated understanding of molecular structure-function. In the present study, students used either hand-held physical models, a 3D-like molecular imaging tutorial (Jmol), both tools, or neither tool in a single discussion class meeting to supplement lecture material about the structure and function of a membrane-binding protein. We hypothesized that students using both physical models and 3D-like imagery would exhibit superior understanding of molecular structure-function relationships, as compared to students who used either tool alone, and that students using either tool alone would perform better than those using neither tool. Furthermore, because of the gender differences in spatial visualization skills [21–23], we examined gender differences in the effectiveness of these tools.</p><!><p>Curricular materials in this study were developed as part of the NSF-CCLI funded Connecting Researchers, Educators and Students (CREST) project. The University of Wisconsin-Madison is one of several institutional partners participating in the CREST project. UW-Madison Biology Core Curriculum (Biocore) honors undergraduate students worked with faculty researchers and educators to develop classroom materials, including hand-held physical models and a 3D-like molecular imaging tutorial. Biocore is a four semester undergraduate honors biology program that emphasizes group learning and the process of science [30]. Cell and molecular biology (Biocore 383) is the second lecture course in the program sequence. Students apply to the program as freshman and begin as sophomores, thus the majority of students in the cell and molecular biology course are sophomores. Students attend three 50-minute lectures and one required 50-minute discussion section per week. Lectures were taught by three faculty members to the entire group of 111 students and discussion sections were led by three graduate Teaching Assistants (TAs) for groups of 15-16 students. However, during the discussion section when our treatment was applied, the faculty member in charge of this courses segment, E.W.D., was present and led all seven discussion sections, with the section TAs assisting. Our research protocol was approved by the UW-Madison Education Research Institutional Review Board (protocol 2012-1060).</p><!><p>Prior to the start of the semester, we used a backwards-design research framework to align three conceptual goals (i.e. what do we want students to understand?) with learning outcomes (i.e. what do we want students to be able to know or do?), instructional activities, and assessments (Appendix A). E.W.D., M.A.H., and three senior undergraduate Biocore students who had previously taken Biocore 383 designed materials used to supplement E.W.D.'s lecture material on a membrane-binding protein, Cdc42-interacting protein 4 (CIP4). CIP4's role in regulating cell membrane shape was used as a model system to help students achieve key molecular structure-function learning outcomes. CIP4 is an intracellular protein involved in regulating plasma membrane structure, primarily during endocytosis, but also in protrusions in developing neurons [31]. Our three conceptual goals were for students to understand: 1) how biomolecular structure affects function, 2) that endocytosis requires several different proteins, and 3) that "one protein ≠ one function" but depends on where it is located in the cell and on what other proteins/lipids/small molecules are present in that part of the cell.</p><!><p>The following instructional activities were designed based on our three conceptual goals: a worksheet with information on CIP4, PDB images of CIP4, and questions for students to answer as they used the tools (Appendix B), an online clickable endocytosis landscape (http://cbm.msoe.edu/includes/modules/coatedPit/coatedPit.html), CIP4 hand-held physical models, and a Jmol molecular imaging tutorial (http://cbm.msoe.edu/includes/modules/jmolFBAR/template_1.html). The clickable endocytosis landscape incorporates an original watercolor by David S. Goodsell of an endocytosing vesicle which was created as part of the CREST Project (Fig. 1A). Students were able to electronically interact with 13 proteins in the landscape, thus providing a 3D-like view of each, both during and outside of discussion section. Four CIP4 physical models were designed to highlight different features of the protein and were built using 3D printing technologies (Fig. 2). All of the content contained in the physical models and Jmol tutorial was present in the worksheet and/or clickable landscape, including PDB images of CIP4, but the physical models and Jmol tutorial allowed for a 3D or 3D-like visualization, respectively, of the protein. E.W.D. began all discussion sections by providing a brief overview of CIP4's role in forming an elongated vesicle, using bananas to demonstrate how the CIP4 dimers orient (multimerize) around the vesicle (Fig. 1B). In sections using the physical models, he also used the physical CIP4 models in this overview. Students were then instructed to work their way through the worksheet questions, using all of the tools available. For sections using the physical models, each of the four CIP4 models was rotated between groups of 2-4 students so that each group had an approximately equivalent amount of time to examine each model. Students were encouraged to discuss the material with their peers. Students given access to the Jmol tutorial were instructed not to share the website address for the tutorial with peers outside of their discussion section.</p><!><p>In spring 2014, students were randomly assigned by discussion section to one of four treatment groups: physical models only, Jmol tutorial only, physical models and Jmol tutorial, and neither physical models nor Jmol tutorial. Each treatment was assigned to two different discussion sections, except the Jmol tutorial only treatment, which was only assigned to one section (Table 1). Again, all four treatment groups had access to the worksheet with PDB images (Appendix B) and the online clickable endocytosis landscape (Fig. 1A), and saw E.W.D. stack bananas that represented the multimerized CIP4 dimers (Fig. 1B).</p><p>Early in our analysis, we decided to remove Jmol as a treatment factor, leaving two levels of treatment: physical models and no physical models. The three primary reasons for the removal are as follows: 1) Based on classroom observations, we noted that, when Jmol was available, students spent little to no time using it during discussion section. This could be due, in part, to the fact that in those sections for which the Jmol tutorial was available, E.W.D. did not spend time introducing Jmol to the sections. 2) The online clickable landscape provided students in all four treatment groups an opportunity to manipulate 3D-like imagery to a limited extent, so we felt that we were not able to evaluate the effects of using 3D-like interactive computer tools alone. 3) Due to the imbalance in our design (there were 7 sections with four treatment combinations), the treatment of Jmol with no physical models could only be assigned to a single section. Thus, there was a strong possibility of a confounding between TA/discussion section and treatment effect. Therefore, our final models do not include Jmol.</p><p>Treatment group and sample size information are summarized in Table 1. Informed consent was obtained from ~95% of enrolled Biocore 383 students. Students who did not consent or were not present for either the CIP4 instructional activities or the assessment were not included in the study.</p><!><p>Baseline performance was assessed by the averages of exams 1 and 2, which were both taken before the CIP4 instructional materials were used. A quiz was designed to assess student understanding of how biomolecular structure affects function (Appendix C) and students took this quiz one week following treatment. The quiz was graded independently by R.F.L. and M.A.H., who were blinded to treatment group, using a detailed rubric with a maximum of 27 points (Appendix C). This 27 point rubric was designed to deeply assess a maximal level of student understanding; therefore a high score was not obtained by most students. There was a high degree of interrater reliability (Pearson correlation = 0.94), so the average of the two scores was used.</p><!><p>Students ranked how well instructional tools led to gains in their understanding of molecular structure/function relationships, as well as several other self-reported measures, using a 5-point Likert scale. Several open ended questions were also included. These were not scored but relevant comments are cited below. The survey was administered using Qualtrics Online Survey Software during the last week of class. Five students were not present for the SALG survey.</p><!><p>Data were analyzed using the software environment R (version 3.1.3) [32,33] . The two main performance variables measured were quiz score and SALG score. In the study, the two treatments were "applied" to discussion sections. The indicator of TA is also associated with section. All student covariates are unique to individual students. The proper analysis of a nested design of this type requires use of a mixed model with an error term for section and another error term for the individual student.</p><p>Initial analyses (initial model described below) indicated that the section error term had negligible magnitude and was insignificant. Thus, for the remainder of our analyses, we dropped the section error term and used standard multiple regression for analysis.</p><p>In our primary multiple regression analysis, we considered a number of covariates in addition to the treatment effect. These were TA, student gender, average of exams 1 and 2, GPA, and enrollment in a related lab course. We also considered an interaction between treatment and gender. Backwards elimination was used to remove non-significant terms (p >0.10, for instance GPA was removed). R squared values are reported for completeness. In order to directly determine the effect of physical models on each gender, the model was fit two ways: one with females as baseline, and the other with males as baseline.</p><!><p>Our initial mixed model analysis of quiz score indicated that the error term for section had zero variance. A formal likelihood ratio test (not shown) confirmed that the null hypothesis of zero section variance could not be rejected. In addition, in examining the residuals of the model with section excluded, we grouped the residuals by section and found no patterns that could suggest the importance of a section effect. Thus, as noted above, our primary analysis relied on multiple regression with no nested effects.</p><!><p>Although we decided to remove the Jmol treatment factor due to design issues discussed above, we carried out a preliminary multiple regression analysis with this variable included in order to fully explore the data and assess this decision. There was a Jmol treatment effect of −7.3 points and an interaction between the Jmol and physical model use, accounting for an increase in 5.9 points, after accounting for significant covariates (Table 2). This indicates that students who did use the Jmol tutorial, but did not use physical models, had lower quiz scores. However, this group is composed of a single discussion section, and because of the possible confounding between the treatment group and TA/discussion, we do not have confidence in these results. This supports our decision (presented above) to exclude Jmol as a treatment factor based on design.</p><!><p>The primary multiple regression analysis indicated that there was a treatment effect, a gender effect, and an interaction between the two (p<0.10) after accounting for significant covariates (Table 3; Fig. 3). One of the covariates, the average of exams 1 and 2, was positively related to quiz score (p=0.004), and the TA "A" corresponded to lower quiz scores than the other two TAs (p=0.06). There were no gender differences in performance on exams 1 and 2, using TA as a covariate (data not shown).</p><p>The students who did not have access to physical models had a mean quiz score of 9.1 points (out of 27 possible points), and the group with physical models had a mean of 12.1 points For students who did not use physical models, males without physical models scored approximately 2.5 points higher than females. For females, physical models significantly increased quiz scores by about 5 points (p=0.001), while the increase in male scores was only about 1 point and was not significant (p=0.60, Fig. 3).</p><!><p>We used the final model described above to analyze SALG data. The multiple regression analysis indicated that there was a treatment effect, a gender effect, and an interaction between the two (p<0.10) after accounting for significant covariates (Table 4; Fig. 4) on self-reported gain in the understanding in response to the statement "My understanding that certain proteins may have context-specific functions in different cell types". Similar to quiz performance, this effect was specific to females (Fig. 4, p=0.006) while for males it was not significant (p=0.98). There were no significant effects of physical model use on any other survey questions.</p><!><p>We found that short term use of hand-held physical models improved female students' ability to predict how specific changes in biomolecular structure affect function, as measured by quiz score. For students who did not use physical models, there was a trend for males to perform better than females, while physical models increased quiz scores in females such that they performed equivalently to their male peers. Interestingly, females also reported larger gains in their understanding of the context-specific function that proteins can have in different types of cells. For students who did not use physical models, there was a trend for males to rate their gains as higher than females, while females who used the physical models reported learning gains that were equivalent to those of their male peers.</p><p>In addition to the physical models, students had numerous other tools to support their achievement of the three learning outcomes. All students had access to the clickable endocytosis landscape, which was interactive in that it allowed students to electronically manipulate 3D-like protein images. Students' comments about these other curricular materials on the open ended questions on the SALG survey were generally positive. For example, one student wrote that, "The cellular landscape was extremely helpful in visualizing what was going on." Some students, however, pointed out that the static nature of the physical models was a key limitation in their understanding of dynamic inter-molecular interactions: "The handheld molecules were not helpful because you could not see how it interacted with the other proteins, since it was just the F-BAR molecule". Indeed, while students with higher spatial visualization ability learn better from static images than students with low spatial visualization ability, dynamic animations have been shown to help students with lower spatial visualization skills compensate [34]. In the current study, the worksheet with PDB images and the 3D-like clickable landscape were sufficient tools only for males, suggesting that there is something specific about the physical models that enhanced understanding in females.</p><p>Models could provide a mental scaffold for students to better visualize molecular structures (although only significantly for females in our study), and could also actually improve spatial visualization ability. The superior gain in understanding of molecular structure-function relationships documented in this study for females who used physical models may be due to the models helping females better visualize the 3D structures represented in the 2D images of molecular structures. Males tend to have higher spatial visualization skills than females [21–23] and the physical models may help females overcome lower spatial visualization ability by providing them with the 3D structure, thus providing a mental scaffold to learn the content related to this structure. This idea is supported by the correlation between spatial visualization and students' learning, particularly in chemistry [35,36]. Wu & Shah (2004) hypothesize that "manipulating these models could help students understand the underlying concepts of visual representations" (p. 478).</p><p>In addition to providing a 3D representation for low spatial ability, physical models may also directly improve spatial visualization ability in other contexts. The use of physical chemistry models improved spatial visualization ability and performance on high Bloom's taxonomy questions [24]. Furthermore, students who have opportunities to hold and manipulate hand held chemistry models perform better on chemistry concept assessments, even on assessments that do not directly relate to visualizations [36]. These findings emphasize the importance of identifying and providing students with tools that allow them to improve their spatial visualization skills; however, it is not known whether the one-time use of physical models in the present study improved spatial visualization skills in female students.</p><p>Interestingly, the difference between males and females in their self-reported learning gains to the SALG question "My understanding that certain proteins may have context-specific functions in different cell types" mirrored the differences found in quiz performance, when grouped by exposure to physical models. Perhaps even one-time exposure to physical models provides students, particularly females, with a memorable mental scaffold on which to organize their conceptual understanding at the molecular level. Student reflections from the SALG survey provide some evidence for this assumption; one student wrote, "The physical models don't necessarily explain it better but they help to recall the information." This suggests that physical models help students to create lasting mental representations at the molecular level.</p><p>When instructors use learning tools such as physical models, they should be mindful of assumptions they make about students' comprehension of external representations of molecular structure. Students need basic prior knowledge and visualization abilities when interpreting external representations, particularly complex representations based on PDB images [35,37,38]: "visualizations benefit those learners who have high spatial abilities more than those who have low visuospatial abilities" (Wu and Shah 2004 p. 485). It is important to note that the students in this study were honors biology and biochemistry students, so even if there was a gender difference in spatial visualization ability, most students presumably had high spatial visualization ability compared to a typical undergraduate biology population. Our findings suggest that in these honors students, the physical models were accessible to both male and female students.</p><p>Cognitive Load Theory (CLT) may also provide insights into the benefits of physical models to student learning. An understanding of cellular processes has been identified as potentially "troublesome" content knowledge because it requires an expert level of understanding of spatial scale and dynamic interactions [3–5]. According to CLT, experts use complex, long-term memories to greatly expand the capacity of their working memory [39]. Physical models may facilitate students', particularly female students', assimilation of complex biomolecular structures into organized, long term memory units or "schemata", thus reducing cognitive load when learning about new functions for familiar biomolecular structures.</p><p>There were several limitations of the present study that should be noted and improved upon in future studies if possible. First, because of the problem with the imbalance of discussion section, lack of time for students to use Jmol in discussion section, and other study design flaws, we were not able to examine the effects of the 3D-like imagery provided by the Jmol tutorial. Additionally, the approach of assigning treatment to discussion section is limiting from a design perspective. However, due to the nature of the instructional tools and importance of group work in the Biocore program, this could not be avoided. E.W.D. attempted to teach the same concepts in the same manner in each discussion section, but because of student questions, he provided unique clarifications and examples for some discussion sections. Despite these challenges, we still saw a significant effect of physical model use on quiz score and self-reported understanding of context-specific protein functions in females. It should be noted, however, that the generalizability of these findings is limited, as they represent data from only one semester of an honors undergraduate class. We are carrying out follow up studies to investigate the relationship between spatial visualization ability, gender, and use of physical models on student learning. Additionally, future studies should investigate whether these findings are reproducible and perhaps more pronounced, in non-honors and non-majors courses.</p><p>Future studies are needed to further inform our understanding of curricular tools that help students understand molecular structure-function relationships. Undergraduate biology and biochemistry instructors would benefit from investigation of the following questions: Would multiple exposures to a variety of physical models and/or interactive 3D-like images improve students' ability to translate 2D molecular images into accurate 3D mental models? Would this improved spatial visualization ability help students understand molecular structure-function relationships at a deeper level, so that they achieve a more expert-like appreciation for the dynamic molecular interactions implied in typical 2D external representations? Are the benefits of using interactive 3D-like computer imagery equivalent to the benefits of physical models, or is a combination of these two tools optimal? Researchers should keep in mind that evidence addressing these and other related questions may very well be gender-dependent.</p><p>In conclusion, we found that a single exposure to hand-held physical models improved the understanding of protein structure-function relationships in female students, increasing quiz performance and self-reported understanding to levels equivalent to those of their male peers. These findings add to an invaluable and growing body of literature on identifying and closing gender gaps in STEM.</p>