We have completed the pilots of the ICaP common assignments. In the Fall of 2017, ICaP administrators in conjunction with the grad student-run Pedagogical Initiatives Committee (PIC) began developing six different common assignments to trial run in English 106 and 106-I courses during the Spring 2018 semester. This was a decision made largely in response to the Council of Writing Program Administrator’s (CWPA) external review of our program a year prior, which suggested a common assignment could be a good way to introduce more consistency to our diverse syllabus approach system while at the same time preserving instructor freedom.
Since then, a team led by and consisting entirely of graduate students across the English department has collectively developed and taught, as well as rated and analyzed samples of, six different common assignments piloted by 39 instructors and completed by more than 780 Purdue composition students. From the outset, this project strived to be a grassroots, bottom-up effort so that those with the most skin in the game had a seat at the table. At every stage of this project, we tried to engage with grad students and lecturers in order to cultivate a localized assessment initiative attuned to the actual experiences of ICaP instructors.
We learned a great deal from these pilots, and we have taken that knowledge into account to make evidence-based decisions about changes to the common assignment options going forward. Given that participation was voluntary for these pilots, our findings should be interpreted with a certain amount of caution. More specifically, there are issues with sample size and randomization, which impact generalizability. Nonetheless, the data we did gather and analyzed helped to start assembling the larger picture, and we used appropriate caution when making final decisions. As the common assignment component of ICaP becomes mandatory this semester, the assignments have been updated to address the most pressing issues we encountered during these pilots.
The assessment report, linked below, contains a full write up of the conditions leading to the advent of the common assignments, the process undertaken developing them, the findings of initial assessment efforts, and the changes we made and recommend making moving forward. We share this report to maintain the grassroots spirit and transparency of a truly bottom-up assessment project we set out with. Any questions, comments, or concerns should be directed to Daniel Ernst at ernst9@purdue.edu. Thanks.
The information below answers most of the questions from the assessment Group-Think-Share session. We transcribed these questions from the whiteboards and added questions received via email. On Tuesday, September 4, we will share a handout that outlines the protocol for turning in your completed common assignments.
We have included some general guidelines for you and students as you begin the common assignments in the General Questions section. If you have further questions, please let Derek Sherman know via sherma11@purdue.edu.
How do we structure assignment sheets and rubrics? What actually needs to be on the assignment sheets vs. rubrics?
To be transparent with students about the common assignment, we ask that you align your assignment sheet AND rubric to the outcomes listed in your common assignment’s instructor’s guide found here. Please use the common assignment outcomes-based rubric as a starting point in the creation of your assignment sheet and rubric. As long as your assignment sheet and rubric show students the outcomes that determine their grade, your assignment and rubric are aligned with ICaP assessment.
How should students submit assignments? In what format?
It’s easiest for everyone if students submit assignments via Blackboard in a .doc or .docx format. Even if you don’t use Blackboard extensively, please ask students to submit their assignments there so that you can easily download and share completed work with us. Electronic submissions on Blackboard and in a .doc or .docx format make anonymizing for assessment purposes easier.
Do we need to share our rubric and assignment sheet as part of assessment?
We would like you to share your rubric and assignment sheet with us so that we can understand how various instructors approached these common assignments. However, we will use the outcomes-basic rubric for assessment purposes.
Would students recognize a logical progression of assignments?
This depends on how each instructor frames their course. Since we do not want to force a standardized curriculum on English 106/108 instructors, instructors must be transparent with their students on how this assignment is scaffolded into the curriculum. If students are made aware of this progression in the syllabus and classroom instruction, students would most certainly recognize a logical progression of skills and assignments.
Can the research essay be argumentative?
Yes, the research-based essay can be argumentative. However, it is not required to be argumentative.
What is the relevance of the research-based essay beyond 106?
ICaP outcomes address research writing in several ways, so this essay is meant to accompany our diverse teaching approaches. ICaP has a strong foundation of teaching research-based skills that programs on campus search for in their students, whereas other “writing” programs lack in teaching research-based skills. This assignment, therefore, was built to appeal to faculty and programs who want students with research-based skills. This assignment introduces students to academic research to better prepare them for future research-based assignments. Since Purdue is also striving for more undergraduate research, focusing on research helps students join that movement, not to mention ICaP as a whole. this also helps ICaP promote itself as an establisher of research-based skills.
Should instructors use solely secondary or primary sources — or will a mix of both source types be acceptable?
Instructors may choose to include all primary or secondary sources, or they may choose to have a mix of both. No matter what, students should be expected to summarize, analyze, and synthesize at least five sources in total, according to the research-based essay instructor guide, in whatever format you, the instructor, expects.
How students will find, use, and integrate CREDIBLE sources?
Credibility varies depending on the student/class’s topic and what type of research the instructor chooses/values. For secondary research, students become acquainted with Purdue Libraries’ databases to search for peer-reviewed articles, thus engrossing students in academic-based research. Primary research, however, varies depending on the research method and participant(s) chosen; therefore, guidance from the instructor should be given to determine what equates to credible research. Often, only interviews or surveys are feasible given the limited time for 106/108.
How are analysis, synthesis, and summary evaluated differently?
These are the three skills that often accompany academic-based work; therefore, for ICaP’s assessment purposes we are looking at these skill holistically (see ICaP outcome three: “Critically think about writing and rhetoric through reading, analysis, and reflection”). By holistically, we want to see if students are able to summarize, analyze, and synthesize academic sources cohesively into one academic paper. We are not seeking to isolate these skills or assess them individually.
Is there still a pretest and posttest for the rhetorical analysis essay?
Yes, please have students complete a pretest without any instruction. After teaching the rhetorical analysis unit, students should complete a second rhetorical analysis. Each essay should be approximately three pages in length.
How do we frame “rhetorical?” Is it important to focus on terms and strategies?
The big issue here is that we want to make sure students go beyond analyzing solely the text and the author’s use of rhetorical/literary devices (e.g., ethos, pathos, logos, simile, metaphor, etc.). In this sense, students should focus on analyzing how the text and the author(s) use of rhetorical/literary devices affect the context and audience. The text, consequently, should be not viewed in isolation but as working against and amongst the author, audience, and larger context.
What are the preferable methods for draft collection?
Since our assessment includes evaluating students’ rough drafts and whether or not they have addressed instructors’ comments and have shown growth per ICaP Outcome Four (i.e., “Provide constructive feedback to others and incorporate feedback into their writing”), it is important that students save a copy of their rough drafts. Please ask students to save a draft in a word document (e.g., ResearchEssay_Draft) and then create a new document for their final essay (e.g., ResearchEssay_Final).
What should be the portfolio’s medium?
We are looking into Purdue’s new portfolio tool and other options and will have an update as soon as we can. We’re hoping to offer several methods. We welcome your suggestions.
What is ICaP’s most important outcome for the professional email?
Rhetorical awareness of the situation is the most important outcome with other outcomes such as purpose and persuasion built in.
How flexible in terms of when during the semester can it be assigned? The time?
Per our discussion at Convocation, the timing is up to the instructor. If you can fit the assignment into an already existing assignment, please go ahead and include it whenever you teach that assignment. As Alisha Karabinus noted, this assignment was originally a kickstarter for rhetorical awareness, but you may include it wherever you see fit.
What are the two audiences?
In the first email, you will write to an instructor about a missed assignment or exam you would like to attempt to make up. You will create your own reasons for the make-up.
200 words is a lot for an email?
You are welcome to suggest a lower maximum as long as the focus remains on achieving the outcomes on the rubric (i.e., addressing the audiences effectively, etc).
We would like to see student samples?
Currently, we don’t have any samples that fit our two new audiences. Because this assignment was updated this summer, our previous samples no longer coincide with this update. The professional emails that students produce in Fall 2018 may serve as samples for the future.
What are the results from the previous semester’s assessment?
How is the APE exercise relevant to the professional email assignment?
Linda Haynes’s APE exercise is designed to be a fun, low-stakes activity which teaches editing and efficiency. While it certainly isn’t an essential activity, many instructors have had success working it in during this unit as a way to make the editing process visible and accessible.
How is ICaP quantifying positive improvement for the professional email?
In terms of student improvement, growth cannot be quantified because the two audiences are different scenarios and we cannot have a measurement of growth without a pretest. Because raters from the professional email’s prior assessment varied in their definition of an effective email, we are hoping that this more focused audience approach will help us establish a more consistent definition of an “effective email.” If a more consistent definition is established, we may be able to move forward in quantifying growth with a pretest and posttest in the future. However, our goal right now is to measure achievement of the outcomes and not individual student growth.
How do we make a short assignment feel more substantial?
The email is a shorter assignment, but it is also effective in teaching rhetorical situations and catering one’s message to one’s audience. Instructors, therefore, may choose to teach this as an introduction to rhetorical awareness at the beginning of the semester or they may embed it within a current unit. Ultimately, scaffolding this assignment in with others is the best way to make this assignment feel more substantial.
Is it pretest and posttest?
No, this is two separate emails to two separate audiences.
What are the number of drafts required?
This is up to the instructor. Instructors may guide students through sample emails first and then accept only one draft. Or, instructors may conduct peer review sessions on multiple drafts. The choice is up to the instructor.
How many class session should be dedicated to the professional email?
This, too, is up to the instructor. This assignment is not meant to take up a considerable amount of time; therefore, scaffolding it into a current unit may help help in establishing how many class sessions you want to dedicate to this common assignment.
Again, if you have any other questions, please let Derek Sherman know at sherma11@purdue.edu.
Hi there. It’s your friendly Assessment Research Coordinator, Daniel Ernst, checking in with an update on things we’ve learned so far from some of the 2018 ICaP common assignment pilots. A group of ICaP instructors recently read and rated a very popular first-year writing assignment, the rhetorical analysis. Raters not only came to consensus about their ratings, but found strong evidence of student improvement.
For this assessment, we randomly selected 23 pairs of essays from a pool of 60 submitted by instructors. Each pair included two rhetorical analysis essays, one written as a diagnostic pre-test and one written as a post-test to conclude a four week class unit on rhetoric. In total, 46 essays (23 pre-tests and 23 post-tests) were read and rated at least twice by eight graduate student instructors and one professor. Essays were de-identified to prevent raters from knowing who wrote them, which class or instructor they were written for, and whether they were a pre or post test. Raters used a simple rubric built from ICaP outcomes one and three, which focus on rhetorical and analytical knowledge and critical thinking skills.
There is a highly significant (p < .001) improvement in mean scores between the pre and post test essays. We can confidently say the improvement in mean scores is not likely due to chance but instead likely due to the effect of the treatment: the class and concepts taught. Additionally, the improvement is not just significant but meaningful: the Cohen’s d value of 1.04 indicates the distribution average improved by one standard deviation from pre to post test. This means that a pre test essay scoring at the 84th percentile of all pre tests would score at just the 50th percentile of all post tests. Finally, the post test mean score (7.63 +/- .41) sits right at the midpoint (7.5) of our rating scale (3-12), indicating a distribution of student performance around the true mean of our scale.
Although the sample is limited in certain ways (size, variety, degree of randomness), we are seeing evidence of significant and meaningful growth in writing quality on rhetorical analysis assignments over the course of a single unit in ICaP courses. The evidence is bolstered by the strength of the rating instrument, which almost perfectly sorted pre and post test writing and scored the mean of post test essays at its exact midpoint, as well as the high correlation coefficient obtained by the nine raters using the ICaP Outcomes-derived scaled rubric (.73). As we begin to develop methods for assessment which will be applied program-wide, these results suggest the ICaP Outcomes and Detailed Learning Objectives can serve as source material for designing assignment rubrics, at least for the outcomes on rhetorical and analytical knowledge.
To be sure, we should expect such strong results due to the design of this specific assessment. Any writing done before learning concepts and skills and then measured again after should demonstrate significant improvement. But by building our rubric and scale directly from the ICaP Outcomes, we also show that instructors are meeting the outcomes related to rhetorical and analytical knowledge as our program currently articulates them. Content knowledge about rhetorical concepts and ability to critically analyze texts are fundamental to any writing class. We’re pleased not only with the scores, but the conversations our readers had as we rated sample essays and discussed the rubric ICaP staff and I developed.
We are currently preparing a full report on the common assignment pilots that will use results and data from the rating sessions to make evidence-based decisions on the future direction of program-wide common assignments and the culture of assessment within ICaP. The next steps going forward include revising the rhetorical analysis and other assignment instructor guides for our second generation of common assignments, as well as refining rubrics and assignment sheets. We welcome feedback and questions from any participating instructors or those interested.
Thanks again to all instructors and raters involved in this assessment, including but not limited to Parva Panahi, Mac Boyle, Deena Varner, Libby Chernouski, Julia Smith, Joe Forte, Carrie Grant, April Pickens, and Bradley Dilger.