Month: August 2023

Posted on

Hack your marking and ease the cognitive load

Amongst the tasks involved in teaching, marking can be particularly exhausting and overwhelming. Marking is itself tiring and draining, and marking deteriorates as tiredness sets in and attention spans wane. Worst of all, marking necessarily comes at the end – at the end of the unit, the end of the course, the end of the semester. It comes at the point when you are most thinly stretched. Marking typically spills over into the time allotted for the next task, so that it is not unusual to have to start preparing the next lecture, unit, or syllabus while still snowed under with a huge amount of marking from the last one.

There are, however, some ways to do a portion of the cognitively hard work of marking ahead of time and thereby make your marking more fair, less taxing, and easier to manage. This post suggests a couple of ways to ease the cognitive load of marking by doing a significant portion of the decision-making before the first student even submits their assignment.

1. Multiple-Choice

Using multiple-choice questions on tests or quizzes for some part of your assessment portfolio is one way to do the hard work before you get to the marking stage. I have been completely persuaded by Rob Loftis’ excellent paper, “Beyond Information Recall: Sophisticated Multiple-Choice Questions in Philosophy”. Amongst his arguments, Loftis points out that most of the subjective and cognitively difficult tasks involved in assessment get front-loaded in multiple-choice assessments: subjective decision-making occurs at the stage of composing the questions, rather than at the stage of marking each individual answer or response. By the time the students take the test, the cognitively difficult and subjective decision-making part of assessment has been completed. Moreover, exactly the same subjective decision-making is shared equally by every student who answers a particular question. That is, students are not subject to the unfairness of repeated and distinct subjective decision-making processes every time a question is marked.

David DiBattista’s paper “Making the Most of Multiple-Choice Questions: Getting Beyond Remembering” points out that multiple-choice tests make it possible to cover a relatively wide scope of material in a relatively short test. He points out that it is furthermore relatively easy – and quick – to mark the tests, even in the context of large or online courses. Learning Management Systems (or LMS systems such as Canvas, D2L, Moodle, and Blackboard) or Optical Readers facilitate the automated marking of multiple-choice tests, which can nonetheless be tailored to focus on higher-order skills such as analysis, understanding, application, and evaluation. I note that I am also persuaded by Loftis to offer students the opportunity to explain their answers. This does add to the cognitive load of marking, but not substantially so. It has helped me catch unnoticed ambiguities in my questions in the past, without penalizing students for it.

The trick, of course, is to make time to compose multiple-choice questions early enough, and at regular intervals throughout the term. But even if you end up composing the test or the set of questions at the last minute, you will still have moved the cognitively difficult, subjective, decision-making task earlier in the assessment process, and prior to the ‘marking’ stage. In fact, you will have moved it prior to the student submission stage.

2. Rubrics

A rubric generally details and specifies the assessment criteria for an assignment. An analytic rubric provides specific criteria, and describes levels of achievement for each criterion. From the instructor’s perspective, rubrics enable timely feedback, and make marking more consistent and fair. Preparing a detailed rubric in advance eases the cognitive load of marking. At the marking stage, the grader can replace a cognitively complex and holistic assessment such as, “What grade does this essay deserve?” with a narrower and cognitively easier set of assessments. Assigning rubric levels according to set descriptions is a narrower and easier task, especially if the task is repeated for dozens or hundreds of assignments. Descriptive feedback is more easily attached to each assignment, and the process of marking is made easier overall for the marker.

From the student’s perspective, rubrics clarify expectations and marking criteria, all the more so if distributed in advance of the assignment deadline. According to Jönsson and Panadero’s chapter, “The Use and Design of Rubrics to Support Assessment for Learning”, rubrics are transparent, make it easier for students’ to use feedback, facilitate peer assessment, and they may reduce student anxiety or support self-regulated learning. The feedback provided by rubrics can be easier for students to process and respond to. The transparency and clarification of criteria provided by a rubric can help students with peer- and self-assessment of their work, and these in turn can reduce student anxiety surrounding assignment submissions.

3. Self- and Peer-Assessment

Of course, nothing will ease the cognitive burden of assessment quite like getting someone else to do it! But more seriously, students supporting each other’s learning through guided peer-assessment can provided genuinely helpful feedback in a timely manner, which again eases the cognitive burden related to marking. In this case, it eases the cognitive burden of providing specific, timely, and relevant feedback, which is a key part of the assessment process. I am by no means suggesting students give each other heavily weighted final grades or even unguided holistic assessments of each other’s work. They don’t necessarily have the training or the experience to be able to recognize what makes a strong (or weak) paper on its own merits.

Best practices in peer assessment would use an anonymized and guided system to help students offer each other feedback, and might give them credit for doing so. In order for peer-assessment to front-load the cognitive burden of marking, clear assessment criteria – perhaps a rubric, or perhaps a peer assessment form- should be prepared in advance. In “The Role of Self and Peer Assessment in Higher Education”, Pérez et. al. demonstrate that well-guided peer assessments can concord with lecturer assessments of the same work, using the same guided criteria. But even peer assessments that only provided content feedback on a draft can provide the student with useful and valuable information prior to a final submission, and would ease the cognitive load of the instructor marking the subsequent submission.

Peer assessment can be designed to use rubrics, or can be designed to provide feedback at a mid-point in a scaffolded assignment. In a recent course where I was a student, we were given the opportunity to submit an early draft for peer review (anonymized through Canvas), and only if we completed the peer review process were we eligible for detailed feedback on the final submission from the professor. Whether the peer assessment generates a mark, or a completion mark, or merely generates feedback, peer-assessment can help provide students with feedback in a format they can use, and thereby reduce the feedback burden of marking.

Guided self-assessments as part of a scaffolded assignment can likewise help students generate feedback and recognize where they have strayed from the stated criteria.

Scaling Up

A note of caution about self- and peer-assessment: they do not scale up as easily as multiple-choice and rubrics. On a large scale, self- and peer-assessment can still be used to help students provide feedback to each other, can be part of a scaffolding process, and can allow students the opportunity to learn from each other. But, when used to generate marks, self- and peer-assessment can be more easily manipulated in ways that might violate academic integrity. Accordingly, my suggestion would be to use peer- and self-assessment to facilitate students providing each other anonymous feedback, and to credit it with a small completion mark or credit worthy of the effort.

Multiple-choice and rubrics, on the other hand, scale up very well. They make the assessment more fair and less subjective on an individual assessment level, and they ease the cognitive load at the final stage of marking.

Posted on

Caring for your future self by reverse engineering the syllabus

There are only so many weeks in a semester. There is also a certain amount of material that must be covered, assignments that must be set, and marking deadlines that each course must meet. So, there will inevitably be a few heavier weeks from the instructor’s perspective, just like there will inevitably be some heavier parts of the semester for students. Some of these are likely to occur near the end of the course once most of the material is on the table. This is how so many of us end up with an overwhelming end of semester push, or an impossible week of marking and covering new material, or too short a marking turnaround time. 

Sometimes, in calculating how to manage and spread out such weeks, it can be tempting to compartmentalize our teaching responsibilities as though we were not also people with lives outside of the classroom. It can be equally tempting to assume the same of our students: that ours is their only class, that their only responsibilities are as students, or even that their priorities are course-related. 

None of these assumptions is fair, either to ourselves or to our students. 

So, one purpose of this post is to serve as a reminder at the course design stage that we all have obligations, responsibilities, deadlines, and expectations beyond any given course. And part of the purpose of this post is to serve as a reminder that we cannot predict the future, nor know every obstacle and emergency that will arise in a given semester. And if all I convince you to do is acknowledge these facts as part of your instructional design process, then this post will have been useful. 

But if you want some advice on how to take account of these features, then read on to learn a bit more about how to reverse engineer your syllabus to hopefully meet more of your own needs as a complete person, and more of your students’ needs as people with lives outside of your class. Partly, this is an exercise in brainstorming about foreseeable conflicts in an attempt to better prepare for them. 

Start with the most mechanical, most fixed features of the course itself: class meetings and times, university holidays, grade submission deadlines. You may wish to use a syllabus date generator such as this one: http://wcaleb.rice.edu/syllabusmaker/generic/ 

Next, add personal dates and appointments. I put these next because you are the only one who can account for these. Your birthday. Important family events (cousin’s wedding? Spouse’s birthday? Parent’s anniversary?) Concert tickets. Grant deadlines. Medical appointments. Conference dates. Travel dates. Daycare pick up. Vet appointments. Note which of these are absolutely fixed, and which are more flexible. After all, you have a life outside of the classroom, and you are best placed to know what that entails. 

Third, add department, professional, college obligations and dates. You likely have less control over these, and potentially less prior warning about them, but they are worth trying to account for even if only roughly. Is there an open house that you’ll be expected to attend? Are there set department meeting times on Tuesday afternoons? Internal scholarship deadlines when students will be asking for reference letters? If you are not starting a new job, you may be in a position to anticipate an approximate pace and time commitment. But even in a new position, some preliminary approximation should be possible, and some fixed expectations should be available. These are reasonable questions to ask.

After these 3 steps, you might be able to recognize some weeks that are already heavy and stressful, prior to your scheduling any course materials or deadlines. Try to flag these. If nothing else, it is useful to be able to anticipate when an intensive week is coming. But ideally, you might be able to schedule some easier teaching weeks to coincide with these heavier commitments weeks, or at least avoid scheduling your heaviest teaching weeks to coincide with already heavy weeks. 

Now, start to draft readings and assignments, but do so by working backwards. When is your marking deadline? How many days or weeks will you need to mark and provide feedback for a given class size or assessment style? How many classes of new material between assignments or tests? How many days or weeks between returning assignments with feedback and starting the next assignment? In each cycle of new material, assessment, and providing feedback, try to schedule some wiggle room in case something comes up. 

On your already flagged heavy weeks, consider scheduling a film viewing or a review session if it’s pedagogically appropriate. Schedule material that you are very comfortable with. Do not schedule your midterm with a 3-day marking turnaround on the week of your parents’ 50th wedding anniversary, nor in the same week as that completely new material. 

At this stage, you may have a preliminary outline of the course and a good preliminary idea of how the pacing will work for you. Highlight the heavy weeks and post reminders in your calendar or other system of organization.

I advise at this stage that you spend a little time thinking through these same issues from a student perspective, which has the added bonus of continuing the project of managing your own pacing and workload. Admittedly, I spend less time on this, but I want to notice ahead of time if I am assigning the two longest and densest readings in the course back to back, or if one week is both a heavy reading week and a heavy assessment week. 

Part of my process is to add a ‘number of pages’ column to the reading list for the course. Obviously, page count is not the only relevant indicator, but it is a tangible metric that students can use to anticipate heavier and (somewhat) lighter weeks. Again, I try to pace the heavy weeks, and sometimes draw out a longer reading over two classes, or replace an excessively long reading with shorter pieces. Making this information available in the syllabus will help students to do the same sort of semester planning.