Tag: Bloom’s Taxonomy

Posted on

Effective, Fair, and Efficient: Using Multiple Choice in the Humanities

Photo by Pixabay on Pexels.com Image description: A black and white photo of a room with seven identical white doors. Large black and white patterned wallpaper covers the wall around the doors.

Presented as a workshop at the American Association of Philosophy Teachers (AAPT) “How We Teach” conference, July 14, 2021

Jennifer Szende

In July 2021, I led a virtual workshop session on multiple choice testing in philosophy as part of the AAPT series on “How We Teach”. This is an adapted version of the handout I distributed with the presentation. Multiple choice testing is sometimes dismissed as too easy for students, too open to dishonesty, or too difficult to design for instructors. Here, I give an argument in favour of multiple choice testing, I respond to some concerns, and I offer some tips and best practices resources for effective multiple choice testing in philosophy. Much of what I say here will be relevant to other academic disciplines and other testing scenarios in addition to academic philosophy.

Why use multiple choice?

There are many good reasons to include multiple choice within a balanced assessment portfolio. I focus on effectiveness, fairness, and efficiency.

Effectiveness: Multiple Choice questions can be an effective way to assess the learner’s ability to recall, understand, apply, analyze, and evaluate. Standardized tests typically use case studies and sight passages to assess student’s understanding, application of concepts, analysis, and evaluation of novel information. So, to an extent, many of us are familiar with multiple choice testing that is designed to assess skills beyond information recall. One frequent objection to those types of tests is that they assess ‘test taking ability’ or ‘familiarity with the test format’ rather than assessing analysis or understanding. Keeping this worry in mind, I have tended to design my tests as open book and without time limit. Open book because I am very happy to build formative assessments that force students to look at the course material in a new light, and give students the opportunity to examine what they find in that new light. If they read a passage for the first time, or re-read it for a subsequent time, in order to answer the question, the test has done its job.

Fairness: Sometimes, assessing students can be unavoidably subjective. For essays and presentations, the bias and subjectivity is mostly located at the stage of marking or assessing student work, with some biases and subjectivity located in the design of the assignment. Rubrics can help standardize the subjectivity across students (thereby increasing fairness), but a level of subjectivity remains. Think of cases where TAs and instructors standardize each other’s ‘A’ paper, ‘B’ paper, and ‘C’ paper, or cases where students appeal a grade by comparing marks and assignments with other students in the class. For multiple choice, the subjectivity of assessment is located at the stage of writing questions, rather than at the stage of marking questions. As a result, the subjectivity and bias are more fairly distributed across all test takers (Loftis 2019). I have taken Rob Loftis’s advice seriously, and have taken to offering students a space in which to explain their answer. I don’t read these explanations for correct answers, but find I am often able to give partial or full credit to students who misunderstood the question but demonstrate understanding of the material, and other times these responses help me to recognize and rectify (with full credit) questions that were unintentionally ambiguous.

Efficiency: A multiple choice test allows you to assess a lot of material in a relatively short assessment, which is furthermore easy to mark. Multiple-choice tests can cover a large scope of material in a relatively small/short test. They are easy to mark, even for large and online courses. See David DiBattista’s argument here. In some cases, the Learning Management System (LMS) or scantron system can be used to automatically mark the test, or mark it pending instructor review and approval. In particular, the cognitive burden of marking is reduced. Reducing the cognitive burden of marking is no small feat, even if much of the cognitive burden shifts to the stage of test design. When I have large (90+ student) classes, these tests allow me to save some of myself for other types of student engagements and assessments. In the case of LMS tests with automatically generated feedback, there is a possibility to give immediate feedback to learners, so that the student gets an explanation of the correct answer.

So, why would philosophers avoid using MC?

It’s too difficult for the instructor! Constructed answer questions are much easier to produce (DiBattista and Kurzawa 2011). The easiest multiple-choice questions to produce assess information recall, and many teachers aren’t interested in assessing information recall. Genuinely challenging, formative multiple-choice questions, especially those that assess understanding, analysis, application, or evaluation, can be challenging and time consuming to write/design.

  • Practice writing questions in a variety of styles, for a range of skills.
  • Use some of the question-writing tips offered here or in the further resources linked below.
  • Pace yourself throughout the term. Write 1-3 questions per week, or per lecture. Schedule time to write questions after each lecture, when the material and discussion are fresh in your mind.

Multiple Choice is too easy for my students, or too low on Bloom’s taxonomy (DiBattista and Kurzawa 2011; DiBattista 2008; Loftis 2019). Many instructors worry that students will just use a search function to find the answers. The solution is to design the test/write questions with this worry in mind.

  • First, ask yourself: “What is the purpose of the test?” Are you assessing whether students have attended lecture/read the material? Whether they have understood the material? Whether they can apply a concept to a novel situation? It might turn out that you want to assess information recall in a particular instance. But, if so, it might be an appropriate occasion on which to set a time limit (with appropriate extensions for students who need it), or it might work best for an in-class test. If, however, you want to assess understanding, analysis, or application, remove the time limit and design questions to be open book. Invite students to take the time to look up the answers. You may wish to use paraphrasing to avoid searchable terms. Alternatively, you may actually choose to have your students look it up, perhaps using a search function. If they haven’t reviewed the material very closely yet, maybe the test is a good way to get them to read key passages.
  • MC can be formative, medium to high on Bloom’s taxonomy, and can provide a valid measure of student achievement.
  • Skills that can be tested with MC: Recall; Understanding; Apply; Analyze; Evaluate? (Loftis)

Academic dishonesty. Lots of worries arise on teaching forums about students paying someone else to write the test, working together, or copying each other. If that is your worry, design with it in mind. But also, learn a bit more about triggers of academic dishonesty, and try to design your evaluation to avoid these.

  • Again, consider: ‘What is the purpose of the test?’ Choose an appropriate assessment strategy for the thing being tested. Multiple choice tests can be formative, and the purpose of testing might be to familiarize students with key concepts. The process of looking up the answer and reading through the questions might be exactly what you want to test. Consider providing a provision and permission for students to work on these questions together, such as an unmarked fill in the blank ‘I worked on this test with the following person/people…. ‘.
  • Use low stakes multiple choice testing. Frequent (open book?) tests worth 2-5% with the lowest marks dropped are less likely to lead students to feel under pressure than one-time exams worth 30-40%.
  • Use randomization. Learning Management Software such as D2L/Brightspace, Blackboard, or Canvas allow multiple forms of randomization in testing. Build question ‘pools’ or ‘banks’ with a larger number of questions on each topic than will appear on the test. The LMS will randomly generate a set of questions, and will randomize the order that they appear in for each student (within parameters set by the instructor or test designer). The LMS can even randomize the order that the options appear in within each multiple choice question, which encourages closer reading of the question.
  • Consider using untimed test and/or open book tests. Design a test that will require looking up (some? Most?) answers, and give students time and permission to do so. If the test is designed to be open book, looking up the answer will not constitute cheating.

General best practices for Multiple Choice:

Image of Bloom’s Taxonomy from Vanderbilt University Center for Teaching. Bloom’s taxonomy ranks cognitive skills. From bottom to top, Bloom lists: remember, understand, apply, analyze, evaluate, create.

Some MC question writing strategies:

What follows are a few question-writing strategies that I have used in the past to generate questions. I keep this list handy when I am trying to generate 1-2 questions each week based on the discussion. I review my lecture notes or power points, any examples discussed in class – especially those raised by students – and try to write a question stem and the correct answer before generating distractor responses also based on lecture, discussion, or written material.

  1. Paraphrase, and use the paraphrase rather than quotations in the stem or the multiple choice options:
    • Paraphrase the thesis of an article.
    • Paraphrase definitions for key terms.
    • Paraphrase key objections.
  2. Use key terms and key concepts in multiple-choice, but try to use them in novel situations/case studies/ examples.
  3. Use comparisons/contrasts/lists drawn from course material or discussions.
  4. What does the example show?
    • Example from the reading: What point is Author making when they use X?
    • Example from the news/ film/ popular culture: What would Author say about X?
    • Example from the news/film/popular culture: Which Author would make which of the following claims?
  5. Who (which Author) would agree with [paraphrase]?
  6. How might Author A respond to Author B’s question/quote/example/concern?
  7. Author A and Author B agree about X.
    • True or False?
    • Which reason would each give for X?