Is knowledge worth testing any more? Is testing knowledge ‘Authentic’?

Bypassing the Remembering and Understanding steps when answering exam questions using Google.

Bypassing the Remembering and Understanding steps when answering exam questions using Google.

I was challenged recently at a Learning and Teaching conference about how I assess students in Biosciences. Apparently my assessment methods are ‘not authentic’ and ‘overly test knowledge’ on the basis that almost every module on my course is in part, assessed by a formal exam. There was some mention of a dodgy Einstein quote derived from “[I do not] carry such information in my mind since it is readily available in books. The value of a college education is not the learning of many facts but the training of the mind to think.” I’m sure context is everything in this quote, and very much doubt he was talking about fundamental knowledge underpinning established theories in physics. That got me thinking: Is knowledge worth testing anymore given that everything is accessible from Google? Furthermore: Seth Godin in his TEDx talk on education said, “There is zero value in memorizing anything, ever again. Anything worth memorizing can be looked up.” Sugato Mitra in “Living in the age where knowing is obsolete” suggested that answers to Physics tests could be googled and discussed, rather than answered alone in an exam. Eric Mazur suggested that any answer to a test that can be googled is not an authentic test. There are many other examples, as outlined very nicely here (sorry, this blog by @webofsubstance has since disappeared). There are two inter-related themes going on here: Knowledge is ‘out there’ and accessible so there is little benefit to memorising it. Even if knowledge is good, testing of knowledge is bad. Related to this final point, all assessments should be ‘authentic’.

As a relative novice to ‘authentic assessment’, I was keen to find out more. In the style of the above experts (excluding Einstein), I Googled it, and came with a link via Wikipedia on the definition of authentic assessment. A brief search suggested that the article is written by an expert on the topic, so I’m confident in the author being an authority on the subject. I have now read this and will discuss this document in relation to my assessment methods, which are really quite traditional (also known as ‘bad’) In ‘authentic assessment’, the assessment tool should mimic real-life situations. So having knowledge and applying it to a situation seems OK, but not testing the knowledge directly, which I think is what Eric Mazur is getting at. However if someone cannot complete an ‘authentic assessment’, what exactly is limiting the student’s progress? Is it an inability to recall the required knowledge to apply it in context, or inability to apply their acquired knowledge in context? If Google is allowed in the examination, as proposed by some experts, then we are focussing on the application of knowledge. Sorry, did I say knowledge? It’s just that it’s not necessarily knowledge is it? It’s more of a ‘state of transient observation’ whilst the information is on screen. Either way, the student got to the answer in a real-life connected-to-the-internet sort of way, so full marks all round. Well done. I’m not convinced.

My view of assessment, and in particular ‘authentic assessment’ in science, is that it should rely upon some known facts, and students should be able to apply those facts to conclude or deduce something, like an authentic scientist. As a research scientist, I have had some ideas over the years, and none of them have come about by not knowing anything about the topic. In fact I’d go as far to say that ALL of what I’d call half-decent project ideas have come from deep understanding two or more disparate topics in great detail, and making some sort of interacting link between the two. So the question is: Am I assessing in a manner that is authentic to science? The classic inauthentic/traditional test is a MCQ test. It is not very often in real life that you have to take an educated punt at 4 possible answers, three of which are deliberately contrived to trip you up. However it can test knowledge. I find a close correlation between MCQ scores and more analytic problems, but rarely can MCQs be designed in such a way. Where does the traditional University essay-style exam fit in? Authentic or not? It is dependent on the wording and what is being tested, and yes, many exams at lower levels do not really test anything other than pure recall of the lecturers’ notes. However any scientific or research report writing, where the introduction is a descriptive knowledge base, ideally with hypothesis based on knowledge, and a discussion containing critical analysis of evidence is essentially what many of us aim to replicate in traditional essay-style exams. So an essay question along the lines of:

Discuss the role of X and Y the process of Z. In answering this question, explain the relative contributions of X (and/or Y) on Z in some relevant setting.

This is a fairly typical degree-level essay exam. Some knowledge recall, in that for good marks, you need to define X and Y before putting them in context of process Z. In the second part, the students have to make some judgements based on the relative importance based on knowledge, which is not possible from memorising a 2D flow diagram. This exam structure is used below in one of my cancer-related exams that I might ask at Final year degree or Master’s-level Molecular Pathology students. Forgive the technical detail on this, but it is really very very interesting…

Discuss the role of p53 and pRb on regulating the cell cycle. In answering this question, explain the relative contributions of inactivation of genes coding for p53 and pRb in Cervical cancer vs Colon cancer (Feel free to skip this bit, but  for those who are bothered or interested):

p53 and pRb are proteins that stop cells dividing at different stages of the cell cycle (that is, time from division to next division) by inter-related and distinct mechanisms. Loss of these (by gene inactivation, or protein degradation) promotes tumour growth. In cervical cancer, genes coding pRb and p53 are very rarely inactivated because Human Papilloma virus (HPV) inactivates/degrades p53/pRb protein, so no there is no selective pressure on inactivating the genes coding for p53/pRb protein. In contrast, p53 gene is usually mutated (inactivated) in Colon cancer, and pRb is either inactive, or something controlling pRb ‘messes up’ the cell cycle control process…

Is this an authentic assessment? Well, according to many experts in education, and the expert definition of authentic assessment, no it isn’t, given that I can get the answer from Google and that it is not a ‘real-life’ situation. I can get nicely written reviews on the topic, and even Wikipedia distils this down to the basics quite nicely, so no, this is not authentic at all. Here are the 5 ‘not-at-all-trying-to-induce-a-false-dichotomy’ features of Traditional vs. Authentic assessment by Jon Mueller, as obtained via Wikipedia.

Traditional Assessment vs. Authentic Assessment:

1) Selecting a Response         vs.                   Performing a Task

2) Contrived                            vs.                     Real-life

3) Recall/Recognition           vs.                    Construction/Application

4) Teacher-structured          vs.                     Student-structured

5) Indirect Evidence            vs.                      Direct Evidence

In my essay, students are 1) performing a task which is authentic to any scientist. Yes, they are selecting a response from memory and relaying that information, but they must apply their knowledge of why mutations arise in a certain manner. It is a 2) real-life task, in that it could represent the key elements of a justification/intro for a research proposal or research paper studying these genes in either cancer type. Yes, this is all on Google, but how would you know what to search for without the appropriate knowledge-base and understanding of how to put that information in to context? Although the first part of the exam question above relies on the traditional  recall. 3) Recall of knowledge “Discuss the role of p53 and pRb on regulating the cell cycle”, the question then asks students to apply this knowledge, and hopefully construct their own hypothesis of why certain events occur. They can only properly apply the knowledge if they understand two important concepts that have been taught previously, but are not specifically asked for. They must also recall and relate the causative events in Colon cancer. They must then ‘explain the relative contributions of inactivation of genes coding for p53 and pRb in Cervical cancer vs Colon cancer‘. So although both pathways are ‘messed up’ differently in each tumour type, the net effect is the same: Cells grow fast and refuse to die. That is an important point to appreciate, and this essay allows me to assess whether the students have grasped this concept. So how about 4) Teacher-structured vs. Student-structured (whatever that means): As helpfully defined by Jon Mueller, “A student’s attention will understandably be focused on and limited to what is on the test. In contrast, authentic assessments allow more student choice and construction in determining what is presented as evidence of proficiency.” Well, it’s an exam, designed and written by myself. I really don’t understand how a student-structured task makes it more authentic, other than to tick the ideologically ‘non-traditional’ box. I suppose I could leave it up to the students to write about a tumour type of their choice, or cell signalling pathway of their choice, but that might miss some conceptual nuances that I want to test. As the teacher/’expert’, I know that these pathways are relevant to ALL cancers, unlike a student’s possible choice of some random cell signalling pathway. I know that studying these two tumours highlights crucial differences in the mechanisms of how these important regulatory pathways are bypassed in cancer. Studying say, Colon vs. Pancreatic wouldn’t cover these nuances, but the students don’t know that. Finally, what about 5) Indirect evidence or direct evidence: Again, as helpfully defined by Jon Mueller “What thinking led the student to pick that answer? We really do not know. At best, we can make some inferences about what that student might know and might be able to do with that knowledge. The evidence is very indirect, particularly for claims of meaningful application in complex, real-world situations.” I know why a student has come to the final conclusion at the end of my essay question as I have tested the underlying fundamental knowledge first, and seen how they have applied their interpretation of the fundamental knowledge to a situation that requires synthesis of ideas from distinct parts of the curriculum. So yes, direct evidence is used to assess the student on application of knowledge. So the startling conclusion is that essay-style questions under test/exam conditions can be ‘quite authentic’, even if the answer can be obtained directly from Google. It also leads us to conclude that being able to Google the answer to tests is not necessarily a good way of assessing students. Finally, if a quote appears to be obviously ridiculous, it probably is ridiculous. There are so many situations where we cannot reasonably be attached to the internet. Fundamental knowledge recall will always be needed and hence, it’s probably worth ‘testing’ whether students have this prior to progression academic progression. As a scientist, here are just a few:

1) Understanding subsequent teaching sessions that rely on prior knowledge.

2) Listening to a conference paper presentation and wondering how X relates to Y, and whether it might also relate to Z, which the authors had not considered (but will form the basis of my next research project).

3) The PhD voice viva examination. A must for any independent research scientist.

4) Any voice viva examination at any level, so a must for any of my BSc or MSc graduates.

5) Being interviewed on how your recent research findings are relevant to X by the media, and how they might relate to something else (usually quite obscure…).

6) A lab meeting with the professor: “How does X relate to Y. What is Y? Why are you studying Y if you don’t know what Y is…? Get out of my office…”

7) Being interviewed for a job, where getting an interview may in part be based on your knowledge base and ‘decent grades’.

“Oh hang on, I’ll just ask Google”. Yes, that will go down well…  

PS: I rarely use Wikipedia as an initial research tool, but given the Godin, Mitra and Mazur quotes, I thought I would give it a go. Sorry. I won’t do it again.

PPS: In future posts I’ll discuss how I get students to answer questions based on interpretation of data from scientific research papers under exam conditions. I’d go as far as to say that this is also ‘somewhat authentic‘.

Advertisements

About TheOtherDrX

Senior Lecturer in Biosciences. MSc Biosciences course leader and lecturer on topics such as Cell Biology, Moleular Pathology and Genetics. I manage a research team of PhD students and post-doctoral scientists working on novel anti-tumour drug combinations, nanotech-based delivery of anti-tumour agents, and artificial scaffolds for 3D cell culture studies as a replacement for animal-based studies. I also do a bit of STEM public engagement work with my Geiger counter.
This entry was posted in Assessment, Student engagement and tagged , , , , , , , , . Bookmark the permalink.

5 Responses to Is knowledge worth testing any more? Is testing knowledge ‘Authentic’?

  1. I think some Universities are definitely missing out on lecturers like yourself- passionate, motivated, believe in what you do and how you should do it! People just want to be lazy and rely on ‘others’ far to often in my opinion. Having to answer a essay-style question at degree level really does test how much you know about a topic and if you could apply it in real-life situations. Because, as you said, coming up with research ideas doesn’t just come from a google search (as I believe this would actually take forever!!), but from knowing topic areas in great depth and with great interest. Very interesting post Dr Cross! I can’t wait to read more!

  2. Multiple choice questions can be useful too, when they are well designed. Have a look at Joe Kirby’s guide to writing them on his ‘Pragmatic Education’ blog. I’ve started using them for recently after reading Hirsch and Kirby and realizing that they can be complex and demanding. They are also very objective.

    • TheOtherDrX says:

      I completely agree. I gave the MCQ a bit of a bashing as the blog was in part about authentic assessment, and I don’t class them as particularly authentic assessments. However I don’t think all assessment should be authentic, so I do use MCQ up to 2nd year degree (I would do more but I’m banned from negative marking, even the ‘fair way’ of knocking off 1/3 mark per wrong answer!). Yes, questions must be well written to test understanding of complex concepts and it is certainly something I’ll look into (I’ll check Joe Kirkby blog), however I think I can make decent judgement of student understanding by a series of mini essays of say, 1 page scribbling for each. MCQ marking keeps me sane though…
      I use MCQ a lot more during lectures as formative assessment with my low-tech iCARDs (see earlier blogpost)

      • I use negative marking within a question, but not across them. They don’t know how many responses are correct in each question. They gain a mark for each correct one, but lose a mark for each incorrect one.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s