Crowdsourcing Evaluation

I happened to discover this article, and at first I got really excited about it:

http://www.technologyreview.com/view/519001/first-trial-of-crowdsourced-grading-for-computer-science-homework/

Then I started reading it, and got a little disappointed. After having spent some time reading through Alfie Kohn’s website, and discussing his article The Case Against Grades with numerous people, I realized something. I realized that I am reading this article with an entirely different perspective than I would have had just a week or two ago, and even that perspective is vastly different from the perspective I would have had this time last year. Where to begin? I suppose a brief summary of this article would be in order.

Luca de Alfaro and Michael Shavlovsky, of the Jack Baskin School of Engineering at the University of California Santa Cruz, have developed an online tool called CrowdGrader. The purpose of this program is to allow instructors to “crowdsource” the grading of assignments, redistributing the burden of grading to include peer review by classmates. Students submit an assignment via the website, and the assignment is anonymously presented to five other students for review and feedback. In order to guarantee that students receive quality feedback, and to provide an incentive for student involvement, a portion of the grade that each student earns (25%) is based on the feedback that he or she provides as a reviewer.

Wow.

So how about we start by going back in time a bit…say one year ago. I am going to do my best to imagine my reaction to this article, had I read it then. I think that my initial reaction would have been one of disgust. What laziness is this?! The job of an instructor is to teach students and then to evaluate their performance! The teacher should be providing feedback, since the teacher is the expert, after all. Besides, think of the potential for cheating. Anonymous or not, students can find ways to collaborate to ensure high grades. Now, rather than just doing their job correctly, teachers will have to spend time policing students, checking the quality of their feedback, and providing feedback on student feedback. They should just spend that time grading assignments!

My assessment: Article Rejected!

OK, now we fast forward a bit, but not quite to present day. I have been participating in a class called Contemporary Pedagogy, I have been reading a lot about education in the U.S., and I have been part of some discussions on student empowerment, student-directed learning, etc. Now I am starting to feel that my eyes are being opened. There are some problems with our educational system, and while I knew this, I did not really have a good understanding of what some of those problems might be. Maybe this is not such a bad idea after all! I mean, here we are getting students involved in the evaluation process, right? Here we have the potential to make the role of the teacher a little less authoritarian, and the classroom environment a little more collaborative. Instead of just turning in an assignment and waiting to be told a grade, students are being challenged to assess the work for themselves, to think about what constitutes quality work, and to consider what feedback is most appropriate. Students have the chance to learn from one another, and can see how others approached the same problem in a possibly different manner. Awesome!

My assessment: Article Accepted!

As it turns out, I just found this article today, which means that my perspective is rather different than the aforementioned scenarios. The very first thing that I noticed about this article is that the software is called CrowdGrader. Yes, I did use italics, bold type and an underline on the word “grader”. I just finished reading Alfie Kohn, for crying out loud! I am now suspicious of the words grade, grades, graded, grading, and grader. Do not get me wrong…I recognize that there is value to found in assessing the quality of learning, but I am starting to question the wisdom of always trying to place a number on it. Still, I read on. As before, I like the idea of allowing students to evaluate one another’s work and the challenge of providing an evaluation or analysis of the quality. I think that the collaborative aspect is valuable as well. Perhaps some assignments could be reviewed independently, while others reviewed as part of a group effort? Of course, this also depends on the approach used. Are the students given strict criteria, or even an answer key, and put to the work of grading? If so, then they are not really being empowered, or challenged to think. They are just doing the job that is often assigned to Teaching Assistants. What seems more problematic, however, is the suggested need to enforce compliance by factoring a “reviewing grade capturing their reviewing effort and accuracy” into their grade for the assignment. Certainly, if none of the students actually bother to review assignments, or if those reviews are of very poor quality, then this effort is going to be in vain…but I still wonder if that is the best way to encourage genuine engagement? If there is any truth to be found in the argument that Alfie Kohn makes against the use of grades, then it seems that this entire effort (as presented) amounts to little more than a new spin on the same old approach. There is certainly potential here to use technology to empower students and facilitate collaboration, if applied to a group of intrinsically-motivated students. In a classroom setting that engages students and seeks to involve them in the direction of the class, this could be a great tool. Used in the traditional classroom setting/approach, however, it seems like little more than a parlor trick to give the perception that students are being empowered. Furthermore, if used improperly, it will require the application of extrinsic motivation to avoid a breakdown of the system.

My assessment: Requires Revision…

I find it interesting to consider how my views have been changing recently. I have been introduced to a lot of information pertaining to education, and my viewpoints repeatedly challenged by ideas and approaches that are new to me. It has been important for me to recognize that I am easily drawn to such new ideas, especially when the idea is well-presented, and/or the argument strikes me as logical and reasonable. I am often hasty in my judgements, and quick to champion a new cause or view. I try to keep that in mind as I seek to integrate all of this new information and determine what will be incorporated into my new perspective, and what will be cast aside. I look forward to seeing how my opinions might change as I continue to learn more about education and what it means to truly teach.

Category(s): Education

Comments are closed.