Both sides of auto-grading argument miss the point

A recent story in the New York Times covers a software program by nonprofit EdX that will soon be available for free to any institution that wants to use it. Using sophisticated machine learning algorithms to train its artificial intelligence, the software will grade essays and short response questions and provide nearly instant feedback. Naturally there are strong supporters for the new software touting it for “freeing professors for other tasks” (like what?). And just as naturally there are strong critics who have formed a group called Professors Against Machine Scoring Essays in High-Stakes Assessment. From the group’s petition:

Let’s face the realities of automatic essay scoring. Computers cannot “read.” They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organization, clarity, and veracity, among others.

While criticism is certainly warranted, I find the quote to be somewhat bullish. Can these people really claim that they understand how they are able to read and measure the essentials of effective written communication well enough that they can look at a computer and say with confidence, “that can not do what I am doing, and here’s why”? It very well may be that current AI programs do not have the ability to comprehend written communication to a degree necessary to assign grades, but to make the argument that the software shouldn’t be used because “computers cannot ‘read'”, as if that were a self-evident fact is just poor communication.

Now to be fair, I disagree with the supporters of the software as well.

“There is a huge value in learning with instant feedback,” Dr. Agarwal said. “Students are telling us they learn much better with instant feedback.”

Ok, well, not that part, I agree with that part in principle. But what kind of feedback? Supposedly the software can generate a grade and also comments whether or not the essay was “on topic”. So a student could get instant feedback, which is great, and then edit and modify, which is great, and resubmit, which is also great… and then what? What would they be learning?

I promise to be highly skeptical of any answer to that question that isn’t “how to write an essay that receives high marks from an automatic grading AI”.

All this talk about feedback. What about feedback for the professor? I find reading through 60 essays just as tedious and time consuming as the next out-of-place grad student in a department that doesn’t value teaching, but I also recognize that reading those essays is a valuable way for me to gauge how I’m doing. Are the concepts that I think are important showing up? Are there any major communication issues? What about individuals, are some struggling, what can I do to help? How will I learn my students’ personalities and how that might affect their personal engagement with the material? How will I learn to be a better educator?

Granted, even though 60 feels overwhelming, it’s nowhere near 200 or more. I can’t even imagine trying to read through that many assignments myself. I’m confident that if I were force to I would not emerge with my sanity intact. This problem does not go unaddressed.

With increasingly large classes, it is impossible for most teachers to give students meaningful feedback on writing assignments, he said. Plus, he noted, critics of the technology have tended to come from the nation’s best universities, where the level of pedagogy is much better than at most schools.

“Often they come from very prestigious institutions where, in fact, they do a much better job of providing feedback than a machine ever could,” Dr. Shermis said. “There seems to be a lack of appreciation of what is actually going on in the real world.”

An “A” for recognizing the problem. But the proposed solution is nothing more than a patch. In fact, it’s worse, because it is a tool that will enable the continual ballooning of class size. And to what expense? Why don’t you rethink your solution and have it on my desk in the morning. I can’t promise instant feedback, but maybe, just maybe, the feedback provided will be the start to moving in a direction that actually addresses the underlying problems, rather than just using technology to hide them.

Digital amplifier: the tweet heard ’round the world

Sometimes, in the face based modern world we live in, it feels like we’re living in the future. But all it takes is the watchful eye of the Internet, and specifically, its uncanny, sometimes disruptive tendency to amplify lurking social ills to remind us we are still very much in the past.

Last week, PyCon nearly ended quietly, without causing much of ruckus, as all good annual gatherings of open source software developers strive for. The organizers of PyCon understand the importance of diversity in the technology field, a currently white male dominated field, have worked hard to create an environment that is open and welcome to everyone, and in case there’s any confusion, they have a published code of conduct.

So when Adria Richards grew frustrated with two men making lewd jokes behind her at a closing talk she snapped their picture and tweeted

Moments later, PyCon staff saw her tweet, responded and escorted the two men into the hallway. The situation was resolved with minimal disruption. It would have all been finished, and we wouldn’t be still talking about it now if it hadn’t been for the first inappropriate response to the, up until this point, fairly minor ordeal.

The company for which the two men were working for, and representing at PyCon, made the decision to fire one of them. The company sited multiple contributing factors, not just the joke, but the timing was extreamly poor on their part if they really didn’t want to connect the termination to the joke incident.

And then the Internet exploded.

Adria Richards got a man fired. A man who had three children to feed. The Internet was not pleased. And to show its displeasure it sent Adria death threats, rape threats, racial epithets and suggested that she consider suicide. A group of hackers, some claiming to be Anonymous, initiated a series of DDOS attacks on her employer’s servers demanding that they fire her for retribution.

And because SendGrid, the company employing Adria, had no spine, they gave into the mob and publicly fired her. It was the easy thing to do, after all.

Justice served?

Bloggers the tech world over chimed in with their support or critique. Many asking whether she should have posted the photo of the two men and how she should have handled the incident differently, in a more lady-like fashion. Many jumped on a post by Amanda Blum that proved Richards “acted out” like this on more than one occasion, though Blum mentioned that she does not like Adria personally, and criticized her actions at PyCon, she did bring up the point that

Within 24 hours, Adria was being attacked with the vile words people use only when attacking women.

And this is the real issue, I think. And the bashful excuses from members of the tech community (both men and women) that “this is just how tech conferences are”, and “she should have a thicker skin”. The voices that suggest she shouldn’t have responded because the lewd comments were likely not directed at her seem to miss the point completely.

But at least we’re talking about it. Soon after the event the organizers of PyCon put the Code of Conduct up on GitHub, a popular open source hosting service, and invited members of the community to collaborate on changes in light of recent events. The community responded by adding language to the policy that prohibits public shaming. This is not unreasonable, and probably desirable and consistent with a “innocent until proven guilty” mentality. But unless a clear, easy communication path is given to report incidents as quickly and efficiently as twitter, in a private manner is provided, this could also be seen as a measure to silence others who may feel the need to speak out about poor conduct, but for whatever reason (and there are many) do not feel comfortable addressing the individuals directly.

The issue is not limited to sex or race, it is a larger one. Folks who are empowered by the status quo, whether they’re conscious of their priveledge or not, do not like the status quo challenged. Christie Koehler blogged about the incident from that perspective

It’s not easy because the tactics available to those who oppose institutional oppression are limited and judged by the very institution that is oppressive.

Those who benefit from the status quo, whether they realize it or not, have a vested interested in maintaining that status quo. That means working to ensure that any threat to it is rendered ineffectual. The best way to do that is to discredit the person who generated the threat. If the threat is the reporting of a transgressive act that the dominant social class enjoys with impunity, then the reaction is to attack the person who reported it.

And when it comes down to it, the vast majority of the negative backlash against Richards and her company (and none that I’ve heard of towards PlayHaven, the company that actually fired the male developer and started the whole fiasco) comes down to defending the status quo with a passion. People will fight for their place of privilege. They will fight hard and they will fight dirty.

And the very sordid nature of their fight will continue to prove unequivocally why we need to keep challenging the status quo until we create a world that is welcoming to all.

More reading:
Why Asking What Adria Richards Could have done different is the wronge question
Adria Richards did Everything Exactly Right

Stranger in a Commonplace Land

As I began reading the two introduction essays by Janet Murray and Lev Manovich to The New Media Reader I first was a bit overwhelmed with the length of each.  This immediately made me think of an article that was reverenced in the previous reading, “Is Google Making us stupid?“: was the fact that I initially gawked at so many words and pages a result of my immersion in a world of near-instant informational gratification and 140 character thoughts? The thing is, I have no problems whatsoever reading a 500 page novel, if it’s interesting and indeed there were certainly pieces of each introduction piece that jumped out at me:

All creativity can be understood as taking in the world as a problem. The problem that preoccupies all of the authors in this volume is the pullulating consciousness that is the direct result of 500 years of print culture. – Janet Murray

The concept of defining a unifying model that describes all of creativity is quite appealing to me.  “The world as a problem” seems at the same time both a grossly over simplified, and a perfectly succinct description of creativity  as I see it, and particular to my field of engineering.  Murray than goes on to draw contrasts between “engineers” and “disciplinary humanists” which particularly piqued my interest because I often feel like an outsider looking in when talking to other engineers about humanistic concepts, but also an outsider when trying to explain how I see engineering to “disciplinary humanists”.   The second essay   provided a nugget that helped direct my thoughts on this curious feeling of duplicity

Human-computer interface comes to act as a new form through which all older forms of cultural production are being mediated. – Lev Manovich

Whether we like it or not, this is becoming the reality.  We now get our books, music, movies and even long distance personal interaction mediated by a computer and the interface they provide us.  The thing is, any good engineer knows that if a piece of technology is doing its job, it should be transparent to the user.  While reading both of these essays I found myself thinking: why are we trying to force so much focus on the “new” in “new media”?  Is our doing so an indication that we as engineers still have more work to do to make the current technology transparent (I think we do) or is society so transfixed by “new” technology for some other reason that we are refusing to let it become as transparent as it could be?

Manovich, I think would disagree on that point, at least in the U.S. as one of his arguments for the late start of new media exhibits in the U.S. was in part do to the rapid assimilation of new technology so that it became ubiquitous before we had time to reflect upon its potential impacts.  As I’m writing that I feel myself rethinking my own view, because I don’t want to suggest that we not reflect upon the impact of technology that we now take for granted, in fact I have often felt we need to do much more reflecting, and I agree wholeheartedly that we have adopted some technologies that have drastically changed our day-to-day lives (who plans things in advance any more when you can just text your friends last minute to find out where people are?) that may consequences far extending the superficial sphere of their direct influences (if we don’t plan our days, are we losing our skill at thinking into the future and acting accordingly in general? Are we becoming a species obsessed with living in the moment and unable to live any other way?)

I’m in danger of rambling now, but I now have a better understanding of why I found it difficult to focus on the entirety of both essays.  Everything around each nugget either seemed redundant, overly descriptive, or a distraction from the thought process that had started forming in my head.  If good technology should be transparent to the user, why are we spending so much time worrying about it? And what are the consequences if we don’t?

The Freedom to be “Technologically Elite”

Also in response to Kim’s recent post.  I think the conversation about access and the issue of digital inclusion is a very important one to have, and we need to continue to be aware of how the tools and technologies we use may include or exclude people.  I would like to talk a bit about the concept of the “technological elite” that Kim brought up.  It’s important to be aware that it is possible for an elite minority to control the tools the majority comes to depend and rely on, but that is not currently the case, and in fact, I would argue that it will only become a reality if the majority allows it to happen.  As Jon Udell mentioned in his conversation the Internet itself has always been and continues to be an inherently distributed technology.  No single organization, whether it be corporate or governmental, owns or controls it.  There have been attempts, and there will continue to be attempts to restrict freedoms and tighten control, like the recent attempted SOPA/PIPA legislation, but it is our responsibility to continue to be aware of those attempts and fight them.

Many popular software tools in use, including WordPress which powers this blog, are free and open source.  This means that anyone can take a look at the source code to learn what is going on behind the scenes, and in many cases, modify and improve that tool for your own or public use.  The language that WordPress is written in, PHP is not only open-source, but there are a plethora of free tutorials online for anyone interested in learning how to program in the language.  The database used by WordPress to store and retrieve content, MySQL is currently open source, though the project itself was originally proprietary (Another relational database management system, PostgreSQL, has been open source for the entirety of its live-time and in many cases can be used as a drop-in replacement for MySQL). The majority of servers powering the internet run some version of the Linux operating system, itself freely available and open source.

Each of these projects, at the various layers the build up to form the tools that we use are generally well documented with enough information freely available to allow anyone who wants to become an expert in their use and design.  Now of course, not everyone will become an expert, and they experts for any one project are not necessarily experts in any other.  But specialization has allowed us to advance as a society in a way that would not be possible without it.

And because I love food:

When many of us began specializing in fields that did not involve agriculture and food production, we became dependent on those who did for our very survival.  Yet I can’t remember the last time I’ve heard anyone call farmers members of the “Agricultural Elite”.  Like the Internet tools I’ve mentioned, any of us have the agency to become experts in farming if we so choose.