The other day when I picked my three year old son up from daycare, I noticed his teacher had taped an image from a kid’s magazine up on the wall. Beneath it were all of these cute little musings and observations made by the daycare students. “This picture shows a boy and a girl.  The boy is sitting on a ball.  The girl is being silly.”

When I got to my son’s write-up, it was nearly a page long and he had something to say about everything. 

“I think the girl is laughing because the boy has shark teeth on the bottom of his shoes.” (He did!) “The girl has curly hair and the boy likes that her shoes are pink. He likes playing ball games and wants her to be his friend. His skin is a little bit lighter than hers and he has a sweater on…”

The son of a scientist and a writer; he is both observant and prolific.

Devi Parikh, an assistant professor in the Department of Electrical and Computer Engineering at Virginia Tech, has recently won a Google Faculty Award to take a very similar idea and use it to teach computers a little bit of common sense.  Although, instead of using toddlers and magazines to generate data, she is using Amazon Mechanical Turk Workers (“Turkers”) and clip art. Through this visual-outsourcing, hundreds of thousands of vetted users from all over the world can create abstract scenes to teach computers object recognition, properties, relationships and poses. These extremely rich scenes touch on something that has never been done before — turning semantic visual features into language.

We all know that (most) kids are afraid of bears. A scene depicting a scared child next to a large, snarling bear would surely make sense to us, but to a computer this is a real head-scratcher.  Parikh hopes to use these user-created clip art scenes and depictions to teach computers much of the information we as humans take for granted. In doing so, she will advance computer vision and enhance a variety of machine learning applications, such as autonomous driving.

And as a mother of two young children and oftentimes distracted minivan driver, anything that brings autonomous driving closer to reality definitely gets the green light.