For a translator to
turn one language (say, English) into another (say, Greek), she has to
be able to understand both languages and what common meanings they point
to, because English is not very similar to Greek.
It turns out that a similar task must be carried out within our own
minds when translating visual information into linguistic information — a
task Alon Hafri, an assistant professor in the Department of
Linguistics and Cognitive Science at the University of Delaware and
director of the Perception and Language (PAL) Lab,
is trying to better understand. His research is on the connection
between language and visual perception, or, in other words, between what
we say and what we see.
Understanding these two activities — seeing and speaking — may at
first seem trivial, Hafri said. Indeed, we're so good at these things
that we do them without thinking. But seeing and speaking are some of
the most complex processes that the human mind carries out. Hafri’s goal
is to understand how they work and how those two systems share
information.
Hafri gives a seemingly simple example: a cat on a mat. If we saw a
cat on a mat, we'd have no problem describing the scene. Or if someone
said, "Look, a cat on a mat," we'd easily be able to identify it out in
the world. Yet at first glance, an image of a cat on a mat and the
sentence “a cat on a mat” have nothing in common: the image has colors,
edges, shapes and locations, while the sentence has sounds, words,
phrases and such.
“If you think about the problem that the mind has to solve when
you're going from a visual scene to a sentence that you produce about
it, there's nothing in common really between words in a certain order
and a visual image,” Hafri said. “The kind of information that they
start with is very different. Something I'm really interested in is how
visual information gets translated — sort of like an English-to-Greek
translator would do — into a format that language can use, such that we
can talk about what we see. And vice versa, so that we can recognize the
things people are talking about in the world. The kind of questions I
ask are, well, how does that happen? What is the translation process?”
Watch a video on symmetrical meaning