How does the brain encode visual information?

The task of the visual system is a big one: to take an almost infinite amount of information in the world around us and to extract just the bits that are useful. Our eyes and brains are so good at this that we often take it for granted. But anyone tricked by a visual illusion knows that what we think the visual system is doing is often not at all what it is actually doing. The clearest example of this is the fact that our eyes and brains do not operate like a camera, taking photographs and developing film. Our delusion that this is how our visual systems operate can be quickly dispelled if you try and read this text with your peripheral vision. You can't, simply because different regions of your visual field are given different priorities by the brain. The fovea, or central part of your visual field, is what you use to read text. Reading actually involves scanning text with your fovea and then piecing it all together. So you can see that our perception of the world is very different from a photograph, where one can get all parts of an image in focus at the same time (if you have the right subject, aperture size and lighting!). Even more remarkable is the fact that our visual systems seem to be an amalgamation of several "sub" visual systems -- one dedicated to color, another to form and a third to motion and depth (although the real story isn't as simple). People fight about the details, but what is clear is that different brain cells at different stages in the visual system often respond to very different aspects of the visual scene. I am interested in how different brain cells compute information to give rise to perception and cognition. We made a breakthrough in 2013 when we discovered that neurons in the part of the brain implicated in "high-level" object recognition comprises several parallel pathways, each comprising of several stages: parallel, mulit-stage processing of faces, colors and places. In 2016, we showed that the same principles govern the organization of both human and macaque monkey visual cortex, providing some of the first insight into the deep structure of object-processing cortex. The work suggests that object cortex consists of about 3 or 4 relatively discrete sub-areas. You can read about the macaque work here and here, and the human work here.

Figure 9 from Lafer-Sousa et al, 2016. Comparison of the functional topography in monkey IT and human VVP suggests a broad homology across these primate species.

***

I've been especially interested in the neural computations for color and motion, and on the intersection between neuroscience, cognitive science and art:

1. Color Processing

This picture is from a 2001 paper in which I show that the color cells in the earliest cortical stage of processing are Double-Opponent (cover image; cover caption) . These important cells are called for in neural models of color vision, yet their existence was much disputed. In this cover illustration, each row shows the receptive field maps for a single cell. Green parts show the part of the retina that was excited by M cones (the "green" cones); red parts show the part of the retina that was excited by L cones (the "red" cones). You can see that each cell has green and red parts that are spatially offset.

The picture above is from a 2006 paper in which I confirm the original report of 2001 described above, and show that double-opponent cells come in two flavors, red-green (really red-cyan) and blue-yellow (cover image; cover caption).

Red-cyan and blue-yellow color cells are located in primary visual cortex, the first stage of visual processing in the cortex. They are the likely building blocks for specific hues, like red, pink, orange, purple etc, although specific hues themselves are probably represented not in primary visual cortex but rather in areas subsequent in the hierarchical processing of color, that receive input from primary visual cortex.

Double-opponent cells are also the probable basis for local color contrast; here you can watch a movie of a real recording of a double-opponent cell, discussed in more detail below. The power of chromatic contrast, shown in this demonstration, unravels the popular notion that color is simply a feature of physical light and places its basis squarely in the brain. Have you ever wondered why red looks redder against a green background? It probably is because these double opponent cells are wired up to relay local color-contrast signals. Put simply, these cells' "preferred" stimulus is a spot of one color, red or green, on a background of the opposite color, green or red.From Conway, BR (2001). See also Conway et al (2002).

The painting strategies used by artists such as Cezanne, Matisse, and Milton Avery offer insight into visual neuroscience: Cezanne and Matisse, both recognized colorists, developed their paintings in layers, working on the entire surface of the canvas as shown in these unfinished paintings. They did not fill-in line drawings, nor did they completely finish one part of the canvas before moving on to the next part. This allows them to constantly modify their color choices, accounting for the profound impact of local color contrast. The neural mechanisms I study show in turn how this strategy is successful.

A movie of this Double-Opponent cell,

When a green bar was restricted to the "green" surrounding crescent of the receptive field, the cell gave a burst of activity, which sounds like clicks on the audiomonitor, to the onset of the spot. And when the green spot was restricted to the "red" center, the cell gave a a burst of activity when the spot was turned off. The cell gave the opposite pattern of responses to a red bar. The monkey is looking at the yellow dot in the black square--his gaze is indicated by the giggling yellow square, and is not perfectly calibrated (if it were, the yellow square would overlap with the yellow dot). The boundary between the red and green subregions is indicated by the "x" drawn on the monitor, in the movie.

If double-opponent cells do not encode specific hues like pink, purple and orange, which cells do, and in what brain areas? With Doris Tsao, I have been using functional magnetic resonance imaging in the alert macaque to identify extrastriate color areas. I have recently recorded from these functionally identified domaines using targeted single-unit recording (Conway et al., Neuron 2007, cover image , cover caption, Supplementary Material). The neurons within the color domains, which we have dubbed "globs", are almost always strongly selective for specific hues. I hope to further test whether cells in these areas respond along other stimulus dimensions, like luminance boundaries (i.e. form), to gain insight into how different visual features are bound to produce a coherent visual percept. That luminance boundaries and color interact is clear from the afterimage demonstration below.

Afterimage responses are enhanced by boundaries

Stare at the red disc in (a) and after 10 seconds transfer your gaze to the fixation spot at right. Most people report a cyan disc afterimage.

Repeat for (b), below. Most people report a stronger afterimage, one that is bound to the outline and leaks out when the boundary is broken.

2. Motion Processing

I have also been looking at the neural machinery that underlies motion processing. We have found that the earliest stages that underlie conscious perception of visual motion are contrast-sign specific and we have discovered the physiological basis for reverse-phi. Reverse-phi is the phenomena that an object that changes contrast as it moves appears to move in the opposite direction from the physical progression of the object. There are some great visual illusions that demonstrate this. Reverse-phi is a consequence of the fact that the visual system is constrained to reconstruct motion signals from an input stage that is contrast-sign specific. Usually this isn't a problem for visual perception because objects rarely change contrast when they move. But it underscores one of the kinds of rules that the visual system employs to represent visual motion. Certain visual illusions, like the Kitaoka illusion, take advantage of this. You can read about this research: Conway and Livingstone (2003) , Livingstone and Conway (2003) and Conway et al (2005) (cover image; cover caption).

3. Vision and Art

In addition to studying basic neurophysiology of vision, I am interested in how art can be used to tell us something meaningful about how visual information is processed. The marks most artists make are fundamentally constrained by the kinds of computations performed by the visual system. Therefore artwork should be useful in telling us something about how the visual system is wired. Moreover, as an artist myself, I wonder whether the visual system can be useful in understanding how, why and when some art is successful. Check out Livingstone's book: Vision and Art, and stay tuned for my book on Vision and Art Practice.

This picture shows 10 Rembrandt self-portraits -- 5 etchings and 5 paintings. In the paintings Rembrandt depicts himself with the eye on the right of the painting directed at you, and the other eye (the outside eye, on the left of the painting) deviated outwards, or wall-eyed. His eyes show the reverse pattern in the etchings because etchings are mirror images of paintings. Margaret Livingstone and I have asked whether Rembrandt was stereoblind; i.e. is not able to see stereo-depth cues. This might actually be an asset for an artist whose goal is to paint the 3-D world on a flat canvas. The New York Times did a nice story on this idea.

While investigating the role of stereovision in artistic practice, I have become interested in how viewing conditions, i.e. the relationship of the viewer to the subject, shapes image making. These musings are described in a recent Current Opinion article, which you can read about here.

Some of my work in this field is captured in a presentation given to the Center for Arts Science and Technology (CAST) at MIT. And our work on #TheDress drew international interest; some of my thoughts are described in an op-Ed in The Guardian, and in an essay On neuroaesthetics and the trouble with beauty, with Alexander Rehding published in PLoS Biology.

go to Publications