Simple Vision
Think about how you see the world (or hear it or feel it), and how much of your perception is actually to do with your eyes (or other senses) versus how much is created in your mind. It’s not as clear as you might think. Remented explores this tricky idea through simple illustrations that let you think about your own experience, and how your perception may not work quite how you think it does. In this example you will take over the sight and the mind of a very simple, imaginary creature to explore just how the art of sight is about much more than what the eye sees.
A Simple Creature with a Simple Eye
Imagine that you are in the mind of a really simple creature, floating about in the ocean. You have just one sense – a very basic form of vision. You can sense levels of light and darkness right in front of you in low resolution.
Using this basic vision you can see changes of light and shade in the background, and if something should pass directly in front of your eye, you can see a change.
This is what you can see.
What can you tell about the world? Not very much! But look again. As you watch this scene, what can you work out? Maybe you can see that there are many smaller shapes that move across your field, and, occasionally a much larger shape. What is registering those shapes and the movement? Your eye alone cannot determine any of these – it can only report light and darkness in each tiny part of the view. It is your mind that associates nearby areas of similar shade together as a unit, as a shape. If an area of shade appears in one place, and moments later an area of shade appears nearby, but not exactly where you last saw it, you will tend to ‘see’ it is as one shape that has probably moved a little bit. The alternative would be to guess that it is a new set of shaded bits while the previous ones disappeared, which is not impossible, just less likely. Your eye has not given you this information. Your mind has built it. The idea of a shape and a place where it exists in the world is a mental construction, tested against sense data from the eye to update it. Already your sight is made up of raw information about light and dark, plus some guesses you make about that and you are doing this automatically. Now look at this series of still images below and see just how little sense information you are getting from your eye. And now think about how much you have determined about the world in front of you already.
Still images from the eye feed
Eat or be Eaten: Our Creature in Context
- If something is smaller than you, and it is in range you can eat it.
- If something is bigger than you, and it covers you, it will most likely eat you.
So now we have loaded up what you see with a lot more meaning. Food is good, predators are bad. Each of these will trigger an instinct for our creature to obey its basic biological drives for sustenance and survival. These would seem to have no place in seeing, but in fact they add a massive amount of context to the scene. And that context makes what you see suddenly a much, much richer, and more vital source of information for making decisions. Here is the scene again, but with a small guide circle in the middle of your view, that shows when something smaller is in position for you to eat it. If you can, you will eat it (and the screen will flash green to tell you). There is a larger guide that shows you the danger zone for predators. If a predator covers this field – you will get eaten (the screen will flash red). Of course if you were really this creature, you would not have targets laid over your vision, but your mind would know that a certain area in your sight means that something is in range to eat, and that if more than a certain amount of your view is covered you are in danger. These guides on the screen represent the meaning that your mind adds to what you see.
What we have done is add context and meaning to the raw vision, that make fairly low-density information from our eye much richer. You can’t easily un-see these extra layers of information and get the raw feed from your eye anymore. Your sight has become more complex as your mind has added meaning.
Predictions and Decisions
You could say that your mind is building a kind of model of the world, of food and predators within it, and their relation to you. The eye does not create this model, it just sees light and shade. Rather than just vague shapes, you are identifying some as threats and some as opportunities. On top of this we can start to make predictions about where these elements in our model world will be in a moment, and overlay that prediction on top of the raw sense data. In fact we can start to predict information about things that are just out of sight or which are only partially seen. What we see is much richer because it is made up of bottom-up sense data, and top down meaning and prediction. In the illustration below, our creature has clearly identified food and foes and makes predictions about how they are moving. The green circular predictions for food and red for predators represent this extra information in your mind.
You now have the ability to move about in a basic way (use your mouse or tap to move), so that you can swim in any direction, to chase food, or to avoid predators. Remember how limited your raw sight is, and think about how rich our experience is now, with the ability to judge opportunity and danger, and take action (by moving in range of food and avoiding predators). The eye is giving us the same data, but we have layered up with more information and our predictions, and what we now see in our mind’s eye is all of this added together.
This illustrates a much richer experience of perception, even though the raw sense data is still just as limited. You have enough now to try and survive as our vision impaired creature. Just to add even more of a challenge, and to show how little you are dependent on the raw sight from your eye, I have given the creature a tendency to blink occasionally, you can see that your mind’s predictions can carry on pretty well even without the immediate sense data. You know how you expect the world to look, even when your view is interrupted. This is significant, because while you blink, you are cut off from the eye’s view of the world. And yet you do not lose your vision when this happens. What you see, is only partly about what you can see with your eyes (if you see what i mean)!
This has been an introduction to the idea that what you see (illustrated through what our imaginary creature sees), is only partly to do with what your eyes report to the brain. So much of your perception (shape, objects, movement, meaning) is about what your mind makes of this very basic feed, together with other information you hold about the world, and what is important to you right now. With your mind layering this information on top of raw senses, you have really quite a rich perception that your eye cannot provide. You have the ability to predict what will happen before it is seen, make decisions, even when the sense stream is very limited, or when it is missing altogether (such as when you blink). This is quite a bold claim – that what you perceive is as much (maybe more) to do with what you project out into your view of the world, as it is to do with what your senses provide. I also introduced the idea that your mind can use predictions of what we expect to see, to help us make decisions about the world, If the prediction is wrong, you update your view and your next prediction with the new information received from the eye (or other senses).
The Game
Enough talk, here is a game version of the creature with food to eat and predators to avoid. You can move within the world, and you have a tendency to blink. You can run the game in a separate window if you want.
Human Vision
So that’s all very well for the strange little see creature, with its one, simple eye. What about us amazing humans? The point of choosing the creature is that is world is simple, it’s mind is simple, and so are the choices it faces. The principles can help us think about what happens between our eyes own detecting light and colour within our field of view, and us having a meaningful sense of the world, complete with a model of what is out there, where it is, what it means to us.
Our vision is really pretty sophisticated, and a huge amount of our brain is given over to it. All the points that are true for our creature are true for us as well, but many times more complicated. As powerful as it is, our brain is not powerful enough to process all the information available to our eyes and build a perfect picture of the world from scratch in every moment. We do what the creature does – we take what we know already and project back into (our model of) the world our best guess or prediction of what is there and how it is behaving. This, together with new information or corrections gives us our perception of the world.
This definitely needs to be explored some more. The only way I can build up my own understanding is taking some feature or experience at a time and exploring it. I hope to do that soon in other Remented posts.
The Science
There is a huge amount of research about how we see – so much more than I can do justice to here. Neuroscientists have been able to identify specific areas of the human brain, and animal brains, and the networks between them that are responsible for different aspects of sight, and other senses. Researchers have mapped networks of neurons that specialise in responding to colours, lines, shapes, objects, and how interactions between these groups are specialised at such granular tasks as recognising faces, or specific kinds of motion. They have identified that there is more than one pathway from the eye into different parts of the brain, so that they are now pretty confident that one pathway identifies objects and visual properties, while another identifies where objects are in space and in relation to each other. There is evidence that there is a pathway, much faster than the full-perception path, that routes sight to the fear-response parts of the brain to give us a chance to respond quickly to potential danger, before we are even completely sure what the danger is.
Here is the staggering and relatively recent discovery. It turns out that there are as many, if not more, neurons passing from each visual area to the preceding step in the chain, adding more evidence to the idea of there being both top-down, as well as bottom-up perception. Like so many aspects of the brain and mind, this runs against what our intuition tells us about how sense work. Our experience from the inside does a good job of masking the true processes going on, of which we are mostly unaware (and that’s where Remented comes in!).
That’s the Neuroscience angle. But psychologists have been just as fascinated by what we see, and how our senses can be fooled (by things like emotion, our other senses, how consistently a scene changes, how lighting and context can alter our interpretations about the same image). There are significant studies showing how what we believe to be true and our biases can significantly alter how we interpret what is happening in a scene.
Sometimes discoveries are made around how things go wrong for some individuals. For example there are various forms of agnosia, where even though it can be proven that a person’s basic senses are perfectly detecting the world, the mind suffers an inability to form coherent objects, spaces, faces, or make judgements. There can be a failue to add meaning to the raw sense information and make sense of it.
This is a vast area that I am not qualified to lead you through. But I will come back and explore some of these related areas in the Remented way later on.
Here are two areas of research that I will recommend you take a look at:
The late Oliver Sachs, a neurologist who himself suffered from a form of agnosia, talks in his many books and studies about many aspects of perception and the mind generally.
In this TED talk, Oliver discusses what we can learn about perception and vision from hallucinations (which is where the projections a person makes onto the world are so powerful that they displace or distort any sense data from the world.
In his book, Visual Intelligence (from back in 1998, so a little dated) Donald Hoffman attempts to reverse-engineer the mind’s ability to construct a world from its senses (principally from vision). He has some unconventional views that the world we experience is not only constructed in our mind, but is a metaphor for the actual world (in the way that an icon on your computer screen is a metaphor for an app or function). These constructions have allowed us to be highly successful in evolutionary terms, but he claims that this does not make it likely that we are perceiving the world as it actually is.
See his 2015 TED talk here.
Other Experiences
It is very easy for me to start a discussion like this, and get stuck in my own perspective. The point of Remented is that it explores the mind and the brain from the human experience. The experiences I chose may not represent you.
An obvious flaw here is that my focus in this article has been about how we see; so what if you cannot see, or you do not see in the way that others do?
The human brain devotes a huge amount of space and ability to vision, in what is called the visual cortex. It is one of the unique features of the human brain. But it is a bit misleading to call it visual. A lot of it is to do with spacial reasoning, or understanding relationships between things, and the properties of objects. It turns out that those projections of the world and predictions about objects, where they are, what they mean (to you, eg. food, danger) are active in almost the same way with our other senses. If someone is blind or visually impaired, their cortex responds in similar ways to when a sighted person sees something. As with the blinking example above, so much of our perception does not rely on direct sight. Even as a sighted person, you can shut your eyes and be able to tell what kind of space you are in from the way that sounds reverberate, and what kinds of sounds you hear. You can tell if you are inside or outside, how large the space is is, whether you are alone, or with a few or many people. If your world is without vision altogether you will be even better at this. Most of the illustration above for making judgements and predictions about the world would work just as well with other sense. We use these spacial reasoning abilities even for abstract ideas that don’t originate from any sense. When you imagine the relationships between people you know, your family, how something works, and when you make plans by imagining a sequence of tasks, you are using these same pieces of brain machinery, and mental powers.
Just think for a moment how movies, animated cartoons, computers/phones, and more recently virtual and augmented reality deliberately fool our senses in order to make our minds project something that is not there, as though it is. They work because designers and programmers are learning how to plant compelling ideas in your mind, and trigger the senses in a way that enhances those ideas.
You may have been born with exceptional attention to detail, or amazing spacial reasoning that makes you great at solving certain kinds of problems, but poor at noticing subtle signals of people’s behaviour around you. It is unlikely that your eyes work very differently, but how your mind processes, prioritises, and adds meaning to the view can be very different for the person standing right next to you, looking at the same scene.
There are things that can go wrong in brain mechanics. So for example, certain areas of the brain can become damaged (maybe as a result of a stroke or injury), and prevent a person from being able to process their perfectly working sense images from the eye into meaningful objects, or faces. These can be distressing and debilitating. Sometimes the plasticity of the brain (its ability to reconfigure its connections) can compensate for such damage, but often it cannot.
Our beliefs about the world can have a huge affect on what we see in a scene. It can affect what we decide is important for us to place our limited focus on, so that we may miss other details, and we may assess something as being benign, while the person next to us sees something threatening. Conditions and diseases, such as dementia, can cause sufferers to hold a different set of beliefs about where they are, who they are with, what year it is, and thus have a very different projection on the same scene as you are seeing.
I am here visiting some of the reasons why any individuals experience of vision could be wildly different from mine, but still concluding that the ideas illustrated above are relevant, and maybe even explain why one person’s experience of a scene can be so very different from another’s.