A large body of research demonstrates that human faces attract attention. This is also true in video, with eye-tracking studies showing that fixations are clustered on the people in the scene. I will review a number of studies from psychology investigating where observers look when watching people in video, and how this is related to their understanding and judgements of the scene.
For example, participants are quite consistent in their judgements of people having a conversation, even from a brief video clip (“thin-slicing”). We have shown that when watching such clips the individuals who receive the most attention are those who are rated as the most dominant or prestigious in the group. This is one case where participants must make a decision about who to fixate at each point in time, and this appears to vary due to both the behaviour of the actor and the characteristics of the observer. In crowd scenes featuring multiple people, certain targets get fixated more often than others and this is affected by age and attractiveness.
When observing pre-recorded conversation, participants spend most of the time fixating the person speaking, but they can also anticipate the change in speaker. This may be partly due to reading the cues provided by interacting participants (such as their gaze). We can manipulate these cues to study signalling in a naturalistic setting, and we have also shown nuanced differences in observers high in autistic and ADHD-related traits. Importantly, we also find a close correspondence between fixations on pre-recorded videos and the gaze displayed by participants in a real face-to-face interaction, which suggests a high level of ecological validity.
Taken together, these studies show “social attention” operating in complex and dynamic situations. This involves not just looking at other people, but selecting specific individuals, and specific behavioural cues, in order to interpret the scene.