This ongoing project asks whether automatic emotion labels can match the moment-by-moment feeling that human viewers report during films, well enough to ease the slow, costly human annotation that affective neuroscience now depends on. I test this from two sides: one arm reads the words the characters speak, the other reads the brain's response.
Le Thi Thach Thao (Jessica) · National Taiwan University
Scan to open
This pageCode on GitHub
Scroll
01The question
Can a model match the human ratings?
Naturalistic-film fMRI needs a feeling for every moment, and today only humans can label it. Two arms test whether a model can.
HUMANThe feelingthe averaged human consensus, the ground truth.→
TEXTThe wordstranscript scored by BERT and Gemini.→
BRAINThe signalvalence decoded straight from fMRI.
02The data
Emo-FilM, an open affective-neuroscience dataset
The human ground truth is the averaged consensus of 44 annotators over 14 short films, all openly released under CC0.
14
short films
30
participants · 3 T
44
annotators
50
emotion items
1.3 s
TR
Preprocessed BOLD in MNI space. OpenNeuro: fMRI ds004892 (CC0) and annotations ds004872 (CC0). Morgenroth et al. (2025), Scientific Data 12:684.
03A transcription trap
The labels read the wrong words
Whisper transcribes songs, recited verse, and credits as if they were dialogue, and the models score that text. On Spaceman, removing about 94 seconds of song and credits, 93 paired seconds of data, flips the correlation.
BERT −0.29 → −0.11 (phantom negative). Gemini −0.10 → +0.24 (real positive). Found in 5 of 12 films.
Spaceman · same 94 s removedtap to enlarge
Polarized classifiers scored the despairing lyrics as strong negatives; the contrast collapses toward zero once the non-dialogue text is removed.
04The words
Machines agree only where words carry the feeling
Give a strong model the surrounding dialogue and it tracks human valence, but only on talk-driven films.
Agreement with human feeling · four filmstap to enlarge
Per-segment BERT shows no real context gain; its one rise is blurring, not comprehension. Gemini's gains are genuine, and only on talk-driven films.
05The brain, first look
One subject cannot settle the sign
Same film, same pipeline. Human valence decodes positive from the highest-motion subject and negative from the cleanest, so a single brain is not enough.
The +0.21 I almost reported, shown only as the result I corrected. It motivates a properly powered test.
RETRACTED · shown only as the error I caught
Tears of Steel · same film, opposite signstap to enlarge
Same mask, same pipeline, same film. The +0.21 from one subject reverses on another; a single subject cannot establish a sign.
06The brain, the controlled test
A validated decoder finds no occipital valence
Powered to five subjects per film with a positive control, the decoder recovers a planted signal but reads real valence at chance, on both films.
Read from 43,683 voxels of posterior occipital cortex, a visual region, not an affect region.
The region I read · occipitaltap to enlarge
Posterior occipital: a valence-correlated visual signal, not affect.
Validated decoder · two filmstap to enlarge
The positive control passes (left); real human valence sits at its own chance level on both films (right).
07What it tracks
What looked like emotion was motion
Run the same decoder on what is on screen, and the occipital signal follows the film's motion, while valence stays flat.
Occipital signal · what it decodestap to enlarge
Visual features clear chance; valence does not. It is a visual region; it carries the picture.
08Where they meet
Both arms stop at the same wall
Feeling drifts slowly, holding a memory of about 80 to 110 seconds. Short films leave too few independent moments for text or brain to lock on.
Reported as effect sizes with cross-model, cross-film replication. Two arms converging on the same limit is stronger than any single p-value.
·Takeaway
Text-based sentiment, even with a frontier LLM, is not yet a robust substitute for human emotion annotation in naturalistic film. The language and brain arms reach the same wall from two directions. Every figure, effect size, and control is public.
What's next. The affective regions are the next test: whether the insula, vmPFC, and amygdala carry a valence signal occipital does not, and whether surrounding dialogue helps beyond the four films tested.
