Reconstructing images from FMRI data

01 — Stimulus

A subject is shown an image.

The visual cortex responds. What the scanner records is that response — never the image itself.

02 — Acquisition

fMRI measures blood oxygen, not light.

The cortex is sampled as a volume of voxels. Each one is a coarse, delayed proxy for the activity of roughly a million neurons.

Volume 2 sVoxels ~100 kPixels 0

03 — Signal

The recording is three-dimensional and mostly noise.

Haemodynamic lag, head motion and scanner drift all land on top of the response. Any structure worth having is buried underneath them.

04 — Encode

Voxel activity is mapped into a latent space.

A learned encoder collapses tens of thousands of noisy measurements into a few hundred numbers. Dimensionality falls away; structure survives.

In ~100 kOut 512Loss perceptual

05 — Decode

A convolutional generator expands it back into an image.

The latent is grown through successive upsampling layers. A discriminator trained alongside it keeps the output on the manifold of plausible images rather than blurred averages.

Arch CNN + GANTraining adversarial

06 — Resolve

The image comes out of the noise.

Coarse structure first — orientation, mass, the position of edges. Detail arrives last, and only where the signal supported it.

07 — Result

Reconstructed from brain activity alone.

No image was ever given to the model. Everything above was inferred from the measured response of a visual cortex looking at something.