[2305.18274] Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors

Abstract:We present MindEye, a novel fMRI-to-image approach to retrieve and reconstruct viewed images from brain activity. Our model comprises two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). MindEye can map fMRI brain activity to any high dimensional multimodal latent space, like CLIP image space, enabling image reconstruction using generative models that accept embeddings from this latent space. We comprehensively compare our approach with other existing methods, using both qualitative side-by-side comparisons and quantitative evaluations, and show that MindEye achieves state-of-the-art performance in both reconstruction and retrieval tasks. In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that MindEye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters. Furthermore, we show that MindEye can better preserve low-level image features in the reconstructions by using img2img, with outputs from a separate autoencoder. All code is available on GitHub.

# link_2ola4aara7.pdf ## Metadata - PDFFormatVersion=1.5 - IsLinearized=false - IsAcroFormPresent=false - IsXFAPresent=false - IsCollectionPresent=false - IsSignaturesPresent=false - CreationDate=D:20231010035228Z - Creator=LaTeX with hyperref - ModDate=D:20231010035228Z - Custom.PTEX.Fullbanner=This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) kpathsea version 6.3.5 - Producer=pdfTeX-1.40.25 - Trapped=False ## Contents ### Page 1 Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion PriorsPaul S. Scotti*,1,2, Atmadeep Banerjee*,2, Jimmie Goode†,

saved by

• Karthik Suresh
• Selena Yang

related reading

• Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priorsmedarc-ai.github.io
• MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Datamedarc-ai.github.io
• AI to Interpret Your Mindjonathanxu.com
• Retrieving and reconstructing conceptually similar images from fMRI with latent diffusion models and a neuro-inspired brain decoding model - IOPscienceiopscience.iop.org
• 2306.16934.pdfarxiv.org
• [2506.06898] NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imageryarxiv.org
• Toward accessible, real-time brain decoding: Introducing ENIGMA | Alljoined Blogalljoined.com
• Stable Diffusion with Brain Activitysites.google.com
• Semantic reconstruction of continuous language from non-invasive brain recordings | Nature Neurosciencenature.com
• Learned Spectrumtheodore.net
• 2310.16532.pdfarxiv.org
• Attentionally modulated subjective images reconstructed from brain activity | bioRxivbiorxiv.org