Resources for 2024 Summer School on AI and Cognitive Neuroscience

Russ Poldrack

All code available at https://github.com/poldrack/AI_in_CogNeuro. All code shared under an MIT License. All other materials shared under a CC-BY license unless otherwise noted.

Lectures

1: What is "AI" and where did it come from?

2: What questions can we ask using AI/ML methods?

3: AI tools for fMRI analysis

4: Foundation models for fMRI

5: Alignment of AI models and neural data

6: AI-assisted coding

7: Doing reproducible research in the context of AI

Jupyter notebooks

Note: some of these notebooks are rather messy!

data_setup_haxby.ipynb

decoding_haxby.ipynb

cv_variability.ipynb

perceptron.ipynb

predicting_rare_things.ipynb

assessing_performance.ipynb

leakage.ipynb

response_estimation.ipynb

Reports for Haxby Subject 1

fMRIPrep report

MRIQC anatomical report

MRIQC functional report

Example group reports for OpenNeuro ds000030

MRIQC group anat example

MRIQC group BOLD example

Readings

Reproducibility

Gael Varoquaux. Cross-validation failure. NeuroImage, 2017, ff10.1016/j.neuroimage.2017.06.061ff. ffhal-01545002f

Varoquaux G, Cheplygina V. Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ Digit Med. 2022 Apr 12;5(1):48. doi: 10.1038/s41746-022-00592-y. PMID: 35413988; PMCID: PMC9005663.

Haibe-Kains, B., Adam, G.A., Hosny, A.Êet al.ÊTransparency and reproducibility in artificial intelligence.ÊNatureÊ586, E14ÐE16 (2020). https://doi.org/10.1038/s41586-020-2766-y

Gundersen, O.E., Gil, Y. and Aha, D.W. (2018), On Reproducible AI: Towards Reproducible Research, Open Science, and Digital Scholarship in AI Publications. AI Magazine, 39: 56-68.Êhttps://doi.org/10.1609/aimag.v39i3.2816

Open Science Collaboration. PSYCHOLOGY. Estimating the reproducibility of psychological science. Science. 2015 Aug 28;349(6251):aac4716. doi: 10.1126/science.aac4716. PMID: 26315443.

MŸller VI, Cieslik EC, Serbanescu I, Laird AR, Fox PT, Eickhoff SB. Altered Brain Activity in Unipolar Depression Revisited: Meta-analyses of Neuroimaging Studies. JAMA Psychiatry. 2017 Jan 1;74(1):47-55. doi: 10.1001/jamapsychiatry.2016.2783. PMID: 27829086; PMCID: PMC5293141.

Button KS, Ioannidis JP, Mokrysz C, Nosek BA, Flint J, Robinson ES, Munaf˜ MR. Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013 May;14(5):365-76. doi: 10.1038/nrn3475. Epub 2013 Apr 10. Erratum in: Nat Rev Neurosci. 2013 Jun;14(6):451. PMID: 23571845.

Poldrack RA, Baker CI, Durnez J, Gorgolewski KJ, Matthews PM, Munaf˜ MR, Nichols TE, Poline JB, Vul E, Yarkoni T. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nat Rev Neurosci. 2017 Feb;18(2):115-126. doi: 10.1038/nrn.2016.167. Epub 2017 Jan 5. PMID: 28053326; PMCID: PMC6910649.

Carp J. On the plurality of (methodological) worlds: estimating the analytic flexibility of FMRI experiments. Front Neurosci. 2012 Oct 11;6:149. doi: 10.3389/fnins.2012.00149. PMID: 23087605; PMCID: PMC3468892.

Botvinik-Nezer et al., Variability in the analysis of a single neuroimaging dataset by many teams. Nature. 2020 Jun;582(7810):84-88. doi: 10.1038/s41586-020-2314-9. Epub 2020 May 20. PMID: 32483374; PMCID: PMC7771346.

Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns (N Y). 2023 Aug 4;4(9):100804. doi: 10.1016/j.patter.2023.100804. PMID: 37720327; PMCID: PMC10499856.

Verstynen T, Kording KP. Overfitting to 'predict' suicidal ideation. Nat Hum Behav. 2023 May;7(5):680-681. doi: 10.1038/s41562-023-01560-6. Epub 2023 Apr 6. PMID: 37024723.

AI

S. Bubeck et al., Sparks of Artificial General Intelligence: Early experiments with GPT-4

Kautz, H. A.Ê2022. ÒÊThe third AI summer: AAAI Robert S. Engelmore Memorial Lecture.ÓÊAI MagazineÊ43:Ê105Ð125.Êhttps://doi.org/10.1002/aaai.12036

M Minsky, Steps Toward Artificial Intelligence, 1961

W. S. McCulloch and W. Pitts, A logical calculus of the ideas immanent in nervous activity, 1943

A. M. Turing, Computing Machinery and Intelligence,ÊMind, Volume LIX, Issue 236, October 1950, Pages 433Ð460,Êhttps://doi.org/10.1093/mind/LIX.236.433

Rosenblatt, F. (1958).Ê"The perceptron: A probabilistic model for information storage and organization in the brain".ÊPsychological Review.Ê65Ê(6): 386Ð408.Êdoi:10.1037/h0042519.ÊISSNÊ1939-1471.ÊPMIDÊ13602029.

Finnie-Ainsley et al., The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming, 2022

Deep Learning

Belkin M, Hsu D, Ma S, Mandal S. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proc Natl Acad Sci U S A. 2019 Aug 6;116(32):15849-15854. doi: 10.1073/pnas.1903070116. Epub 2019 Jul 24. PMID: 31341078; PMCID: PMC6689936.

A. Doering et al., The neuroconnectionist research programme, 2022.

Krizhevsky, A., Sutskever, I. and Hinton, G. E. ImageNet Classification with Deep Convolutional Neural Networks. 2012

Vaswani et al., Attention Is All You Need, 2017

Power et al., Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, 2022

Templeton, et al., "Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet", Transformer Circuits Thread, 2024.

AI-assisted coding

Poldrack, Lu, and Begus, AI-assisted coding: Experiments with GPT-4. 2023.

Decoding

Hamdan S, Love BC, von Polier GG, Weis S, Schwender H, Eickhoff SB, Patil KR. Confound-leakage: confound removal in machine learning leads to leakage. Gigascience. 2022 Dec 28;12:giad071. doi: 10.1093/gigascience/giad071. Epub 2023 Sep 30. PMID: 37776368; PMCID: PMC10541796.

Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science. 2001 Sep 28;293(5539):2425-30. doi: 10.1126/science.1063736. PMID: 11577229.

Naselaris T, Kay KN, Nishimoto S, Gallant JL. Encoding and decoding in fMRI. Neuroimage. 2011 May 15;56(2):400-10. doi: 10.1016/j.neuroimage.2010.07.073. Epub 2010 Aug 4. PMID: 20691790; PMCID: PMC3037423.

Poldrack RA, Huckins G, Varoquaux G. Establishment of Best Practices for Evidence for Prediction: A Review. JAMA Psychiatry. 2020 May 1;77(5):534-540. doi: 10.1001/jamapsychiatry.2019.3671. PMID: 31774490; PMCID: PMC7250718.

Brown et al., ADHD-200 Global Competition: diagnosing ADHD using personal characteristic data can outperform resting state fMRI measurements, 2012

Varoquaux G, Raamana PR, Engemann DA, Hoyos-Idrobo A, Schwartz Y, Thirion B. Assessing and tuning brain decoders: Cross-validation, caveats, and guidelines. Neuroimage. 2017 Jan 15;145(Pt B):166-179. doi: 10.1016/j.neuroimage.2016.10.038. Epub 2016 Oct 29. PMID: 27989847.

Hosseini M, Powell M, Collins J, Callahan-Flintoft C, Jones W, Bowman H, Wyble B. I tried a bunch of things: The dangers of unexpected overfitting in classification of brain data. Neurosci Biobehav Rev. 2020 Dec;119:456-467. doi: 10.1016/j.neubiorev.2020.09.036. Epub 2020 Oct 6. PMID: 33035522.

Cognitive Neuroscience

Yarkoni T, Poldrack RA, Nichols TE, Van Essen DC, Wager TD. Large-scale automated synthesis of human functional neuroimaging data. Nat Methods. 2011 Jun 26;8(8):665-70. doi: 10.1038/nmeth.1635. PMID: 21706013; PMCID: PMC3146590.

Poldrack, R. A. (2010). Mapping Mental Function to Brain Structure: How Can Cognitive Neuroimaging Succeed? Perspectives on Psychological Science, 5(6), 753-761. https://doi.org/10.1177/1745691610388777

Newell, A. (1973). You can't play 20 questions with nature and win : projective comments on the papers of this symposium.

Poldrack RA, Yarkoni T. From Brain Maps to Cognitive Ontologies: Informatics and the Search for Mental Structure. Annu Rev Psychol. 2016;67:587-612. doi: 10.1146/annurev-psych-122414-033729. Epub 2015 Sep 21. PMID: 26393866; PMCID: PMC4701616.

Poldrack, R. A., Halchenko, Y. O., & Hanson, S. J. (2009). Decoding the Large-Scale Structure of Brain Function by Classifying Mental States Across Individuals. Psychological Science, 20(11), 1364-1372. https://doi.org/10.1111/j.1467-9280.2009.02460.x

Poldrack RA, Kittur A, Kalar D, Miller E, Seppa C, Gil Y, Parker DS, Sabb FW, Bilder RM. The cognitive atlas: toward a knowledge foundation for cognitive neuroscience. Front Neuroinform. 2011 Sep 6;5:17. doi: 10.3389/fninf.2011.00017. PMID: 21922006; PMCID: PMC3167196.

Markiewicz CJ, Gorgolewski KJ, Feingold F, Blair R, Halchenko YO, Miller E, Hardcastle N, Wexler J, Esteban O, Goncavles M, Jwa A, Poldrack R. The OpenNeuro resource for sharing of neuroscience data. Elife. 2021 Oct 18;10:e71774. doi: 10.7554/eLife.71774. PMID: 34658334; PMCID: PMC8550750.

Poldrack RA et al, The past, present, and future of the brain imaging data structure (BIDS). Imaging Neuroscience 2024; 2 1Ð19. doi: https://doi.org/10.1162/imag_a_00103

Varoquaux G, Schwartz Y, Poldrack RA, Gauthier B, Bzdok D, Poline JB, Thirion B. Atlases of cognition with large-scale human brain mapping. PLoS Comput Biol. 2018 Nov 29;14(11):e1006565. doi: 10.1371/journal.pcbi.1006565. PMID: 30496171; PMCID: PMC6289578.

Walters J, King M, Bissett PG, Ivry RB, Diedrichsen J, Poldrack RA. Predicting brain activation maps for arbitrary tasks with cognitive encoding models. Neuroimage. 2022 Nov;263:119610. doi: 10.1016/j.neuroimage.2022.119610. Epub 2022 Sep 3. PMID: 36064138; PMCID: PMC9981816.

fMRI

Esteban O, Birman D, Schaer M, Koyejo OO, Poldrack RA, Gorgolewski KJ. MRIQC: Advancing the automatic prediction of image quality in MRI from unseen sites. PLoS One. 2017 Sep 25;12(9):e0184661. doi: 10.1371/journal.pone.0184661. PMID: 28945803; PMCID: PMC5612458.

Esteban O, Markiewicz CJ, Blair RW, Moodie CA, Isik AI, Erramuzpe A, Kent JD, Goncalves M, DuPre E, Snyder M, Oya H, Ghosh SS, Wright J, Durnez J, Poldrack RA, Gorgolewski KJ. fMRIPrep: a robust preprocessing pipeline for functional MRI. Nat Methods. 2019 Jan;16(1):111-116. doi: 10.1038/s41592-018-0235-4. Epub 2018 Dec 10. PMID: 30532080; PMCID: PMC6319393.

Dadi et al., Fine-grain atlases of functional modes for fMRI analysis, 2020

Mumford JA, Turner BO, Ashby FG, Poldrack RA. Deconvolving BOLD activation in event-related designs for multivoxel pattern classification analyses. Neuroimage. 2012 Feb 1;59(3):2636-43. doi: 10.1016/j.neuroimage.2011.08.076. Epub 2011 Sep 5. PMID: 21924359; PMCID: PMC3251697.

Jacob S Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, Kendrick N Kay (2022) Improving the accuracy of single-trial fMRI response estimates using GLMsingle eLife 11:e77599

Coding Practices

D Boswell and T. Foucher, The Art of Readable Code

D. Thomas and A. Hunt, The Pragmatic Programmer

M. Fowler, Refactoring

R. Martin, Clean Code

J. Osterhout, A Philosophy of Software Design

Naming Things: The Hardest Problem in Software Engineering