32 Deep Learning as a Model of the Brain

Learning Objectives By the end of this chapter, you will be able to:

Understand the revolutionary proposal to use deep learning networks as scientific models of brain information processing.

Explain the core methodology of cognitive computational neuroscience: comparing representations between brains and models.

Master the concept of Representational Similarity Analysis (RSA) as a key tool for this comparison.

Analyze the success of this approach in explaining the visual system and its implications for other cognitive domains.

Appreciate the bidirectional relationship where AI serves as a “computational microscope” for neuroscience.

32.1 22.1 A New Kind of Brain Theory

Figure 32.1: Cognitive neuroscience and deep learning take parallel approaches to understanding intelligent systems, with remarkable correspondences between brain hierarchies and network architectures.

For centuries, neuroscientists have described the brain using metaphors and qualitative models. We’ve said the cortex is like a switchboard, a computer, or a logic engine. But these were just words. We lacked a way to build a functional, quantitative model that could actually do what the brain does, and whose internal workings we could then compare to the brain itself.

Deep learning has changed everything. For the first time, we have a class of models that can perform complex, brain-like tasks—such as object recognition or language understanding—at a level that rivals human performance. This has led to a revolutionary proposal at the heart of modern cognitive computational neuroscience:

The core hypothesis is that goal-driven deep neural networks are candidate models of information processing in biological brains.

This is not just an analogy. It’s a testable scientific hypothesis. The idea is that if we train a deep network to solve the same problem that a brain region solves (e.g., recognizing objects), the network might learn a solution that is functionally similar to the brain’s own algorithm. We can then treat the AI model as a “computational microscope” to understand the brain’s strategy.

32.2 22.2 The Method: Comparing Representations

How can we test this hypothesis? We can’t compare the “wetware” of the brain to the “software” of an AI model directly. Instead, we compare their representations.

A representation is the pattern of activity across a set of neurons (or model units) in response to a stimulus. The key idea is to see if the geometry of these representations is similar in both the brain and the model.

The Bookshelf Analogy For Comparing Representations Imagine you want to know if two people, Alice and Bob, think about movies in the same way. You can’t read their minds directly. Instead, you give them both the same 50 movie DVDs and ask them to arrange them on a bookshelf.

Alice arranges them by genre (sci-fi on the left, comedies in the middle, dramas on the right).

Bob arranges them by director (all of Spielberg’s films together, all of Nolan’s films together).

By comparing their bookshelves, you can see that they have very different representational geometries for movies. Alice thinks “Star Wars” is similar to “Blade Runner” (both sci-fi), while Bob thinks “Star Wars” is similar to “Indiana Jones” (both Lucas).

Representational Similarity Analysis (RSA) is a formal method for comparing these “bookshelves.” We show the brain and the AI model the same set of stimuli (e.g., 100 different images) and measure the resulting patterns of activity. We then ask: do the things that create similar patterns in the brain also create similar patterns in the model?

22.2.1 Representational Similarity Analysis (RSA)

RSA is the workhorse method for comparing brains and models. The process is as follows: 1. Stimulus Set: Select a set of stimuli (e.g., 100 images of different objects). 2. Measure Brain Activity: Show these images to a human or animal subject while recording neural activity (e.g., with fMRI or electrophysiology) from a specific brain region. 3. Measure Model Activity: Feed the exact same images into a deep learning model and record the activations of a specific layer. 4. Create Representational Dissimilarity Matrices (RDMs): For both the brain and the model, create a matrix that captures the pairwise “dissimilarity” between the activity patterns for every pair of images. A common metric is (1 - correlation). This matrix is a signature of the representational geometry. 5. Compare RDMs: Finally, compare the brain’s RDM to the model’s RDM. A high correlation between the two matrices suggests that the brain region and the model layer represent information in a similar way.

RSA Workflow Figure 22.1: The workflow for Representational Similarity Analysis (RSA). The same set of stimuli is presented to both a biological subject and an AI model. The resulting activity patterns are used to create RDMs, which are then compared to quantify the similarity of their representational geometries.

32.3 22.3 The Evidence: Modeling the Ventral Visual Stream

Figure 32.2: The visual cortex hierarchy (V1→V2→V4→IT) maps remarkably onto CNN layer structure, with both systems progressing from simple features to complex object representations.

The most stunning success of this approach has been in modeling the ventral visual stream, the pathway in the primate brain responsible for object recognition.

Neuroscientists have long known that this pathway is a hierarchy: - V1 (early visual cortex) represents simple edges. - V4 represents more complex shapes and textures. - Inferotemporal (IT) cortex represents whole objects.

Researchers found that the layers of a deep convolutional neural network (CNN) trained on object recognition (like AlexNet or VGG) map beautifully onto this hierarchy. - Early layers of the CNN have representations that are highly similar to V1. - Middle layers have representations that are highly similar to V4. - Top layers have representations that are highly similar to IT cortex.

This was a profound result. It showed that a system optimized for a single goal (object classification) had discovered a solution—a cascade of hierarchical feature extraction—that was remarkably similar to the one that evolution had discovered in the primate brain. For the first time, we had a candidate model that could not only predict what the brain was doing, but also offered a hypothesis for why: because it is an optimized solution to a specific computational problem.

Figure 32.3: Cognitive benchmarks evaluate both biological and artificial systems on reasoning, memory, and attention tasks, revealing parallels and differences.

32.4 22.4 The Implications: A New Era for Neuroscience

The ability to use deep learning models as testable theories is having a profound impact on neuroscience.

Moving from “Where” to “How”: For decades, brain imaging has been very good at telling us where in the brain things happen (e.g., in the fusiform face area). Deep learning models now allow us to ask how the computations are performed. We can dissect the model’s layers and circuits to form concrete hypotheses about the algorithms being implemented by the neural circuits.
Causality and Control: Once a model accurately predicts brain activity, we can use it to play “what if” games. What if we lesion certain units in the model? Does it reproduce the deficits seen in patients with brain damage? We can also use models to design optimal stimuli to drive specific neural populations, giving experimentalists powerful new tools.
Understanding Complexity: The brain is a massively complex, nonlinear dynamical system. Our intuitive, verbal theories are often insufficient to explain its emergent properties. Deep learning provides a formal language for modeling this complexity, allowing us to understand how intelligence can emerge from the interaction of millions of simple units.

Figure 32.4: Working memory in biological prefrontal cortex and attention mechanisms in transformers both maintain task-relevant information through selective gating.

32.5 22.5 The Future: A Unified Science of Intelligence?

The convergence of cognitive neuroscience and deep learning is still in its early days, but it points toward a future where the study of biological and artificial intelligence is no longer separate. - Neuroscience will continue to guide AI: As we push the boundaries of AI, the brain will remain the ultimate source of inspiration for building truly general, efficient, and robust intelligence. Principles like continual learning, energy efficiency, and causal reasoning are the next great frontiers for AI, and the brain is our only working example. - AI will accelerate neuroscience: The increasing sophistication of AI models will allow us to build more and more accurate simulations of brain systems, leading to a deeper understanding of everything from perception to consciousness, and accelerating the development of treatments for neurological and psychiatric disorders.

This virtuous cycle is creating a new, unified science of intelligence, where insights from minds and machines are woven together to answer the oldest questions we have about the nature of thought itself.

32.6 Exercises

Conceptual Questions

DL as Brain Models: Explain the revolutionary proposal to use deep learning networks as scientific models of the brain. How does this differ from previous approaches to modeling brain function? What makes deep learning models particularly suitable for this purpose?
Representational Geometry: The chapter uses the “bookshelf analogy” to explain representational geometry. In your own words, explain what a representation is in the context of both neural systems and artificial networks. Why is comparing the “geometry” of representations (rather than individual neurons or units) a more meaningful way to compare brains and models?
The Hierarchy Correspondence: Describe the remarkable correspondence found between the layers of CNNs and the hierarchy of the ventral visual stream (V1 → V4 → IT). What does this correspondence suggest about the computational strategy both systems use for object recognition? What are the limitations of this correspondence?
Causality and Prediction: The chapter mentions that once a model accurately predicts brain activity, we can use it to play “what if” games (e.g., lesioning units, optimizing stimuli). Explain why this is powerful for neuroscience. How does this approach complement traditional experimental methods?

Computational Problems

RSA Implementation: Implement a complete Representational Similarity Analysis pipeline:
- Generate or use a set of stimuli (e.g., 50 images)
- Simulate “brain” responses (e.g., 100-dimensional vectors with structured similarity)
- Simulate “model” responses with varying degrees of similarity to the brain
- Compute RDMs for both
- Calculate the correlation between RDMs
- Visualize the RDMs as heatmaps and create a brain-model similarity matrix
- Test: how does the similarity measure change as you make the model more/less brain-like?
Layer-by-Layer Correspondence: Using a pre-trained CNN (e.g., VGG or ResNet):
- Extract activations from multiple layers for a set of images
- Compute RDMs for each layer
- Create synthetic “brain region” RDMs that are more similar to early, middle, or late layers
- For each brain region, identify which model layer has the highest representational similarity
- Visualize the results: plot similarity as a function of layer depth for each brain region
- Discuss: what does it mean when a brain region matches multiple layers or no layers well?
Optimal Stimulus Synthesis: Implement a simple “neural encoding” model:
- Train a linear regression model to predict synthetic “neural responses” from image features (e.g., pixels or CNN activations)
- Use gradient ascent on the image space to synthesize an image that maximally activates a target “neuron”
- Visualize the resulting optimal stimulus
- Compare optimal stimuli for different “neurons” (early vs. late in a hierarchy)
- Discuss: how could this technique be used to design better neuroscience experiments?
Model Comparison: Compare different model architectures as brain models:
- Implement or use multiple models: a simple feedforward network, a CNN, and a recurrent network
- Generate predictions for the same set of stimuli
- Create synthetic brain data with known properties (e.g., hierarchical, recurrent dynamics)
- Calculate RSA similarity between each model and the “brain”
- Determine which model is the best predictor
- Discuss: what architectural features make a model more brain-like?

Discussion Questions

Levels of Explanation: David Marr proposed that computational systems can be understood at three levels: computational (what is being computed), algorithmic (how is it computed), and implementational (how is it physically realized). Where do deep learning models as brain theories fit in this framework? Do they primarily address the computational level, the algorithmic level, or both? What questions do they leave unanswered?
The Limits of Similarity: While CNNs show remarkable similarity to the visual cortex, they also differ in important ways (e.g., robustness to adversarial examples, sample efficiency, recurrent processing). Discuss: Is it more scientifically valuable to study the similarities or the differences between brains and models? How should discrepancies inform our understanding of both biological and artificial intelligence?
The Future of Neuroscience: The chapter suggests we are moving toward a “unified science of intelligence” where the study of biological and artificial systems converges. Do you think this is a realistic and desirable future? What might be gained and what might be lost if neuroscience becomes increasingly dominated by AI-inspired computational models? How can we ensure that this approach complements rather than replaces other methods (anatomy, physiology, behavior)?

Chapter Summary This chapter explored the powerful convergence of cognitive neuroscience and deep learning, framing it as a new, quantitative approach to understanding the brain.

The Core Hypothesis: The central idea is that deep neural networks, when optimized to perform brain-like tasks, can serve as testable scientific models of the brain’s own computational algorithms.

The Primary Method: We learned about Representational Similarity Analysis (RSA), a key technique for comparing the “representational geometry” of brain regions and model layers to see if they organize information in a similar way.

The Key Evidence: The most compelling success story is the modeling of the ventral visual stream, where the hierarchical layers of a CNN trained on object recognition map directly onto the known hierarchy of the primate visual cortex.

The Broader Implication: This approach is transforming neuroscience, moving it from describing where computation happens to generating testable hypotheses about how it happens. Deep learning models are becoming a “computational microscope” for exploring the brain’s algorithms.

The Virtuous Cycle: This bidirectional relationship, where neuroscience inspires AI and AI provides tools to advance neuroscience, is creating a new, unified science of intelligence.

Knowledge Connections Looking Back - Chapter 4 (Perception): The hierarchical model of the ventral visual stream discussed here is the biological foundation for the CNN models analyzed in this chapter. - Chapter 14 (Bridging Bio & AI): This chapter provides a deep dive into the methodology of the “virtuous cycle” described in the previous chapter, showing exactly how the comparison between brains and AI is done in practice.

Looking Forward - Chapter 21 (AI for Neuro Discovery): We will further explore the practical applications of using AI as a tool for neuroscience, moving beyond representation comparison to areas like neural decoding and simulation.

32.7 References

Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016). Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific Reports, 6(1), 27755.

Diedrichsen, J., & Kriegeskorte, N. (2017). Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis. PLoS Computational Biology, 13(4), e1005508.

Güçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27), 10005-10014.

Khaligh-Razavi, S. M., & Kriegeskorte, N. (2014). Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Computational Biology, 10(11), e1003915.

Kriegeskorte, N., Mur, M., & Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2, 4.

Lindsay, G. W. (2020). Convolutional neural networks as a model of the visual system: Past, present, and future. Journal of Cognitive Neuroscience, 33(10), 2017-2031.

Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., … & Kording, K. P. (2019). A deep learning framework for neuroscience. Nature Neuroscience, 22(11), 1761-1770.

Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., … & DiCarlo, J. J. (2018). Brain-Score: Which artificial neural network for object recognition is most brain-like? bioRxiv, 407007.

Yamins, D. L., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience, 19(3), 356-365.

Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23), 8619-8624.

Zhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C., DiCarlo, J. J., & Yamins, D. L. (2021). Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3), e2014196118.

--- title: "Deep Learning as a Model of the Brain" number-sections: true number-depth: 2 --- > **Learning Objectives** > By the end of this chapter, you will be able to: > > - **Understand** the revolutionary proposal to use deep learning networks as scientific models of brain information processing. > - **Explain** the core methodology of cognitive computational neuroscience: comparing representations between brains and models. > - **Master** the concept of Representational Similarity Analysis (RSA) as a key tool for this comparison. > - **Analyze** the success of this approach in explaining the visual system and its implications for other cognitive domains. > - **Appreciate** the bidirectional relationship where AI serves as a "computational microscope" for neuroscience. <div style="page-break-before:always;"></div> ## 22.1 A New Kind of Brain Theory ![Cognitive neuroscience and deep learning take parallel approaches to understanding intelligent systems, with remarkable correspondences between brain hierarchies and network architectures.](../shared/images/ch22/cognitive_dl_comparison.svg){#fig-cognitive-dl width="100%"} For centuries, neuroscientists have described the brain using metaphors and qualitative models. We've said the cortex is like a switchboard, a computer, or a logic engine. But these were just words. We lacked a way to build a functional, quantitative model that could actually *do* what the brain does, and whose internal workings we could then compare to the brain itself. Deep learning has changed everything. For the first time, we have a class of models that can perform complex, brain-like tasks---such as object recognition or language understanding---at a level that rivals human performance. This has led to a revolutionary proposal at the heart of modern cognitive computational neuroscience: **The core hypothesis is that goal-driven deep neural networks are candidate models of information processing in biological brains.** This is not just an analogy. It's a testable scientific hypothesis. The idea is that if we train a deep network to solve the same problem that a brain region solves (e.g., recognizing objects), the network might learn a solution that is functionally similar to the brain's own algorithm. We can then treat the AI model as a "computational microscope" to understand the brain's strategy. ## 22.2 The Method: Comparing Representations How can we test this hypothesis? We can't compare the "wetware" of the brain to the "software" of an AI model directly. Instead, we compare their **representations**. A representation is the pattern of activity across a set of neurons (or model units) in response to a stimulus. The key idea is to see if the *geometry* of these representations is similar in both the brain and the model. > **The Bookshelf Analogy For Comparing Representations** > Imagine you want to know if two people, Alice and Bob, think about movies in the same way. You can't read their minds directly. Instead, you give them both the same 50 movie DVDs and ask them to arrange them on a bookshelf. > > - **Alice** arranges them by genre (sci-fi on the left, comedies in the middle, dramas on the right). > - **Bob** arranges them by director (all of Spielberg's films together, all of Nolan's films together). > > By comparing their bookshelves, you can see that they have very different *representational geometries* for movies. Alice thinks "Star Wars" is similar to "Blade Runner" (both sci-fi), while Bob thinks "Star Wars" is similar to "Indiana Jones" (both Lucas). > > **Representational Similarity Analysis (RSA)** is a formal method for comparing these "bookshelves." We show the brain and the AI model the same set of stimuli (e.g., 100 different images) and measure the resulting patterns of activity. We then ask: do the things that create similar patterns in the brain also create similar patterns in the model? ### 22.2.1 Representational Similarity Analysis (RSA) RSA is the workhorse method for comparing brains and models. The process is as follows: 1. **Stimulus Set**: Select a set of stimuli (e.g., 100 images of different objects). 2. **Measure Brain Activity**: Show these images to a human or animal subject while recording neural activity (e.g., with fMRI or electrophysiology) from a specific brain region. 3. **Measure Model Activity**: Feed the exact same images into a deep learning model and record the activations of a specific layer. 4. **Create Representational Dissimilarity Matrices (RDMs)**: For both the brain and the model, create a matrix that captures the pairwise "dissimilarity" between the activity patterns for every pair of images. A common metric is (1 - correlation). This matrix is a signature of the representational geometry. 5. **Compare RDMs**: Finally, compare the brain's RDM to the model's RDM. A high correlation between the two matrices suggests that the brain region and the model layer represent information in a similar way. ![RSA Workflow](../shared/images/ch19/rep_similarity_analysis.svg) *Figure 22.1: The workflow for Representational Similarity Analysis (RSA). The same set of stimuli is presented to both a biological subject and an AI model. The resulting activity patterns are used to create RDMs, which are then compared to quantify the similarity of their representational geometries.* ## 22.3 The Evidence: Modeling the Ventral Visual Stream ![The visual cortex hierarchy (V1→V2→V4→IT) maps remarkably onto CNN layer structure, with both systems progressing from simple features to complex object representations.](../shared/images/ch22/visual_cortex_cnn_parallel.png){#fig-visual-cortex-cnn width="100%"} The most stunning success of this approach has been in modeling the **ventral visual stream**, the pathway in the primate brain responsible for object recognition. Neuroscientists have long known that this pathway is a hierarchy: - **V1** (early visual cortex) represents simple edges. - **V4** represents more complex shapes and textures. - **Inferotemporal (IT) cortex** represents whole objects. Researchers found that the layers of a deep convolutional neural network (CNN) trained on object recognition (like AlexNet or VGG) map beautifully onto this hierarchy. - **Early layers** of the CNN have representations that are highly similar to **V1**. - **Middle layers** have representations that are highly similar to **V4**. - **Top layers** have representations that are highly similar to **IT cortex**. This was a profound result. It showed that a system optimized for a single goal (object classification) had discovered a solution---a cascade of hierarchical feature extraction---that was remarkably similar to the one that evolution had discovered in the primate brain. For the first time, we had a candidate model that could not only predict *what* the brain was doing, but also offered a hypothesis for *why*: because it is an optimized solution to a specific computational problem. ```{python} #| echo: false import numpy as np import matplotlib.pyplot as plt from scipy.stats import spearmanr from scipy.spatial.distance import pdist, squareform def simulate_brain_and_model_data(n_stimuli=50): """ Simulate simplified fMRI data from brain ROIs and activations from CNN layers. The simulation is designed to show a hierarchical correspondence. """ # Brain ROIs roi_names = ["V1", "V4", "IT"] brain_data = {} # Model Layers layer_names = ["Conv1", "Conv3", "FC1"] model_data = {} # Stimulus features (e.g., complexity) stimulus_features = np.linspace(0, 1, n_stimuli) for i, roi in enumerate(roi_names): # Early regions are sensitive to low-level features, later regions to high-level noise = np.random.randn(n_stimuli, 100) * 0.5 signal = np.sin(stimulus_features * (i + 1) * np.pi).reshape(-1, 1) brain_data[roi] = signal + noise for i, layer in enumerate(layer_names): noise = np.random.randn(n_stimuli, 100) * 0.5 signal = np.sin(stimulus_features * (i + 1) * np.pi).reshape(-1, 1) model_data[layer] = signal + noise return brain_data, model_data def compute_rdm(activations): """Compute a Representational Dissimilarity Matrix (RDM).""" return squareform(pdist(activations, metric='correlation')) def run_rsa_comparison(): """Run a full RSA comparison and visualize the results.""" brain_data, model_data = simulate_brain_and_model_data() brain_rdms = {name: compute_rdm(data) for name, data in brain_data.items()} model_rdms = {name: compute_rdm(data) for name, data in model_data.items()} n_rois = len(brain_rdms) n_layers = len(model_rdms) similarity_matrix = np.zeros((n_rois, n_layers)) roi_keys = list(brain_rdms.keys()) layer_keys = list(model_rdms.keys()) for i, roi_key in enumerate(roi_keys): brain_rdm_flat = brain_rdms[roi_key][np.triu_indices(brain_rdms[roi_key].shape[0], k=1)] for j, layer_key in enumerate(layer_keys): model_rdm_flat = model_rdms[layer_key][np.triu_indices(model_rdms[layer_key].shape[0], k=1)] corr, _ = spearmanr(brain_rdm_flat, model_rdm_flat) similarity_matrix[i, j] = corr # Visualization fig, ax = plt.subplots(figsize=(8, 6)) im = ax.imshow(similarity_matrix, cmap='viridis', vmin=0, vmax=1) fig.colorbar(im, label='Representational Similarity (Spearman ρ)') ax.set_xticks(np.arange(n_layers)) ax.set_yticks(np.arange(n_rois)) ax.set_xticklabels(layer_keys) ax.set_yticklabels(roi_keys) ax.set_xlabel('Model Layers') ax.set_ylabel('Brain Regions (ROIs)') ax.set_title('Brain-Model Representational Similarity') for i in range(n_rois): for j in range(n_layers): ax.text(j, i, f"{similarity_matrix[i, j]:.2f}", ha="center", va="center", color="w") plt.show() run_rsa_comparison() ``` ![Cognitive benchmarks evaluate both biological and artificial systems on reasoning, memory, and attention tasks, revealing parallels and differences.](../shared/images/ch22/cognitive_benchmark_tasks.png){#fig-cognitive-benchmarks width="100%"} ## 22.4 The Implications: A New Era for Neuroscience The ability to use deep learning models as testable theories is having a profound impact on neuroscience. 1. **Moving from "Where" to "How"**: For decades, brain imaging has been very good at telling us *where* in the brain things happen (e.g., in the fusiform face area). Deep learning models now allow us to ask *how* the computations are performed. We can dissect the model's layers and circuits to form concrete hypotheses about the algorithms being implemented by the neural circuits. 2. **Causality and Control**: Once a model accurately predicts brain activity, we can use it to play "what if" games. What if we lesion certain units in the model? Does it reproduce the deficits seen in patients with brain damage? We can also use models to design optimal stimuli to drive specific neural populations, giving experimentalists powerful new tools. 3. **Understanding Complexity**: The brain is a massively complex, nonlinear dynamical system. Our intuitive, verbal theories are often insufficient to explain its emergent properties. Deep learning provides a formal language for modeling this complexity, allowing us to understand how intelligence can emerge from the interaction of millions of simple units. ![Working memory in biological prefrontal cortex and attention mechanisms in transformers both maintain task-relevant information through selective gating.](../shared/images/ch22/working_memory_attention.png){#fig-working-memory-attention width="100%"} ## 22.5 The Future: A Unified Science of Intelligence? The convergence of cognitive neuroscience and deep learning is still in its early days, but it points toward a future where the study of biological and artificial intelligence is no longer separate. - **Neuroscience will continue to guide AI**: As we push the boundaries of AI, the brain will remain the ultimate source of inspiration for building truly general, efficient, and robust intelligence. Principles like continual learning, energy efficiency, and causal reasoning are the next great frontiers for AI, and the brain is our only working example. - **AI will accelerate neuroscience**: The increasing sophistication of AI models will allow us to build more and more accurate simulations of brain systems, leading to a deeper understanding of everything from perception to consciousness, and accelerating the development of treatments for neurological and psychiatric disorders. This virtuous cycle is creating a new, unified science of intelligence, where insights from minds and machines are woven together to answer the oldest questions we have about the nature of thought itself. ## Exercises ### Conceptual Questions 1. **DL as Brain Models**: Explain the revolutionary proposal to use deep learning networks as scientific models of the brain. How does this differ from previous approaches to modeling brain function? What makes deep learning models particularly suitable for this purpose? 2. **Representational Geometry**: The chapter uses the "bookshelf analogy" to explain representational geometry. In your own words, explain what a representation is in the context of both neural systems and artificial networks. Why is comparing the "geometry" of representations (rather than individual neurons or units) a more meaningful way to compare brains and models? 3. **The Hierarchy Correspondence**: Describe the remarkable correspondence found between the layers of CNNs and the hierarchy of the ventral visual stream (V1 → V4 → IT). What does this correspondence suggest about the computational strategy both systems use for object recognition? What are the limitations of this correspondence? 4. **Causality and Prediction**: The chapter mentions that once a model accurately predicts brain activity, we can use it to play "what if" games (e.g., lesioning units, optimizing stimuli). Explain why this is powerful for neuroscience. How does this approach complement traditional experimental methods? ### Computational Problems 1. **RSA Implementation**: Implement a complete Representational Similarity Analysis pipeline: - Generate or use a set of stimuli (e.g., 50 images) - Simulate "brain" responses (e.g., 100-dimensional vectors with structured similarity) - Simulate "model" responses with varying degrees of similarity to the brain - Compute RDMs for both - Calculate the correlation between RDMs - Visualize the RDMs as heatmaps and create a brain-model similarity matrix - Test: how does the similarity measure change as you make the model more/less brain-like? 2. **Layer-by-Layer Correspondence**: Using a pre-trained CNN (e.g., VGG or ResNet): - Extract activations from multiple layers for a set of images - Compute RDMs for each layer - Create synthetic "brain region" RDMs that are more similar to early, middle, or late layers - For each brain region, identify which model layer has the highest representational similarity - Visualize the results: plot similarity as a function of layer depth for each brain region - Discuss: what does it mean when a brain region matches multiple layers or no layers well? 3. **Optimal Stimulus Synthesis**: Implement a simple "neural encoding" model: - Train a linear regression model to predict synthetic "neural responses" from image features (e.g., pixels or CNN activations) - Use gradient ascent on the image space to synthesize an image that maximally activates a target "neuron" - Visualize the resulting optimal stimulus - Compare optimal stimuli for different "neurons" (early vs. late in a hierarchy) - Discuss: how could this technique be used to design better neuroscience experiments? 4. **Model Comparison**: Compare different model architectures as brain models: - Implement or use multiple models: a simple feedforward network, a CNN, and a recurrent network - Generate predictions for the same set of stimuli - Create synthetic brain data with known properties (e.g., hierarchical, recurrent dynamics) - Calculate RSA similarity between each model and the "brain" - Determine which model is the best predictor - Discuss: what architectural features make a model more brain-like? ### Discussion Questions 1. **Levels of Explanation**: David Marr proposed that computational systems can be understood at three levels: computational (what is being computed), algorithmic (how is it computed), and implementational (how is it physically realized). Where do deep learning models as brain theories fit in this framework? Do they primarily address the computational level, the algorithmic level, or both? What questions do they leave unanswered? 2. **The Limits of Similarity**: While CNNs show remarkable similarity to the visual cortex, they also differ in important ways (e.g., robustness to adversarial examples, sample efficiency, recurrent processing). Discuss: Is it more scientifically valuable to study the similarities or the differences between brains and models? How should discrepancies inform our understanding of both biological and artificial intelligence? 3. **The Future of Neuroscience**: The chapter suggests we are moving toward a "unified science of intelligence" where the study of biological and artificial systems converges. Do you think this is a realistic and desirable future? What might be gained and what might be lost if neuroscience becomes increasingly dominated by AI-inspired computational models? How can we ensure that this approach complements rather than replaces other methods (anatomy, physiology, behavior)? <div style="page-break-before:always;"></div> > **Chapter Summary** > This chapter explored the powerful convergence of cognitive neuroscience and deep learning, framing it as a new, quantitative approach to understanding the brain. > > - **The Core Hypothesis**: The central idea is that deep neural networks, when optimized to perform brain-like tasks, can serve as testable scientific models of the brain's own computational algorithms. > - **The Primary Method**: We learned about **Representational Similarity Analysis (RSA)**, a key technique for comparing the "representational geometry" of brain regions and model layers to see if they organize information in a similar way. > - **The Key Evidence**: The most compelling success story is the modeling of the **ventral visual stream**, where the hierarchical layers of a CNN trained on object recognition map directly onto the known hierarchy of the primate visual cortex. > - **The Broader Implication**: This approach is transforming neuroscience, moving it from describing *where* computation happens to generating testable hypotheses about *how* it happens. Deep learning models are becoming a "computational microscope" for exploring the brain's algorithms. > - **The Virtuous Cycle**: This bidirectional relationship, where neuroscience inspires AI and AI provides tools to advance neuroscience, is creating a new, unified science of intelligence. > **Knowledge Connections** > **Looking Back** > - **Chapter 4 (Perception)**: The hierarchical model of the ventral visual stream discussed here is the biological foundation for the CNN models analyzed in this chapter. > - **Chapter 14 (Bridging Bio & AI)**: This chapter provides a deep dive into the *methodology* of the "virtuous cycle" described in the previous chapter, showing exactly how the comparison between brains and AI is done in practice. > > **Looking Forward** > - **Chapter 21 (AI for Neuro Discovery)**: We will further explore the practical applications of using AI as a tool for neuroscience, moving beyond representation comparison to areas like neural decoding and simulation. ## References Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016). Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. *Scientific Reports*, 6(1), 27755. Diedrichsen, J., & Kriegeskorte, N. (2017). Representational models: A common framework for understanding encoding, pattern-component, and representational-similarity analysis. *PLoS Computational Biology*, 13(4), e1005508. Güçlü, U., & van Gerven, M. A. (2015). Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. *Journal of Neuroscience*, 35(27), 10005-10014. Khaligh-Razavi, S. M., & Kriegeskorte, N. (2014). Deep supervised, but not unsupervised, models may explain IT cortical representation. *PLoS Computational Biology*, 10(11), e1003915. Kriegeskorte, N., Mur, M., & Bandettini, P. A. (2008). Representational similarity analysis-connecting the branches of systems neuroscience. *Frontiers in Systems Neuroscience*, 2, 4. Lindsay, G. W. (2020). Convolutional neural networks as a model of the visual system: Past, present, and future. *Journal of Cognitive Neuroscience*, 33(10), 2017-2031. Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., ... & Kording, K. P. (2019). A deep learning framework for neuroscience. *Nature Neuroscience*, 22(11), 1761-1770. Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., ... & DiCarlo, J. J. (2018). Brain-Score: Which artificial neural network for object recognition is most brain-like? *bioRxiv*, 407007. Yamins, D. L., & DiCarlo, J. J. (2016). Using goal-driven deep learning models to understand sensory cortex. *Nature Neuroscience*, 19(3), 356-365. Yamins, D. L., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014). Performance-optimized hierarchical models predict neural responses in higher visual cortex. *Proceedings of the National Academy of Sciences*, 111(23), 8619-8624. Zhuang, C., Yan, S., Nayebi, A., Schrimpf, M., Frank, M. C., DiCarlo, J. J., & Yamins, D. L. (2021). Unsupervised neural network models of the ventral visual stream. *Proceedings of the National Academy of Sciences*, 118(3), e2014196118.