22 Lab 22: Deep Learning as a Model of the Brain
22.1 Learning Objectives
- Understand the core hypothesis of using deep neural networks as models of brain information processing
- Implement Representational Similarity Analysis (RSA) to compare brain and model representations
- Apply RSA methodology to simulated brain and model data
- Analyze the hierarchical correspondence between brain regions and model layers
- Connect computational methods to cognitive neuroscience research
22.2 Prerequisites
- Reading: Chapter 22: Cognitive Neuroscience & Deep Learning
- Libraries: NumPy, Matplotlib, SciPy
- Concepts: Representational geometry, RSA, visual hierarchy
22.3 Setup
Run this cell to import the required libraries and configure the environment:
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import spearmanr
from scipy.spatial.distance import pdist, squareform
np.random.seed(42)
plt.rcParams['figure.figsize'] = (12, 8)22.4 Part 1: Understanding Representational Similarity Analysis
Representational Similarity Analysis (RSA) is the workhorse method for comparing brains and models. The key idea is to compare the geometry of representations rather than individual neurons.
22.4.1 The RSA Pipeline
- Measure activity patterns from brain regions (or model layers) in response to stimuli
- Compute Representational Dissimilarity Matrices (RDMs) - pairwise dissimilarities between all stimulus pairs
- Compare RDMs between brain and model to assess representational similarity
22.5 Part 2: Simulating Brain and Model Data
In this section, we simulate fMRI data from brain regions of interest (ROIs) and activations from CNN layers. The simulation is designed to demonstrate hierarchical correspondence between brain regions (V1, V4, IT) and model layers (Conv1, Conv3, FC1).
Run this cell to define the simulation function:
def simulate_brain_and_model_data(n_stimuli=50):
"""
Simulate simplified fMRI data from brain ROIs and activations from CNN layers.
The simulation is designed to show a hierarchical correspondence.
Parameters
----------
n_stimuli : int
Number of stimuli to simulate (default: 50)
Returns
-------
brain_data : dict
Dictionary with ROI names as keys and (n_stimuli x n_units) arrays as values
model_data : dict
Dictionary with layer names as keys and (n_stimuli x n_units) arrays as values
"""
# Brain ROIs
roi_names = ["V1", "V4", "IT"]
brain_data = {}
# Model Layers
layer_names = ["Conv1", "Conv3", "FC1"]
model_data = {}
# Stimulus features (e.g., complexity)
stimulus_features = np.linspace(0, 1, n_stimuli)
for i, roi in enumerate(roi_names):
# Early regions are sensitive to low-level features, later regions to high-level
noise = np.random.randn(n_stimuli, 100) * 0.5
signal = np.sin(stimulus_features * (i + 1) * np.pi).reshape(-1, 1)
brain_data[roi] = signal + noise
for i, layer in enumerate(layer_names):
noise = np.random.randn(n_stimuli, 100) * 0.5
signal = np.sin(stimulus_features * (i + 1) * np.pi).reshape(-1, 1)
model_data[layer] = signal + noise
return brain_data, model_data22.6 Part 3: Computing Representational Dissimilarity Matrices
An RDM captures the pairwise dissimilarity between activity patterns for all stimulus pairs. We use correlation distance (1 - correlation) as our dissimilarity metric.
Run this cell to define the RDM computation function:
def compute_rdm(activations):
"""
Compute a Representational Dissimilarity Matrix (RDM).
Parameters
----------
activations : ndarray
(n_stimuli x n_units) array of activity patterns
Returns
-------
rdm : ndarray
(n_stimuli x n_stimuli) symmetric dissimilarity matrix
"""
return squareform(pdist(activations, metric='correlation'))22.7 Part 4: Running RSA Comparison
Now we bring everything together to run a full RSA comparison. We: 1. Simulate brain and model data 2. Compute RDMs for each brain region and model layer 3. Calculate Spearman correlations between RDMs 4. Visualize the brain-model similarity matrix
Run this cell to execute the complete RSA analysis:
def run_rsa_comparison():
"""
Run a full RSA comparison and visualize the results.
This function:
1. Simulates brain and model data with hierarchical structure
2. Computes RDMs for all brain regions and model layers
3. Calculates representational similarity (Spearman correlation)
4. Visualizes the brain-model similarity matrix
"""
brain_data, model_data = simulate_brain_and_model_data()
brain_rdms = {name: compute_rdm(data) for name, data in brain_data.items()}
model_rdms = {name: compute_rdm(data) for name, data in model_data.items()}
n_rois = len(brain_rdms)
n_layers = len(model_rdms)
similarity_matrix = np.zeros((n_rois, n_layers))
roi_keys = list(brain_rdms.keys())
layer_keys = list(model_rdms.keys())
for i, roi_key in enumerate(roi_keys):
brain_rdm_flat = brain_rdms[roi_key][np.triu_indices(brain_rdms[roi_key].shape[0], k=1)]
for j, layer_key in enumerate(layer_keys):
model_rdm_flat = model_rdms[layer_key][np.triu_indices(model_rdms[layer_key].shape[0], k=1)]
corr, _ = spearmanr(brain_rdm_flat, model_rdm_flat)
similarity_matrix[i, j] = corr
# Visualization
fig, ax = plt.subplots(figsize=(8, 6))
im = ax.imshow(similarity_matrix, cmap='viridis', vmin=0, vmax=1)
fig.colorbar(im, label='Representational Similarity (Spearman rho)')
ax.set_xticks(np.arange(n_layers))
ax.set_yticks(np.arange(n_rois))
ax.set_xticklabels(layer_keys)
ax.set_yticklabels(roi_keys)
ax.set_xlabel('Model Layers')
ax.set_ylabel('Brain Regions (ROIs)')
ax.set_title('Brain-Model Representational Similarity')
for i in range(n_rois):
for j in range(n_layers):
ax.text(j, i, f"{similarity_matrix[i, j]:.2f}", ha="center", va="center", color="w")
plt.show()
# Run the comparison
run_rsa_comparison()22.7.1 Interpreting the Results
The heatmap shows the representational similarity between brain regions and model layers. You should observe: - V1 shows highest similarity with early layers (Conv1) - V4 shows highest similarity with middle layers (Conv3) - IT shows highest similarity with late layers (FC1)
This diagonal pattern demonstrates the hierarchical correspondence between visual cortex and CNN layers.
22.8 Exercises
22.8.1 Exercise 1: RSA Pipeline Implementation
Implement a complete RSA pipeline from scratch: - Generate a set of 50 stimuli with known similarity structure - Create simulated “brain” responses (100-dimensional vectors) - Create simulated “model” responses with varying similarity to brain - Compute RDMs for both and calculate their correlation - Visualize both RDMs as heatmaps
22.8.2 Exercise 2: Parameter Exploration
Explore how RSA results change with different parameters: - Vary the noise level in the simulation - Change the number of stimuli - Modify the signal structure (different frequency patterns) - Document how each parameter affects brain-model similarity
22.8.3 Exercise 3: Real CNN Features
Using a pre-trained CNN (e.g., VGG16 from torchvision): - Extract activations from multiple layers for a set of images - Compute RDMs for each layer - Create synthetic “brain region” RDMs targeting early/middle/late layers - Identify which model layer best matches each brain region
22.8.4 Exercise 4: Comparing Distance Metrics
Compare different dissimilarity metrics for RDM computation: - Correlation distance (1 - Pearson r) - Euclidean distance - Cosine distance - Mahalanobis distance - Discuss which metric is most appropriate for brain-model comparison
22.8.5 Exercise 5: Statistical Testing
Implement statistical testing for RSA results: - Use permutation testing to assess significance - Calculate confidence intervals for similarity scores - Compare noise ceilings (upper bounds on predictability)
22.9 Challenge Problems
22.9.1 Challenge 1: Layer-wise Correspondence Analysis
Implement a comprehensive layer-wise analysis: - Use a deep CNN with 10+ layers - Compute RDMs for all layers - Create a similarity matrix showing all pairwise layer-region correspondences - Identify the optimal layer for each brain region
22.9.2 Challenge 2: Searchlight RSA
Implement searchlight RSA for fMRI data: - Define a spherical searchlight that moves through the brain - Compute local RDMs at each searchlight location - Create whole-brain maps of model similarity - Identify brain regions with highest model correspondence
22.9.3 Challenge 3: Cross-validated RSA
Implement cross-validated RSA to improve reliability: - Split stimuli into training and test sets - Compute RDMs separately for each split - Compare cross-validated similarity estimates - Discuss advantages over standard RSA
22.10 Discussion Questions
22.10.1 Question 1
Why is comparing representational geometry (via RDMs) more meaningful than comparing individual neurons directly? Consider the differences in how brains and artificial networks implement computations.
22.10.2 Question 2
What are the limitations of using simulated data to validate RSA methods? How might results differ with real neural data?
22.10.3 Question 3
The diagonal pattern in the similarity matrix suggests hierarchical correspondence. What would it mean if you observed a different pattern (e.g., all brain regions matching late layers)?
22.10.4 Question 4
How could you extend this analysis to study individual differences in brain-model correspondence? What might cause some individuals to show higher similarity to the model than others?
22.11 Summary
This lab explored Representational Similarity Analysis (RSA) as a key method for comparing brain and model representations.
Key Takeaways: - RSA compares representational geometry rather than individual units - RDMs capture the pairwise dissimilarity structure of representations - Hierarchical correspondence between visual cortex and CNN layers demonstrates convergent solutions - Statistical testing and cross-validation improve reliability of RSA results - RSA provides a powerful tool for using AI as a “computational microscope” for neuroscience