14 Lab 14: Sequence Models: RNN -> Attention -> Transformer
14.1 Learning Objectives
- Understand core concepts from Chapter 14
- Implement key algorithms and techniques
- Apply methods to practical problems
- Analyze results and draw insights
- Connect to broader NeuroAI themes
14.2 Prerequisites
- Reading: Chapter 14: Sequence Models
- Libraries: NumPy, Matplotlib, transformers, bertopic
- Concepts: RNNs, LSTMs, and temporal processing
14.3 Setup
Run this cell to import the necessary libraries and configure the environment.
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(42)
plt.rcParams['figure.figsize'] = (12, 8)14.4 Part 1: Topic Modeling with BERTopic
Topic modeling discovers latent themes in document collections. BERTopic combines transformer embeddings with clustering to create interpretable topics.
14.4.1 Exercise 1.1: Topic Discovery
Run this cell to see how BERTopic extracts topics from a collection of documents.
# BERTopic example (conceptual)
from bertopic import BERTopic
# Initialize with pre-trained embeddings
topic_model = BERTopic(embedding_model="all-MiniLM-L6-v2")
# Fit on documents
topics, probs = topic_model.fit_transform(documents)
# Inspect discovered topics
topic_model.get_topic_info()
# Returns: Topic 0: ["neural", "network", "deep", "learning", ...]
# Topic 1: ["brain", "cortex", "neuron", "synapse", ...]Instructions: 1. Prepare a list of documents (strings) covering different topics 2. Fit the BERTopic model on your documents 3. Examine the discovered topics using get_topic_info() 4. Try different embedding models and compare results
14.4.2 Exercise 1.2: Topic Analysis
- What topics does the model discover in your document collection?
- How do the topics relate to neuroscience and AI concepts?
- Experiment with different numbers of documents and observe topic quality.
14.5 Part 2: Named Entity Recognition (NER)
Named Entity Recognition identifies and classifies named entities (people, organizations, locations, dates) in text. Modern NER uses transformer-based sequence labeling.
14.5.1 Exercise 2.1: Transformer-based NER
Run this cell to extract named entities from text using a pre-trained transformer model.
# Transformer NER example (conceptual)
from transformers import pipeline
ner_pipeline = pipeline("ner", model="dslim/bert-base-NER")
text = "Apple CEO Tim Cook announced new products in Cupertino"
entities = ner_pipeline(text)
# [{'word': 'Apple', 'entity': 'B-ORG', 'score': 0.99},
# {'word': 'Tim', 'entity': 'B-PER', 'score': 0.99},
# {'word': 'Cook', 'entity': 'I-PER', 'score': 0.99},
# {'word': 'Cupertino', 'entity': 'B-LOC', 'score': 0.98}]Instructions: 1. Run the NER pipeline on the example text 2. Examine the entity types and confidence scores 3. Try with your own text containing various entity types 4. Experiment with different pre-trained NER models
14.5.2 Exercise 2.2: Entity Type Analysis
- Which entity types are most reliably detected?
- How does the model handle ambiguous entities (e.g., “Apple” as fruit vs. company)?
- Try biomedical or clinical text and observe performance differences.
14.6 Part 3: Sentiment Analysis
Sentiment analysis determines the emotional tone of text, classifying it as positive, negative, or neutral.
14.6.1 Exercise 3.1: Transformer-based Sentiment Analysis
Run this cell to analyze sentiment in text samples.
# Transformer sentiment analysis (conceptual)
from transformers import pipeline
sentiment_pipeline = pipeline("sentiment-analysis")
texts = [
"I love this product! Best purchase ever.",
"Terrible experience. Would not recommend.",
"It's okay, nothing special."
]
for text in texts:
result = sentiment_pipeline(text)
print(f"{text[:40]}... -> {result[0]['label']} ({result[0]['score']:.2f})")
# Output:
# I love this product! Best purchase ever... -> POSITIVE (0.99)
# Terrible experience. Would not recommend... -> NEGATIVE (0.99)
# It's okay, nothing special... -> NEGATIVE (0.72) # Neutral texts are tricky!Instructions: 1. Run the sentiment pipeline on the example texts 2. Observe how the model handles neutral/ambiguous statements 3. Test with texts containing negation, sarcasm, or implicit sentiment 4. Compare results across different sentiment models
14.6.2 Exercise 3.2: Sentiment Challenges
Test the model with challenging cases:
# Test challenging sentiment cases
challenging_texts = [
"Not bad at all!", # Negation - should be positive
"Great, another bug...", # Sarcasm - should be negative
"The battery lasted 2 hours", # Implicit - context dependent
"Sick beat!", # Domain shift - positive in music context
"Better than the old version but worse than competitors" # Comparison
]
for text in challenging_texts:
result = sentiment_pipeline(text)
print(f"{text} -> {result[0]['label']} ({result[0]['score']:.2f})")- Which challenges does the model handle well?
- Where does it struggle?
- How might you improve sentiment detection for these cases?
14.7 Part 4: Visualizing Sentiment Analysis Challenges
Run this cell to visualize the relative difficulty of different sentiment analysis challenges for ML models.
import numpy as np
import matplotlib.pyplot as plt
def visualize_sentiment_challenges():
"""Visualize challenges in sentiment analysis."""
challenges = ['Negation', 'Sarcasm', 'Implicit', 'Domain\nShift', 'Comparison']
examples = [
'"Not bad at all"\n(Positive)',
'"Great, another bug"\n(Negative)',
'"Battery: 2 hours"\n(Neg. for phones)',
'"Sick beat!"\n(Positive for music)',
'"Better than X"\n(Needs context)'
]
difficulty = [0.6, 0.85, 0.7, 0.5, 0.75] # Difficulty for models
colors = ['#3498db', '#e74c3c', '#f39c12', '#2ecc71', '#9b59b6']
fig, ax = plt.subplots(figsize=(10, 5))
bars = ax.bar(challenges, difficulty, color=colors, edgecolor='black', linewidth=1.5)
# Add example text on bars
for bar, example in zip(bars, examples):
height = bar.get_height()
ax.text(bar.get_x() + bar.get_width()/2., height + 0.02, example,
ha='center', va='bottom', fontsize=8, style='italic')
ax.set_ylabel('Difficulty for Models')
ax.set_title('Sentiment Analysis Challenges')
ax.set_ylim(0, 1.15)
ax.axhline(y=0.5, color='gray', linestyle='--', alpha=0.5, label='Baseline difficulty')
plt.tight_layout()
return plt
visualize_sentiment_challenges()
plt.show()
14.8 Exercises
14.8.1 Exercise 4.1: Pipeline Comparison
Compare different transformer models for NER and sentiment analysis. Which models perform best on scientific/medical text?
14.8.2 Exercise 4.2: Custom Dataset
Create a small dataset of neuroscience-related sentences and apply all three techniques (topic modeling, NER, sentiment analysis).
14.8.3 Exercise 4.3: Error Analysis
Identify and categorize the types of errors each model makes. Are there patterns related to domain-specific terminology?
14.8.4 Exercise 4.4: Neuroscience Connection
How do these NLP techniques relate to how the brain processes language? Consider: - Topic modeling vs. semantic memory organization - NER vs. proper name processing in the temporal pole - Sentiment analysis vs. emotional processing in the limbic system
14.8.5 Exercise 5
Discuss NeuroAI connections and future research directions.
14.9 Challenge Problems
14.9.1 Challenge 1: Biomedical NER
Fine-tune a transformer model for biomedical named entity recognition using a dataset like BC5CDR or JNLPBA.
14.9.2 Challenge 2: Multi-task Learning
Build a model that performs NER and sentiment analysis simultaneously, sharing representations between tasks.
14.9.3 Challenge 3: Healthcare Time Series
Apply sequence models to clinical time-series data (e.g., EHR records) to predict patient outcomes.
14.10 Discussion Questions
14.10.1 Question 1
How does topic modeling relate to how the brain organizes semantic knowledge in the anterior temporal lobe and angular gyrus?
14.10.2 Question 2
What are the implications of NER for building knowledge graphs and question answering systems in neuroscience?
14.10.3 Question 3
How might sentiment analysis be applied to clinical notes to detect patient distress or depression indicators?
14.11 Summary
Explored topic modeling, named entity recognition, and sentiment analysis through hands-on implementation with transformer models.
Key Takeaways: - BERTopic combines embeddings with clustering for interpretable topics - Transformer-based NER achieves high accuracy using sequence labeling - Sentiment analysis faces challenges with negation, sarcasm, and implicit meaning - These NLP techniques have direct parallels in brain language processing systems