15 Lab 15: Large Language Models & Fine-Tuning

Open In Colab

15.1 Learning Objectives

Understand LoRA fine-tuning and parameter-efficient adaptation
Implement agent tool use and function calling patterns
Apply quantization techniques for efficient model deployment
Analyze LLM evaluation using judge-based approaches
Connect to broader NeuroAI themes

15.2 Prerequisites

Reading: Chapter 15: Large Language Models
Libraries: transformers, peft, torch
Concepts: Transformers, fine-tuning, quantization

15.3 Setup

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Set random seed for reproducibility
torch.manual_seed(42)

15.4 Part 1: LoRA Fine-Tuning

LoRA (Low-Rank Adaptation) enables efficient fine-tuning by training small adapter matrices while freezing the main model weights. This dramatically reduces the number of trainable parameters.

15.4.1 Exercise 1.1: Loading a Base Model with LoRA Adapter

Run this cell to see how to load a pre-trained base model and attach a LoRA adapter.

# Example: Loading a base model with a LoRA adapter
from peft import PeftModel
from transformers import AutoModelForCausalLM

# Load base model (frozen)
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Load LoRA adapter (tiny, task-specific)
model = PeftModel.from_pretrained(base_model, "username/my-lora-adapter")
# The adapter might be only 10-50MB for a 14GB base model!

Key Parameters to Explore:

Use Case	Rank (r)	Target Modules	Trainable Params
Light customization	4-8	q_proj, v_proj	~0.05%
General fine-tuning	16-32	q, k, v, o_proj	~0.2%
Complex adaptation	64-128	All linear layers	~1%

NeuroScience Connection: LoRA is analogous to how the brain learns new skills without overwriting existing ones. The frozen pre-trained model is like the brain’s vast, stable knowledge base (neocortex), while the PEFT adapter is like a small, plastic subnetwork (hippocampus) that is rapidly modified to encode new skills.

15.5 Part 2: AI Agents and Tool Use

AI Agents extend LLMs from reactive text generators to autonomous systems capable of reasoning, planning, using tools, and taking actions.

15.5.1 Exercise 2.1: Agent with Tool Definitions

Run this cell to understand how agents are configured with tools using structured JSON schemas.

# Example: Agent with tools (conceptual)
tools = [
    {"name": "web_search", "description": "Search the internet",
     "parameters": {"query": "string"}},
    {"name": "calculator", "description": "Perform math calculations",
     "parameters": {"expression": "string"}},
]

# Model decides to use a tool
response = model.generate(
    prompt="What is the population of France divided by 3?",
    tools=tools
)
# Output: {"tool": "web_search", "params": {"query": "population of France"}}
# ... execute tool, get result, feed back to model ...

Agent Memory Systems: - Short-term (Context Window): Conversation history within current context - Long-term (External Storage): Vector databases, knowledge graphs - Episodic: Records of past interactions and learned preferences - Working Memory: Scratchpads for intermediate reasoning steps

NeuroScience Connection: AI agents implement a computational version of executive function. The prefrontal cortex (PFC) is central to executive function, and agent architectures mirror several PFC capabilities including planning, working memory, and cognitive control.

15.6 Part 3: Model Quantization

Quantization enables running powerful models on consumer hardware by reducing precision from 32-bit floats to 8-bit or 4-bit integers.

15.6.1 Exercise 3.1: Loading a Model in 4-bit Precision

Run this cell to see how quantization dramatically reduces memory requirements.

# Load a 7B model in 4-bit precision (example syntax)
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    load_in_4bit=True,
    device_map="auto"
)
# Model now uses ~3.5GB instead of ~28GB!

Quality vs. Size Trade-offs:

Model	Format	Size	Perplexity	Relative Quality
Llama-2-7B	FP16	14 GB	5.47	100% (baseline)
Llama-2-7B	Q8_0	7 GB	5.48	~99.8%
Llama-2-7B	Q5_K_M	4.8 GB	5.52	~99%
Llama-2-7B	Q4_K_M	4.1 GB	5.58	~98%
Llama-2-7B	Q4_0	3.8 GB	5.68	~96%

NeuroScience Connection: The brain does not use infinite-precision representations either. Neural firing rates are inherently noisy, and information is encoded in population-level patterns. The brain achieves robust computation through redundancy and error-tolerant coding, similar to quantization principles.

15.7 Part 4: LLM Evaluation

Evaluating language models is complex because generated text can be fluent but factually wrong, or correct but awkwardly phrased.

15.7.1 Exercise 4.1: LLM-as-Judge Evaluation

Run this cell to see how strong LLMs can be used to evaluate responses from other models.

# Example prompt for LLM-as-judge
evaluation_prompt = """
You are an expert evaluator. Rate the following response on a scale of 1-10.

Question: {question}
Response: {response}

Rate on:
1. Accuracy (is the information correct?)
2. Helpfulness (does it answer the question?)
3. Clarity (is it well-written?)

Provide scores and brief justification.
"""

Evaluation Best Practices: - For Fine-tuning: Use perplexity, task-specific metrics, and LLM-as-judge - For Pre-deployment: Run benchmarks, red-teaming, and A/B testing - For Production: Track user engagement and periodic human review

NeuroScience Connection: The brain’s evaluation of language involves multiple systems: - Fluency Processing (Broca’s Area): Syntactic processing, analogous to automated fluency checks - Semantic Integration (Wernicke’s Area): Semantic fit, similar to BERTScore - Pragmatic Evaluation (Prefrontal Cortex): Higher-level judgments about helpfulness and appropriateness

15.8 Exercises

15.8.1 Exercise 1

Implement a basic LoRA fine-tuning pipeline using a small model like GPT-2. Compare the number of trainable parameters with and without LoRA.

15.8.2 Exercise 2

Create a simple agent with two tools (e.g., calculator and web search simulation) and trace the decision process for a multi-step query.

15.8.3 Exercise 3

Load a quantized model and compare inference speed and memory usage against the full-precision version.

15.8.4 Exercise 4

Design an evaluation rubric for LLM responses and implement it as a judge prompt. Test it on responses from different models.

15.8.5 Exercise 5

Discuss how the brain’s complementary learning systems (hippocampus and neocortex) relate to LoRA’s architecture of frozen weights plus trainable adapters.

15.9 Challenge Problems

15.9.1 Challenge 1

Implement a complete fine-tuning pipeline with LoRA for a specific domain (e.g., medical Q&A or code explanation). Track training metrics and evaluate on a held-out test set.

15.9.2 Challenge 2

Build a multi-agent system with specialized agents (researcher, coder, writer) that collaborate on a complex task. Implement an orchestrator that delegates and synthesizes outputs.

15.9.3 Challenge 3

Compare different quantization methods (INT8, INT4, AWQ) on the same model and analyze the quality degradation across different task types (reasoning, creative writing, factual recall).

Open In Colab

15.10 Discussion Questions

15.10.1 Question 1

How does LoRA’s approach of freezing base weights while training small adapters relate to the brain’s ability to learn new skills without catastrophic forgetting?

15.10.2 Question 2

What are the parallels between AI agent architectures and the brain’s executive function? How do both systems handle planning, working memory, and error correction?

15.10.3 Question 3

Why does quantization work so well for neural networks? What does this tell us about the information encoding strategies that both artificial and biological neural systems might use?

15.10.4 Question 4

How does the LLM-as-judge evaluation paradigm compare to how humans evaluate language quality? What cognitive processes are captured or missed by automated evaluation?

15.11 Summary

Explored Large Language Models through practical implementation of key techniques:

Key Takeaways: - LoRA Fine-Tuning: Efficient adaptation with 99%+ fewer trainable parameters - Agent Architectures: Tool use, memory systems, and planning modules - Quantization: 4-bit models that preserve 95%+ quality while reducing size 4x - LLM Evaluation: Multi-level assessment from automated metrics to human judgment - NeuroAI Connections: Parallels between artificial systems and biological neural mechanisms

--- title: "Lab 15: Large Language Models & Fine-Tuning" --- ::: {.content-visible when-format="html"} [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ricyoung/NeuroAI-Handbook_v3/blob/main/workbook/notebooks/lab15_large_language_models.ipynb) ::: ## Learning Objectives 1. **Understand** LoRA fine-tuning and parameter-efficient adaptation 2. **Implement** agent tool use and function calling patterns 3. **Apply** quantization techniques for efficient model deployment 4. **Analyze** LLM evaluation using judge-based approaches 5. **Connect** to broader NeuroAI themes ## Prerequisites - **Reading**: Chapter 15: Large Language Models - **Libraries**: transformers, peft, torch - **Concepts**: Transformers, fine-tuning, quantization ## Setup ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer from peft import PeftModel # Set random seed for reproducibility torch.manual_seed(42) ``` ## Part 1: LoRA Fine-Tuning **LoRA (Low-Rank Adaptation)** enables efficient fine-tuning by training small adapter matrices while freezing the main model weights. This dramatically reduces the number of trainable parameters. ### Exercise 1.1: Loading a Base Model with LoRA Adapter Run this cell to see how to load a pre-trained base model and attach a LoRA adapter. ```python # Example: Loading a base model with a LoRA adapter from peft import PeftModel from transformers import AutoModelForCausalLM # Load base model (frozen) base_model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf") # Load LoRA adapter (tiny, task-specific) model = PeftModel.from_pretrained(base_model, "username/my-lora-adapter") # The adapter might be only 10-50MB for a 14GB base model! ``` **Key Parameters to Explore**: | Use Case | Rank (r) | Target Modules | Trainable Params | |----------|----------|----------------|------------------| | Light customization | 4-8 | q_proj, v_proj | ~0.05% | | General fine-tuning | 16-32 | q, k, v, o_proj | ~0.2% | | Complex adaptation | 64-128 | All linear layers | ~1% | **NeuroScience Connection**: LoRA is analogous to how the brain learns new skills without overwriting existing ones. The frozen pre-trained model is like the brain's vast, stable knowledge base (neocortex), while the PEFT adapter is like a small, plastic subnetwork (hippocampus) that is rapidly modified to encode new skills. ## Part 2: AI Agents and Tool Use **AI Agents** extend LLMs from reactive text generators to autonomous systems capable of reasoning, planning, using tools, and taking actions. ### Exercise 2.1: Agent with Tool Definitions Run this cell to understand how agents are configured with tools using structured JSON schemas. ```python # Example: Agent with tools (conceptual) tools = [ {"name": "web_search", "description": "Search the internet", "parameters": {"query": "string"}}, {"name": "calculator", "description": "Perform math calculations", "parameters": {"expression": "string"}}, ] # Model decides to use a tool response = model.generate( prompt="What is the population of France divided by 3?", tools=tools ) # Output: {"tool": "web_search", "params": {"query": "population of France"}} # ... execute tool, get result, feed back to model ... ``` **Agent Memory Systems**: - **Short-term (Context Window)**: Conversation history within current context - **Long-term (External Storage)**: Vector databases, knowledge graphs - **Episodic**: Records of past interactions and learned preferences - **Working Memory**: Scratchpads for intermediate reasoning steps **NeuroScience Connection**: AI agents implement a computational version of **executive function**. The prefrontal cortex (PFC) is central to executive function, and agent architectures mirror several PFC capabilities including planning, working memory, and cognitive control. ## Part 3: Model Quantization **Quantization** enables running powerful models on consumer hardware by reducing precision from 32-bit floats to 8-bit or 4-bit integers. ### Exercise 3.1: Loading a Model in 4-bit Precision Run this cell to see how quantization dramatically reduces memory requirements. ```python # Load a 7B model in 4-bit precision (example syntax) from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained( "meta-llama/Llama-2-7b-hf", load_in_4bit=True, device_map="auto" ) # Model now uses ~3.5GB instead of ~28GB! ``` **Quality vs. Size Trade-offs**: | Model | Format | Size | Perplexity | Relative Quality | |-------|--------|------|------------|------------------| | Llama-2-7B | FP16 | 14 GB | 5.47 | 100% (baseline) | | Llama-2-7B | Q8_0 | 7 GB | 5.48 | ~99.8% | | Llama-2-7B | Q5_K_M | 4.8 GB | 5.52 | ~99% | | Llama-2-7B | Q4_K_M | 4.1 GB | 5.58 | ~98% | | Llama-2-7B | Q4_0 | 3.8 GB | 5.68 | ~96% | **NeuroScience Connection**: The brain does not use infinite-precision representations either. Neural firing rates are inherently noisy, and information is encoded in population-level patterns. The brain achieves robust computation through redundancy and error-tolerant coding, similar to quantization principles. ## Part 4: LLM Evaluation Evaluating language models is complex because generated text can be fluent but factually wrong, or correct but awkwardly phrased. ### Exercise 4.1: LLM-as-Judge Evaluation Run this cell to see how strong LLMs can be used to evaluate responses from other models. ```python # Example prompt for LLM-as-judge evaluation_prompt = """ You are an expert evaluator. Rate the following response on a scale of 1-10. Question: {question} Response: {response} Rate on: 1. Accuracy (is the information correct?) 2. Helpfulness (does it answer the question?) 3. Clarity (is it well-written?) Provide scores and brief justification. """ ``` **Evaluation Best Practices**: - **For Fine-tuning**: Use perplexity, task-specific metrics, and LLM-as-judge - **For Pre-deployment**: Run benchmarks, red-teaming, and A/B testing - **For Production**: Track user engagement and periodic human review **NeuroScience Connection**: The brain's evaluation of language involves multiple systems: - **Fluency Processing (Broca's Area)**: Syntactic processing, analogous to automated fluency checks - **Semantic Integration (Wernicke's Area)**: Semantic fit, similar to BERTScore - **Pragmatic Evaluation (Prefrontal Cortex)**: Higher-level judgments about helpfulness and appropriateness ## Exercises ### Exercise 1 Implement a basic LoRA fine-tuning pipeline using a small model like GPT-2. Compare the number of trainable parameters with and without LoRA. ### Exercise 2 Create a simple agent with two tools (e.g., calculator and web search simulation) and trace the decision process for a multi-step query. ### Exercise 3 Load a quantized model and compare inference speed and memory usage against the full-precision version. ### Exercise 4 Design an evaluation rubric for LLM responses and implement it as a judge prompt. Test it on responses from different models. ### Exercise 5 Discuss how the brain's complementary learning systems (hippocampus and neocortex) relate to LoRA's architecture of frozen weights plus trainable adapters. --- ## Challenge Problems ### Challenge 1 Implement a complete fine-tuning pipeline with LoRA for a specific domain (e.g., medical Q&A or code explanation). Track training metrics and evaluate on a held-out test set. ### Challenge 2 Build a multi-agent system with specialized agents (researcher, coder, writer) that collaborate on a complex task. Implement an orchestrator that delegates and synthesizes outputs. ### Challenge 3 Compare different quantization methods (INT8, INT4, AWQ) on the same model and analyze the quality degradation across different task types (reasoning, creative writing, factual recall). --- ::: {.content-visible when-format="html"} [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ricyoung/NeuroAI-Handbook_v3/blob/main/workbook/notebooks/lab15_large_language_models.ipynb) ::: ## Discussion Questions ### Question 1 How does LoRA's approach of freezing base weights while training small adapters relate to the brain's ability to learn new skills without catastrophic forgetting? ### Question 2 What are the parallels between AI agent architectures and the brain's executive function? How do both systems handle planning, working memory, and error correction? ### Question 3 Why does quantization work so well for neural networks? What does this tell us about the information encoding strategies that both artificial and biological neural systems might use? ### Question 4 How does the LLM-as-judge evaluation paradigm compare to how humans evaluate language quality? What cognitive processes are captured or missed by automated evaluation? --- ## Summary Explored Large Language Models through practical implementation of key techniques: **Key Takeaways**: - **LoRA Fine-Tuning**: Efficient adaptation with 99%+ fewer trainable parameters - **Agent Architectures**: Tool use, memory systems, and planning modules - **Quantization**: 4-bit models that preserve 95%+ quality while reducing size 4x - **LLM Evaluation**: Multi-level assessment from automated metrics to human judgment - **NeuroAI Connections**: Parallels between artificial systems and biological neural mechanisms