Lab 05 · A small GPT, pseudocode, and API calls again
Read a GPT. Say it in pseudocode.
Walk through a small GPT with a guide, practise the pseudocode the midterm asks for, and finish the API calls from last week.
What are we doing?
Three things, each building on something you have already done.
On Tuesday we saw how attention lets a model look back. Today you see the whole thing built: a guided tour of Andrej Karpathy's notebook, which makes a small GPT from an empty page. Homework 5 starts from that tour.
Then pseudocode. The midterm is on paper, and several questions ask you to write out steps with no computer. We will look at four answers to a question from Homework 4 and decide together what earns full marks.
Last, the API calls from Lab 4. Many course keys did not work last week, so we will run that notebook again, and then write one API call as pseudocode.
A 70-minute plan
Three parts. The homework begins in the first one.
- 0–25 minutes: step through the guided tour together, then open the notebook.
- 25–45 minutes: pseudocode. Vote on four answers, then compare.
- 45–65 minutes: run the Lab 4 API notebook with your key, then write one API call as pseudocode.
- 65–70 minutes: start Homework 5.
Part 1
Read a GPT
Build a GPT, gently
A step-through guide to the notebook in five moves: text into numbers, the guessing game, a lookup table, attention, and the full model. The notebook's own model runs inside the page, so you can watch it learn and write. Every slide names the notebook cell it is about.
Open the guided tour → The notebookOpen gpt_dev.ipynb in Colab
Andrej Karpathy's notebook, ready to run. Do the tour first, then read the notes and the outputs here. Run the cells down to the first babble.
Open in Colab → Tuesday's explainerLook back
The attention idea in 14 slides: the shopping analogy, query, key and value, and a real model's attention. Homework 5's by-hand question uses exactly this.
Open the explainer →Optional, two hours: watch Karpathy build the notebook line by line, with the tour as your map.
Part 2
Pseudocode
Pseudocode, by example
Four answers to the word-tokenizer question from Homework 4. Vote Good, OK or Not yet on each one, then see how we would grade it and how to repair it. The examples were written for this session; none is a classmate's answer.
Open the session → New · on paper, no AINine pseudocode problems
Short problems from Weeks 1 to 5: accuracy, gradient descent, overfitting, tokenizers, an RNN loop, temperature and batches. Some ask you to write, some to complete, order or find a mistake.
Open the nine problems → Check after attemptingAnswers to the nine
One full-marks answer for each problem, what we look for, and a common slip to avoid.
Open the answers → Midterm practice · questions firstWhat counts as good pseudocode?
The short expectations guide and the practice questions. Sections A and B are about pseudocode. Attempt one without notes or AI.
Open the practice questions → Check after attemptingPractice answer key
Compare the ideas and the sequence of steps, not the surface syntax. Several forms of pseudocode can express the same correct procedure.
Open the answer key →Part 3
API calls, second try
Open the API lab in Colab
The Lab 4 notebook again. Store your key as a Colab secret, check your budget, and send one chat request. Stop after section 3 today. The embeddings and the five-model comparison are there if you finish early.
Open a runnable copy → Get your course keyYale SOM Token Fund
Sign in and copy the OpenRouter key beginning with sk-or-. The separate token beginning with ytf is for the portal itself and cannot call a model.
Write that API call as pseudocode
Put the seven steps of a chat request in order, compare pseudocode with the notebook's code, then write a three-model loop on paper.
Open the exercise →For what an API is, how to keep a key safe, and a downloadable copy of the notebook, see the Lab 4 page.
Optional
Going further
From the tiny GPT to the models you call
Four steps up from the notebook's 209,729 parameters. Train a model 50 times larger on the same Shakespeare. Read stories from a model that uses word pieces. Watch a billion-parameter open model at four points in its training. Then call the kind of model you used in Lab 4.
Open in Colab →