Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog. — Kunal

Lecture 1: Overview

2025-12-26
lecture
assignment
Goals
- mechanics -- implementations
- mindset -- maximize hardware utilization
- intuitions
(overview of the class and the parts of training)
efficiency is the design driving decision
- compute constrained
- data processing; tokenization; architecture motivated by reducing memory/flops
- training: single epoch; scaling laws: reduce compute
- alignment: smaller base models to be tunable
- data constrained soon
tokenization
- tiktokenizer ui; I've made something very similar for myself in the past
- play with encoding/decoding to confirm it roundtrips
- compression ratio -- how many bytes does a token represent
- motivation
  - simplest, unicode code points
    - compression ratio ~ 1.5
    - not a very effective use of budget
    - vocab size is huge
  - byte based
    - single or multiple bytes
    - leads to long sequences; naive attention is quadratic, not very efficient
    - most elegant thing
    - compression ratio 1
  - word tokenizer
    - very classic in NLP
    - split string into sequences of segments
    - vocabulary size is unbounded -- can always get new segments
    - some rare words are very rare
    - new words can get lost, can break perplexity
  - byte pair encoding
    - 1994, Philip Gage
    - for data compression
    - used in NLP since 2015
    - then used in GPT2 in 2019
    - gpt2 -- word based tokenization, then bpe on each segment
      - start with each byte as a token
      - successively merge most common pairs
margins
- for programming, can we use real tokens?
  - experiment with using real tokenization with Python and others
- how does the performance compare?
- I implemented this with a trie in c in the past, curious to see what stacks up
- meta
  - the stanford notebook presentation format is really nice
  - I had a very primitive version of this for my JupyterCon talk