Source: raw/Anthropic_Researcher_explains_how_to_train_LLMs.md Creator: Matthew Berman, interviewing an Anthropic researcher captioned “Jeremy Aburman” ^[ambiguous: Berman says “no, we are not related”, which suggests the surname is Berman] · URL: https://www.youtube.com/watch?v=Sjdvu4nHfDk · Duration: not stated · Platform: YouTube

A beginner walkthrough of how a language model is built: a neural network as a set of tunable “knobs”, pretraining as next-word prediction over the internet, post-training that turns an autocomplete engine into an assistant, and reinforcement learning that teaches it to reason. The researcher, who says he is self-taught, demos each stage on a free Google Colab T4 GPU with a small open-weight Qwen model. The practical value is the demo: you can repeat it yourself in an afternoon at no cost and see the difference between a base model and an instruct model.

Key Takeaways

  • A model is a file of numbers plus a short program. “A language model is really just a download from Hugging Face which has the numbers”, and a Python script that says how they are wired. A 2-trillion-parameter model is 2 trillion numbers.
  • Training is nudging knobs to reduce loss. PyTorch tracks how much each parameter affects the error (backpropagation), and each step moves every parameter slightly in the direction that lowers it; the step size is the learning rate.
  • A language model’s last layer is a probability for every possible next token. Training makes the token that actually came next a bit more likely, which makes every other token a bit less likely.
  • A base model is internet autocomplete. Given nothing, the base Qwen model in the demo produced code and then Chinese text, which hints at its training mix. Given “Matthew Berman said”, it invented a quote.
  • The assistant is a formatting trick, then training on it. Prefilling “User: … Assistant:” makes a base model answer like an assistant. Post-training on many examples in a fixed format (Qwen’s M start / M end markers ^[inferred: the ChatML <|im_start|> / <|im_end|> tokens]) plus a stop token teaches it the role and when to stop. The chat messages you send are converted into that same text format under the hood.
  • Reasoning came from RL, not more internet text. The internet shows finished work, not the thinking behind it. o1 was the first public reasoning model; DeepSeek’s R1 paper showed publicly how to train one with RL.
  • Two RL methods, explained simply:
    • Rejection sampling: answer a problem many times, keep the correct answers, train on them. It works, but ignores failures and reinforces noise tokens in good answers.
    • GRPO (group relative policy optimization): answer about 100 times, score each answer for format and correctness, subtract the group average to get each answer’s “advantage”, then make every token of above-average answers more likely and below-average ones less likely.
  • Why on-policy RL forgets less. The model trains on text it generated itself, so updates are small; training on someone else’s reasoning shifts many more weights and can degrade other skills.
  • Why pretraining and post-training stay separate. Once a model has an assistant persona, training it on raw text (his example: a code corpus) makes it start imitating that text and lose the assistant behaviour.
  • His view on new discoveries: reasoning over long horizons lets models build “new abstractions on the fly”, so he expects them to discover new things. That is his opinion.

Try It

  1. Run the base-vs-instruct comparison yourself. Open Google Colab, pick the free T4 GPU, load a small Qwen base model (the demo used a 0.5B model, “500 million knobs”) and its instruct version. Give both “Why is the sky blue?” and compare. Ask Claude or ChatGPT to write the notebook; the researcher and Berman both suggest this.
  2. Print what the model actually sees. Pass a chat message list through the tokenizer’s chat template and print the result. You will see the system message and role markers that the API adds for you, which explains why system-prompt wording matters.
  3. Test “let’s think step by step” on a base or instruct model. The demo’s multiplication (398 × 923 + 2,901) went wrong when answered directly and got closer when the answer was prefilled with “Let’s think step by step”.
  4. For deeper study, the self-improving training loop in Karpathy’s AutoResearch walkthrough trains a small GPT-style model end to end.

Open Questions

  • The researcher’s name is uncertain from the captions (see above).
  • The exact Qwen model is captioned “Qwen to 0.5B” and “Q two”; it is probably a Qwen 2 or 2.5 0.5B checkpoint.
  • The notebook was promised in the video description, which is not in the raw file.