Recurrent neural networks#

In this session we learn about language modeling. We look at a general definition of a language model and then zoom in on left-to-right LMs, for which discuss training, prediction and evaluation. We study recurrent neural networks as a first instance of neural language models.

Learning goals for this session#

  1. become familiar with language modeling

    1. causal (left-to-right) models

    2. training, prediction, evaluation

  2. meet a first neural LM: recurrent neural networks

  3. implement a character-level RNN

    1. loss from next-word surprisal

    2. teaching forcing

    3. autoregressive (greedy) decoding

Slides#

Here are the slides for this session.

Practical exercises#

There is one notebook with hands-on exercises for RNNs.