Recurrent Neural Network

By Prof. Seungchul Lee
Industrial AI Lab at POSTECH

Table of Contents

1. Recurrent Neural Network (RNN)

  • RNNs are a family of neural networks for processing sequential data

1.1. Feedforward Network and Sequential Data

  • Separate parameters for each value of the time index
  • Cannot share statistical strength across different time indices

In [1]:
<center><iframe src="" 
width="560" height="315" frameborder="0" allowfullscreen></iframe></center>

1.2. Representation Shortcut

  • Input at each time is a vector
  • Each layer has many neurons
  • Output layer too may have many neurons
  • But will represent everything simple boxes
  • Each box actually represents an entire layer with many units

1.3. Finite-Response Model

  • Something that happens today only affects the output of the system for $N$ days into the future
  • $N$ is the width of the system

  • Problem: Increasing the “history” makes the network more complex
$$Y_t = f\left(X_t, X_{t-1}, \cdots, X_{t - N} \right)$$

1.4. In Theory, We Want Infinite Memory

  • Required: Infinite response systems
    • What happens today can continue to affect the output forever
    • Possibly with weaker and weaker influence
$$Y_t = f\left(X_t, X_{t-1}, \cdots, X_{t - \infty} \right)$$

  • Recursive
    • Output contains information about the entire past
$$Y_t = f\left(X_t, X_{t-1}, \cdots, X_{t - \infty} \right) \quad \Longrightarrow \quad Y_t = f\left(X_t, Y_{t-1} \right)$$

1.5. An Alternate Model for Infinite Response Systems

  • The state-space model
$$ \begin{align*} h_t &= f(x_t, h_{t-1})\\ y_t &= g(h_t) \end{align*} $$
  • This is a recurrent neural network
  • State summarizes information about the entire past
  • Single Hidden Layer RNN (Simplest State-Space Model)

  • Multiple Recurrent Layer RNN