Recurrent Neural Network

By Prof. Seungchul Lee
http://iai.postech.ac.kr/
Industrial AI Lab at POSTECH

Table of Contents

1. Recurrent Neural Network (RNN)¶

• RNNs are a family of neural networks for processing sequential data

1.1. Feedforward Network and Sequential Data¶

• Separate parameters for each value of the time index
• Cannot share statistical strength across different time indices

In [1]:
%%html
<center><iframe src="https://www.youtube.com/embed/oYglxfBtSQk?rel=0"
width="560" height="315" frameborder="0" allowfullscreen></iframe></center>


1.2. Representation Shortcut¶

• Input at each time is a vector
• Each layer has many neurons
• Output layer too may have many neurons
• But will represent everything simple boxes
• Each box actually represents an entire layer with many units

1.3. Finite-Response Model¶

• Something that happens today only affects the output of the system for $N$ days into the future
• $N$ is the width of the system

• Problem: Increasing the “history” makes the network more complex
$$Y_t = f\left(X_t, X_{t-1}, \cdots, X_{t - N} \right)$$

1.4. In Theory, We Want Infinite Memory¶

• Required: Infinite response systems
• What happens today can continue to affect the output forever
• Possibly with weaker and weaker influence
$$Y_t = f\left(X_t, X_{t-1}, \cdots, X_{t - \infty} \right)$$

• Recursive
• Output contains information about the entire past
$$Y_t = f\left(X_t, X_{t-1}, \cdots, X_{t - \infty} \right) \quad \Longrightarrow \quad Y_t = f\left(X_t, Y_{t-1} \right)$$

1.5. An Alternate Model for Infinite Response Systems¶

• The state-space model
\begin{align*} h_t &= f(x_t, h_{t-1})\\ y_t &= g(h_t) \end{align*}
• This is a recurrent neural network
• State summarizes information about the entire past
• Single Hidden Layer RNN (Simplest State-Space Model)

• Multiple Recurrent Layer RNN