Homework 04
Due Wed., 10/04/2023, 4:00 PM
http://iailab.kaist.ac.kr/
Industrial AI Lab at KAIST
For your handwritten solution, scan or take a picture of them (you can write it in markdown if you want).
For your code, only .ipynb file will be graded.
- Please write your NAME and student ID on your .ipynb files. ex) IljeokKim_20202467_HW02.ipynb
Please compress all the files to make a single .zip file
- Please write your NAME and student ID on your .zip files. ex) DogyeomPark_20202467_HW02.zip
- Submit it to KLMS
Do not submit a printed version of your code. It will not be graded.
Problem 1: Understanding Backpropagation¶
In this problem, we are going to compute the gradient using the chain rule and dynamic programming, and update the weights $\omega \rightarrow \omega^+$. After that, the weights are updated through 1 back-propagation and compared with the error before the update.
Neural Network Model
- The artificial neural network structure: an input layer, a hidden layern, and an output layer.
- All neurons ($h_1$, $h_2$, $\sigma_1$ and $\sigma_2$) in the hidden and output layers use the sigmoid function as activation function.
- The red number means the initial weight values, the blue number means the input values, and the ground truth means the actual values.
- The loss function is the mean square error (MSE). Use
1/2 MSEfor calculation convenience. For example, $E = \frac{1}{2}\sum(\text{target} - \text{output})^2$ - Learning rate is set to 0.9.
Step 1: Forward Propagation¶
(1) [hand written] Write and calculate $z_1$, $z_2$, $h_1$, $h_2$, $z_3$, $z_4$, $\sigma_1$, $\sigma_2$, and $E_{\text{total}}$ of forward propagation.
Step 2: BackPropagation 1¶
(2) [hand written] update $\omega_5$, $\omega_6$, $\omega_7$, $\omega_8$ $\rightarrow$ $\omega_5^+$, $\omega_6^+$, $\omega_7^+$, $\omega_8^+$ of back-propagation.
Step 3: BackPropagation 2¶
(3) [hand written] update $\omega_1$, $\omega_2$, $\omega_3$, $\omega_4$ $\rightarrow$ $\omega_1^+$, $\omega_2^+$, $\omega_3^+$, $\omega_4^+$ of back-propagation.
Step 4: Check the Result for Weight Update¶
(4) [hand written] Write and calculate $E_{\text{total}}$ with the updated weights, and compare it to the previous error.
Problem 2: Multi-Layer Perceptron¶
You will do binary classification for nonlinearly seperable data using MLP. Plot the given data first.
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
%matplotlib inline
N = 200
M = 2*N
gamma = 0.01
G0 = np.random.multivariate_normal([0, 0], gamma*np.eye(2), N)
G1 = np.random.multivariate_normal([1, 1], gamma*np.eye(2), N)
G2 = np.random.multivariate_normal([0, 1], gamma*np.eye(2), N)
G3 = np.random.multivariate_normal([1, 0], gamma*np.eye(2), N)
train_X = np.vstack([G0, G1, G2, G3])
train_y = np.vstack([np.ones([M,1]), np.zeros([M,1])])
train_X = np.asmatrix(train_X)
train_y = np.asmatrix(train_y)
print(train_X.shape)
print(train_y.shape)
plt.figure(figsize = (8, 6))
plt.plot(train_X[:M,0], train_X[:M,1], 'b.', alpha = 0.4, label = 'A')
plt.plot(train_X[M:,0], train_X[M:,1], 'r.', alpha = 0.4, label = 'B')
plt.axis('equal')
plt.xlim([-1, 2]); plt.ylim([-1, 2]);
plt.grid(alpha = 0.15)
plt.legend(fontsize = 12)
plt.show()
(1) Design a perceptron model which has a single layer, and train it to show the accuracy.
- Hidden layer with no nonlinear activiation function
model = tf.keras.models.Sequential([
## your code here
])
model.summary()
## Your code here
#
(2) Plot the classifier (decision boundary).
## Your code here
#
(3) What is the highest accuracy you can get? Discuss the result.
## write down your discussion here
#
#
(4) Modify a perceptron model which has 2 layers, and train it to show the accuracy.
- Hidden layer: sigmoid function
## Your code here
model = tf.keras.models.Sequential([
## your code here
])
model.summary()
## Your code here
#
(5) Plot two linear classification boundaries in the input space.
## your code here
#
(6) Plot one linear classification boundary in z space (or values in the hidden layer).
## your code here
#
Problem 3: ANN Regression¶
In this problem, you are asked to use TensorFlow to implement the linear regression algorithm. By doing this, we hope that you are getting familiar with the syntax of TensorFlow.
# Data Generation
m = 5000
data_x = np.linspace(-3, 3, m)
data_y = 0.8*data_x + 2 + np.random.randn(m)*0.3
plt.figure(figsize = (10,8))
plt.plot(data_x, data_y, '.', alpha = 0.4)
plt.axis('equal')
plt.show()
We will build the simplest ANN model as shown in the following figure in order to find the best line fit (i.e., linear regression) for the given training data set. Note that there is no hidden layer, and both the input and output layer have only one neuron.
(1) Define the AI model.
model = tf.keras.models.Sequential([
## your code here
])
model.summary()
## Your code here
#
(2) Find the estimated weight and bias from the trained model.
## Your code here
#
(3) Plot the linear regression.
## Your code here
#
