Machine Learning for Mechanical Engineering

Regression


Prof. Seungchul Lee
http://iailab.kaist.ac.kr/
Industrial AI Lab at KAIST
  • For your handwritten solutions, please scan or take a picture of them. Alternatively, you can write them in markdown if you prefer.

  • Only .ipynb files will be graded for your code.

    • Ensure that your NAME and student ID are included in your .ipynb files. ex) SeungchulLee_20241234_HW01.ipynb
  • Compress all the files into a single .zip file.

    • In the .zip file's name, include your NAME and student ID.

    ex) SeungchulLee_20241234_HW01.zip

    • Submit this .zip file on KLMS
  • Do not submit a printed version of your code, as it will not be graded.

Problem 1

Suppose you receive the binary signal. The signal is corrupted with noises while transmitting through channels. We want to get rid of noise from the original signal through the $L_1$ optimization. The mathematical problem statement is given:


$$ \begin{array}{Icr}\begin{align*} y = x + \omega\\ x \in \{ 0, 1 \}\\ \omega \text{ is noise} \end{align*}\end{array} \quad \Longrightarrow \begin{array}{I} \quad \text{Recover original signal } x \text{ from corrupted signal } y \end{array} $$

Step 1. Data generation

(a) First, generate the original signal of 200 data points as shown in the below figure. This can be simply done using ones and zeros commands in Python.

In [ ]:
# Write your code here
#

(b) Next, corrupt the original signal with Gaussian noise. This can be done using randn command in Python. One of realizations of the corrupted signal will be given:

In [ ]:
# Write your code here
#

Step 2. $L_1$ optimization

Note that the signal is sparse (in a sense of frequency domain). Therefore we can apply the $L_1$ optimization. We will optimize the $L_1$ loss function with the $L_2$ constraints.

(c) Plot the reconstructed signal. In addition, by changing $\beta$ in the lecture slide, explain how $\beta$ affects in an optimization process. One of reconstructed signal with a proper value of $\beta$ is shown below:

In [ ]:
# Write your code here
#

Problem 2

The regularized least-squares problem has the form


$$ \min_{\theta} \;\lVert A\theta -y\rVert_2^2 + \lambda \lVert \theta \rVert_2^2$$

(a) Show that the solution is given by

$$ \hat{\theta} = \left( A^T A + \lambda I_n \right)^{-1} A^T y $$

  • Do not use the method of Lagrangian multipliers



(b) Write down a gradient descent algorithm for the given optimization problem.


* Hint: Note that $$ \;\lVert A\theta -y\rVert_2^2 = (A\theta - y)^T(A\theta - y)$$
Then, you can differentiate the above equation to compute the gradient. Likewise, you can compute the gradient of the regularizer.

(c) Based on the result of (b), describe the role of regularizer term.

  • Hint: Gradient $g$ is computed by $ g = g_{\text{projection}} + g_{\text{regularizer}} $.

(d) Describe results of (a) and (b) have the same meaning.


(e) Find and draw an approximated curve of the given data points below in Python using your gradient descent algorithm.


In [ ]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# 10 data points
n = 10
x = np.linspace(-4.5, 4.5, 10).reshape(-1, 1)
y = np.array([0.9819, 0.7973, 1.9737, 0.1838, 1.3180, -0.8361, -0.6591, -2.4701, -2.8122, -6.2512]).reshape(-1, 1)

plt.figure(figsize = (10, 8))
plt.plot(x, y, 'o', label = 'Data')
plt.xlabel('X', fontsize = 15)
plt.ylabel('Y', fontsize = 15)
plt.grid(alpha = 0.3)
plt.show()
In [ ]:
# Write your code here
#
[[ 3.48274701e-01]
 [-2.58951123e+00]
 [-4.55286474e-01]
 [ 1.85022226e+00]
 [ 1.06250369e-01]
 [-4.43328786e-01]
 [-9.25753472e-03]
 [ 3.63088178e-02]
 [ 2.35143849e-04]
 [-9.24099978e-04]]

Problem 3: Image Panorama with Regression

We want to demonstrate an image panorama as an example of linear regression. A panorama is any wide-angle view or representation of a physical space.



In [ ]:
%%html
<center><iframe src="https://www.youtube.com/embed/86rnwu3ZFbE?rel=0"
width="560" height="315" frameborder="0" allowfullscreen></iframe></center>
In [ ]:
# import library
import numpy as np
import matplotlib.pyplot as plt
import cv2
In [ ]:
# load images
imag1 = cv2.imread('./data_files/1.jpg')
imag1 = cv2.cvtColor(imag1, cv2.COLOR_BGR2RGB)
imag2 = cv2.imread('./data_files/2.jpg')
imag2 = cv2.cvtColor(imag2, cv2.COLOR_BGR2RGB)
imag3 = cv2.imread('./data_files/3.jpg')
imag3 = cv2.cvtColor(imag3, cv2.COLOR_BGR2RGB)
In [ ]:
## your code here
#
In [ ]:
## your code here
#
In [ ]:
## your code here
#