한국소성가공학회 전문교육

Part 1: 인공지능 기초


Prof. Seungchul Lee
http://iai.postech.ac.kr/
Industrial AI Lab at POSTECH

Table of Contents

1. Machine Learning and Deep Learning




1.1. Taxonomy of AI



1.2. Scikit Learn

  • Machine Learning in Python
  • Simple and efficient tools for data mining and data analysis
  • Accessible to everybody, and reusable in various contexts
  • Built on NumPy, SciPy, and matplotlib
  • Open source, commercially usable - BSD license
  • https://scikit-learn.org/stable/index.html




1.3. Supervised Learning

  • Given training set $\left\{ \left(x^{(1)}, y^{(1)}\right), \left(x^{(2)}, y^{(2)}\right),\cdots,\left(x^{(m)}, y^{(m)}\right) \right\}$
  • Want to find a function $f_{\omega}$ with learning parameter, $\omega$
    • $f_{\omega}$ desired to be as close as possible to $y$ for future $(x,y)$
    • $i.e., f_{\omega}(x) \approx y$
  • Define a loss function
$$\ell \left(f_{\omega} \left(x^{(i)}\right), y^{(i)}\right)$$
  • Solve the following optimization problem:
$$ \begin{align*} \text{minimize} &\quad \frac{1}{m} \sum_{i=1}^{m} \ell \left(f_{\omega} \left(x^{(i)}\right), y^{(i)}\right)\\ \text{subject to} &\quad \omega \in \boldsymbol{\omega} \end{align*} $$


  • Function approximation between inputs and outputs


  • Once it is learned,

2. Regression

  • A set of statistical processes for estimating the relationships between a dependent variable and one or more independent variables

2.1. Linear Regression



2.2. Multivariate Linear Regression



2.3. Nonlinear Regression



2.4. Feature Selection

  • Multivariate regression


$$ \hat{y} = \theta_0 + \theta_{1}x_1 + \theta_{2}x_2 + \theta_{3}x_3 + \cdots $$

  • The process of selecting a subset of relevant features (variables, predictors) for use in model construction.
  • Feature selection techniques are used for several reasons:
    • simplification of models to make them easier to interpret,
    • shorter training times,
    • to avoid the curse of dimensionality,
    • improve data's compatibility with a learning model class,
    • encode inherent symmetries present in the input space.

2.5. Correlation Coefficient

  • $+1 \to$ close to a straight line

  • $-1 \to$ close to a straight line

  • Indicate how close to a linear line, but

  • No information on slope

$$0 \leq \left\lvert \text{ correlation coefficient } \right\rvert \leq 1$$$$\hspace{1cm}\begin{array}{Icr}\leftarrow\\ (\text{uncorrelated})\end{array} \quad \quad \quad \begin{array}{Icr}\rightarrow \\ (\text{linearly correlated})\end{array}$$
  • Does not tell anything about causality

2.6. Correlation Coefficient Plot



2.7. Python

2.7.1. Linear Regression

In [1]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# data points in column vector [input, output]
x = np.array([0.1, 0.4, 0.7, 1.2, 1.3, 1.7, 2.2, 2.8, 3.0, 4.0, 4.3, 4.4, 4.9]).reshape(-1, 1)
y = np.array([0.5, 0.9, 1.1, 1.5, 1.5, 2.0, 2.2, 2.8, 2.7, 3.0, 3.5, 3.7, 3.9]).reshape(-1, 1)

# to plot
plt.figure(figsize=(10, 6))
plt.title('Linear Regression', fontsize=15)
plt.xlabel('X', fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.plot(x, y, 'ko', label="data")
plt.xlim([0, 5])
plt.grid(alpha=0.3)
plt.axis('scaled')
plt.show()
In [3]:
from sklearn.linear_model import LinearRegression

reg = LinearRegression()
reg.fit(x,y)
Out[3]:
LinearRegression()
In [4]:
print(reg.coef_)       # Coef
print(reg.intercept_)  # Bias

[[0.67129519]]
[0.65306531]
In [5]:
# to plot
plt.figure(figsize=(10, 6))
plt.title('Linear Regression', fontsize=15)
plt.xlabel('X', fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.plot(x, y, 'ko', label="data")

# to plot a straight line (fitted line)
xp = np.arange(0, 5, 0.01).reshape(-1, 1)
yp = reg.coef_*xp + reg.intercept_

plt.plot(xp, yp, 'r', linewidth=2, label="$L_2$")
plt.legend(fontsize=15)
plt.axis('scaled')
plt.grid(alpha=0.3)
plt.xlim([0, 5])
plt.show()

2.7.2. Nonlinear Regression

In [6]:
n = 100            
x = -5 + 15*np.random.rand(n, 1)
noise = 10*np.random.randn(n, 1)
y = 10 + 1*x + 2*x**2 + noise

plt.figure(figsize=(10, 6))
plt.title('Nonlinear Regression', fontsize=15)
plt.xlabel('X', fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.plot(x, y, 'o', markersize=4, label='actual')
plt.xlim([np.min(x), np.max(x)])
plt.grid(alpha=0.3)
plt.legend(fontsize=15)
plt.show()
In [7]:
from sklearn.kernel_ridge import KernelRidge

reg = KernelRidge(kernel='rbf', gamma=0.1)
reg.fit(x, y)
Out[7]:
KernelRidge(gamma=0.1, kernel='rbf')
In [8]:
p = reg.predict(x)
In [9]:
plt.figure(figsize=(10, 6))
plt.title('Nonlinear Regression', fontsize=15)
plt.xlabel('X', fontsize=15)
plt.ylabel('Y', fontsize=15)
plt.plot(x, y, 'o', markersize=4, label='actual')
plt.plot(x, p, 'ro', markersize=4, label='predict')
plt.grid(alpha=0.3)
plt.legend(fontsize=15)
plt.xlim([np.min(x), np.max(x)])
plt.show()

3. Classification

  • where $y$ is a discrete value
    • develop the classification algorithm to determine which class a new input should fall into
  • To find a classification boundary
  • We will learn
    • Support Vector Machine (SVM)
    • Logistic Regression


3.1. Linear Classification


3.2. Non-linear Classification


3.3. Python

3.3.1. SVM

In [10]:
x1 = 8*np.random.rand(100, 1)
x2 = 7*np.random.rand(100, 1) - 4

g0 = 0.8*x1 + x2 - 3
g1 = g0 - 1
g2 = g0 + 1

C1 = np.where(g1 >= 0)[0]
C2 = np.where(g2 < 0)[0]

X1 = np.hstack([x1[C1],x2[C1]])
X2 = np.hstack([x1[C2],x2[C2]])
n = X1.shape[0]
m = X2.shape[0]
X = np.vstack([X1, X2])
y = np.vstack([np.zeros([n, 1]), np.ones([m, 1])])

plt.figure(figsize=(10, 6))
plt.plot(x1[C1], x2[C1], 'ro', label='C1')
plt.plot(x1[C2], x2[C2], 'bo', label='C2')
plt.xlabel('$x_1$', fontsize = 20)
plt.ylabel('$x_2$', fontsize = 20)
plt.legend(loc = 4)
plt.xlim([0, 8])
plt.ylim([-4, 3])
plt.show()
In [11]:
from sklearn.svm import SVC

clf = SVC(kernel='linear')
clf.fit(X, np.ravel(y))
Out[11]:
SVC(kernel='linear')
In [12]:
print(clf.coef_)
print(clf.intercept_)

[[-0.76177006 -0.93761421]]
[2.85068138]
In [13]:
xp = np.linspace(0,8,100).reshape(-1,1)
yp = -clf.coef_[0,0]/clf.coef_[0,1]*xp - clf.intercept_/clf.coef_[0,1]

plt.figure(figsize=(10, 6))
plt.plot(X[0:n, 0], X[0:n, 1], 'ro', label='C1')
plt.plot(X[n:-1, 0], X[n:-1, 1], 'bo', label='C2')
plt.plot(xp, yp, '--k', label='SVM')
plt.xlabel('$x_1$', fontsize = 20)
plt.ylabel('$x_2$', fontsize = 20)
plt.legend(loc = 4)
plt.xlim([0, 8])
plt.ylim([-4, 3])
plt.show()

3.3.2. Logistic Regression

In [14]:
m = 500

X0 = np.random.multivariate_normal([0, 0], np.eye(2), m)
X1 = np.random.multivariate_normal([10, 10], np.eye(2), m)

X = np.vstack([X0, X1])
y = np.vstack([np.zeros([m,1]), np.ones([m,1])])

plt.figure(figsize=(10, 6))
plt.plot(X0[:,0], X0[:,1], '.b', label='Class 0')
plt.plot(X1[:,0], X1[:,1], '.k', label='Class 1')

plt.title('Data Classes', fontsize=15)
plt.legend(loc='lower right', fontsize=15)
plt.xlabel('X1', fontsize=15)
plt.ylabel('X2', fontsize=15)
plt.xlim([-10,20])
plt.ylim([-4,14])
plt.grid(alpha=0.3)
plt.show()
In [15]:
from sklearn.linear_model import LogisticRegression

clf = LogisticRegression()
clf.fit(X, np.ravel(y))
Out[15]:
LogisticRegression()
In [16]:
print(clf.coef_)
print(clf.intercept_)

[[0.93938153 0.90472328]]
[-9.3291099]
In [17]:
xp = np.linspace(-10,20,100).reshape(-1,1)
yp = -clf.coef_[0,0]/clf.coef_[0,1]*xp - clf.intercept_/clf.coef_[0,1]

plt.figure(figsize=(10, 6))
plt.plot(X0[:,0], X0[:,1], '.b', label='Class 0')
plt.plot(X1[:,0], X1[:,1], '.k', label='Class 1')
plt.plot(xp, yp, '--k', label='Logistic')
plt.xlim([-10,20])
plt.ylim([-4,14])

plt.title('Data Classes', fontsize=15)
plt.legend(loc='lower right', fontsize=15)
plt.xlabel('X1', fontsize=15)
plt.ylabel('X2', fontsize=15)
plt.grid(alpha=0.3)
plt.show()
In [18]:
pred = clf.predict_proba([[0,6]])
pred
Out[18]:
array([[0.98017467, 0.01982533]])

4. Steps for Machine Learning

4.1. Model Evaluation

  • Adding more features will always decrease the loss
  • How do we determine when an algorithm achieves “good” performance?


  • A better criterion:
    • Training set (e.g., 70 %)
    • Testing set (e.g., 30 %)
  • Performance on testing set called generalization performance

4.2. Supervised Learning

  • Workflow



  • Workflow in more detail



5. Supervised Learning vs. Unsupervised Learning




6. Clustering

  • Data clustering is an unsupervised learning problem

  • Given:

    • $m$ unlabeled examples $\{x^{(1)},x^{(2)}\cdots, x^{(m)}\}$
    • the number of partitions $k$
  • Goal: group the examples into $k$ partitions


$$\{x^{(1)},x^{(2)},\cdots,x^{(m)}\} \quad \Rightarrow \quad \text{Clustering}$$


6.1. K-means