Optimization
Table of Contents
$\;\;\;$(source: http://nautil.us/blog/to-save-drowning-people-ask-yourself-what-would-light-do)
3 key components
Procedures
In mathematical expression
$$\begin{align*}
\min_{x} \quad &f(x) \\
\text{subject to} \quad &g_i(x) \leq 0, \qquad i=1,\cdots,m
\end{align*}
$$
$\;\;\; $where
Remarks) equivalent
Generalization for multivariate function $f:\mathbb{R}^n \rightarrow \ \mathbb{R}$
$$
x = \begin{bmatrix}
x_1 \\
x_2 \\
\vdots \\
x_n
\end{bmatrix} \quad \quad \quad \quad \nabla _x f(x) = \begin{bmatrix}
\partial f(x) \over \partial x_1 \\
\vdots\\
\partial f(x) \over \partial x_n
\end{bmatrix}
$$
Direct solution
$$
\begin{align*}
f(x) &= 2x_1^2+ x_2^2 + x_1 x_2 -6 x_1 -5 x_2\\\\
\Longrightarrow \nabla _x f(x) &= \begin{bmatrix}
4x_1+x_2-6\\
2x_2 + x_1 -5
\end{bmatrix} = \begin{bmatrix}0\\0 \end{bmatrix}\\\\
\therefore x^* &= \begin{bmatrix}
4 & 1\\
1 & 2
\end{bmatrix} ^{-1} \begin{bmatrix}
6 \\ 5\\
\end{bmatrix} = \begin{bmatrix}
1 \\ 2\\
\end{bmatrix}
\end{align*}
$$
Iterative methods
$$ x \leftarrow x - \alpha \nabla _x f(x) \quad \quad \text{for some step size } \alpha > 0$$
Example
$$
\begin{align*}
\min& \quad (x_1-3)^{2} + (x_2-3)^{2}\\\\
=\min& \quad \frac{1}{2} \begin{bmatrix} x_1 & x_2 \end{bmatrix}
\begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} -
\begin{bmatrix} 6 & 6 \end{bmatrix}
\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} + 18
\end{align*}
$$
import numpy as np
H = np.matrix([[2, 0],[0, 2]])
g = -np.matrix([[6],[6]])
x = np.zeros((2,1))
alpha = 0.2
for i in range(25):
df = H*x + g
x = x - alpha*df
print(x)
$\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \quad \qquad \qquad \bullet \, \text{Random initialization}$
$\qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \qquad \quad \qquad \qquad \bullet \, \text{Multiple trials}$
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')