Optimization Fundamentals


By Prof. Seungchul Lee
http://iailab.kaist.ac.kr/
Industrial AI Lab at KAIST

Table of Contents


1. Optimization Basics

In [1]:
from IPython.display import YouTubeVideo
YouTubeVideo('SembyQdDGo8', width = "560", height = "315")
Out[1]:

1.1. Terminology

1.2. Optimality Condition


  • If function f is C1 continous, then the necessary condition for $x^*$ to be a local minimum is

$$ f'(x^*)=0 $$


  • If function f is C2 continous, then the necessary condition for $x^*$ to be a strong local minimum is

$$ f'(x^*) = 0 \, \text{ and } \, f''(x^*) > 0 $$


1.3. Convex Function


$$ f \left( ax_1+(1-a)x_2 \right)\; \leq \; a f(x_1)+(1-a) f(x_2) \quad \text{when}\quad 0 \leq a \leq 1$$



Properties of Convex Function


$$ \begin{align*} f(y) & \geq f(x) + f'(x)(y-x)\\ f''(x) &\geq 0 \end{align*} $$


2. Optimization Models

In [2]:
from IPython.display import YouTubeVideo
YouTubeVideo('SembyQdDGo8?si=rraTYYhQnMMJS_5Z&start=625', width = "560", height = "315")
Out[2]:

2.1. General Optimizatin Model


$$\begin{align*} \text{minimize} \quad & f(x)\\ \text{subject to} \quad & x\in F \end{align*}$$


  • Design variable $x$: element of an arbitary set $F$

  • Objective function $f:F\rightarrow \mathbb{R} $

  • (Global) optimal solution $x^*: x^*\in F$ and $\forall x\in F, f(x^*)\leq f(x)$

  • Local optimal solution $\tilde{x}$ with respect to neighborhood $\mathcal{N}(\tilde{x})\subseteq F: \tilde{x} \in F$ and $\forall x\in \mathcal{N}(\tilde{x}), f(\tilde{x})\leq f(x)$


2.2. Negative Null Form


$$\begin{align*} \text{minimize} \quad &f(x)\\ \text{subject to} \quad &g(x)\leq 0\\ &h(x)=0\\ &\quad \; x\in S\\\\ \end{align*}$$

  • Optimization form of many nonlinear optimization software

  • Objective function $f$, inequality constraint $g$, and equality constraint $h$ can be linear or nonlinear


2.3. Standard Form


$$\begin{align*} \text{minimize} \quad &c^Tx\\ \text{subject to} \quad & Ax = b\\ \quad & \;\;\, x\geq 0\\ \quad & \;\;\, x\in S\\\\ \end{align*}$$


2.4. Canonical Form


$$\begin{align*} \text{minimize} \quad &c^Tx\\ \text{subject to} \quad & Ax\leq b\\ \quad & \;\;\, x\geq 0\\ \quad & \;\;\, x\in S\\\\ \end{align*}$$

  • Design variable $x$ can be all continuous (LP), partially integer (MILP), all integer (ILP), or all binary (ZOILP).

2.5. Typical Forms in Many Numerical Solvers

  1. Linear programming

$$\begin{align*} \min_{x} \quad & c^Tx \\ \text{subject to} \quad & Ax \leq b\\ & A_{eq}x = b_{eq}\\ & l \leq x \leq u \end{align*} $$


  1. Integer programming

$$\begin{align*} \min_{x} \quad & c^Tx \\ \text{subject to} \quad & Ax \leq b\\ & A_{eq}x = b_{eq}\\ & l \leq x \leq u\\ & \color{red}{x \in \mathbb{Z} \;\text{(integer)}} \end{align*} $$


  1. Mixed integer linear programming

$$\begin{align*} \min_{x,z} \quad & c^T \begin{bmatrix} x \\ z \end{bmatrix} \\ \text{subject to} \quad & A \begin{bmatrix} x \\ z \end{bmatrix} \leq b \\ & A_{eq} \begin{bmatrix} x \\ z \end{bmatrix} = b_{eq}\\ & l \leq \begin{bmatrix} x \\ z \end{bmatrix} \leq u \\ & x \in \mathbb{R} \;\text{(real)} \\ & z \in \mathbb{Z} \end{align*} $$


  1. Binary integer programming

$$\begin{align*} \min_{x} \quad & c^Tx \\ \text{subject to} \quad & Ax \leq b \\ & A_{eq}x = b_{eq} \\ & l \leq x \leq u \\ & \color{red}{x \in \{0,1\}} \end{align*} $$


  1. Quadratic programming

$$\begin{align*} \min_{x} \quad & \frac{1}{2}x^THx + c^Tx \\ \text{subject to} \quad & Ax \leq b \\ & A_{eq}x = b_{eq} \\ & l \leq x \leq u \end{align*} $$

3. Commom Modeling Techniques

In [3]:
from IPython.display import YouTubeVideo
YouTubeVideo('SembyQdDGo8?si=qi-8JszbxlMSp6tB&start=1583', width = "560", height = "315")
Out[3]:

3.1. From Canonical to Standard Form


$$\begin{align*} \text{minimize}\quad & c^Tx & \text{minimize} \quad & c^Tx\\ \text{subject to} \quad & Ax \color{red}{\leq} b \qquad \Longrightarrow & \text{subject to} \quad & Ax \color{red}{=} b\\ \quad & x \geq 0 && x \geq 0\\ \quad & x \in S && x\in S\\\\ \end{align*}$$


$$ \begin{align*} Ax \leq b, x\geq 0 \quad \implies \quad \begin{bmatrix} A & \mathbb{1} \end{bmatrix} \begin{bmatrix} x\\ \color{red}{\omega} \end{bmatrix}=b,\quad \begin{bmatrix} x\\ \color{red}{\omega} \end{bmatrix} \geq 0 \end{align*}$$


  • $\color{red}{\omega\geq 0}$ is a slack variable.

3.2. From Discrete to Binary Transformation


$$\begin{align*} x\in D=\{d_1, d_2, \cdots, d_n\}\qquad \Longrightarrow \qquad &x = d_1 y_1 + d_2 y_2 + \cdots + d_n y_n\\ & \sum_{k=1}^{n} y_i = 1, \quad y_i \in \{0, 1\} \end{align*}$$


3.3. From Logical (or Boolean) to Binary Transformation


$$\begin{align*} p \wedge q,\;\; p, q\in \{\text{true, false}\} \qquad &\Longrightarrow \qquad x_1x_2=1,\;\; x_1,x_2\in \{0, 1\}\\\\ p \vee q,\;\; p,q\in \{\text{true, false}\} \qquad &\Longrightarrow \qquad x_1+x_2\geq1,\;\; x_1,x_2\in \{0, 1\}\\\\ \end{align*}$$


3.4. Equality $\rightarrow$ Inequality Transformation


$$ h(x)=0 \qquad \Longrightarrow \qquad \begin{cases} \;\;\, h(x) \leq 0\\ -h(x) \leq 0 \end{cases}$$


3.5. From Max to Min Transformation


$$\begin{align*} &\max \; f(x) \qquad \Longrightarrow \qquad \min \;(-f(x)) \\\\ &\max \; f(x) \qquad \Longrightarrow \qquad \min \left(\frac{1}{f(x)}\right) \end{align*}$$


  • The second method only if $\;\forall x,\; f(x)\neq 0$.

3.6. Moving Constraints to Objective Functions

  • Quadratic penalty method

$$\begin{align*} \text{minimize} & \quad f(x) & \text{minimize} & \quad f(x) + \sum_{i=1}^{l} \gamma_i\{\max(0, g_i(x))\}^2 + \sum_{i=1}^{m} \mu_i \{h_i(x)\}^2\\ \text{subject to} & \quad g(x) \leq 0 \quad \qquad \Longrightarrow \quad & \text{subject to} & \quad x \in S\\ &\quad h(x) = 0\\ &\quad x\in S \end{align*}$$


3.7. From Multiple to Single Objective

  • Weighted sum method

$$\min \; f_1(x), f_2(x), \cdots, f_m(x) \qquad \Longrightarrow \qquad \min \; F(x)=\omega_1 f_1(x) + \omega_2 f_2(x) + \cdots + \omega_m f_m(x)$$


  • Weights $\omega_1, \omega_2, \cdots, \omega_m$ reflect the designer's preference to the objectives.

  • Difficult to determine good values for $\omega_1, \omega_2, \cdots, \omega_m$ if the objectives are not normalized.


  • Goal programming

$$\begin{align*} \min \; f_1(x), f_2(x), \cdots, f_m(x) \qquad \Longrightarrow \quad \qquad \min \; &\beta \\\\ \text{subject to}\; &f_1(x), f_2(x),\cdots, f_m(x)\leq \beta \end{align*}$$

In [ ]:
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')