Inequality


By Prof. Seungchul Lee
http://iailab.kaist.ac.kr/
Industrial AI Lab at KAIST

Table of Contents

In [1]:
from IPython.display import YouTubeVideo
YouTubeVideo('CO0JK3JwSu4', width = "560", height = "315")
Out[1]:

1. First Example


$$\begin{align*}\min& \; f(x)=x + \frac{1}{x}\\ \text{s.t.} &\; x\in (0, \infty)\\\\ \end{align*}$$

  • Approach 1

$$\begin{align*}\frac{d f}{d x}=1 - \frac{1}{x^2} = 0\;\;\; \implies \;\;\; x^* = 1\\\\ \end{align*}$$
  • Approach 2


  • Approach 3

$$\begin{align*}f(x) = x + \frac{1}{x}\;\; \geq \;\; 2\sqrt{x\cdot\frac{1}{x}} \;\; = \;\; 2\\\\ \end{align*}$$

2. Arithmetic Mean-Geometric Mean Inequality (AGM)

In [2]:
from IPython.display import YouTubeVideo
YouTubeVideo('CO0JK3JwSu4?si=4rMfQ8lKFv7bdmOo&start=224', width = "560", height = "315")
Out[2]:

Let $x_1, x_2, \cdots, x_n$ be positive numbers


$$ \begin{align*} G & = \left( x_1 x_2 \cdots x_n \right)^\frac{1}{n}\\ \\ A & = \frac{x_1+x_2+\cdots+x_n}{n}\\\\ \end{align*} $$

Then,

$$ \begin{align*} G&\leq A\\ \\ (x_1 x_2 \cdots x_n)^\frac{1}{n} \; &\leq \; \frac{x_1+x_2+\cdots +x_n}{n}\\\\ \end{align*} $$

with equality if and only if $\;x_1 = x_2 = \cdots = x_n$


Generalized AGM Inequality (GAGM)



$$ x_1^{a_1} x_2^{a_2} \cdots x_n^{a_n}\; \leq \; a_1x_1 + a_2x_2 + \cdots + a_nx_n\\\\ $$

where

$$ \begin{align*} x_i &\geq 0\\ \sum_i^n a_i &= 1\\ a_i &\geq 0 \end{align*} $$

with equality iff $\;x_1=x_2=\cdots=x_n$


2.1. Convex Function


A function $f(x)$ is said to be convex if


$$f(a_1x_1+\cdots+a_nx_n)\;\leq\;a_1f(x_1)+\cdots+a_nf(x_n)$$

where


$$ \begin{align*} a_i &\geq 0 \\ \sum_i^na_i &= 1\\\\ \end{align*} $$

When $n = 2$


$$ f \left( ax_1+(1-a)x_2 \right)\; \leq \; a f(x_1)+(1-a) f(x_2) $$

2.2. Example of GAGM (or Proof)


Consider the function of $f(x)=-\log{x}$.

  1. Show that $f(x)=-\log{x}$ is convex.

  2. Prove the generalized AGM inequality.


Solution


Jensen's Inequality when $f(x)$ is convex:

$$ f\left( \sum_{i} a_i x_i \right) \leq \sum_{i} a_i f(x_i) $$

where $a_i \geq 0$ and $\sum_{i} a_i = 1$.


Now, applying this to $f(x) = -\log(x)$:


$$ -\log\left( \sum_{i} a_i x_i \right) \leq \sum_{i} a_i (-\log(x_i)) = - \sum_{i} a_i \log(x_i) = -\log\left( \prod_{i} x_i^{a_i} \right) $$

Taking the exponential of both sides:


$$ \sum_{i} a_i x_i \geq \prod_{i} x_i^{a_i} $$

This represents the generalized arithmetic-geometric mean inequality.

2.3. Example of AGM


  • Find the minimum of

$$f(x,y)=\frac{12}{x} + \frac{18}{y} + xy,\quad \forall x, y > 0 $$
  • and the values of $x, y$ at the minimum

Solution


$$ \begin{align*} \left(\frac{12}{x}\cdot \frac{18}{y}\cdot xy\right)^\frac{1}{3} &\leq \frac{1}{3} \left(\frac{12}{x}+\frac{18}{y}+xy\right)\\\\ \implies \; 3(12\cdot18)^\frac{1}{3} =18 &\leq \frac{12}{x} + \frac{18}{y}+xy\\\\\\ \text{minimum at}\;\; \frac{12}{x}=\frac{18}{y}=xy \; &\implies \; x=2, y=3\\ \end{align*} $$

2.4. Example of AGM


  • Find the maximum of

$$ f(x,y)=xy \cdot (72 - 3x - 4y) \quad \text{over} \quad x, y > 0 $$
  • and the values of $x, y$ at the maximum

Solution


$$ \begin{align*} (3x\cdot 4y \cdot (72 - 3x -4y))^\frac{1}{3} &\leq \frac{1}{3}(3x + 4y + 72 -3x -4y)\\\\ \qquad x \cdot y \cdot (72 - 3x -4y) &\leq \left(\frac{72}{3}\right)^3 \cdot \frac{1}{3} \cdot \frac{1}{4} = 1152\\\\ \text{maximum at}\;\; 3x = 4y = 72 -3x -4y \; &\implies \; x = 8, y=6 \end{align*} $$

2.5. Example of AGM


  • Find the minimum of
$$ f(x,y) = 4x + \frac{x}{y^2} + \frac{4y}{x} $$
  • and the values of $x, y$ at the minimum

Solution


$$ \begin{align*} \left(4x\cdot \frac{x}{y^2} \cdot \frac{2y}{x} \cdot \frac{2y}{x} \right)^\frac{1}{4}\; &\leq \; \frac{1}{4} \left(4x + \frac{x}{y^2} + \frac{2y}{x} + \frac{2y}{x} \right)\\\\ 4 \cdot 2 = 8 &\; \leq \; 4x + \frac{x}{y^2} + \frac{4y}{x}\\\\ \text{minimum at}\;\; 4x = \frac{x}{y^2} = \frac{2y}{x} \; &\implies \; x = \frac{1}{2}, y = \frac{1}{2} \end{align*} $$

2.6. Example of AGM


  1. Minimize
$$ f(x) = x^2 + \frac{1}{x^2} + 4x + \frac{4}{x} $$
  1. Verify your answer by plotting $f(x)$

Wrong Solution


$$ \begin{align*} \left(x^2\cdot\frac{1}{x^2}\cdot4x\cdot\frac{4}{x}\right)^\frac{1}{4}\; &\leq\;\frac{1}{4}\left(x^2+\frac{1}{x^2}+4x+\frac{4}{x}\right)\\ 8\; &\leq\; x^2+\frac{1}{x^2}+4x+\frac{4}{x}\\\\ \text{at}\quad x^2=\frac{1}{x^2}=4x&=\frac{4}{x}\;\implies\; x^2=\frac{1}{x^2}\;\implies\;x=\pm1\\ &\,\;\quad\qquad\quad\;\;\;4x=\frac{4}{x}\;\implies\;x=\pm1\\ &\,\;\quad\qquad\quad\;\;\;x^2=4x\;\implies\;x=0, 4\\\\ \nexists\;x\; &\text{s.t. holding an equality} \end{align*} $$

Correct Solution


$$ \begin{align*} \left(x^2\cdot\frac{1}{x^2}\cdot x\cdot x\cdot x\cdot x\cdot\frac{1}{x}\frac{1}{x}\frac{1}{x}\frac{1}{x}\right)^\frac{1}{10}\;&\leq\;\frac{1}{10}\left(x^2+\frac{1}{x^2}+4x+\frac{4}{x}\right)\\ \Rightarrow\; 10 \;&\leq\; x^2+\frac{1}{x^2}+4x+\frac{4}{x}\\\\ \text{Equality at}\; x^2=\frac{1}{x^2}=x=\frac{1}{x}\;&\implies\;x=1 \end{align*} $$
In [ ]:
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0.1, 3, 100)
y = x**2 + 1/(x**2) + 4*x + 4/x

plt.plot(x, y, 'k', linewidth = 2)
plt.xlabel('x')
plt.ylabel('f(x)')
plt.grid(alpha = 0.2)
plt.xlim([0, 2])
plt.show()

3. Geometric Mean – Harmonic Mean Inequality (GHM)

In [3]:
from IPython.display import YouTubeVideo
YouTubeVideo('CO0JK3JwSu4?si=KAPH3MbaRd5PIkYN&start=1343', width = "560", height = "315")
Out[3]:

Definition of harmonic mean


$$H \triangleq\left(\frac{\frac{1}{x_1}+\cdots+\frac{1}{x_n}}{n}\right)^{-1}$$
$$ \begin{align*} H &\leq G\\\\ \left(\frac{\frac{1}{x_1}+\cdots+\frac{1}{x_n}}{n}\right)^{-1} &\leq (x_1\cdots x_n)^\frac{1}{n} \end{align*} $$

Proof


$$ \begin{align*} \left(\frac{\frac{1}{x_1}+\cdots+\frac{1}{x_n}}{n}\right)^{-1} &\leq (x_1\cdots x_n)^\frac{1}{n}\\\\ \frac{1}{(x_1\cdots x_n)^\frac{1}{n}}=\left(\frac{1}{x_1}\cdots\frac{1}{x_n}\right)^\frac{1}{n}&\leq\frac{1}{n}\left(\frac{1}{x_1}+\cdots+\frac{1}{x_n}\right)\\\\ \implies \quad \frac{n}{\frac{1}{x_1}+\cdots+\frac{1}{x_n}}&\leq(x_1\cdots x_n)^\frac{1}{n} \end{align*} $$

3.1. Harmonic, Geometric, and Arithmetic Means (HGAM)


$$ \begin{align*} H\leq & G\leq A \qquad\text{equality when}\;x_1=\cdots=x_n\\\\\\ \left(\frac{\frac{1}{x_1}+\cdots+\frac{1}{x_n}}{n}\right)^{-1} & \leq (x_1\cdots x_n)^\frac{1}{n}\leq \frac{1}{n}(x_1+\cdots+ x_n) \end{align*} $$

3.2. Example of HGAM


Show

$$\left(\frac{1}{x_1}+\cdots+\frac{1}{x_n}\right) (x_1+\cdots+ x_n)\geq n^2$$

Solution


$$ \begin{align*} \implies \quad \frac{n}{\frac{1}{x_1}+\cdots+\frac{1}{x_n}}&\leq\frac{1}{n}(x_1+\cdots+ x_n)\\ n^2&\leq\left(\frac{1}{x_1}+\cdots+\frac{1}{x_n}\right)(x_1+\cdots+ x_n)\\\\ \text{equality when}&\;x_1=\cdots=x_n \end{align*} $$

3.3. Example of HGAM


  • Cut R into $n$ pieces $(R_1,R_2,\cdots,R_n)$
  • Then connect them in parallel
  • Find the maximum of $R_{eq}$

Solution


When two resistors are connected in parallel


$$R_{eq} = \frac{1}{\frac{1}{R_1} + \frac{1}{R_2}} = \frac{R_1 R_2}{R_1 + R_2}$$

If $n$ pieces of resistors are connected in parallel, the equvalent $R_{eq}$ is


$$ \begin{align*} R_{eq}=\frac{1}{\frac{1}{R_1}+\cdots+\frac{1}{R_n}}\\\\ R_1+R_2+\cdots+R_n=R \end{align*} $$

We know that


$$ \begin{align*} &\left(\frac{1}{R_1}+\frac{1}{R_2}+\cdots+\frac{1}{R_n}\right)(R_1+R_2+\cdots+R_n)\geq n^2\\\\ &\implies\quad\frac{R}{n^2}\geq R_{eq}\\\\ &\text{Equality at}\;R_1=R_2=\cdots =R_n \end{align*} $$

4. Cauchy’s Inequality

In [4]:
from IPython.display import YouTubeVideo
YouTubeVideo('CO0JK3JwSu4?si=rtBxfPaigl43fmGl&start=1858', width = "560", height = "315")
Out[4]:

$$ (x_1y_1 + \cdots + x_ny_n)^2\; \leq \; \left(x_1^2 + \cdots + x_n^2 \right)\left(y_1^2 + \cdots + y_n^2 \right)\\\\ $$

with equality iff $\exists \; a $ such that $x_i = ay_i, \; \forall i$


Proof


$$ \begin{align*} x\;=\;\begin{bmatrix} x_1\\x_2\\ \vdots \\ x_n\end{bmatrix} \qquad y\;=\;\begin{bmatrix} y_1\\y_2\\ \vdots \\ y_n\end{bmatrix} \end{align*} $$
$$ \begin{align*} \rVert x \lVert_2 = \sqrt{\sum x_i^2} = &\sqrt{\langle x, x\rangle} \, , \quad \text{2 norm and inner product} \end{align*} $$
$$ \begin{align*} \lvert \langle x, y \rangle \rvert &\leq \lVert x \rVert_2 \cdot \lVert y \rVert_2 \quad \text{(why ?)}\\\\ \implies \; \rvert (x_1y_1 + \cdots + x_ny_n) \lvert &\leq \sqrt{x_1^2 + \cdots + x_n^2}\;\sqrt{y_1^2 + \cdots + y_n^2} \end{align*} $$

With equalty when $x \parallel y$ or $\cos\theta = \pm 1$


4.1. Example of Cauchy’s Inequality


  • Find the largest and smallest values of

$$ f(x,y,z) = 2x + 3y + 6z $$
  • among the points $(x, y, z)$ with $x^2 + y^2 + z^2 = 1$
  • and $(x, y, z)$ at these two points

Solution


$$ \begin{align*} \quad\;\;(2x+3y+6z)^2 & \leq(2^2+3^2+6^2)\;(x^2+y^2+z^2)=49\\ \\ \implies \; \lvert 2x+3y+6z \rvert \;& \leq \; 7 \end{align*} $$



$$\text{Equality at} \quad \begin{bmatrix} 2\\3\\6\end{bmatrix}\; \parallel \; \begin{bmatrix} x\\y\\z\end{bmatrix}$$
$$ \begin{align*} 2x+3y+6z &=7 \\ 2(2t)+3(3t)+6(6t) &= 49t = 7 \\ t =\frac{1}{7}\; &\implies\; x_{\max}^*=\frac{1}{7}\;\begin{bmatrix} 2\\3\\6\end{bmatrix}\\\\ 2x+3y+6z &= -7\\ &\implies x_{\min}^* = -\frac{1}{7} \; \begin{bmatrix} 2\\3\\6\end{bmatrix} \end{align*} $$

4.2. Example of Cauchy’s Inequality


  • Find the smallest values of
$$ f(x,y) = x^2 + y^2\\ \text{s.t.}\;3x-y=20 $$

Solution


$$ \begin{align*} (3x-y)^2 \; & \leq \; (3^2+(-1)^2)\;(x^2+y^2)\\ 40=\frac{20^2}{10}\;&\leq\;x^2+y^2 \end{align*} $$



$$ \begin{align*} \text{Equality at} \quad \begin{bmatrix} 3\\-1\end{bmatrix}\; \parallel \; \begin{bmatrix} x\\y\end{bmatrix} \;\implies \;x=3t&=6\\ y=-t&=-2\\ t&=2 \end{align*} $$


4.3. Example of Cauchy’s Inequality


$$ \begin{align*} \min\quad&\frac{1}{x}+\frac{1}{y}+\frac{1}{z}\\ \text{s.t.}\quad &x+y+z=9\\ &x,y,z\geq0 \end{align*} $$

Solution


$$ \begin{align*} (1+1+1)^2\;&\leq\;(x+y+z)\left(\frac{1}{x}+\frac{1}{y}+\frac{1}{z}\right)\\ &=\left(\sqrt{x}^2+\sqrt{y}^2+\sqrt{z}^2 \right)\left(\left(\frac{1}{\sqrt{y}}\right)^2+\left(\frac{1}{\sqrt{y}}\right)^2+\left(\frac{1}{\sqrt{z}}\right)^2\right)\\ \implies\;1&\leq\frac{1}{x}+\frac{1}{y}+\frac{1}{z} \end{align*} $$
$$ \begin{align*} \text{Equality at} \quad \begin{bmatrix} x\\y\\z\end{bmatrix}\; \parallel \; \begin{bmatrix} 1/x\\1/y\\1/z\end{bmatrix} \;\implies\;&x^2=y^2=z^2=t^2\\ &x=y=z=3\\ \end{align*} $$

5. Young’s Inequality

In [5]:
from IPython.display import YouTubeVideo
YouTubeVideo('CO0JK3JwSu4?si=KSedRIr_l4UX6rHA&start=2451', width = "560", height = "315")
Out[5]:

$$ xy\leq\frac{x^p}{p}+\frac{y^q}{q} $$

where

$$ \begin{align*} \frac{1}{p}+\frac{1}{q} &=1\\ p,q &>1\\ x,y &>0 \end{align*} $$
Equality at $$x^p=y^q$$

Proof


$$ \begin{align*} x_1^{a_1}x_2^{a_2}&\leq a_1x_1+a_2x_2\;:\;\text{GAGM}\\\\ \end{align*} $$

Set

$$ \begin{align*} x_1=x^p \quad a_1=\frac{1}{p}\\ x_2=y^p \quad a_2=\frac{1}{q} \end{align*} $$

Then

$$$$$$ \begin{align*} x_1^\frac{1}{p}x_2^\frac{1}{q} &\leq\frac{1}{p}x_1+\frac{1}{q}x_2\\\\ \implies \quad x\cdot y &\leq\frac{1}{p}x^p+\frac{1}{q}y^q\\\\ \text{equal at}&\;x^p=y^q \end{align*} $$

Example

$$xy\leq\frac{x^2}{2}+\frac{y^2}{2}\;\;\text{(True?)}$$
In [ ]:
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')