Linear Transformation and Eigen Analysis

By Prof. Seungchul Lee
iSystems Design Lab
http://isystems.unist.ac.kr/
UNIST

Table of Contents

1. Matrix and Transformation

Vector

$$ \vec x = \begin{bmatrix} x_{1}\\ x_{2}\\ x_{3} \end{bmatrix} $$



Matrix and Transformation

$$ M= \begin{bmatrix} m_{11} & m_{12} & m_{13}\\ m_{21} & m_{22} & m_{23}\\ m_{31} & m_{32} & m_{33}\\ \end{bmatrix} $$


$$\begin{array}\ \vec y& = &M \vec x\\ \begin{bmatrix}\space \\ \space \\ \space \end{bmatrix} & = &\begin{bmatrix} & & \\ & & \\ & &\end{bmatrix}\begin{bmatrix} \space \\ \space \\ \space \end{bmatrix} \end{array} $$


$$\begin{array}\ \text{Given} & & \text{Interpret}\\ \text{Transformation} & \longrightarrow & \text{matrix}\\ \text{matrix} & \longrightarrow & \text{Transformation}\\ \end{array} $$


$$\begin{array}{c}\ \vec x\\ \text{input} \end{array} \begin{array}{c}\ \quad \text{transformation}\\ \implies \end{array} \quad \begin{array}{l} \vec y\\ \text{output} \end{array}$$


$$\text{transformation} =\text{rotate + stretch/compress}$$

1.1. Rotation

Rotation : $R(\theta)$



$$ \vec y = R(\theta) \vec x$$

Find matrix $R(\theta)$

$$ \begin{bmatrix} \cos(\theta)\\ \sin(\theta) \end{bmatrix}= R(\theta) \begin{bmatrix} 1\\ 0 \end{bmatrix}\\ $$
$$ \begin{bmatrix} -\sin(\theta)\\ \cos(\theta) \end{bmatrix}= R(\theta) \begin{bmatrix} 0\\ 1 \end{bmatrix} $$
$$\begin{array}\\ \Longrightarrow &\begin{bmatrix} \cos(\theta)& -\sin(\theta)\\ \sin(\theta)& \cos(\theta) \end{bmatrix}& =& R(\theta) \begin{bmatrix} 1& 0\\ 0& 1 \end{bmatrix}& =& R(\theta)\\ \\ \\ &\begin{array}\\ M\vec{x}_1 = \vec{y}_1\\ M\vec{x}_2 = \vec{y}_2\\ \end{array}& =& M \begin{bmatrix} \vec{x}_1 & \vec{x}_2 \end{bmatrix}& =& \begin{bmatrix} \vec{y}_1 & \vec{y}_2 \end{bmatrix} \end{array}$$

1.2. Stretch/Compress

Stretch/Compress (keep the direction)



$$\begin{array}\\ \vec y = &k\vec x\\ & \uparrow\\ & \text{scalar (not matrix)}\\ \\ \vec y = &k I \vec x & \text{where } I = \text{ Identity martix}\\ \\ \vec y = &\begin{bmatrix}k&0\\0&k\end{bmatrix}\vec x \end{array}$$

Example

T: stretch $a$ along $\hat x$-direction & stretch $b$ along $\hat y$-direction

compute the corresponding matrix $A$

$$\hat y = A \hat x$$



$$\begin{array}\\ \begin{bmatrix}ax_1\\ bx_2\end{bmatrix}& = A\begin{bmatrix}x_1\\ x_2\end{bmatrix} \Longrightarrow A = \,?\\ & = \begin{bmatrix}a & 0\\ 0 & b\end{bmatrix}\begin{bmatrix}x_1\\ x_2\end{bmatrix} \end{array}$$

$$\begin{array}\\ A\begin{bmatrix}1\\0\end{bmatrix} & = \begin{bmatrix}a\\0\end{bmatrix} \\ A\begin{bmatrix}0\\1\end{bmatrix} & = \begin{bmatrix}0\\b\end{bmatrix} \\ A\begin{bmatrix}1 & 0\\ 0 &1\end{bmatrix} & = A = \begin{bmatrix}a & 0\\0 & b\end{bmatrix} \\ \end{array}$$

More importantly, by looking at $A = \begin{bmatrix}a & 0\\0 & b\end{bmatrix}$, can you think of a transformation T?

Decomposition

T = rotate + stretch

  1. rotate $\theta$, then
  2. stretch



1.3. Projection

P: Projection onto $\hat x$ - axis

$$\begin{array}{c}\\ & P & \\ \begin{bmatrix}x_1\\x_2\end{bmatrix} & \implies & \begin{bmatrix}x_1\\ 0\end{bmatrix}\\ \vec x & & \vec y \end{array}$$



$$\vec y = P\vec x = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}\begin{bmatrix} x_1 \\ x_2 \end{bmatrix} = \begin{bmatrix} x_1 \\ 0 \end{bmatrix}$$



$$ \begin{array}\\ P \begin{bmatrix} 1 \\ 0 \end{bmatrix} & = \begin{bmatrix} 1 \\ 0 \end{bmatrix}\\ P \begin{bmatrix} 0 \\ 1 \end{bmatrix} & = \begin{bmatrix} 0 \\ 0 \end{bmatrix}\\ P \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} & = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix} \end{array} $$

1.4. Multiple Transformations

$T_1$ : transformation 1 : $M_1$

$T_2$ : transformation 2 : $M_2$

$T$ : Do transforamtion 1, followed by transformation 2

$$ \therefore \; M = M_2 M_1$$

$$ \begin{array}{c}\\ &T_1&&T_2\\ \vec x & \longrightarrow & \vec y & \longrightarrow & \vec z\\ \end{array} $$


$$ \begin{array}\\ \vec y & = M_1\vec x\\ \vec z & = M_2\vec y & = M_2M_1\vec x\\ & & = M \vec x \end{array} $$

Example

P: Projection onto vector = $\begin{bmatrix} \cos \theta\\ \sin \theta \end{bmatrix}$



$$\begin{array}\\ P\begin{bmatrix} 1\\ 0\end{bmatrix} & = \begin{bmatrix}\cos^2 \theta \\ \cos \theta \sin \theta\end{bmatrix}\\ P\begin{bmatrix} 0\\ 1\end{bmatrix} & = \begin{bmatrix} \sin \theta \cos \theta \\ \sin^2 \theta\end{bmatrix}\\ P\begin{bmatrix}1 & 0 \\ 0 & 1\end{bmatrix} & = \begin{bmatrix}\cos^2 \theta & \sin \theta \cos \theta\\ \cos \theta \sin \theta & \sin^2 \theta\end{bmatrix} \end{array}$$

Another way to find this projection matrix


$$\begin{array}\\ &R(-\theta)& &\begin{bmatrix} 1 & 0 \\ 0 & 0\end{bmatrix} & & R(\theta)\\ \vec x & \implies & \vec x' & \implies & \vec x'' & \implies & \vec y\\ \end{array}$$$$\begin{array}\\ \vec y & = R(\theta) \begin{bmatrix} 1 & 0 \\ 0 & 0\end{bmatrix} R(-\theta) \vec x\\ & = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos \theta \end{bmatrix} \begin{bmatrix} 1 & 0 \\ 0 & 0\end{bmatrix} \begin{bmatrix} \cos\theta & \sin\theta \\ -\sin\theta & \cos \theta \end{bmatrix} \\ & =\begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos \theta \end{bmatrix} \begin{bmatrix} \cos\theta & \sin\theta \\ 0 & 0 \end{bmatrix} \\ & = \begin{bmatrix} \cos^2 \theta & \cos \theta \sin \theta \\ \sin\theta \cos \theta & \sin^2 \theta \end{bmatrix}\\ \end{array}$$

2. Linear Transformation

  • Superposition


$$T(x_1+x_2) = T(x_1)+T(x_2)$$
  • Homogeniety



$$T(kx) = kT(x)$$

Linear vs. Non-linear

$$\begin{array}{c}\\ \text{linear}& & \text{non-linear}\\ f(x) = 0 & & f(x) = x + c\\ f(x) = kx & & f(x) = x^2\\ f(x(t)) = \frac{dx(t)}{dt} & & f(x) = \sin x\\ f(x(t)) = \int_{a}^{b} x(t)dt & & \end{array}$$

Linear Transformation

If $\vec {v}_1$ and $\vec {v}_2$ are basis, and we know $T(\vec {v}_1) = \vec {w}_1$ and $T(\vec {v}_2) = \vec {w}_2$

Then, for any $\vec x$

$$\begin{array}{l} \vec x & = a_1\vec v_1 + a_2\vec v_2 & (a_1 \;\text{and } a_2 \;\text{unique})\\ \\ T(\vec x) & = T(a_1\vec v_1 + a_2\vec v_2) \\ & = a_1T(\vec v_1) + a_2T(\vec v_2)\\ & = a_1\vec {\omega}_1 + a_2\vec {\omega}_2\\ \\ \implies &\text{T: linear} \end{array}$$

3. Eigenvalue and Eigenvector



$$ A \vec v = \lambda \vec v$$



$$ \begin{array}\\ \lambda & = &\begin{cases} \text{positive}\\ 0\\ \text{negative} \end{cases}\\ \lambda & : & \text{stretched vector}\\ &&\text{(same direction with } \vec x)\\ A & : &\text{transformed vector}\\ &&(\text{generally rotate + stretch}) \end{array}$$

$$A \vec x \text{ parallel to } \vec x$$

Example 1

$A = \begin{bmatrix} 1 & 0 \\ 0 & 0 \end{bmatrix}$ : projection onto $\hat x$- axis

Find eigenvalues and eigenvectors.



$$ \vec y = \begin{bmatrix} 0 \\ 0 \end{bmatrix} = A \vec x = 0 \cdot \vec x$$ $$\lambda_1 = 0 \space \;\text{and}\; \space \vec {v}_1 = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$$


$$ \vec y = \begin{bmatrix} 1 \\ 0 \end{bmatrix} = A \vec x = 1 \cdot \vec x$$ $$\lambda_2 = 1 \space \;\text{and}\; \space \vec {v}_2 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$$
In [1]:
import numpy as np

A = np.array([[1, 0],
              [0, 0]])
D, V = np.linalg.eig(A)
print('D :', D)
print('V :', V)
D : [ 1.  0.]
V : [[ 1.  0.]
 [ 0.  1.]]

Example 2

$\begin{array}\\ A = \begin{bmatrix} 2 & 0 \\ 0 & 1 \end{bmatrix} : & \text{stretch by 2 along } \vec x \text{- axis}\\ &\text{stretch by 1 along } \vec y \text{- axis} \end{array}$

Find eigenvalues and eigenvectors.



$$\lambda_1 = 2 \;\text{and}\; \vec {v}_1 = \begin{bmatrix} 1 \\ 0 \end{bmatrix} $$


$$\lambda_2 = 1\; \text{and} \;\vec {v}_2 = \begin{bmatrix} 0 \\ 1 \end{bmatrix} $$
In [2]:
A = np.array([[2, 0],
              [0, 1]])
D, V = np.linalg.eig(A)

idx = np.argsort(-D)
D = D[idx]
V = V[idx]

print('D :', D)
print('V :', V)
D : [ 2.  1.]
V : [[ 1.  0.]
 [ 0.  1.]]

Example 3

Projection onto the plane. Find eigenvalues and eigenvectors.



For any $\vec x$ in the plane, $P\vec x = \vec x \Rightarrow \lambda = 1$

For any $\vec x$ perpendicular to the plane, $P\vec x = \vec 0 \Rightarrow \lambda = 0$

Example 4

  • What kind of a linear transformation?



$$\begin{bmatrix} x_2\\ x_1 \end{bmatrix} = \begin{bmatrix} 0 & 1\\ 1 & 0 \end{bmatrix} \begin{bmatrix} x_1\\ x_2 \end{bmatrix}$$



  • Eigenvalues and eigenvectors?

    • can $\vec x$ be an eigenvector?



$$A\vec x = \vec x, \quad \lambda = 1$$

$$A\vec x = -\vec x, \quad \lambda = -1$$
  • Side note : Matrix $A$ can be seen as a multiple transformations

    $$A = R(45) M R(-45)$$

    $$\begin{array}\\ R(45) & = \begin{bmatrix} \cos \frac{\pi}{4} & -\sin \frac{\pi}{4} \\ \sin\frac{\pi}{4} & \cos \frac{\pi}{4} \end{bmatrix} = \frac{1}{\sqrt{2}}\begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}\\ M & : \text{ mirror along } \hat x \text{- axis}, \begin{bmatrix} 1 & 1 \\ 0 & -1 \end{bmatrix}\\ A & = \left(\frac{1}{\sqrt 2}\right)^2\begin{bmatrix} 1 & -1 \\ 1 & 1 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 0 & -1 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ -1 & 1 \end{bmatrix} \end{array}$$

Example 5

$$ \begin{array}\\ A = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix} \end{array}$$
$$ \begin{array}\\ A = R\left(\frac{\pi}{2}\right) = R(90^{\circ}) = \begin{bmatrix}\cos \frac{\pi}{2} & -\sin \frac{\pi}{2}\\ \sin \frac{\pi}{2} & \cos \frac{\pi}{2}\end{bmatrix} \end{array} $$



  • Side note: Multiple transformations



$$A = \begin{bmatrix} -1 & 0 \\ 0 & 1 \end{bmatrix}\begin{bmatrix} 0 & 1 \\ 1 & 0 \end{bmatrix} = \begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}$$
  • Eigenvalues: complex number



$$\begin{array}\\ \Rightarrow & \text{det}(A - \lambda I) & = 0\\ & \left | \begin{array}\\ -\lambda & -1\\ 1 & -\lambda \end{array} \right | & = \lambda ^2 + 1 = 0\\ & & \therefore \space \lambda = \pm i \end{array}$$



  • What is the physical meaning?

How to Compute Eigenvalue & Eigenvector



$$ A \vec{v} = \lambda \vec v = \lambda I \vec v$$

$$ \begin{array} \implies & A\vec v - \lambda I \vec v = (A - \lambda I)\vec v = 0\\ \\ \implies & A - \lambda I = 0 \text{ or }\\ &\vec v = 0 \text{ or } \\ & (A - \lambda I)^{-1} \text{ does not exist }\\ \\ \implies & \text{det}(A - \lambda I) = 0 \end{array} $$
In [3]:
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')