**Markov Chain**

By Prof. Seungchul Lee

http://iai.postech.ac.kr/

Industrial AI Lab at POSTECH

http://iai.postech.ac.kr/

Industrial AI Lab at POSTECH

Table of Contents

- Most classifiers ignored the sequential aspects of data

- Consider a system which can occupy one of $N$ discrete states or categories $q_t \in \{S_1,S_2,\cdots,S_N\}$

- We are interested in stochastic systems, in which state evolution is random

- Any joint distribution can be factored into a series of conditional distributions

$$p(q_0,q_1,\cdots,q_T ) = p(q_0) \; p(q_1 \mid q_0) \; p( q_2 \mid q_1 q_0 ) \; p( q_3 \mid q_2 q_1 q_0 ) \cdots$$

$$p(q_0,q_1,\cdots,q_T ) = p(q_0) \; p(q_1 \mid q_0) \; p( q_2 \mid q_1 ) \; p( q_3 \mid q_2 ) \cdots$$

- (Assumption) for a Markov process, the next state depends only on the current state:

- More clearly

- Given current state, the past does not matter
- The state captures all relevant information from the history
- The state is a sufficient statistic of the future

- For a Markov state 𝑠 and successor state 𝑠′, the state transition probability is defined by

- State transition matrix $P$ defines transition probabilities from all states $s$ to all successor states $s'$.

Example: Markov chhain episodes

- sample episodes starting from $S_1$

In [1]:

```
import numpy as np
P = [[0, 0, 1],
[1/2, 1/2, 0],
[1/3, 2/3, 0]]
```

In [2]:

```
print(P[1][:])
```

In [3]:

```
s = np.random.choice(3, 1, p = P[1][:])[0]
print(s)
```

In [4]:

```
# sequential processes
# sequence generated by Markov chain
# S1 = 0, S2 = 1, S3 = 2
# starting from 0
x = 0
S = []
S.append(x)
for i in range(20):
x = np.random.choice(3, 1, p = P[x][:])[0]
S.append(x)
print(S)
```

It represents passive stochastic behavior

$\;\;$ 1) a finite set of $N$ states, $S = \{ S_1, \cdots , S_N \}$

$\;\;$ 2) a state transition probability, $P = \{ p_{ij} \}_{M \times M},\quad 1 \leq i,j \leq M$

$\;\;$ 3) an initial state probability distribution, $\pi = \{ \pi_i\}$

Example of Markov Chain

- Starting from $S_1$ = Class 1

In [5]:

```
P = [[0, 0.5, 0, 0, 0, 0.5, 0],
[0, 0, 0.8, 0, 0, 0, 0.2],
[0, 0, 0, 0.6, 0.4, 0, 0],
[0, 0, 0, 0, 0, 0, 1],
[0.2, 0.4, 0.4, 0, 0, 0, 0],
[0.1, 0, 0, 0, 0, 0.9, 0],
[0, 0, 0, 0, 0, 0, 1]]
# sequential processes
# sequence generated by Markov chain
# [C1 C2 C3 Pass Pub FB Sleep] = [0 1 2 3 4 5 6]
name = ['C1','C2','C3','Pass','Pub','FB','Sleep']
# starting from 0
x = 0
S = []
S.append(x)
for i in range(5):
x = np.random.choice(7, 1, p = P[x][:])[0]
S.append(x)
print(S)
```

In [6]:

```
episode = []
for i in S:
episode.append(name[i])
print(episode)
```

- (1-step transition probabilities) For a Markov chain on a finite state space, $S = \{ S_1, \cdots , S_N \}$, with transition probability matrix $P$ and initial distribution $\pi = \{ \pi_i^{0} \}$ (row vector) then the distribution of $X(1)$ is given by

- (2-step transition probabilities) For a Markov chain on a finite state space, $S = \{ S_1, \cdots , S_N \}$, with transition probability matrix $P$ and initial distribution $\pi = \{ \pi_i^{0} \}$ (row vector) then the distribution of $X(2)$ is given by

- (n-step transition probabilities) For a Markov chain on a finite state space, $S = \{ S_1, \cdots , S_N \}$, with transition probability matrix $P$ and initial distribution $\pi = \{ \pi_i^{0} \}$ (row vector) then the distribution of $X(n)$ is given by

- $P^n$: n-step transition probabilities

- Key recursion

In [7]:

```
# state probability distribution
P = [[0, 0, 1],
[1/2, 1/2, 0],
[1/3, 2/3, 0]]
u = [0, 1, 0]
P = np.asmatrix(P)
u = np.asmatrix(u)
for i in range(100):
u = u*P
print(u)
```

In [8]:

```
u = [0, 1, 0]
u = u*P**10
print(u)
```

- Steady-state behavior

- Does $p_{ij}(n)=P[X_n = j \mid X_0 = i] $ converge to some $\pi_j$ ?

- Take the limit as $n \rightarrow \infty$

- Need also $\sum_j \pi_j = 1$

- How to compute (https://brilliant.org/wiki/stationary-distributions/)
- Eigen-analysis
- Fixed-point iteration (we already did)

In [9]:

```
# eigenvalue = 1 and associated eigenvector
d, v = np.linalg.eig(P.T)
print(d)
```

In [10]:

```
v[:,2]/np.sum(v[:,2])
```

Out[10]:

In [11]:

```
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```