Super-resolution and Deblurring
By Prof. Seungchul Lee
http://iailab.kaist.ac.kr/
Industrial AI Lab at KAIST
http://iailab.kaist.ac.kr/
Industrial AI Lab at KAIST
Table of Contents
1. Image Restoration¶
- Image restoration tries to recover original image from degraded one with prior knowledge of degradation process.
The sources of corruption in digital images arise during image acquisition (digitization) and transmission.
- Imaging sensors can be affected by ambient conditions.
- Interference can be added to an image during transmission.
2. Inverse Problem¶
- The reconstruction is the inverse of the acquisition.
- Inverse problems involve modeling of degradation and applying the inverse process in order to recover the original image from inadequate observations.
- The observations contain incomplete information about the target parameter or data due to physical limitations of the measurement devices.
- Consequently, solutions to inverse problems are non-unique.
3. Image Super-resolution¶
- Restore High Resolution (HR) image from Low Resolution (LR) image
- There are numerous learning-based SR approaches.
3.1. HR and LR Images¶
Download data from here
In [1]:
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
In [2]:
from google.colab import drive
drive.mount('/content/drive')
In [3]:
train_lr = np.load('/content/drive/MyDrive/DL_Colab/DL_data/SR_train_lr.npy')
train_hr = np.load('/content/drive/MyDrive/DL_Colab/DL_data/SR_train_hr.npy')
test_lr = np.load('/content/drive/MyDrive/DL_Colab/DL_data/SR_test_lr.npy')
n_train = train_lr.shape[0]
n_test = test_lr.shape[0]
print ("The number of training LR images : {}, shape : {}".format(n_train, train_lr.shape))
print ("The number of training HR images : {}, shape : {}".format(n_train, train_hr.shape))
print ("The number of testing LR images : {}, shape : {}".format(n_test, test_lr.shape))
In [4]:
idx = np.random.randint(n_train)
plt.figure(figsize = (8, 6))
plt.subplot(1,2,1)
plt.imshow(train_lr[idx][:,:,0], 'gray')
plt.title('Low-resolution image')
plt.axis('off')
plt.subplot(1,2,2)
plt.imshow(train_hr[idx][:,:,0], 'gray')
plt.title('High-resolution image')
plt.axis('off')
plt.show()
3.2. Build a FCN Model¶
In [5]:
inputs = tf.keras.Input(shape = (112, 112, 1))
# 3x3 convolutional layer
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(inputs)
# first residual block
x_skip = x
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Add()([x_skip, x])
# second residual block
x_skip = x
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Add()([x_skip, x])
# third residual block
x_skip = x
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Add()([x_skip, x])
# upsampling layer
x = tf.keras.layers.Conv2DTranspose(filters = 16,
kernel_size = (4,4),
strides = (2,2),
padding = 'SAME',
activation = 'relu')(x)
# 3x3 convolutional layer
outputs = tf.keras.layers.Conv2D(filters = 1,
kernel_size = (3,3),
padding = 'SAME',
activation = 'sigmoid')(x)
model = tf.keras.Model(inputs, outputs)
3.3. Training¶
In [6]:
model.compile(optimizer = 'adam',
loss = 'mean_absolute_error',
metrics = ['mean_squared_error'])
In [7]:
model.fit(train_lr, train_hr, batch_size = 16, epochs = 30)
Out[7]:
3.4. Testing¶
In [8]:
test_x = test_lr[[3]]
test_sr = model.predict(test_x)
plt.figure(figsize = (8, 6))
plt.subplot(1,2,1)
plt.imshow(test_x[0][:,:,0], 'gray')
plt.title('Low-resolution image')
plt.axis('off')
plt.subplot(1,2,2)
plt.imshow(test_sr[0][:,:,0], 'gray')
plt.title('Super-resolved image')
plt.axis('off')
plt.show()
In [9]:
train_blur = np.load('/content/drive/MyDrive/DL_Colab/DL_data/deblurring_train_blur.npy')
train_deblur = np.load('/content/drive/MyDrive/DL_Colab/DL_data/deblurring_train_deblur.npy')
test_blur = np.load('/content/drive/MyDrive/DL_Colab/DL_data/deblurring_test_blur.npy')
n_train = train_blur.shape[0]
n_test = test_blur.shape[0]
print ("The number of training blur images : {}, shape : {}".format(n_train, train_blur.shape))
print ("The number of training deblur images : {}, shape : {}".format(n_train, train_deblur.shape))
print ("The number of testing blur images : {}, shape : {}".format(n_test, test_blur.shape))
In [10]:
idx = np.random.randint(n_train)
plt.figure(figsize = (8, 6))
plt.subplot(1,2,1)
plt.imshow(train_blur[idx][:,:,0], 'gray')
plt.title('Blurred image')
plt.axis('off')
plt.subplot(1,2,2)
plt.imshow(train_deblur[idx][:,:,0], 'gray')
plt.title('Deblurred image')
plt.axis('off')
plt.show()
4.2. Build a FCN Model¶
In [11]:
inputs = tf.keras.Input(shape = (224, 224, 1))
# 3x3 convolutional layer
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(inputs)
# first residual block
x_skip = x
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Add()([x_skip, x])
# second residual block
x_skip = x
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Add()([x_skip, x])
# third residual block
x_skip = x
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Conv2D(filters = 16,
kernel_size = (3,3),
padding = 'SAME',
activation = 'relu')(x)
x = tf.keras.layers.Add()([x_skip, x])
# 3x3 convolutional layer
outputs = tf.keras.layers.Conv2D(filters = 1,
kernel_size = (3,3),
padding = 'SAME',
activation = 'sigmoid')(x)
model = tf.keras.Model(inputs, outputs)
4.3. Training¶
In [12]:
model.compile(optimizer = 'adam',
loss ='mean_absolute_error',
metrics = ['mean_squared_error'])
In [13]:
model.fit(train_blur, train_deblur, batch_size = 16, epochs = 30)
Out[13]:
4.4. Testing¶
In [14]:
test_x = test_blur[[1]]
test_deblur = model.predict(test_x)
plt.figure(figsize = (8, 6))
plt.subplot(1,2,1)
plt.imshow(test_x[0][:,:,0], 'gray')
plt.title('Blurred image')
plt.axis('off')
plt.subplot(1,2,2)
plt.imshow(test_deblur[0][:,:,0], 'gray')
plt.title('Deblurred image')
plt.axis('off')
plt.show()
In [15]:
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')