Fully Convolutional Networks for Segmentation
By Prof. Seungchul Lee
http://iai.postech.ac.kr/
Industrial AI Lab at POSTECH
http://iai.postech.ac.kr/
Industrial AI Lab at POSTECH
Table of Contents
0. Video LecturesĀ¶
InĀ [1]:
%%html
<center><iframe src="https://www.youtube.com/embed/8-PA11R3e9c?start=1915&rel=0"
width="560" height="315" frameborder="0" allowfullscreen></iframe></center>
InĀ [2]:
%%html
<center><iframe src="https://www.youtube.com/embed/4vyohdppEoY?rel=0"
width="560" height="315" frameborder="0" allowfullscreen></iframe></center>
1. SegmentationĀ¶
- Segmentation task is different from classification task because it requires predicting a class for each pixel of the input image, instead of only 1 class for the whole input.
- Classification needs to understand what is in the input (namely, the context).
- However, in order to predict what is in the input for each pixel, segmentation needs to recover not only what is in the input, but also where.
- Segment images into regions with different semantic categories. These semantic regions label and predict objects at the pixel level
2. Fully Convolutional Networks (FCN)Ā¶
- FCN is built only from locally connected layers, such as convolution, pooling and upsampling.
- Note that no dense layer is used in this kind of architecture.
- Network can work regardless of the original image size, without requiring any fixed number of units at any stage.
To obtain a segmentation map (output), segmentation networks usually have 2 parts
- Downsampling path: capture semantic/contextual information
- Upsampling path: recover spatial information
- The downsampling path is used to extract and interpret the context (what), while the upsampling path is used to enable precise localization (where).
- Furthermore, to fully recover the fine-grained spatial information lost in the pooling or downsampling layers, we often use skip connections.
- Given a position on the spatial dimension, the output of the channel dimension will be a category prediction of the pixel corresponding to the location.
InĀ [1]:
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
InĀ [2]:
train_imgs = np.load('./data_files/images_training.npy')
train_seg = np.load('./data_files/seg_training.npy')
test_imgs = np.load('./data_files/images_testing.npy')
n_train = train_imgs.shape[0]
n_test = test_imgs.shape[0]
print ("The number of training images : {}, shape : {}".format(n_train, train_imgs.shape))
print ("The number of segmented images : {}, shape : {}".format(n_train, train_seg.shape))
print ("The number of testing images : {}, shape : {}".format(n_test, test_imgs.shape))
InĀ [3]:
idx = np.random.randint(n_train)
plt.figure(figsize = (15,10))
plt.subplot(1,3,1)
plt.imshow(train_imgs[idx])
plt.axis('off')
plt.subplot(1,3,2)
plt.imshow(train_seg[idx][:,:,0])
plt.axis('off')
plt.subplot(1,3,3)
plt.imshow(train_seg[idx][:,:,1])
plt.axis('off')
plt.show()
3.2. From CAE to FCNĀ¶
- CAE
- FCN
- VGG16
- Skip connections to fully recover the fine-grained spatial information lost in the pooling or downsampling layers
InĀ [4]:
model_type = tf.keras.applications.vgg16
base_model = model_type.VGG16()
base_model.trainable = False
base_model.summary()
InĀ [5]:
map5 = base_model.layers[-5].output
# sixth convolution layer
conv6 = tf.keras.layers.Conv2D(filters = 4096,
kernel_size = (7,7),
padding = 'SAME',
activation = 'relu')(map5)
# 1x1 convolution layers
fcn4 = tf.keras.layers.Conv2D(filters = 4096,
kernel_size = (1,1),
padding = 'SAME',
activation = 'relu')(conv6)
fcn3 = tf.keras.layers.Conv2D(filters = 2,
kernel_size = (1,1),
padding = 'SAME',
activation = 'relu')(fcn4)
# Upsampling layers
fcn2 = tf.keras.layers.Conv2DTranspose(filters = 512,
kernel_size = (4,4),
strides = (2,2),
padding = 'SAME')(fcn3)
fcn1 = tf.keras.layers.Conv2DTranspose(filters = 256,
kernel_size = (4,4),
strides = (2,2),
padding = 'SAME')(fcn2 + base_model.layers[14].output)
output = tf.keras.layers.Conv2DTranspose(filters = 2,
kernel_size = (16,16),
strides = (8,8),
padding = 'SAME',
activation = 'softmax')(fcn1 + base_model.layers[10].output)
model = tf.keras.Model(inputs = base_model.inputs, outputs = output)
InĀ [6]:
model.summary()
4.3. TrainingĀ¶
InĀ [7]:
model.compile(optimizer = 'adam',
loss = 'categorical_crossentropy',
metrics = 'accuracy')
InĀ [8]:
model.fit(train_imgs, train_seg, batch_size = 5, epochs = 5)
Out[8]:
4.4. TestingĀ¶
InĀ [9]:
test_x = test_imgs[[1]]
test_seg = model.predict(test_x)
seg_mask = (test_seg[:,:,:,1] > 0.5).reshape(224, 224, 1).astype(float)
plt.figure(figsize = (14,14))
plt.subplot(2,2,1)
plt.imshow(test_x[0])
plt.axis('off')
plt.subplot(2,2,2)
plt.imshow(seg_mask, cmap = 'Blues')
plt.axis('off')
plt.subplot(2,2,3)
plt.imshow(test_x[0])
plt.imshow(seg_mask, cmap = 'Blues', alpha = 0.5)
plt.axis('off')
plt.show()
InĀ [10]:
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')