XAI: eXplainable AI
Table of Contents
1. Black Box AI¶
- Black box AI: AI produces insights based on a data set, but the end-user doesn’t know how
- Many machine learning and deep learning models share ‘black box’ problem
- AI does not provide reasons behind the decision or prediction it makes
- The reliability of AI models may be questioned
- XAI which humans can understand the decisions or predictions made by the AI
Why XAI?
- XAI can be used to increase the interpretability of AI by enabling description of the expected outcome and potential bias of the model
- Depending on the AI performance, XAI results can be used in various ways:
- AI performance < Human performance
- XAI suggests improvement directions for AI models
- AI performance ≈ Human performance
- XAI identifies the principles behind AI model learning
- AI performance > Human performance
- XAI enables acquiring new knowledge from AI
- AI performance < Human performance
Model-Specific XAI
- Model-Specific XAI: only applicable to specific algorithms that provides explanations by using the intrinsic structure of the model
- Examples: Class Activation Mapping (CAM) & Gradient-CAM for Convolution Neural Network (CNN) models
Model-Agnostic XAI
- Model Agnostic XAI: applicable to any machine learning algorithms and work on the black box model
- Obtain explanations by perturbing and mutating the input data and obtaining sensitivity of the performance of theses mutations with respect to the original data performance
- Examples:
- SHapley Additive exPlanations (SHAP)
- Local Interpretable Model-agnostic Explanations (LIME)
2. SHapley Additive exPlanations (SHAP)¶
- SHAP: a game theoretic approach to explain the output of any machine learning model
- Compute feature importance on each predicted value using Shapley value
- According to the SHAP value, the contribution of each feature can be expressed as the degree of change in the overall performance when the contribution of that feature is excluded
- Unlike general permutation method, SHAP calculates the model influence by considering the dependencies between features
from google.colab import drive
drive.mount('/content/drive')
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import tensorflow as tf
2.1. Build and Predict Machine Learning and Deep Learning Models¶
Data Description
- Injection molding dataset consisting of 36 mold shapes
- This dataset consists of 5 process features (control parameters), 32 mold shape features, and the weight of the molded part (ground truth).
dataset = pd.read_excel('/content/drive/MyDrive/linc/data_files/train_dataset.xlsx',
index_col = 0,
engine = 'openpyxl')
dataset
X = dataset.drop('Weight (g)', axis = 1)
y = dataset.loc[:, 'Weight (g)']
X = (X - X.min()) / (X.max() - X.min())
- train/test split
from sklearn.model_selection import train_test_split
train_x, test_x, train_y, test_y = train_test_split(X, y, test_size = 0.30, random_state = 42)
print('train_x: {}, train_y: {}'.format(train_x.shape, train_y.shape))
print('test_x: {}, test_y: {}'.format(test_x.shape, test_y.shape))
from sklearn.ensemble import RandomForestRegressor
Randomforest train
rf = RandomForestRegressor(random_state = 1)
rf.fit(train_x, train_y)
Prediction for testset
pred_y = rf.predict(test_x)
fig, ax = plt.subplots(figsize=(10, 10))
ax.scatter(test_y, pred_y, edgecolors=(0, 0, 0))
ax.plot([y.min(), y.max()], [y.min(), y.max()], "r--", lw=4)
ax.set_xlabel("Measured")
ax.set_ylabel("Predicted")
plt.show()
2.2.2. SHAP¶
SHAP Implementation
!pip install shap
import shap
explainer = shap.TreeExplainer(rf)
shap_values = explainer.shap_values(test_x)
Average of Absolute SHAP Values of Entire Test Data (Feature Importance)
shap.summary_plot(shap_values, test_x, plot_type = "bar")
tf.random.set_seed(42)
model = tf.keras.models.Sequential([
tf.keras.layers.Input(shape = (train_x.shape[1],)),
tf.keras.layers.Dense(units = 10, activation = 'relu'),
tf.keras.layers.Dense(units = 10, activation = 'relu'),
tf.keras.layers.Dense(units = 10, activation = 'relu'),
tf.keras.layers.Dense(units = 10, activation = 'relu'),
tf.keras.layers.Dense(units = 10, activation = 'relu'),
tf.keras.layers.Dense(units = 1, activation = None)
])
Model define and train
model.summary()
model.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.001),
loss = 'mse')
loss = model.fit(train_x, train_y, epochs = 50)
Prediction for testset
pred_y = model.predict(test_x)
fig, ax = plt.subplots(figsize = (6, 6))
ax.scatter(test_y, pred_y, edgecolors = (0, 0, 0))
ax.plot([test_y.min(), test_y.max()], [test_y.min(), test_y.max()], "r--", lw = 4)
ax.set_xlabel("Measured")
ax.set_ylabel("Predicted")
plt.show()
2.3.2. SHAP¶
SHAP Implementation
import shap
explainer_shap = shap.DeepExplainer(model = model,
data = train_x.to_numpy())
shap_values = explainer_shap.shap_values(test_x.to_numpy())
Average of Absolute SHAP Values of Entire Test Data (Feature Importance)
shap.summary_plot(shap_values, test_x.to_numpy(),
feature_names = X.columns)
3. Class Activation Maps (CAM)¶
Attention
Visualizing and Understanding Convolutional Networks
Source
- Source paper from http://cnnlocalization.csail.mit.edu/
- Source code from https://github.com/metalbubble/CAM
3.1. CNN with a Fully Connected Layer¶
The conventional CNN can be conceptually divided into two parts. One part is feature extraction and the other is classification. In the feature extraction process, convolution is used to extract the features of the input data so that the classification can be performed well. The classification process classifies which group each input data belongs to by using the extracted features from the input data.
When we visually identify images, we do not look at the whole image; instead, we intuitively focus on the most important parts of the image. CNN learning is similar to the way humans focus. When its weights are optimized, the more important parts are given higher weights. But generally, we are not able to recognize this because the generic CNN goes through a fully connected layer and makes the features extracted by the convolution layer more abstract.
Issues on CNN (or Deep Learning)
Deep learning performs well comparing with any other existing algorithms
But works as a black box
- A classification result is simply returned without knowing how the classification results are derived → little interpretability
When we visually identify images, we do not look at the whole image
Instead, we intuitively focus on the most important parts of the image
When CNN weights are optimized, the more important parts are given higher weights
Class activation map (CAM)
- We can determine which parts of the image the model is focusing on, based on the learned weights
- Highlighting the importance of the image region to the prediction
3.2. CAM: CNN with a Global Average Pooling¶
- shed light on how it explicitly enables the convolutional neural network to have remarkable localization ability
- the heatmap is the class activation map, highlighting the importance of the image region to the prediction
The deep learning model is a black box model. When input data is received, a classification result of 1 or 0 is simply returned for the binary classification problem, without knowing how the classification results are derived. Meanwhile, The class activation map (CAM) is capable of interpreting the results of the classification. We can determine which parts of the image the model is focusing on. Through an analysis of which part of the image the model is focusing on, we are able to interpret which part of the image is considered important.
The class activation map (CAM) is a modified convolution layer. It directly highlights the important parts of the spatial grid of an image. As a result, we can see the emphasized parts of the model. The below figure describes the procedure for class activation mapping.
The feature maps of the last convolution layer can be interpreted as a collection of visual spatial locations focused on by the model. The CAM can be obtained by taking a linear sum of the features. They all have different weights and thus can obtain spatial locations according to various input images through a linear combination. For a given image, $f_k(x,y)$ represents the feature map of unit $k$ in the last convolution layer at spatial location $(x,y)$. For a given class $c$, the class score, $S_c$, is expressed as the following equation.
$$S_c = \sum_k \omega_k^c \sum_{x,y} f_k(x,y)= \sum_{x,y} \sum_k \omega_k^c \; f_k(x,y)$$
where $\omega_k^c$ the weight corresponding to class $c$ for unit $k$. The class activation map for class $c$ is denoted as $M_c$.
$$M_c(x,y) = \sum_k \omega_k^c \; f_k(x,y)$$
$M_c$ directly indicates the importance of the feature map at a spatial grid $(x,y)$ of the class $c$. Finally the output of the softmax for class $c$ is,
$$P_c = \frac{\exp\left(S_c\right)}{\sum_c \exp\left(S_c\right)}$$
In case of the CNN, the size of the feature map is reduced by the pooling layer. By simple up-sampling, it is possible to identify attention image regions for each label.
Limitations of Class Activation Maps (CAM)
Requires a Global Average Pooling layer
Unable to visualize feature maps from different layers (other than the last)
3.3. CAM with NEU¶
Download NEU steel surface defects images and labels
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import cv2
from google.colab import drive
drive.mount('/content/drive')
# Change file paths if necessary
train_x = np.load('/content/drive/MyDrive/DL_Colab/DL_data/NEU_train_imgs.npy')
train_y = np.load('/content/drive/MyDrive/DL_Colab/DL_data/NEU_train_labels.npy')
test_x = np.load('/content/drive/MyDrive/DL_Colab/DL_data/NEU_test_imgs.npy')
test_y = np.load('/content/drive/MyDrive/DL_Colab/DL_data/NEU_test_labels.npy')
n_train = train_x.shape[0]
n_test = test_x.shape[0]
print ("The number of training images : {}, shape : {}".format(n_train, train_x.shape))
print ("The number of testing images : {}, shape : {}".format(n_test, test_x.shape))
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(filters = 32,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (200, 200, 1)),
tf.keras.layers.MaxPool2D((2,2)),
tf.keras.layers.Conv2D(filters = 64,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (100, 100, 32)),
tf.keras.layers.MaxPool2D((2,2)),
tf.keras.layers.Conv2D(filters = 64,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (50, 50, 64)),
tf.keras.layers.MaxPool2D((2,2)),
tf.keras.layers.Conv2D(filters = 64,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (25, 25, 64)),
tf.keras.layers.GlobalAveragePooling2D(),
tf.keras.layers.Dense(6, activation = 'softmax')
])
model.summary()
model.compile(optimizer = 'adam',
loss = 'sparse_categorical_crossentropy',
metrics = 'accuracy')
model.fit(train_x, train_y, epochs = 15)
# accuracy test
test_loss, test_acc = model.evaluate(test_x, test_y)
# get max pooling layer and fully connected layer
conv_layer = model.get_layer(index = 6)
fc_layer = model.layers[8].get_weights()[0]
# Class activation map
my_map = tf.matmul(conv_layer.output, fc_layer)
CAM = tf.keras.Model(inputs = model.inputs, outputs = my_map)
test_idx = [7]
test_image = test_x[test_idx]
pred = np.argmax(model.predict(test_image), axis = 1)
predCAM = CAM.predict(test_image)
attention = predCAM[:,:,:,pred]
attention = np.abs(np.reshape(attention,(25,25)))
resized_attention = cv2.resize(attention,
(200*5, 200*5),
interpolation = cv2.INTER_CUBIC)
resized_test_x = cv2.resize(test_image.reshape(200,200),
(200*5, 200*5),
interpolation = cv2.INTER_CUBIC)
plt.figure(figsize = (6, 9))
plt.subplot(3,2,1)
plt.imshow(test_x[test_idx].reshape(200,200), 'gray')
plt.axis('off')
plt.subplot(3,2,2)
plt.imshow(attention)
plt.axis('off')
plt.subplot(3,2,3)
plt.imshow(resized_test_x, 'gray')
plt.axis('off')
plt.subplot(3,2,4)
plt.imshow(resized_attention, 'jet', alpha = 0.5)
plt.axis('off')
plt.subplot(3,2,6)
plt.imshow(resized_test_x, 'gray')
plt.imshow(resized_attention, 'jet', alpha = 0.5)
plt.axis('off')
plt.show()
3.4. Grad-CAM: Gradient-weighted Class Activation Maps¶
Does not require a particular architecture (as long as we can differentiate)
Uses gradients to determine weighting of each feature map
CAM
$$\sum_k \omega^c_k f_k$$
Grad-CAM
$$ReLU \left(\sum_{k} \alpha^c_k f_k \right) \quad \text{where} \quad \alpha^c_k = \frac{1}{Z} \sum_{x,y} \frac{\partial z_c}{\partial f_k(x,y)}$$
model = tf.keras.models.Sequential([
tf.keras.layers.Conv2D(filters = 32,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (200, 200, 1)),
tf.keras.layers.MaxPool2D((2,2)),
tf.keras.layers.Conv2D(filters = 64,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (100, 100, 32)),
tf.keras.layers.MaxPool2D((2,2)),
tf.keras.layers.Conv2D(filters = 64,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (50, 50, 64)),
tf.keras.layers.MaxPool2D((2,2)),
tf.keras.layers.Conv2D(filters = 64,
kernel_size = (3,3),
activation = 'relu',
padding = 'SAME',
input_shape = (25, 25, 64)),
tf.keras.layers.MaxPool2D((2,2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(units = 64, activation = 'relu'),
tf.keras.layers.Dense(units = 6, activation = 'softmax')
])
model.compile(optimizer = 'adam',
loss = 'sparse_categorical_crossentropy',
metrics = 'accuracy')
model.fit(train_x, train_y, epochs = 15)
model.summary()
test_idx = [7]
test_image = test_x[test_idx]
conv_layer = model.get_layer(index = 6)
grad_model = tf.keras.models.Model(inputs = model.inputs, outputs = [conv_layer.output, model.output])
with tf.GradientTape() as tape:
desired_conv_layer_output, preds = grad_model(test_image)
pred_index = tf.argmax(preds[0])
class_channel = preds[:, pred_index]
# compute gradient via tensorflow GradientTape()
grads = tape.gradient(class_channel, desired_conv_layer_output)
pooled_grads = tf.reduce_mean(grads, axis = (0, 1, 2))
heatmap = tf.matmul(desired_conv_layer_output[0], pooled_grads[..., tf.newaxis])
heatmap = tf.squeeze(heatmap)
attention_grad = np.abs(np.reshape(heatmap,(25,25)))
resized_attention_grad = cv2.resize(attention_grad,
(200*5, 200*5),
interpolation = cv2.INTER_CUBIC)
resized_test_x = cv2.resize(test_image.reshape(200,200),
(200*5, 200*5),
interpolation = cv2.INTER_CUBIC)
plt.figure(figsize = (6, 9))
plt.subplot(3,2,1)
plt.imshow(test_x[test_idx].reshape(200,200), 'gray')
plt.axis('off')
plt.subplot(3,2,2)
plt.imshow(attention_grad)
plt.axis('off')
plt.subplot(3,2,3)
plt.imshow(resized_test_x, 'gray')
plt.axis('off')
plt.subplot(3,2,4)
plt.imshow(resized_attention_grad, 'jet', alpha = 0.5)
plt.axis('off')
plt.subplot(3,2,6)
plt.imshow(resized_test_x, 'gray')
plt.imshow(resized_attention_grad, 'jet', alpha = 0.5)
plt.axis('off')
plt.show()
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')