AI for Mechanical Engineering

Autoencoder (AE)

Problem 1

  1. Make an autoencoder model where latent space is 10-D with the MNIST dataset.

Download files

In [ ]:
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [ ]:
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

mnist_train_images = np.load('/content/drive/MyDrive/ML_Colab/ML_data/mnist_train_images_rev.npy')
mnist_train_labels = np.load('/content/drive/MyDrive/ML_Colab/ML_data/mnist_train_labels_rev.npy')
mnist_test_images = np.load('/content/drive/MyDrive/ML_Colab/ML_data/mnist_test_images_rev.npy')
mnist_test_labels = np.load('/content/drive/MyDrive/ML_Colab/ML_data/mnist_test_labels_rev.npy')

train_labels = mnist_train_labels
train_imgs = mnist_train_images
test_labels = mnist_test_labels
test_imgs = mnist_test_images

## write your code here
#

import tensorflow as tf

encoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(500, activation = 'relu', input_shape = (784,)),
    tf.keras.layers.Dense(300, activation = 'relu'),
    tf.keras.layers.Dense(10, activation = 'relu')
    ])

decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(300, activation = 'relu', input_shape = (10,)),
    tf.keras.layers.Dense(500, activation = 'relu'),
    tf.keras.layers.Dense(784, activation = 'relu')
    ])

autoencoder = tf.keras.models.Sequential([encoder, decoder])
autoencoder.compile(optimizer = tf.keras.optimizers.Adam(0.001),
                    loss = 'mean_squared_error',
                    metrics = ['mse'])
training = autoencoder.fit(train_imgs, train_imgs, batch_size = 256, epochs = 500, verbose = 0)
  1. Show 5 random reconstructed images.
In [ ]:
random_indices = np.random.choice(len(test_imgs), 5, replace=False)
decoded_imgs = autoencoder.predict(test_imgs[random_indices])

plt.figure(figsize=(20, 4))
for i, idx in enumerate(random_indices):

    # Display original
    ax = plt.subplot(2, 5, i + 1)
    plt.imshow(test_imgs[idx].reshape(28, 28), cmap='gray')
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)

    # Display reconstruction
    ax = plt.subplot(2, 5, i + 1 + 5)
    plt.imshow(decoded_imgs[i].reshape(28, 28), cmap='gray')
    ax.get_xaxis().set_visible(False)
    ax.get_yaxis().set_visible(False)
plt.show()
1/1 [==============================] - 0s 29ms/step
  1. Classify 10 digit classes by using MLP on latent space. Note that accurary above 65% is good enough.
In [ ]:
## write your code here
#

train_encodings = encoder.predict(train_imgs)
test_encodings = encoder.predict(test_imgs)

input_dim = 10
output_dim = 10

mlp_classifier = tf.keras.models.Sequential([
    tf.keras.layers.Dense(64, input_dim=input_dim, activation='relu'),
    tf.keras.layers.Dense(output_dim, activation='softmax')
])

mlp_classifier.compile(optimizer=tf.keras.optimizers.Adam(0.001),
                       loss='sparse_categorical_crossentropy',
                       metrics=['accuracy'])

history = mlp_classifier.fit(train_encodings, train_labels, epochs=50, verbose=0)

loss, accuracy = mlp_classifier.evaluate(test_encodings, test_labels)
print(f'Classifier Accuracy: {accuracy * 100:.2f}%')
32/32 [==============================] - 0s 5ms/step
4/4 [==============================] - 0s 7ms/step
4/4 [==============================] - 0s 4ms/step - loss: 0.3961 - accuracy: 0.8700
Classifier Accuracy: 87.00%

Problem 2

Mechanical systems invariably exhibit vibrations during their operation. Consequently, vibration analysis stands as one of the foremost and traditional methods for diagnosing mechanical systems. In the context of this problem, we will employ the logistic regression algorithm to identify anomalies within a mechanical system.

Data information

  • File format: npy
  • Information: signal, label
  • Sampling rate: 12,800 Hz
  • File length (time): 0.78 sec
  • Labels
    • A (Normal)
    • B (Abnormal)

Download the datasets

In [ ]:
import tensorflow as tf
import numpy as np
import scipy
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
%matplotlib inline
In [ ]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [ ]:
## data load

signal = np.load('/content/drive/MyDrive/DL_Colab/DL_data/rotating_machinery_signal.npy')
label = np.load('/content/drive/MyDrive/DL_Colab/DL_data/rotating_machinery_label.npy')

print(signal.shape)
print(label.shape)
(2000, 10000)
(2000,)
In [ ]:
m = label.shape[0]

for _ in range(2):
  idx = np.random.randint(m)
  plt.plot(signal[idx])

  if label[idx] == True:
    plt.title('B (Abnormal)')
  else:
    plt.title('A (Normal)')

  plt.show()

Now, we need to extract features from the raw signal.

  • Use the following statistical features
    • peak, rms, kurtosis, crest factor, impulse factor, shape factor, skewness, smr, peak-peak

The equations are as follows:


$$\begin{align*} \text{Peak Value } &= \max_{n = 1, \dots, N} x_n \\ \\ \text{RMS } &= \sqrt{\frac{1}{N}\sum_{n=1}^N x_n^2} \\ \\ \text{Kurtosis } &= \frac{\frac{1}{N}\sum_{n=1}^N (x_n-\bar{x})^4}{\text{Var}^2} \\ \\ \text{Var } &= \frac{1}{N}\sum_{n=1}^N (x_n-\bar{x})^2 \\\\ \text{Crest Factor } &= \frac{\text{Peak}}{\text{RMS}}\\\\ \text{Impulse Factor } &= \frac{\text{Peak}}{\text{Mean}} \\ \\ \text{Mean } &= \frac{1}{N}\sum_{n=1}^N \lvert x_n \rvert \\\\ \text{Shape Factor } &= \frac{\text{RMS}}{\text{Mean}}\\\\ \text{Skewness } &= \frac{\frac{1}{N}\sum_{n=1}^N(x_n-\bar{x})^3}{\text{Var}^{3/2}}\\\\ \text{SMR } &= \left(\frac{1}{N}\sum_{n=1}^N\sqrt{x_n} \right)^2 \\\\ \text{Peak-Peak Value } &= \max_{n = 1, \dots, N} x_n-\min_{n = 1, \dots, N} x_n \end{align*}$$$$x_i: \text{signal data}$$

The function for feature extraction has been preconfigured for your convenience. The function operates with 'data x' as input and produces a horizontally stacked feature vector as its output.

In [ ]:
def extfeat(x):
    fvector = []

    # time domain feature
    peak = np.max(np.abs(x))
    fvector.append(peak)

    rms = np.sqrt(np.mean(x**2))
    fvector.append(rms)

    kurtosis = scipy.stats.kurtosis(x)
    fvector.append(kurtosis)

    crest_factor = fvector[0]/fvector[1]
    fvector.append(crest_factor)

    impulse_factor = fvector[0]/(np.sum(np.abs(x))/len(x))
    fvector.append(impulse_factor)

    shape_factor = fvector[1]/(np.sum(np.abs(x))/len(x))
    fvector.append(shape_factor)

    skewness = scipy.stats.skew(x)
    fvector.append(skewness)

    smr = (np.sum(np.sqrt(np.abs(x)))/len(x))**2
    fvector.append(smr)

    pp = np.max(x) - np.min(x)
    fvector.append(pp)

    return fvector

feature_name = ['Peak', 'RMS', 'Kurtosis', 'Crest Factor', 'Impulse Factor','Shape Factor','Skewness','SMR', 'Peak-Peak']
In [ ]:
feature_record = []

for idx in range(m):
    feature_record.append(extfeat(signal[idx]))

feature_record = np.array(feature_record)

train_x, test_x, train_y, test_y = train_test_split(feature_record, label, test_size = 1/4, shuffle = True, random_state = 42)
  1. Design your logistic regression model using scikit learn library and train it.
In [ ]:
## your code here
#

from sklearn.linear_model import LogisticRegression

clf = LogisticRegression().fit(train_x, np.ravel(train_y))
  1. Compute the test accuracy.
In [ ]:
## your code here
#

pred = clf.predict(test_x)
accr = np.mean(np.equal(pred, test_y.ravel()))
print("Accuracy : {}%".format(accr*100))
Accuracy : 97.8%
  1. Configure an autoencoder model with the identical structure as depicted in the figure below and proceed with the training process.
In [ ]:
## your code here
#

encoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(4, activation = 'relu', input_shape = (9,)),
    tf.keras.layers.Dense(2, activation = None)
    ])

decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(4, activation = 'relu', input_shape = (2,)),
    tf.keras.layers.Dense(9, activation = None)
    ])

autoencoder = tf.keras.models.Sequential([encoder, decoder])
In [ ]:
autoencoder.compile(optimizer = tf.keras.optimizers.Adam(0.001),
                    loss = 'mean_squared_error',
                    metrics = ['mse'])
In [ ]:
# Train Model & Evaluate Test Data

training = autoencoder.fit(train_x, train_x, batch_size = 32, epochs = 250, verbose = 0)
  1. Compute MSE loss for the test dataset.
In [ ]:
## your code here
#

test_scores = autoencoder.evaluate(test_x, test_x)
16/16 [==============================] - 0s 2ms/step - loss: 0.0102 - mse: 0.0102
  1. Visualize the training dataset in the latent space.
In [ ]:
## your code here
#

latent = encoder.predict(train_x, verbose = 0)

plt.figure(figsize = (6, 6))
plt.scatter(latent[train_y == 0, 0], latent[train_y == 0, 1], label = 'normal')
plt.scatter(latent[train_y == 1, 0], latent[train_y == 1, 1], label = 'abnormal')
plt.title('Latent Space')
plt.xlabel('Z1')
plt.ylabel('Z2')
plt.axis('equal')
plt.show()
  1. Design a logistic regression model to find a linear boundary between normal and abnormal instances within the latent space.
In [ ]:
## your code here
#

from sklearn import linear_model

clf = linear_model.LogisticRegression()
clf.fit(latent, train_y)
Out[ ]:
LogisticRegression()
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
  1. Visualize the test dataset within the latent space along with the linear boundary that separates normal and abnormal instances.
In [ ]:
## your code here
#

test_latent = encoder.predict(test_x, verbose = 0)

w0 = clf.intercept_[0]
w1 = clf.coef_[0,0]
w2 = clf.coef_[0,1]

xp = np.linspace(np.min(test_latent[:,0]), np.max(test_latent[:,0]),100).reshape(-1,1)
yp = - w1/w2*xp - w0/w2

plt.figure(figsize = (6, 6))
plt.scatter(test_latent[test_y == 0, 0], test_latent[test_y == 0, 1], label = 'normal')
plt.scatter(test_latent[test_y == 1, 0], test_latent[test_y == 1, 1], label = 'abnormal')
plt.plot(xp, yp, 'g', linewidth = 4, label = 'Logistic Regression')
plt.title('Logistic Regression')
plt.xlabel('Z1')
plt.ylabel('Z2')
plt.axis('equal')
plt.show()

Problem 3

PCA (Principal Component Analysis)

  • PCA is one of the oldest and most widely used dimensionality reduction algorithms. Its idea is to reduce the dimensionality of a dataset, while preserving as much 'variability' as possible. (but you don't need to fully understand the PCA algorithm for this problem.)

Actually, autoencoder is very similar to the PCA algorithm in terms of 'dimension reduction'. So, in this problem, we are going to build up PCA with autoencoder. While PCA is a linear dimension reduction method, the autoencoder has non-linear activation functions. Therefore, autoencoder without non-linear activation functions can be considered as PCA.

Now, we have 3D data. Run the below cell to load and 3D plot them.


Download the datasets

In [ ]:
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
%matplotlib inline
In [ ]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [ ]:
data = np.load('/content/drive/MyDrive/ML_Colab/ML_data/pca_autoencoder.npy')

fig = plt.figure(figsize = (8, 8))
ax = fig.add_subplot(111, projection = '3d')
ax.scatter(data[:,0], data[:,1], data[:,2])
plt.show()

train_x, test_x = data[:100], data[100:]

PCA result in 2D is shown as follows.

In [ ]:
from sklearn.decomposition import PCA

pca = PCA(n_components = 2)
pca.fit(data)
result = pca.transform(data)

plt.figure(figsize = (6, 6))
plt.plot(result[:,0], result[:,1], 'o')
plt.axis('equal')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.show()
In [ ]:
pca.fit(test_x)
result = pca.transform(test_x)

plt.figure(figsize = (6, 6))
plt.plot(result[:,0], result[:,1], 'o')
plt.axis('equal')
plt.xlabel('PC1')
plt.ylabel('PC2')
plt.show()
  1. Design your linear autoencoder model using ANN. (freely design your own network structure)
  • Hint: you must use (activation = None)
In [ ]:
## your code here
#

n_input = 3
n_latent = 2
n_output = n_input

encoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(n_latent, activation = None, input_shape = (n_input,)),
    ])

decoder = tf.keras.models.Sequential([
    tf.keras.layers.Dense(n_output, activation = None, input_shape = (n_latent,)),
    ])

autoencoder = tf.keras.models.Sequential([encoder, decoder])
In [ ]:
autoencoder.compile(optimizer = tf.keras.optimizers.Adam(0.001),
                    loss = 'mean_squared_error')
In [ ]:
training = autoencoder.fit(train_x, train_x, epochs = 1000, verbose = 0)
  1. After training your model, plot the data on latent space. (You will be able to obtain similar result as PCA!!)
In [ ]:
## your code here
#

result = encoder.predict(test_x)

plt.figure(figsize = (6, 6))
plt.plot(result[:,0], result[:,1], 'o')
plt.axis('equal')
plt.xlabel('Latent dimension 1')
plt.ylabel('Latent dimension 2')
plt.show()
4/4 [==============================] - 0s 2ms/step

Problem 4

The encoder part of an autoencoder is well known as a dimensionality reduction operator. This problem will ask you to implement the autoencoder algorithm for face data. The given data consists of 100 pictures of human faces with size of (50, 40), we will apply an autoencoder to this dataset.

Downlaod the datasets

In [ ]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [ ]:
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

data = np.load('/content/drive/MyDrive/ML_Colab/ML_data/pca_faces.npy')

print(data.shape)
(100, 50, 40)
  1. Plot one random face out of 100 pictures. You might want to run multiple times to see what kinds of faces are in the data set.
In [ ]:
## your code here
#
  1. Apply the autoencoder to the dataset. Build your model with the following structure:
  • first encoder: 500
  • second encoder: 300
  • latent node: 8
  • first decoder: 300
  • second decoder: 500
  • activation = 'relu'
In [ ]:
train_face = data.reshape([100, 50*40])
print(train_face.shape)
(100, 2000)
In [ ]:
## your code here
#
  1. Plot a randomly selected input image alongside its correspoding reconstructed image.
In [ ]:
## your code here
#
1/1 [==============================] - 0s 71ms/step
  1. Reconstruct the image of the individual wearing a crown using your autoencoder, and provide a discussion regarding the result of the reconstructed face. Please note that the reconstruction performance may not be optimal.
In [ ]:
test = train_face[94].reshape(50,40)

## your code here
#
1/1 [==============================] - 0s 59ms/step
  1. Reconstruct the image of the individual wearing sunglasses using your autoencoder and provide a discussion regarding the result of the reconstructed face. Please note that the reconstruction performance may not be optimal.
In [ ]:
test_face = train_face[28]

## your code here
#
1/1 [==============================] - 0s 37ms/step