Artificial Neural Networks (ANN)
Table of Contents
1. Recall Perceptron¶
from IPython.display import YouTubeVideo
YouTubeVideo('W5i9OA0bW-A', width="560", height="315", frameborder="0")
Perceptron
Binary Linear Classification in 2D
Binary Linear Classification in High Dimensions
A perceptron models the decision boundary as a linear equation:
$$\omega_0 + \omega_1 x_1 + \omega_2 x_2 + \cdots + \omega_d x_d = 0$$
This means that a perceptron can only create a straight-line (hyperplane) to separate the data. It is instructive to conceptualize a single perceptron as representing a hyperplane in a geometric space.
2. From Perceptron to Multi-Layer Perceptron (MLP)¶
To explain the concept of a multi-layer perceptron (MLP), we start with a simple perceptron and establish its connection to a single hyperplane. A perceptron functions as a linear classifier by learning a hyperplane that separates data into distinct classes. However, when the data is non-linearly separable, a single hyperplane is insufficient.
In such cases, two possible approaches arise:
- Using multiple hyperplanes (or multiple dividers): This can be achieved by stacking multiple layers of perceptrons, allowing the model to combine hyperplanes and approximate more complex decision boundaries.
- Learning non-linear boundaries (or curved surface): By introducing nonlinear activation functions and multiple layers, the model can learn non-linear transformations that capture complex patterns in the data.
This progression from a single hyperplane to non-linear boundaries forms the foundation of multi-layer perceptrons (MLPs), enabling them to handle non-linear separability.
2.1. Perceptron¶
Neurons compute the weighted sum of their inputs
A neuron is activated or fired when the sum $a$ is positive
$$ \begin{align*} a &= \omega_0 + \omega_1 x_1 + \omega_2 x_2 \\ \\ \hat{y} &= g(a) = \begin{cases} 1 & a > 0\\ 0 & \text{otherwise} \end{cases} \end{align*} $$
- A step function (or sign function) is non-differentiable.
- To address this limitation, we later replace the step function with differentiable non-linear activation functions such as the sigmoid, or tanh functions.
XOR Problem
Minsky-Papert Controversy on XOR
- not linearly separable
- limitation of perceptron
A perceptron is a simple linear classifier that separates data using a single hyperplane. However, there are some classification tasks that a perceptron cannot solve due to its linear nature. A classic example of this limitation is the XOR (exclusive OR) problem.
Idea: Nonlinear Curve Approximated by Multiple Lines
2.2. Multiple Perceptron¶
A single perceptron is often insufficient for complex tasks, as it can only learn a single hyperplane to separate the data. When the data is not linearly separable, a single hyperplane cannot capture the intricate boundaries or patterns in the feature space, necessitating deeper or more complex architectures to model the underlying relationships effectively.
For example, if two perceptrons are stacked, it represents two hyperplanes.
- Switch to differentiable activation function (for example, the sigmoid function)
- In a compact representation
- Combining the summation and the sigmoid function forms another neuron.
2.3. Hidden Layers as Kernel Learning¶
Each neuron applies a nonlinear activation function to its inputs, effectively performing a nonlinear transformation of the data. As a result, the hidden layers can be interpreted as a nonlinear mapping between the input and output spaces, similar to the role of kernel functions in classical machine learning methods.
The Second Way of Looking at Multiple Perceptrons
A nonlinear activation function enables a perceptron to model nonlinear relationships between input and output.
Suppose that data is not linearly separable
Nonlinear mapping + neuron
- User-defined Kernel
- For example,
$$\phi: (x_1, x_2) \rightarrow (x_1, x_2, x_1 x_2)$$
Nonlinear mapping can be represented by another layer (or neurons)
- Learnable Kernel
- Nonlinear activation functions
In machine learning, defining the appropriate kernels for non-linear mapping is crucial for models like logistic regression and kernel-based methods. However, in deep learning, the kernel function is effectively represented by the hidden layers of the neural network, which are learned directly from the data rather than being predefined.
- In traditional machine learning: Kernels (such as polynomial or RBF kernels) are manually chosen to project the input into a higher-dimensional space for better separation.
- In deep learning: The hidden layers act as adaptive feature extractors, learning the non-linear transformations (analogous to kernels) automatically during training.
This flexibility allows deep learning models to discover the best representations and transformations for the data without requiring explicit kernel design.
Multi-Layer Perceptron as a Sequence of Feature Extraction
When multiple hidden layers are stacked, the output of one hidden layer becomes the input to the next. This structure can be viewed as a sequence of feature extraction stages, where each layer transforms its input into a higher-level, more abstract representation.
This process of hierarchical feature extraction enables the Multi-Layer Perceptron (MLP) to capture and learn increasingly complex patterns within the data. Lower layers typically extract basic features, while deeper layers combine these to form more sophisticated and semantically meaningful representations.
A multi-layer perceptron is not merely a collection of neurons but a structured sequence of feature extraction steps. Each layer extracts increasingly abstract and relevant features, allowing the model to handle complex, non-linear tasks that a single-layer perceptron cannot solve.
Intuition Behind Feature Extraction in MLP
- The first layer extracts basic features from the input (e.g., edges or simple patterns in images).
- The hidden layers combine these basic features to create more complex representations (e.g., shapes, contours).
- The final layer maps the abstract features to the output (e.g., predicting a class label or regression value).
By stacking layers, the MLP gradually transforms simple input features into complex features that capture relationships in non-linear data.
Each layer can be thought of as performing a mapping from one feature space to another, progressively refining the representation to make the final decision easier.
This hierarchy of transformations allows the network to move from raw input data to task-specific representations.
2.4. Summary: Two Ways of Looking at Artificial Neural Networks¶
(1) Can represent multiple lines
(2) Can represent nonlinear relationship between input and outputs due to nonlinear activation function
3. Logistic Regression in a Form of Neural Network¶
Here, we will demonstrate a multi-layer perceptron (MLP) using a logistic regression example. In this demonstration, we will observe the hyperplanes and the features learned in the hidden layer at the end of the training process.
- When we extend logistic regression to a multi-layer perceptron, the hidden layer transforms the input features through non-linear activations, creating intermediate feature spaces.
- At the end of learning, we can visualize how the hidden layer features and multiple hyperplanes evolve to form more complex decision boundaries.
This approach will illustrate how the MLP builds upon logistic regression by stacking multiple transformations to handle non-linear separability.
Let's start with logistic regression for a linearly separable case.
$$ \begin{align*} y^{(i)} &\in \{0, 1\}\\\\ y &= \sigma (\omega_0 + \omega_1 x_1 + \omega_2 x_2) \end{align*} $$
After training, $\omega_0 + \omega_1 x_1 + \omega_2 x_2 = 0$ will represent a linear classification boundary.
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
%matplotlib inline
#training data gerneration
m = 1000
x1 = 8*np.random.rand(m, 1)
x2 = 7*np.random.rand(m, 1) - 4
g = 0.8*x1 + x2 - 3
C1 = np.where(g >= 0)[0]
C0 = np.where(g < 0)[0]
N = C1.shape[0]
M = C0.shape[0]
m = N + M
X1 = np.hstack([x1[C1], x2[C1]])
X0 = np.hstack([x1[C0], x2[C0]])
train_X = np.vstack([X1, X0])
train_y = np.vstack([np.ones([N,1]), np.zeros([M,1])])
train_X = np.asmatrix(train_X)
train_y = np.asmatrix(train_y)
plt.figure(figsize = (6, 4))
plt.plot(x1[C1], x2[C1], 'ro', alpha = 0.4, label = 'C1')
plt.plot(x1[C0], x2[C0], 'bo', alpha = 0.4, label = 'C0')
plt.legend(loc = 1)
plt.xlabel(r'$x_1$')
plt.ylabel(r'$x_2$')
plt.show()
LogisticRegression = tf.keras.models.Sequential([
tf.keras.layers.Input(shape=(2,)),
tf.keras.layers.Dense(units = 1, activation = 'sigmoid')
])
LogisticRegression.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.1),
loss = 'binary_crossentropy')
loss = LogisticRegression.fit(train_X, train_y, epochs = 10, verbose = 0)
w = LogisticRegression.layers[0].get_weights()[0]
b = LogisticRegression.layers[0].get_weights()[1]
print(w)
print("\n")
print(b)
x1p = np.arange(0, 8, 0.01).reshape(-1, 1)
x2p = - w[0,0]/w[1,0]*x1p - b[0]/w[1,0]
plt.figure(figsize = (6, 4))
plt.plot(x1[C1], x2[C1], 'ro', alpha = 0.4, label = 'C1')
plt.plot(x1[C0], x2[C0], 'bo', alpha = 0.4, label = 'C0')
plt.plot(x1p, x2p, 'g', linewidth = 3, label = '')
plt.xlabel('$x_1$')
plt.ylabel('$x_2$')
plt.legend(loc = 1)
plt.show()
3.1. Looking at Parameters in Nonlinear Classification¶
Now, let's move on to a non-linearly separable case.
In this scenario, the dataset cannot be separated by a single linear hyperplane. Therefore, we use a multi-layer perceptron (MLP), which approximates complex non-linear decision boundaries by learning multiple hyperplanes and combining them using non-linear activation functions.
Let's see how the MLP performs in this case!
Before that, let's first mention the notation changes commonly used in neural network conventions. In neural network notation, it is common to represent: $\omega_0 \rightarrow b$
- Weights as $\omega$
- Bias as $b$ (previously denoted as $\omega_0$)
$$y = \sigma(\omega_0 + \omega_1 x_1 + \omega_2 x_2) \quad \longrightarrow \quad y = \sigma(b + \omega_1 x_1 + \omega_2 x_2)$$
# training data gerneration
m = 1000
x1 = 10*np.random.rand(m, 1) - 5
x2 = 8*np.random.rand(m, 1) - 4
g = - 0.5*(x1-1)**2 + 2*x2 + 5
C1 = np.where(g >= 0)[0]
C0 = np.where(g < 0)[0]
N = C1.shape[0]
M = C0.shape[0]
m = N + M
X1 = np.hstack([x1[C1], x2[C1]])
X0 = np.hstack([x1[C0], x2[C0]])
train_X = np.vstack([X1, X0])
train_X = np.asmatrix(train_X)
train_y = np.vstack([np.ones([N,1]), np.zeros([M,1])])
plt.figure(figsize = (6, 4))
plt.plot(x1[C1], x2[C1], 'ro', alpha = 0.4, label = 'C1')
plt.plot(x1[C0], x2[C0], 'bo', alpha = 0.4, label = 'C0')
plt.legend(loc = 1, fontsize = 15)
plt.xlabel(r'$x_1$', fontsize = 15)
plt.ylabel(r'$x_2$', fontsize = 15)
plt.axis('equal')
plt.ylim([-4, 4])
plt.show()
As illustrated in the figure, the data is non-linearly distributed. To approximate the non-linear decision boundary using two linear boundaries, a hidden layer with two neurons (plus a bias neuron) is intentionally added.
LogisticRegression = tf.keras.models.Sequential([
tf.keras.layers.Input(shape=(2,)),
tf.keras.layers.Dense(units = 2, activation = 'sigmoid'),
tf.keras.layers.Dense(units = 1, activation = 'sigmoid')
])
LogisticRegression.compile(optimizer = tf.keras.optimizers.Adam(learning_rate = 0.1),
loss = 'binary_crossentropy')
loss = LogisticRegression.fit(train_X, train_y, epochs = 10, verbose = 0)
w1 = LogisticRegression.layers[0].get_weights()[0]
b1 = LogisticRegression.layers[0].get_weights()[1]
w2 = LogisticRegression.layers[1].get_weights()[0]
b2 = LogisticRegression.layers[1].get_weights()[1]
H = train_X*w1 + b1
H = 1/(1 + np.exp(-H))
plt.figure(figsize = (6, 4))
plt.plot(H[0:N,0], H[0:N,1], 'ro', alpha = 0.4, label = 'C1')
plt.plot(H[N:m,0], H[N:m,1], 'bo', alpha = 0.4, label = 'C0')
plt.xlabel('$z_1$', fontsize = 15)
plt.ylabel('$z_2$', fontsize = 15)
plt.legend(loc = 1, fontsize = 15)
plt.axis('equal')
plt.ylim([0, 1])
plt.show()
Here the features $z_1$ and $z_2$ learned in the hidden layer have two notable characteristics:
Bounded Output Range: Both features are restricted to the interval $(0,1)$ due to the use of the sigmoid activation function. This non-linear transformation ensures that the outputs remain within a normalized range, regardless of the input values.
Feature Redistribution: The hidden layer redistributes the input data into a new feature space where the transformed data becomes approximately linearly separable. This transformation allows the multi-layer perceptron to form a linear decision boundary in the higher-dimensional space that corresponds to a non-linear boundary in the original input space.
These learned features (or feature extractions) demonstrate how the hidden layer facilitates the approximation of complex patterns by creating new, informative representations of the input data.
x1p = np.arange(0, 1, 0.01).reshape(-1, 1)
x2p = - w2[0,0]/w2[1,0]*x1p - b2[0]/w2[1,0]
plt.figure(figsize = (6, 4))
plt.plot(H[0:N,0], H[0:N,1], 'ro', alpha = 0.4, label = 'C1')
plt.plot(H[N:m,0], H[N:m,1], 'bo', alpha = 0.4, label = 'C0')
plt.plot(x1p, x2p, 'k', linewidth = 3, label = '')
plt.xlabel('$z_1$', fontsize = 15)
plt.ylabel('$z_2$', fontsize = 15)
plt.legend(loc = 1, fontsize = 15)
plt.axis('equal')
plt.ylim([0, 1])
plt.show()
The result aligns with the linear boundary in the $z$-space, indicating that the transformed features $z_1$ and $z_2$ successfully reshape the input data into a feature space where a linear decision boundary can separate the classes. This transformation validates the role of the hidden layer in projecting non-linearly distributed data into a higher-dimensional space, making it linearly separable.
x1p = np.arange(-5, 5, 0.01).reshape(-1, 1)
x2p = - w1[0,0]/w1[1,0]*x1p - b1[0]/w1[1,0]
x3p = - w1[0,1]/w1[1,1]*x1p - b1[1]/w1[1,1]
plt.figure(figsize = (6, 4))
plt.plot(x1[C1], x2[C1], 'ro', alpha = 0.4, label = 'C1')
plt.plot(x1[C0], x2[C0], 'bo', alpha = 0.4, label = 'C0')
plt.plot(x1p, x2p, 'k', linewidth = 3, label = '')
plt.plot(x1p, x3p, 'g', linewidth = 3, label = '')
plt.xlabel('$x_1$', fontsize = 15)
plt.ylabel('$x_2$', fontsize = 15)
plt.legend(loc = 1, fontsize = 15)
plt.axis('equal')
plt.ylim([-4, 4])
plt.show()
After training, two linear boundaries in the $x$-space (input space) are learned, corresponding to the linear boundaries formed by the hidden layer neurons. These boundaries result in multiple lines that, when combined, approximate the non-linear classification boundary. This demonstrates how the multi-layer perceptron (MLP) constructs a complex decision boundary in the input space by learning and integrating multiple hyperplanes from the hidden layer.
4. Regression in a Form of Neural Network¶
A multi-layer perceptron (MLP) can also be applied to nonlinear regression tasks. Unlike linear regression, which fits a straight line, MLPs can model complex, nonlinear relationships between input features and the output due to their use of multiple layers and nonlinear activation functions.
In this example, the Rectified Linear Unit (ReLU) will be used as the non-linear activation function. Therefore, let’s first take a closer look at ReLU before proceeding to the example.
Rectified linear unit (ReLU activation function)
$$h(x) = \max(0, x) = \begin{cases} 0, &\text{if}\;\; x \leq 0 \\ 1, &\text{if} \;\; x > 0 \end{cases}$$
Key Characteristics of ReLU:
- Piecewise Linear: ReLU outputs zero for negative values of $x$ and passes the value of $x$ unchanged for positive values.
- Sparsity: ReLU introduces sparsity by setting some neuron activations to zero, which can make the network more efficient.
- Avoids Vanishing Gradients: Unlike sigmoid or tanh, ReLU does not squash values to a narrow range, thus avoiding the issue of vanishing gradients in deep networks.
- Non-saturating: The gradient remains constant for positive values, making optimization faster.
$$h(x)$$
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
def relu(x):
return np.maximum(0, x)
xp = np.linspace(-1.5, 1.5, 100)
yp = relu(xp)
plt.figure(figsize = (6, 4))
plt.plot(xp, yp, '--', color = 'red', lw = 3, alpha = 0.5)
plt.xlabel('x')
plt.ylabel('y')
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.grid(alpha = 0.3)
plt.show()
$$h(x-0.5)$$
xp = np.linspace(-1.5, 1.5, 100)
yp = relu(xp - 0.5)
plt.figure(figsize = (6, 4))
plt.plot(xp, yp, '--', color = 'red', lw = 3, alpha = 0.5)
plt.xlabel('x')
plt.ylabel('y')
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.grid(alpha = 0.3)
plt.show()
$$h(-2x)$$
xp = np.linspace(-1.5, 1.5, 100)
yp = relu(-2*xp)
plt.figure(figsize = (6, 4))
plt.plot(xp, yp, '--', color = 'red', lw = 3, alpha = 0.5)
plt.xlabel('x')
plt.ylabel('y')
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.grid(alpha = 0.3)
plt.show()
$$- h(-2x-1)$$
xp = np.linspace(-1.5, 1.5, 100)
yp = -relu(-2*xp - 1)
plt.figure(figsize = (6, 4))
plt.plot(xp, yp, '--', color = 'red', lw = 3, alpha = 0.5)
plt.xlabel('x')
plt.ylabel('y')
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.grid(alpha = 0.3)
plt.show()
The expression $\omega_2 h(\omega_1 x + b_1) + b_2 $, where $h(\cdot)$ represents the ReLU activation function, can describe a variety of transformations of the ReLU function, including horizontal and vertical shifts, as well as reflections across the $x$-axis or $y$-axis, depending on the values and signs of $\omega_1, \omega_2, b_1$, and $b_2$.
Let us consider the task of modeling a multi-layer perceptron (MLP) to approximate the given non-linear functions.
import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
np.random.seed(0)
tf.random.set_seed(0)
def function(x):
return x**3 + 0.1*x**2 - x + 0.1
x = np.linspace(-1.5, 1.5, 1000)
y = function(x)
plt.figure(figsize = (6, 4))
plt.plot(x, y, '--', color = 'red', lw = 3, alpha = 0.5)
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.xlabel('x')
plt.ylabel('y')
plt.grid(alpha = 0.3)
plt.show()
train_x = np.random.uniform(-1.5, 1.5, 1000)
train_y = function(train_x)
Nonlinear Regression as a Linear Combination of ReLU Functions
In an MLP with ReLU activation, the output of the model can be interpreted as a linear combination of ReLU-transformed features. Each hidden layer neuron applies a ReLU transformation to a weighted sum of the input features, and the final output is a linear combination of these transformed features.
Key Insight:
Nonlinear regression with MLPs can be thought of as creating a piecewise linear approximation of the true function, where the linear segments are defined by the ReLU activations in the hidden layer. The network learns how to place these segments and their slopes to best fit the target function.
For a neural network with one hidden layer, the predicted output $ \hat{y} $ is:
$$ \hat{y} = \sum_{j=1}^{m} \omega_j^{(2)} h\left( \sum_{i=1}^{n} \omega_{ij}^{(1)} x_i + b_j^{(1)} \right) + b^{(2)} $$
$\quad$where:
- $ \omega_{ij}^{(1)} $ and $ b_j^{(1)} $ are the weights and biases for the hidden layer.
- $ \omega_j^{(2)} $ and $ b^{(2)} $ are the weights and bias for the output layer.
In the following examples, $b^{(2)}$, the bias term for the output layer, is omitted for simplicity. In other words, the vertical shift transformation is not applied.
Single Neuron with ReLU
$$\hat{y} = \omega^{(2)} \left( h \left( \omega^{(1)} x + b^{(1)} \right) \right)$$
model = tf.keras.models.Sequential([
tf.keras.layers.Input(shape = (1,)),
tf.keras.layers.Dense(units = 1, activation = 'relu'),
tf.keras.layers.Dense(units = 1, use_bias = False)
])
model.compile(optimizer = 'adam',
loss = 'mse')
train_x = train_x.reshape(-1, 1)
train_y = train_y.reshape(-1, 1)
model.fit(train_x, train_y, epochs = 500, verbose = 0)
weights = model.layers[0].get_weights()[0]
biases = model.layers[0].get_weights()[1]
weights2 = model.layers[1].get_weights()[0]
print("Coefficients (Weights):", weights)
print("Intercepts (Biases):", biases)
print("Coefficients (Weights):", weights2)
real_x = np.linspace(-1.5, 1.5, 100)
real_y = function(real_x)
def relu(x):
return np.maximum(0, x)
x1p = np.arange(-2, 2, 0.01).reshape(-1, 1)
x2p = weights2*(relu(weights*x1p + biases))
plt.figure(figsize = (6, 4))
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.plot(real_x, real_y, '--', color = 'red', alpha = 0.5)
plt.plot(x1p, x2p, 'c', linewidth = 3)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(alpha = 0.3)
plt.show()
Two Neurons with ReLU
$$\hat{y} = \omega_1^{(2)} \left( h \left( \omega_1^{(1)} x + b_1^{(1)} \right) \right) + \omega_2^{(2)} \left( h \left( \omega_2^{(1)} x + b_2^{(1)} \right) \right)$$
model_2 = tf.keras.models.Sequential([
tf.keras.layers.Input(shape = (1,)),
tf.keras.layers.Dense(units = 2, activation = 'relu'),
tf.keras.layers.Dense(units = 1, use_bias = False)
])
model_2.compile(optimizer = 'adam',
loss = 'mse')
train_x = train_x.reshape(-1, 1)
train_y = train_y.reshape(-1, 1)
model_2.fit(train_x, train_y, epochs = 3000, verbose = 0)
weights = model_2.layers[0].get_weights()[0]
biases = model_2.layers[0].get_weights()[1]
weights2 = model_2.layers[1].get_weights()[0]
print("Coefficients (Weights):", weights)
print("Intercepts (Biases):", biases)
print("Coefficients (Weights):", weights2)
x1p = np.arange(-2, 2, 0.01).reshape(-1, 1)
x2p = weights2[0]*relu(weights[0][0]*x1p + biases[0])
x3p = weights2[1]*relu(weights[0][1]*x1p + biases[1])
plt.figure(figsize = (6, 4))
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.plot(real_x, real_y, '--', color = 'red', alpha = 0.5)
plt.plot(x1p, x2p, 'b', linewidth = 3, alpha = 0.5)
plt.plot(x1p, x3p, 'k', linewidth = 3, alpha = 0.5)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(alpha = 0.3)
plt.show()
x1p = np.arange(-2, 2, 0.01).reshape(-1, 1)
x2p = weights2[0]*relu(weights[0][0]*x1p + biases[0]) + weights2[1]*relu(weights[0][1]*x1p + biases[1])
plt.figure(figsize = (6, 4))
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.plot(real_x, real_y, '--', color = 'red', alpha = 0.5)
plt.plot(x1p, x2p, 'c', linewidth = 3)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(alpha = 0.3)
plt.show()
Four Neurons with ReLU
$$\hat{y} = \omega_1^{(2)} \left( h \left( \omega_1^{(1)} x + b_1^{(1)} \right) \right) + \omega_2^{(2)} \left( h \left( \omega_2^{(1)} x + b_2^{(1)} \right) \right) + \omega_3^{(2)} \left( h \left( \omega_3^{(1)} x + b_3^{(1)} \right)\right) + \omega_4^{(2)} \left( h \left( \omega_4^{(1)} x + b_4^{(1)} \right) \right) $$
model_4 = tf.keras.models.Sequential([
tf.keras.layers.Input(shape = (1,)),
tf.keras.layers.Dense(units = 4, activation = 'relu'),
tf.keras.layers.Dense(units = 1, use_bias = False)
])
model_4.compile(optimizer = 'adam',
loss = 'mse')
model_4.fit(train_x, train_y, epochs = 4000, verbose = 0)
weights = model_4.layers[0].get_weights()[0]
biases = model_4.layers[0].get_weights()[1]
weights2 = model_4.layers[1].get_weights()[0]
x1p = np.arange(-2, 2, 0.01).reshape(-1, 1)
x2p = weights2[0]*relu(weights[0][0]*x1p + biases[0])
x3p = weights2[1]*relu(weights[0][1]*x1p + biases[1])
x4p = weights2[2]*relu(weights[0][2]*x1p + biases[2])
x5p = weights2[3]*relu(weights[0][3]*x1p + biases[3])
plt.figure(figsize = (6, 4))
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.plot(real_x, real_y, '--', color = 'red', alpha = 0.5)
plt.plot(x1p, x2p, 'b', linewidth = 3, alpha = 0.5)
plt.plot(x1p, x3p, 'k', linewidth = 3, alpha = 0.5)
plt.plot(x1p, x4p, 'g', linewidth = 3, alpha = 0.5)
plt.plot(x1p, x5p, 'y', linewidth = 3, alpha = 0.5)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(alpha = 0.3)
plt.show()
x1p = np.arange(-2, 2, 0.01).reshape(-1, 1)
x2p = weights2[0]*relu(weights[0][0]*x1p + biases[0]) + weights2[1]*relu(weights[0][1]*x1p + biases[1]) + weights2[2]*relu(weights[0][2]*x1p + biases[2]) + weights2[3]*relu(weights[0][3]*x1p + biases[3])
plt.figure(figsize = (6, 4))
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.plot(real_x, real_y, '--', color = 'red', alpha = 0.5)
plt.plot(x1p, x2p, 'c', linewidth = 3)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(alpha = 0.3)
plt.show()
20 Neurons with ReLU
k = 20
model_20 = tf.keras.models.Sequential([
tf.keras.layers.Input(shape = (1,)),
tf.keras.layers.Dense(units = k, activation = 'relu'),
tf.keras.layers.Dense(units = 1, use_bias = False)
])
model_20.compile(optimizer = 'adam',
loss = 'mse')
model_20.fit(train_x, train_y, epochs = 1000, verbose = 0)
weights = model_20.layers[0].get_weights()[0]
biases = model_20.layers[0].get_weights()[1]
weights2 = model_20.layers[1].get_weights()[0]
x1p = np.arange(-2, 2, 0.01).reshape(-1, 1)
x2p = 0
for i in range(k):
x2p += weights2[i]*relu(weights[0][i]*x1p + biases[i])
plt.figure(figsize = (6, 4))
plt.xlim([-2, 2])
plt.ylim([-4/3, 4/3])
plt.plot(real_x, real_y, '--', color = 'red', alpha = 0.5)
plt.plot(x1p, x2p, 'c', linewidth = 3)
plt.xlabel('x')
plt.ylabel('y')
plt.grid(alpha = 0.3)
plt.show()
Key Points:
- Each curve represents the output of a single ReLU neuron in response to the input feature $x$.
- The activation values are zero for certain ranges of $x$, indicating that the neuron is "inactive" in those regions.
- The piecewise linear segments created by the ReLU activations contribute to forming the overall non-linear regression curve.
This illustrates how the hidden layer's neurons, each applying ReLU activation, form different regions of the input space and collectively approximate the non-linear function.
5. Artificial Neural Networks¶
So far, we have conducted an in-depth examination of how an artificial neural network (ANN) operates at a micro level, focusing on the underlying mechanisms of individual neurons, layers, and their interactions. Having developed these foundational insights and intuitions, we are now in a position to broaden our perspective and analyze the ANN as a cohesive system, considering its overall architecture, functionality, and the role each component plays in contributing to the network's collective decision-making process.
Complex/Nonlinear Universal Function Approximator
ANNs are powerful universal function approximators that can model both linear and nonlinear relationships.
By stacking layers of neurons and using non-linear activation functions, ANNs can represent highly complex mappings, making them suitable for a wide range of tasks, such as image recognition, speech processing, and time-series predictions.
ANN Architecture
ANNs are typically organized as feedforward networks with layers that are fully connected. In these networks:
- Each neuron in a layer receives inputs from all neurons in the previous layer.
- The network propagates information forward, from input to output, without loops.
Each layer performs a linear transformation of the input.
Linear connections alone cannot capture complex relationships; this is where non-linear activation functions become essential.
Each neuron applies a non-linear activation function to the weighted sum of its inputs
These nonlinear neurons allow ANNs to learn non-linear decision boundaries, making them capable of solving complex, non-linearly separable problems.
Hidden Layers and Autonomous Feature Learning
Hidden layers are the intermediate layers between the input and output layers in a neural network.
Each hidden layer learns intermediate representations or features from the input data.
Shallow networks have fewer hidden layers and may struggle with highly complex patterns.
Deep networks with many hidden layers (Deep Neural Networks) can model highly intricate relationships but require large datasets and longer training times.
The neurons in the hidden layers create intermediate transformations that allow the network to construct hierarchical feature representations, leading to improved performance for complex tasks.
Unlike traditional machine learning models that rely on manual feature extraction, ANNs can automatically learn features from raw data.
Each layer in an ANN learns increasingly abstract features.
This autonomous feature learning makes ANNs robust across diverse domains and applications, eliminating the need for domain-specific feature engineering.