Convolutional Neural Networks (CNN)

Table of Contents




1. Convolution

We will explore Convolutional Neural Networks (CNNs); however, before diving into the details, it is essential to first examine the concept of the convolution operation.


In [1]:
from IPython.display import YouTubeVideo
YouTubeVideo('xSuFInvLjBo', width = "560", height = "315")
Out[1]:

1.1. Convolution in Signal Processing

Convolution is a fundamental mathematical operation in signal processing, widely used in applications such as filtering, audio processing, image processing, and system analysis. The operation involves integrating the product of two functions, where one function is flipped and shifted relative to the other. Convolution can be interpreted as a process that combines two signals to produce a third signal, characterizing how the shape of one signal is altered by the influence of another.


Mathematical Formulation

(1) Continuous-Time Convolution

For two continuous functions $f(t)$ (input signal) and $g(t)$ (impulse response), the convolution is defined as:


$$(f * g)(t) = \int_{-\infty}^{\infty} f(\tau) g(t - \tau) d\tau$$

(2) Discrete-Time Convolution

For discrete signals $f[n]$ and $g[n]$, the convolution is expressed as:


$$(f * g)[n] = \sum_{k=-\infty}^{\infty} f[k] g[n - k]$$

Here, $g[n - k]$ represents the flipped and shifted version of $g$, and the summation computes the weighted sum of the overlapping values.


Concept of Convolution in Signal Processing

  • Input Signal: The function being processed (e.g., an image, an audio waveform).

  • Impulse Response (Kernel/Filter): The function that defines how the system modifies the input. Common kernels include smoothing filters, edge detection filters, and band-pass filters.

  • Convolution Output: The result after applying the filter to the input, showing how the original signal is transformed.


Visual Intuition

The convolution operation can be understood as a sliding window process, where one function (usually the filter or kernel) is flipped and shifted across the input. At each position, the product of the overlapping values is summed to compute the output value at that point.

For example:

  • In 1D audio signals, the sliding window represents the flipped and shifted impulse response passing over the time-series input.

  • In 2D image convolutions, the kernel slides over different regions of the image, applying the same process of element-wise multiplication and summation for each spatial location.



1.2. Cross-Correlation

Convolution and cross-correlation are two similar yet distinct operations commonly used in signal processing and deep learning. Both operations involve a sliding window (filter/kernel) applied to an input signal, but they differ in how the filter is applied.


Mathematical Formulation

(1) Continuous-Time Cross-Correlation

The continuous-time cross-correlation of two signals $f(t)$ and $g(t)$ is defined as:


$$(f \star g)(t) = \int_{-\infty}^{\infty} f(\tau) g(t + \tau) d\tau$$

Here, $g(t + \tau)$ represents the shifted version of $g(t)$. Unlike convolution, the function $g(t)$ is not flipped before integration.


(2) Discrete-Time Cross-Correlation

The discrete 1D cross-correlation of $f[n]$ and $g[n]$ is defined as:


$$(f \star g)[n] = \sum_{k=-\infty}^{\infty} f[k] g[n + k]$$

The kernel $g[n]$ is not flipped; instead, it is directly shifted across the input signal.


Visual Intuition

In [2]:
from IPython.display import YouTubeVideo
YouTubeVideo('vrl1YlCvyQo', width = "560", height = "315")
Out[2]:

Intuition Behind the Operations

(1) Convolution Intuition:

  • The signal $f[n]$ is "processed" by the kernel $g[n]$, which acts as a function that modifies or filters the input.

  • By flipping the kernel, convolution ensures that the output respects the system's causality (i.e., the output depends only on present and past inputs, not future values).


(2) Cross-Correlation Intuition:

  • Cross-correlation measures how similar the input signal is to the kernel at different positions.

  • The larger the output value at a given shift, the more similar the signal segment is to the kernel.



1.3. Convolution in 1D

Let's compute the 1D convolution process step-by-step.


(1) Kernel $[1, 3, 0, -1]$ at Position 1:

The kernel overlaps with the first four values of the input signal: $[1, 3, 2, 3]$.


  • Element-wise multiplication:

$$1 \times 1 + 3 \times 3 + 0 \times 2 + (-1) \times 3 = 1 + 9 + 0 - 3 = 7 $$
  • The first output value is 7.

(2) Kernel $[1, 3, 0, -1]$ Shifted by 1 Position:

The kernel now overlaps with the next segment of the input: $[3, 2, 3, 0]$.


  • Element-wise multiplication:

$$ 1 \times 3 + 3 \times 2 + 0 \times 3 + (-1) \times 0 = 3 + 6 + 0 + 0 = 9$$
  • The second output value is 9.


(3) This process repeats until the kernel has slid across the entire input signal.




1.4. Image

In computer vision and machine learning, images are represented as numerical data.


Grayscale Image

A grayscale image is essentially a 2D matrix, where each element corresponds to the intensity of a pixel, typically ranging from 0 (black) to 255 (white) for 8-bit images.



Each number in this matrix tells us how bright or dark a specific pixel is. This numerical representation allows us to perform mathematical operations - such as convolution, filtering, and transformation - to extract features or train neural networks.


Colored Image

Unlike grayscale images, which are represented as 2D matrices, colored images are represented as 3D tensors. Each color image is composed of three separate channels:

  • Red
  • Green
  • Blue

Each channel is itself a 2D matrix containing pixel intensities for that specific color. Together, they form a 3D array.