§ 4. Deep Neural Networks¶
Deep Neural Networks¶
Concept 4.1 : LR can be seen as 1-layer (shallow) neural network.
In LR, we solve
where \(\ell\left(y_{1}, y_{2}\right)=\log \left(1+e^{-y_{1} y_{2}}\right)\) and \(f_{\theta}\) is linear.
We can view \(f_{\theta}(x)=O=a^{\top} x+b\) as a 1-layer (shallow) neural network.
Concept 4.2 : Deep Neural Networks with Nonlinearities
What happens if we stack multiple linear layers?
Problem: This is pointless because composition of linear functions is linear.
( \(O = W_2 h = W_2(W_1 x) = (W_2 W_1) x \leftarrow\) linear in \(x\) )
Solution: use a nonlinear activation function \(\sigma\) to inject nonlinearities.
Output layer \(O=W_2 h = W_2 \sigma(W_1 x) \leftarrow\) nonlinear in \(x\)
Definition 4.3 : Common Activation Functions
-
Rectified Linear Unit (ReLU)
\[ \operatorname{ReLU}(z)=\max (z, 0) \]
-
Sigmoid
\[ \operatorname{Sigmoid}(z)=\frac{1}{1+e^{-z}} \]
-
Hyperbolic Tangent
\[ \tanh (z)=\frac{1-e^{-2 z}}{1+e^{-2 z}} \]
Definition 4.4 : Multilayer Perceptron (MLP)
The multilayer perceptron, also called fully connected neural network, has the form
where \(x \in \mathbb{R}^{n_{0}}, W_{\ell} \in \mathbb{R}^{n_{\ell} \times n_{\ell-1}}, b_{\ell} \in \mathbb{R}^{n_{\ell}}\), and \(n_{L}=1\). To clarify, \(\sigma\) is applied element-wise.
Definition 4.5 : Linear Layer (with Batches)
- Input tensor: \(X \in \mathbb{R}^{B \times n}, B\) batch size, \(n\) number of indices.
- Output tensor: \(Y \in \mathbb{R}^{B \times m}, B\) batch size, \(m\) number of indices.
With weight \(A \in \mathbb{R}^{m \times n}\), bias \(b \in \mathbb{R}^{m}, k=1, \ldots B\), and \(i=1, \ldots, m\) :
Operation is independent across elements of the batch.
If bias=False, then \(b=0\).
Multi-Class Classification¶
Definition 4.6 : Multi-Class Classification Problem
Consider supervised learning with data \(X_{1}, \ldots, X_{N} \in \mathbb{R}^{n}\) and labels \(Y_{1}, \ldots, Y_{N} \in\{1, \ldots, k\}\). (A \(k\) class classification problem.) Assume there exists a function \(f_{\star}: \mathbb{R}^{n} \rightarrow \Delta^{k}\) mapping from data to label probabilities. Here, \(\Delta^{k} \subset \mathbb{R}^{k}\) denotes the set of probability mass functions on \(\{1, \ldots, k\}\).
Define the empirical distribution \(\mathcal{P}(y) \in \mathbb{R}^{k}\) as the one-hot vector:
for \(i=1, \ldots, k\).
Definition 4.7 : Softmax Function
Softmax function \(\mu: \mathbb{R}^{k} \rightarrow \Delta^{k}\) is defined by
for \(i=1, \ldots, k\), where \(z=\left(z_{1}, \ldots, z_{k}\right) \in \mathbb{R}^{k}\). Since
Name "softmax" is a misnomer. "Softargmax" would be more accurate
- \(\mu(z) \not \approx \max (z)\)
- \(\mu(z) \approx \operatorname{argmax}(z)\)
Examples:
Definition 4.8 : Softmax Regression (SR)
In softmax regression (SR):
Choose the model
Minimize KL-Divergence (or cross entropy) from the model \(\mu\left(f_{A, b}\left(X_{i}\right)\right)\) output probabilities to the empirical distribution \(\mathcal{P}\left(Y_{i}\right)\).
Concept 4.9 : Other Expression of Softmax Regression
Definition 4.10 : Cross Entropy Loss
Where \(f \in \mathbb{R}^{k}, y \in \{1, 2, \cdots, k\}\), the cross entropy loss is
Concept 4.11 : SR uses cross entropy loss as loss function.
- SR = linear model \(f_{A, b}\) with cross entropy loss:
-
The natural extension of SR is to consider
\[ \underset{\theta \in \mathbb{R}^{p}}{\operatorname{minimize}} \frac{1}{N} \sum_{i=1}^{N} \ell^{\mathrm{CE}}\left(f_{\theta}\left(X_{i}\right), Y_{i}\right) \Leftrightarrow \underset{\theta \in \mathbb{R}^{p}}{\operatorname{minimize}} \quad \sum_{i=1}^{N} D_{\mathrm{KL}}\left(\mathcal{P}\left(Y_{i}\right) \| \mu\left(f_{\theta}\left(X_{i}\right)\right)\right) \]where \(f_{\theta}\) is a deep neural network.
GPUs in Deep Learning¶
Concept 4.12 : History of GPU Computing
Rendering graphics involves computing many small tasks in parallel. Graphics cards provide many small processors to render graphics.
In 1999, Nvidia released GeForce 256 and introduced programmability in the form of vertex and pixel shaders. Marketed as the first 'Graphical Processing Unit (GPU)'.
Researchers quickly learned how to implement linear algebra by mapping matrix data into textures and applying shaders.
In 2007, Nvidia released 'Compute Unified Device Architecture (CUDA)', which enabled general purpose computing on a CUDA-enabled GPUs.
Unlike CPUs which provide fast serial processing, GPUs provide massive parallel computing with its numerous slower processors.
The 2008 financial crisis hit Nvidia very hard as GPUs were luxury items used for games. This encouraged Nvidia to invest further in 'General Purpose GPUs (GPGPU)' and create a more stable consumer base.
Concept 4.15 : GPUs for Machine Learning
Raina et al.'s 2009* paper demonstrated that GPUs can be used to train large neural networks. (This was not the first to use of GPUs in machine learning, but it was one of the most influential.)
Modern deep learning is driven by big data and big compute, respectively provided by the internet and GPUs.
Krizhevsky et al.'s 2012 landmark paper introduced AlexNet trained on GPUs and kickstarted the modern deep learning boom.
(R. Raina, A. Madhavan, and A. Y. Ng , Large-scale Deep Unsupervised Learning using Graphics Processors, ICML, 2009. / A. Krizhevsky, I. Sutskever, G. E. Hinton, ImageNet Classification with Deep Convolutional Neural Networks, NeurIPS, 2012.)
Concept 4.16 : Deep Learning on GPUs
Steps for training neural network on GPU:
-
Create the neural network on CPU and send it to GPU. Neural network parameters stay on GPU.
- Sometimes you load parameters from CPU to GPU.
-
Select data batch (image, label) and send it to GPU every iteration
- Data for real-world setups is large, so keeping all data on GPU is infeasible.
-
On GPU, compute network output (forward pass)
- On GPU, compute gradients (backward pass)
- On GPU, perform gradient update
-
Once trained, perform prediction on GPU.
- Send test data to GPU.
- Compute network output.
- Retrieve output on CPU.
- Alternatively, neural network can be loaded on CPU and prediction can be done on CPU.