§ 8. CNNs for Other Supervised Learning Tasks¶
Inverse Problem¶
Definition 8.1 : Inverse Problem Model
In inverse problems, we wish to recover a signal \(X_{\text {true }}\) given measurements \(Y\). The unknown and the measurements are related through
where \(\mathcal{A}\) is often, but not always, linear, and \(\varepsilon\) represents small error.
The forward model \(\mathcal{A}\) may or may not be known. In other words, the goal of an inverse problem is to find an approximation of \(\mathcal{A}^{-1}\).
In many cases, \(\mathcal{A}\) is not even be invertible. In such cases, we can still hope to find an mapping that serves as an approximate inverse in practice.
Concept 8.2 : Inverse Problems via Deep Learning
In deep learning, we use a neural network to approximate the inverse mapping
i.e., we want \(f_{\theta}(Y) \approx X_{\text {true }}\) for the measurements \(X\) that we care about.
If we have \(X_{1}, \ldots, X_{N}\) and \(Y_{1}, \ldots, Y_{N}\) (but no direct knowledge of \(\mathcal{A}\) ), we can solve
If we have \(X_{1}, \ldots, X_{N}\) and knowledge of \(\mathcal{A}\), we can solve
If we have \(Y_{1}, \ldots, Y_{N}\) and knowledge of \(\mathcal{A}\), we can solve
Gaussian Denoising¶
Definition 8.3 : Gaussian Denoising
Given \(X_{\text {true }} \in \mathbb{R}^{w \times h}\), we measure
where \(\varepsilon_{i j} \sim \mathcal{N}\left(0, \sigma^{2}\right)\) is IID Gaussian noise. For the sake of simplicity, assume we know \(\sigma\). Goal is to recover \(X_{\text {true }}\) from \(Y\).
Gaussian denoising is the simplest setup in which the goal is to remove noise from the image. In more realistic setups, the noise model will be more complicated and the noise level \(\sigma\) will be unknown.
In 2017, Zhang et al. presented the denoising convolutional neural networks (DnCNNs). They trained a 17-layer CNN \(f_{\theta}\) to learn the noise with the loss
so that the clean recovery can be obtained with \(Y_{i}-f_{\theta}\left(Y_{i}\right)\). (This is equivalent to using a residual connection from beginning to end.)
Image denoising is was an area with a large body of prior work. DnCNN dominated all prior approaches that were not based on deep learning.
Nowadays, all state-of-the-art denoising algorithms are based on deep learning.
(K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising, IEEE TIP, 2017.)
Image Super-Resolution¶
Definition 8.5 : Image Super-Resolution
Given \(X_{\text {true }} \in \mathbb{R}^{w \times h}\), we measure
where \(\mathcal{A}\) is a "downsampling" operator. So \(Y \in \mathbb{R}^{w_{2} \times h_{2}}\) with \(w_{2}<w\) and \(h_{2}<h\). Goal is to recover \(X_{\text {true }}\) from \(Y\).
In the simplest setup, \(\mathcal{A}\) is an average pool operator with \(r \times r\) kernel and a stride \(r\).
In 2015, Dong et al. presented super-resolution convolutional neural network (SRCNN). They trained a 3-layer \(\operatorname{CNN} f_{\theta}\) to learn the high-resolution reconstruction with the loss
where \(\tilde{Y}_{i} \in \mathbb{R}^{w \times h}\) is an upsampled version of \(Y_{i} \in \mathbb{R}^{(w / r) \times(h / r)}\), i.e., \(\tilde{Y}_{i}\) has the same number of pixels as \(X_{i}\), but the image is pixelated or blurry. The goal is to have \(f_{\theta}\left(\tilde{Y}_{i}\right)\) be a sharp reconstruction.
SRCNN showed that simple learning based approaches can match the state-of the art performances of superresolution task.
(C. Dong, C. C. Loy, K. He, and X. Tang, Image super-resolution using deep convolutional networks, IEEE TPAMI, 2015.)
In 2016, Kim et al. presented VDSR. They trained a 20-layer CNN with a residual connection \(f_{\theta}\) to learn the high-resolution reconstruction with the loss
The residual connection was the key insight that enabled the training of much deeper CNNs.
VDSR dominated all prior approaches not based on deep learning. Showed that simple learning based approaches can batch the state-of theart performances of super-resolution task.
(J. Kim, J. K. Lee, and K. M. Lee, Accurate image super-resolution using very deep convolutional networks, CVPR, 2016.)
Other Examples¶
(C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, and W. Shi, Photo-realistic single image super-resolution using a generative adversarial network, CVPR, 2017.)
Example 8.9 : Image Colorization
(R. Zhang, P. Isola, and A. A. Efros, Colorful image colorization, ECCV, 2016.)
Example 8.10 : Image Inpainting
(J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, Generative image inpainting with contextual attention, CVPR, 2018.)
Operations Increasing Spatial Dimensions¶
Concept 8.11 : Operations Increasing Spatial Dimensions
In image classification tasks, the spatial dimensions of neural networks often decrease as the depth progresses.
This is because we are trying to forget location information. (In classification, we care about what is in the image, but we do not where it is in the image.)
However, there are many networks for which we want to increase the spatial dimension:
- Linear layers
- Upsampling
- Transposed convolution
Transposed convolution¶
Concept 8.12 : Linear Operator \(\cong\) Matrix
Core tenet of linear algebra: matrices are linear operators and linear operators are matrices.
Let \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}\) be linear, i.e.,
for all \(x, y \in \mathbb{R}^{n}\) and \(\alpha \in \mathbb{R}\).
There exists a matrix \(A \in \mathbb{R}^{m \times n}\) that represents \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}\), i.e.,
for all \(x \in \mathbb{R}^{n}\).
Let \(e_{i}\) be the \(i\)-th unit vector, i.e., \(e_{i}\) has all zeros elements except entry 1 in the \(i\)-th coordinate.
Given a linear \(f: \mathbb{R}^{n} \rightarrow \mathbb{R}^{m}\), we can find the matrix
representing \(f\) with
for all \(j=1, \ldots, n\), or with
for all \(i=1, \ldots, m\) and \(j=1, \ldots, n\).
Concept 8.13 : Linear Operator \(\ncong\) Matrix
In applied mathematics and machine learning, there are many setups where explicitly forming the matrix representation \(A \in \mathbb{R}^{m \times n}\) is costly, even though the matrix-vector products \(A x\) and \(A^{\top} y\) are efficient to evaluate.
In machine learning, convolutions are the primary example. Other areas, linear operators based on FFTs are the primary example.
In such setups, the matrix representation is still a useful conceptual tool, even if we never intend to form the matrix.
Given a matrix \(A\), the transpose \(A^{\top}\) is obtained by flipping the row and column dimensions, i.e., \(\left(A^{\top}\right)_{i j}=(A)_{j i}\). However, using this definition is not always the most effective when understanding the action of \(A^{\top}\).
Another approach is to use the adjoint view. Since
for any \(x \in \mathbb{R}^{n}\) and \(y \in \mathbb{R}^{m}\), understand the action of \(A^{\top}\) by finding an expression of the form
Example 8.14 : 1D Transpose Convolution
Consider the 1D convolution represented by \(A \in \mathbb{R}^{(n-f+1) \times n}\) defined with a given \(w \in \mathbb{R}^{f}\) and
Then we have
and we have the following formula which coincides with transposing the matrix \(A\).
Definition 8.15 : Transposed Convolution
In transposed convolution, input neurons additively distribute values to the output via the kernel. Before people noticed that this is the transpose of convolution, the names backwards convolution and deconvolution were used.
For each input neuron, multiply the kernel and add (accumulate) the value in the output. Can accommodate strides, padding, and multiple channels.
- Convolution Visualized
- Transpose Convolution Visualized
Definition 8.16 : 2D Transpose Convolution Layer (Formal Definition)
- \(B\) : batch size
- \(C_{\text{in}}\) : # of input channels
- \(C_{\text{out}}\) : # of output channels
- \(m, n\) : # of vertical and horizontal indices of input
- \(f_1, f_2\) : # of vertical and horizontal indices of filter
- Input tensor: \(Y \in \mathbb{R}^{B \times C_{\mathrm{in}} \times m \times n}\)
- Output tensor: \(X \in \mathbb{R}^{B \times C_{\text {out }} \times\left(m+f_{1}-1\right) \times\left(n+f_{2}-1\right)}\)
- Filter \(w \in \mathbb{R}^{C_{\text {in }} \times C_{\text {out }} \times f_{1} \times f_{2}}\)
- Bias \(b \in \mathbb{R}^{C_{\text {out }}}\) (If
bias=False, then \(b=0\).)
def trans_conv(Y, w, b):
c_in, c_out, f1, f2 = w.shape
batch, c_in, m, n = Y.shape
X = torch.zeros(batch, c_out, m + f1 - 1, n + f2 - 1)
for k in range(c_in):
for i in range(Y.shape[2]):
for j in range(Y.shape[3]):
X[:, :, i:i+f1, j:j+f2] += Y[:, k, i, j].view(-1,1,1,1)*w[k, :, :, :].unsqueeze(0)
return X + b.view(1,-1,1,1)
In a matrix representation \(A\) of convolution, the dependencies of the inputs and outputs are represented by the non-zeros of \(A\), i.e., the sparsity pattern of \(A\). If \(A_{i j}=0\), then input neuron \(j\) does not affect the output neuron \(i\). If \(A_{i j} \neq 0\), then \(\left(A^{\top}\right)_{j i} \neq 0\). So if input neuron \(j\) affects output neuron \(i\) in convolution, then input neuron \(i\) affects output neuron \(j\) in transposed convolution.
We can combine this reasoning with our visual understanding of convolution. The diagram simultaneously illustrates the dependencies for both convolution and transposed convolution.
Upsampling¶
Torch.nn.Upsample with mode='bilinear'
linear interpolation is available for 1D data
trilinear interpolation is available for 3D data
(We won't pay attention to the interpolation formula.)
Semantic Segmentation¶
Definition 8.19 : Semantic Segmentation
In semantic segmentation, the goal is to segment the image into semantically meaningful regions by classifying each pixel.
Definition 8.20 : Object Localization
Object localization localizes a single object usually via a bounding box.
Definition 8.21 : Object Detection
Object detection detects many objects, with the same class often repeated, usually via bounding boxes.
Definition 8.22 : Image Segmentation
Instance segmentation distinguishes multiple instances of the same object type.
We will focus on semantic segmentation.
We will use PASCAL Visual Object Classes (VOC) dataset for semantic segmentation. (Dataset also contains labels for object detection.)
There are 21 classes: 20 main classes and 1 "unlabeled" class.
Data \(X_{1}, \ldots, X_{N} \in \mathbb{R}^{3 \times m \times n}\) and labels \(Y_{1}, \ldots, Y_{N} \in\{0,1, \ldots, 20\}^{m \times n}\), i.e., \(Y_{i}\) provides a class label for every pixel of \(X_{i}\).
Concept 8.24 : Loss for Semantic Segmentation
Consider the neural network
such that \(\mu\left(f_{\theta}(X)\right)_{i j} \in \Delta^{k}\) is the probabilities for the \(k\) classes for pixel \((i, j)\).
We minimize the sum of pixel-wise cross-entropy losses
where \(\ell^{C E}\) is the cross entropy loss.
The U-Net architecture:
- Reduce the spatial dimension to obtain high-level (coarse scale) features
- Upsample or transpose convolution to restore spatial dimension.
- Use residual connections across each dimension reduction stage.
Definition 8.26 : Magnetic Resonance Imaging (MRI)
Magnetic resonance imaging (MRI) is an inverse problem in which we partially measure the Fourier transform of the patient and the goal is to reconstruct the patient's image.
So \(X_{\text {true }} \in \mathbb{R}^{n}\) is the true original image (reshaped into a vector) with \(n\) pixels or voxels and \(\mathcal{A}\left[X_{\text {true }}\right] \in \mathbb{C}^{k}\) with \(k \ll n\). (If \(k=n\), MRI scan can take hours.)
Classical reconstruction algorithms rely on Fourier analysis, total variation regularization, compressed sensing, and optimization.
Recent state-of-the-art use deep neural networks.
Definition 8.27 : FastMRI Dataset
A team of researchers from Facebook AI Research and NYU released a large MRI dataset to stimulate datadriven deep learning research for MRI reconstruction.
(J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, M. Bruno, M. Parente, K. J. Geras, J. Katsnelson, H. Chandarana, Z. Zhang, M. Drozdzal, A. Romero, M. Rabbat, P. Vincent, N. Yakubova, J. Pinkerton, D. Wang, E. Owens, C. L. Zitnick, M. P. Recht, D. K. Sodickson, and Y. W. Lui, fastMRI: An open dataset and benchmarks for accelerated MRI, arXiv, 2019.)
Definition 8.28 : Computational Tomography (CT)
Computational tomography (CT) is an inverse problem in which we partially measure the Radon transform of the patient and the goal is to reconstruct the patient's image.
So \(X_{\text {true }} \in \mathbb{R}^{n}\) is the true original image (reshaped into a vector) with \(n\) pixels or voxels and \(\mathcal{A}\left[X_{\text {true }}\right] \in \mathbb{R}^{k}\) with \(k \ll n\). (If \(k=n\), the X -ray exposure to perform the CT scan can be harmful.)
Recent state-of-the-art use deep neural networks.
Concept 8.29 : U-Net is used for inverse problems.
Although U-Net was originally proposed as an architecture for semantic segmentation, it is also being used widely as one of the default architectures in inverse problems, including MRI reconstruction.
(J. Zbontar, F. Knoll, A. Sriram, T. Murrell, Z. Huang, M. J. Muckley, A. Defazio, R. Stern, P. Johnson, M. Bruno, M. Parente, K. J. Geras, J. Katsnelson, H. Chandarana, Z. Zhang, M. Drozdzal, A. Romero, M. Rabbat, P. Vincent, N. Yakubova, J. Pinkerton, D. Wang, E. Owens, C. L. Zitnick, M. P. Recht, D. K. Sodickson, and Y. W. Lui, fastMRI: An open dataset and benchmarks for accelerated MRI, arXiv, 2019.)
U-Net is also used as one of the default architectures in CT reconstruction.
(K. H. Jin, M. T. McCann, E. Froustey, and M. Unser, Deep convolutional neural network for inverse problems in imaging, IEEE TIP, 2017.)