Types of Neural Network

Computer Vision

• Image Classification

Computer Vision

• Object Detection

Computer Vision

• Neural Style Transfer

Image Data

Image Data

Convolutions

Edge Detection

• Vertical Edge Detection

Parameters

Padding

• Pad so that output size is the same as the input size.

Strided convolutions

• n×n (n=7)
• f×f (f=3)
• no padding
• stride s=2

Summary of Convolutions

• n×n image; f×f filter
• padding p; stride s
• Output: [n+2p−fs+1]×[n+2p−fs+1]

Convolutions Over Volume

• https://www.youtube.com/watch?v=HzuFgLnhgqw

Your Turn: Number of Parameters in One Layer

Question: If you have 10 filters that are 3×3×3 in one layer of a neural network, how many parameters does that layer have?

Summary of Notation

If layer l is a convolution layer:

• f[l] = filter size
• p[l] = padding
• s[l] = stride
• n[l]C = number of filters
• Each filter: f[l]×f[l]×n[l−1]C
• Activations: a[l]→n[l]H×n[l]W×n[l]C
• Weights: f[l]×f[l]×n[l−1]C×n[l]C
• bias: b[l]C
• Input: n[l−1]H×n[l−1]W×n[l−1]C
• Output: n[l]H×n[l]W×n[l]C
• n[l]H=n[l−1]W+2p[l]−f[l]s[l]+1 (similar for W)

Pooling Layers

Pooling Layers

Types of Layer in A Convolutional Network

• Convolution

• Pooling

• Fully Connected

Classic CNNs: LeNet-5

• LeNet-5: LeCun et al., 1998. Gradient-based learning applied to document recognition

Classic CNNs: AlexNet

• AlexNet: Krizhevsky et al., 2012. ImageNet Classification with Deep Convolutional Neural Networks

Classic CNNs: VGG-16

• VGG-16: Simonyan & Zisserman 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition

Classic CNNs: ResNets

• ResNets: He et al., 2015. Deep Residual Learning for Image Recognition

Using Keras To Build CNN

Typical keras workflow:

1. Define your training data: input tensors and target tensors
2. Define a network of layers (or models) that maps your inputs to your targets
3. Configure the learning process by choosing a loss function, an optimizer, and some metrics to monitor
4. Iterate on your training data by calling the fit() method of your model

Using Keras To Build CNN

# Define model structure
cnn_model <- keras_model_sequential() %>%
  layer_conv_2d(filters = 32, kernel_size = c(3, 3), 
  activation = "relu", input_shape = input_shape) %>%
  layer_max_pooling_2d(pool_size = c(2, 2)) %>%
  layer_conv_2d(filters = 64, kernel_size = c(3, 3), activation = "relu") %>%
  layer_dropout(rate = 0.25) %>%
  layer_flatten() %>%
  layer_dense(units = 128, activation = "relu") %>%
  layer_dropout(rate = 0.5) %>%
  layer_dense(units = num_classes, activation = "softmax")

Using Keras To Build CNN

• Compile and define loss function, optimizer and metrics to monitor during the training

# Compile model
cnn_model %>% compile(
  loss = loss_categorical_crossentropy,
  optimizer = optimizer_adadelta(),
  metrics = c('accuracy')
)

Using Keras To Build CNN

• Fit the model using training dataset and define epochs, batch size and validation data

# Train model
cnn_history <- cnn_model %>%
  fit(
    x_train, y_train,
    batch_size = batch_size,
    epochs = epochs,
    validation_split = 0.2
  )

• Predict new outcomes using the trained model

# Model prediction
cnn_pred <- cnn_model %>%
  predict_classes(x_test)

Size of the Model

Different Architecture Search Algorithms:

• NASnet: 1800 GPU days (5 yrs on 1 GPU)

• AmoebaNet: 3150 GPU days

• DARTS: 4 GPU days

• ENAS: 1000 x cheaper than standard NAS

Understanding Neural Networks

• Understanding Neural Networks Through Deep Visualization, Video

• Deep Neural Networks are Easily Fooled, Video