Convolutional Neural Network Architecture

The network usually consists of convolutional layer, pooling layer, and fully-connected layer.

• Convolutional layer (input layer – filter – feature map)
  • Filter parameters: number of filters, stride, filter size and amount of zero padding.
  • Learning parameters (weights and biases)

• Activation functions
  • Sigmoid: σ(x)=1/((1+e^(-x)))
  • Rectified linear unit (ReLU): f(x)=max⁡(0,x)
  • Tanh: tanh⁡(x)=2σ(2x)-1

• Pooling layer
  • Pooling parameters: number of filters, filter size, pooling ratio and amount of zero padding.
  • Sensitivity irrelevant
  • Common pooling methods are max, average, and sum.

• Loss function
  • Softmax classifier: L_i=-log(e^(s_(y_i ) )/(∑1_j▒e^(s_j ) ))

where s_j is all incorrect class score and s_(y_i ) is correct class score.

Convolutional Neural Network Training

The network training is a minimizing the loss function process. In other words, it is a process that optimizes weights and biases so that they lead to minimized loss.

• Optimization approach
  • Following and computing the gradients (the slope)

• Backpropagation: computing gradients technique
• Gradient descent: repeatedly evaluating the gradient and then performing a parameter update.
  • Mini-batch Gradient Descent
  • Stochastic Gradient Descent update