Backpropagation in neural Network is vital for applications like image recognition, language processing and more.

Neural networks have shown significant advancements in recent years. From facial recognition tools in smartphone Face ID, to self driving cars, the applications of neural networks have influenced every industry.

This subset of machine learning is comprised of node layers, containing an input layer, one or more hidden layers, and an output layer. Each node is interconnected like human brain and has an associated weight and threshold. Suppose the output value of a node is higher than the specified threshold value, it implies that the node is activated and ready to relay data to the next layer of the neural network. There are various activation functions like Threshold function, Piecewise linear function or Sigmoid function. Further, the activation value of a neuron is calculated with several components, as it represents the weighted sum of the inputs. Its formula is,

Activation = sum (weight * input) + bias

While most deep neural networks are feedforward, i.e., they flow in one direction only, from input to output (like in feedforward neural net), one can also train a neural net model to move in the opposite direction from output to input. This is possible through backpropagation.

Backpropagation: What, Why and How?

Backpropagation is a popular method for training artificial neural networks, especially deep neural networks. It refers to the method of fine-tuning the weights of a neural network on the basis of the error rate obtained in the previous iteration. It was first introduced in 1960s and almost 30 years later (1989) popularized by David Rumelhart, Geoffrey Hinton and Ronald Williams in a paper called "Learning representations by back-propagating errors".

As per an article on Quanta Magazine, backpropagation works in two phases. In the forward phase (forward pass), when the network is given an input, it infers an output, which may be erroneous. The second backward phase (backward pass) updates the synaptic weights using gradient descent algorithms, or other more advanced optimization techniques, bringing the output more in line with a target value. This allows neural network developers to calculate and attribute the error associated with each neuron, to adjust and fit the parameters of the model appropriately – thus enabling to create a more reliable model by increasing its generalization.

According to the paper from 1989, backpropagation 'repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector.' So basically, for carrying backpropagation in neural network, one has to train the model in the supervised learning method, where the error between the system's output and a known expected output is presented to the model and used to modify its internal state. Then, one must update the weights to get the global loss minimum. The method that is used to update the weights of the network is based in the chain-rule. A simplified chain rule formula for backpropagation partial derivatives looks something like this:

The weights are updated either after every sample in training set, in batch or in randomized mini batches.

Types

There are two main types of backpropagation, viz.,

• Static Backpropagation: This produces a mapping of a static input for static output. Here mapping is quite rapid.
• Recurrent Backpropagation: In this, neural network is fed forward until a fixed value is achieved. After that, the error is computed and propagated backward. Here mapping is non-static.

Do we need it?

Backpropagation is an integral part of today's neural networks. Without it, neural networks won't be able to carry out tasks like recognizing images and interpreting natural language.

But one of the key problems of backpropagation is that, after the neural network model learned to make predictions from a dataset, it is prone to the risk of forgetting what it learned when given new training data. This phenomenon is called catastrophic forgetting. It also updates the neural network layers sequentially, making it difficult to parallelize the training process and leading to longer training times.

However, one bright side, backpropagation is still necessary as it is simple, fast and easy to program and flexible. Also, since only numbers of the input are tuned and not any other parameter, user need not require any prior knowledge about the network nor the necessity to learn any special functions. Currently, scientists are working to develop advanced neural networks that offset the bottlenecks of backpropagation.