The Backpropagation Algorithm

Simple neural network

On the right, you see a neural network with one input, one output node and two hidden layers of two nodes each.

Nodes in neighboring layers are connected with weights , which are the network parameters.

Activation function

Each node has a total input , an activation function and an output . has to be a non-linear function, otherwise the neural network will only be able to learn linear models.

A commonly used activation function is the Sigmoid function: .

Error function

The goal is to learn the weights of the network automatically from data such that the predicted output is close to the target for all inputs .

To measure how far we are from the goal, we use an error function E. A commonly used error function is

Forward propagation

We begin by taking an input example (, ) and updating the input layer of the network.

For consistency, we consider the input to be like any other node but without an activation function, so its output is equal to its input, i.e. .

Forward propagation

Now, we update the first hidden layer. We take the output of the nodes in the previous layer and use the weights to compute the input of the nodes in the next layer.

Forward propagation

Then we update the output of the nodes in the first hidden layer. For this we use the activation function, .

Forward propagation

Using these 2 formulas we propagate for the rest of the network and get the final output of the network.

Error derivative

The backpropagation algorithm decides how much to update each weight of the network after comparing the predicted output with the desired output for a particular example. For this, we need to compute how the error changes with respect to each weight .

Once we have the error derivatives, we can update the weights using a simple update rule:

where is a positive constant, the learning rate, which we fine-tune empirically.

[Note] If the error goes down when the weight increases (), increase the weight; if the error goes up when the weight increases (), decrease the weight.

Additional derivatives

To help compute , we additionally store for each node two more derivatives — how the error changes with:

• the total input of the node , and
• the output of the node .

Back propagation

Let's begin backpropagating the error derivatives. Since we have the predicted output of this particular input example, we can compute how the error changes with that output. Given our error function we have:

Back propagation

Now that we have we can get using the chain rule.

where when is the Sigmoid activation function.

Back propagation

As soon as we have the error derivative with respect to the total input of a node, we can get the error derivative with respect to the weights coming into that node.

Back propagation

And using the chain rule, we can also get from the previous layer. We have made a full circle.

Back propagation

All that is left to do is repeat the previous three formulas until we have computed all the error derivatives.

The end.

Feed the next example and repeat: forward pass → error → backward pass → update. Over many iterations the error shrinks and the network learns.