Neural Networks on the Arduino Part 4: Calculation of hidden errors in a neural network

Of all the things I found hard to "believe" "understand" or "grok" was how the hidden errors are calculated in a neural network. If you've done matrix maths the equation seems far too simple to be true. Here I'll illustrate to you, and to myself, how it works.

The problem is that while it is clear what an error is at the output of a neural network, it is not immediately clear what an error is at the hidden layer output. Graphically:

It turns out that a good approach is to use the errors at the output to set the "errors" in the hidden layer.

(By the way you can't just copy the output errors into the hidden layer because there is no guarantee that the number of outputs is the same as the number of hidden nodes. Plus the fact that you'd be arbitrarily jumping over the HiddenToOutput matrix.)

So the idea is that the hidden layer errors are weighted averages of the errors at the output. We make the assumption that an error at the output has been caused by an "error" in the hidden layer. How much the error at the output has been caused by the error in the hidden depends on the weight connecting the two nodes (errors).

For example look at eH2 above, it contributed to errors at eO1 and eO2, and how much it contributed depended on the weights wa and wb. So eH2 will be a weighted average of eO1 and eO2. As explained in the book by Tariq Rashid, we do not need to use an actual weighted average, we can just use the weights directly.

In other words...

eH2 = wa*eO1 + wb*eO2

Luckily this can be nicely done with matrices, and the matrix of the weights for this back propagation of errors is easily created from the forward hidden to output matrix.

I'm about to demonstrate that, by the magic of matrices, the matrix which gives us the hidden "errors" from the output errors is simply the transpose of the forward hidden to output matrix! Lets look again at how the forward hidden to output matrix works:

I've arranged the weights closer to the output nodes so you can see the calculation better. Study and understand how o1 is calculated. Compare the drawing and the matrix versions in the above image.

So. That is how the forward calculation of the outputs work. How do we do the backward propagation of the errors using matrix multiplication? Here's another, this time illustrating the backward propagation of errors from the output to the hidden layer:

• I've drawn the errors flowing rightwards, from eo1 to eh1 etc. This is so you can easily relate the diagram to the matrix multiplication.
• The o in eo2 (for example) stands for output
• The h in eh3 (for example) stands for hidden
• I've drawn the weights which modify the output errors to form the hidden "errors" close to the hidden error nodes
• I've drawn the hidden error calculations at top right of the image.
• I've drawn the matrix multiplication version in the bottom half.
• To be sure the weights have the correct indices try checking the weight which connects o1(eo1) to  h2(eh2) in both diagrams. In both diagrams it is w21.
• Note that the matrix in this diagram is the transpose of the matrix in the previous diagram.

E voila! That is why and how the transpose of the final forward matrix can be used to "invent" the hidden "errors" from the output errors.

Part 5 How To Update The Weights

Neural Networks on the Arduino Part 2: A start to changing the weights

In part one (read it before you read this) we saw how to create a neural network function which could run on an Arduino, but we missed out the most important part: how to train the network. Training here means changing the two matrices which connect the three vectors.

We train the network by back propagation of errors, so the network can reduce the errors and learn to get closer to the correct answer. It is called that because

1. The errors goes backwards from the output towards the input 
2. The errors are propagated into the network

We'll be using the MatrixMath.h functions for the Arduino, here is a simple neural network and the matrices and vectors which represent it (remember weights are stored inside matrices):

Ok, so how to we change the weights to get the outputs closer to the targets? The best explanation I have found so far is in the book by Tariq Rashid...

... so I'll follow that, and translating from Python to Arduino C/C++. Here is the function from the book:

Here is the same thing in my blocky matrix type illustration

Stay with me. I had to draw these diagrams several times before I fully understood what I was doing.

The column vector  Oj (inputs to this layer (outputs from the previous layer)) multiplied by f (a row vector) create the matrix which are the changes to add (the deltas to apply) to the 3 x 2 matrix.

(a is the learning rate and is between 0 and 1 exclusive. A high learning rate (more than 0.5) may mean the network will never find a stable set of weights. A low learning rate may mean that it the training takes longer. In this Arduino version I've found that 0.1 is a good setting.)

The deltaW matrix is what we'll add to the original matrix to modify its weights.

HiddenToOutputMatrix, also obviously a 3 x 2. And we can follow a similar reasoning to the InputToHiddenMatrix, actually identical apart from the number of rows and columns. So shouldn't we put that inside a single function which can be called twice?

Look at the original diagram above, we can split it into two halves, and you can see that the architecture is the same, and only the sizes are different:

  First layer, input to hidden, 4 inputs and 3 outputs

Second layer, hidden to output, 3 inputs and 2 outputs

You can see that there are two layers.

As I said, this means is that it makes sense to have a single back propagation function which can be called twice instead of writing it all out twice. 

Note that in the Python implementation by Tariq Rashid the update of the weights by back propagation was done with a single "line" per layer. Here is one of those "lines":

self.who += self.lrate * 

            numpy.dot((output_errors*final_outputs*(1-final_outputs)),

            numpy.transpose(hidden_outputs))

who = weights-hidden-to-output. Why is there a "dot" in the above function? Well actually in Python this is the outer product. With the outer product two vectors produce a matrix, and in our case produces the matrix of changes to apply to the old matrix. For example:

It is always good to keep your feet on the ground when using matrices for neural networks, otherwise, if you're like me, you'll soon lose clarity about what the rows and columns are. 

In this case the number of rows in a matrix in a neural network layer are the number of inputs to the layer, and the number of columns in the matrix is the number of outputs. In images:

In the diagram above you can see a column vector and a row vector multiplied together to form a matrix. You do this using the outer product. Here it is in Arduino C:

// Outer product, from two vectors form a matrix
// C = A*B
// A is a column vector (vertical and has lots of rows)
// B is a row vector (horizontal and has lots of columns)
// C must have space for mRows and nColumns for this to work.
void OuterProduct(mtx_type* A, mtx_type* B, int mRows, int nColumns, mtx_type* C)
{
    int ra, cb;
    for (ra = 0; ra < mRows; ra++) {
        for(cb = 0; cb < nColumns; cb++)
        {
            // C[ra][cb] = C[ra][cb] + (A[ra] * B[cb]);
            C[(nColumns * ra) + cb] = A[ra] * B[cb];
        }
    }
} 

(The Arduino language is a sort of reduced C/C++ by the way, as far as I can understand. On the other hand the Arduino is so low-cost you can imagine putting Arduino neural networks to work anywhere.)

And here's Part 3: Details and tests of the matrix multiplication.