# Neural Network Linear Algreba

Prior to this topic is the Perceptron and Multilayer Perceptron. This is information about linear algebra specific to Neural Networks.

## Vectors

In an 2-dimensional space, a vector is simply a line, it has a direction and a magnitude. We can represent this vector as: `[x y]`.

In a 3-dimensional space the vector would be three directions and magnitudes: `[x y z]`.

In 4, 5, ... up to N dimensional space, a vector would be a single list of direction and magnitudes, one for each dimension to represent itself.

I have worked with vectors often in game development. A 2D game may have a vector of `[1.3 -7.0]` which could be the direction and speed of a bullet. The two values constitute a direction in an x/y coordinate system and their magnitude is like their force. These vector values can be normalized to within a range of -1 to 1 to get only their direction without their magnitude of force.

Note these are other ways we can notate a vector:

``````[x y]

[x]  // one tall square bracket on each side
[y]

(x)  // one tall pair of parenthsis
(y)``````

### Vector Operations

#### 1) Scalar

Each element is multiplied by the scalar and produces another vector.

``````[2] * 2  = [4]
[3]        [6]``````

#### 2) Element Wise

Each element is added to another vector and produces another vector.

``````[2] + [-1] = [1]
[3]   [ 5]   [8]``````

#### 3) Dot Product

Two vectors multiplied together and summed to produce a scalar.

Each element in the same row multiplied together. The result of each row added together.

``````[2] . [-1] =  ((2 * -1) + (3 * 5)) = 13
[3]   [ 5]``````

## Matrix

A matrix, instead of a linear list of values, is a 2-dimensional grid of values. They are referred to by rows x columns.

This is an example of a 2x3 matrix:

``````[a b c]
[d e f]``````

Here is an example of a matrix implementation in Javascript

``````function Matrix(rows, cols) {
this.rows = rows;
this.cols = cols;
this.matrix = [];
// Initialize our 2d array with 0
for (var i=0; i < rows; i++) {
this.matrix[i] = [];
for (var j=0; j < cols; j++) {
this.matrix[i][j] = 0;
}
}
}
var m = new Matrix(2, 3);``````

### Matrix Operations

#### 1) Scalar

Each element is multiplied by the scalar and produces a matrix of the same size.

``````[ 2 3] * 2 = [ 4  6]
[-4 9]       [-8 18]

[ 2 3] + 2 = [ 4  5]
[-4 9]       [-2 11]``````

Here is an implementation of this method in our Javascript example.

``````// Multiplication
Matrix.prototype.scalarMultiply = function (scalar) {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] *= scalar;
}
}
}
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += scalar;
}
}
}``````

#### 2) Element Wise

Each element is added to the corresponding element in the matrices producting a matrix of the same size.

``````[a b] + [e f] = [a+e b+f]
[c d]   [g h]   [x+g d+h]``````

We can update our scalarMultiple and scalarAdd to work with other matrices.

``````// Multiplication
Matrix.prototype.multiply = function (n) {
if (n instanceof Matrix) {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] *= n.matrix[i][j]; // element wise (Hadamard Product)
}
}
} else {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += n // scalar
}
}
}
}
if (n instanceof Matrix) {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += n.matrix[i][j]; // element wise
}
}
} else {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += n // scalar
}
}
}
}``````

#### 4) Dot Product

A dot product can only be calculated on two matrices of the same size. The number of columns of one must equal the number of rows of the other.

``````A = [a b c]  2x3
[d e f]

B = [g h]  3x2
[i j]
[k l]

A . B``````

These two matrices can be dot producted'ed. But they are not communicative, meaning they cannot be multiplied in either order and produce the same result, `A . B != B . A`.

The resulting matrix will always be the number of rows of A by the number of colums of B. `A . B = 2x2 Matrix`.

To calculate the values of the resultant 2x2 matrix we will perform the Vector Dot Product of each row of A to its corresponding column of B.

• Row 1 in A dot Column 1 in B for position 1,1.
• Row 1 in A dot Column 2 in B for position 1,2.
• Row 2 in A dot Column 1 in B for position 2,1.
• Row 2 in A dot Column 2 in B for position 2,2.
``````A = [a b c]  2x3
[d e f]

B = [g h]  3x2
[i j]
[k l]

A . B = C

C = [p11 p12]
[p21 p22]

C = [ (a*g + b*i + c*k) (a*h + b*j + c*l) ]
[ (d*g + e*i + f*k) (d*h + e*j + f*l) ]``````

We can expand our Javascript example Matrix to have a dot product operation.

``````Matrix.prototype.dot = function (n) {

if (n instanceof Matrix) {
// check size
if (this.cols !== n.rows) {
return undefined;
}

let result = new Matrix(this.rows, n.cols);
for (let i=0; i < result.rows; i++) {
for (let j=0; j < result.cols; j++) {
// dot product of values in col
let sum = 0;
for (let k=0; k < this.cols; k++) {
sum += this.matrix[i][k] * n.matrix[k][j];
}
result.matrix[i][j] = sum;
}
}
return result;
} else {
return undefined;
}
}``````

#### 5) Transpose

Another important operation for matrices in neural networks is the transpose operation. It basically makes the rows the columns and columns the rows.

``````[a b c]
[d e f]

transposed becomes:

[a d]
[b e]
[c f]``````

And we can implement in our Javacript Matrix as:

``````Matrix.prototype.transpose = function() {
let result = new Matrix(this.cols, this.rows);
for (var i=0; i < this.rows; i++) {
for (var j=0; j < this.cols; j++) {
result.matrix[j][i] = this.matrix[i][j];
}
}
return result;
}
``````