Neural Network Linear Algreba
Prior to this topic is the Perceptron and Multilayer Perceptron. This is information about linear algebra specific to Neural Networks.
Vectors
In an 2-dimensional space, a vector is simply a line, it has a direction and a magnitude. We can represent this vector as: [x y].
In a 3-dimensional space the vector would be three directions and magnitudes: [x y z].
In 4, 5, ... up to N dimensional space, a vector would be a single list of direction and magnitudes, one for each dimension to represent itself.
I have worked with vectors often in game development. A 2D game may have a vector of [1.3 -7.0] which could be the direction and speed of a bullet. The two values constitute a direction in an x/y coordinate system and their magnitude is like their force. These vector values can be normalized to within a range of -1 to 1 to get only their direction without their magnitude of force.
Note these are other ways we can notate a vector:
[x y]
[x] // one tall square bracket on each side
[y]
(x) // one tall pair of parenthsis
(y)Vector Operations
1) Scalar
Each element is multiplied by the scalar and produces another vector.
[2] * 2 = [4]
[3] [6]2) Element Wise
Each element is added to another vector and produces another vector.
[2] + [-1] = [1]
[3] [ 5] [8]3) Dot Product
Two vectors multiplied together and summed to produce a scalar.
Each element in the same row multiplied together. The result of each row added together.
[2] . [-1] = ((2 * -1) + (3 * 5)) = 13
[3] [ 5]Matrix
A matrix, instead of a linear list of values, is a 2-dimensional grid of values. They are referred to by rows x columns.
This is an example of a 2x3 matrix:
[a b c]
[d e f]Here is an example of a matrix implementation in Javascript
function Matrix(rows, cols) {
this.rows = rows;
this.cols = cols;
this.matrix = [];
// Initialize our 2d array with 0
for (var i=0; i < rows; i++) {
this.matrix[i] = [];
for (var j=0; j < cols; j++) {
this.matrix[i][j] = 0;
}
}
}
var m = new Matrix(2, 3);Matrix Operations
1) Scalar
Each element is multiplied by the scalar and produces a matrix of the same size.
[ 2 3] * 2 = [ 4 6]
[-4 9] [-8 18]
[ 2 3] + 2 = [ 4 5]
[-4 9] [-2 11]Here is an implementation of this method in our Javascript example.
// Multiplication
Matrix.prototype.scalarMultiply = function (scalar) {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] *= scalar;
}
}
}
// Addition
Matrix.prototype.scalarAdd = function (scalar) {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += scalar;
}
}
}2) Element Wise
Each element is added to the corresponding element in the matrices producting a matrix of the same size.
[a b] + [e f] = [a+e b+f]
[c d] [g h] [x+g d+h]We can update our scalarMultiple and scalarAdd to work with other matrices.
// Multiplication
Matrix.prototype.multiply = function (n) {
if (n instanceof Matrix) {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] *= n.matrix[i][j]; // element wise (Hadamard Product)
}
}
} else {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += n // scalar
}
}
}
}
// Addition
Matrix.prototype.add = function (n) {
if (n instanceof Matrix) {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += n.matrix[i][j]; // element wise
}
}
} else {
for (var i=0; i < rows; i++) {
for (var j=0; j < cols; j++) {
this.matrix[i][j] += n // scalar
}
}
}
}4) Dot Product
A dot product can only be calculated on two matrices of the same size. The number of columns of one must equal the number of rows of the other.
A = [a b c] 2x3
[d e f]
B = [g h] 3x2
[i j]
[k l]
A . BThese two matrices can be dot producted'ed. But they are not communicative, meaning they cannot be multiplied in either order and produce the same result, A . B != B . A.
The resulting matrix will always be the number of rows of A by the number of colums of B. A . B = 2x2 Matrix.
To calculate the values of the resultant 2x2 matrix we will perform the Vector Dot Product of each row of A to its corresponding column of B.
- Row 1 in A dot Column 1 in B for position 1,1.
- Row 1 in A dot Column 2 in B for position 1,2.
- Row 2 in A dot Column 1 in B for position 2,1.
- Row 2 in A dot Column 2 in B for position 2,2.
A = [a b c] 2x3
[d e f]
B = [g h] 3x2
[i j]
[k l]
A . B = C
C = [p11 p12]
[p21 p22]
C = [ (a*g + b*i + c*k) (a*h + b*j + c*l) ]
[ (d*g + e*i + f*k) (d*h + e*j + f*l) ]We can expand our Javascript example Matrix to have a dot product operation.
Matrix.prototype.dot = function (n) {
if (n instanceof Matrix) {
// check size
if (this.cols !== n.rows) {
return undefined;
}
let result = new Matrix(this.rows, n.cols);
for (let i=0; i < result.rows; i++) {
for (let j=0; j < result.cols; j++) {
// dot product of values in col
let sum = 0;
for (let k=0; k < this.cols; k++) {
sum += this.matrix[i][k] * n.matrix[k][j];
}
result.matrix[i][j] = sum;
}
}
return result;
} else {
return undefined;
}
}5) Transpose
Another important operation for matrices in neural networks is the transpose operation. It basically makes the rows the columns and columns the rows.
[a b c]
[d e f]
transposed becomes:
[a d]
[b e]
[c f]And we can implement in our Javacript Matrix as:
Matrix.prototype.transpose = function() {
let result = new Matrix(this.cols, this.rows);
for (var i=0; i < this.rows; i++) {
for (var j=0; j < this.cols; j++) {
result.matrix[j][i] = this.matrix[i][j];
}
}
return result;
}