Simulating a Neural network with python libraries without using any framework(tensorflow, pytorch)
We will be using numpy and pandas to simulate neural network to learn basic gates like AND, OR and XOR.
1.Importing required libraries-
2.Creating a dataframe from dictionary.
3. Initializing random weights.
4. We are writing a simple activation function a lambda function with value as input and weights.
np.where(np.dot(inputs,weights)>0 , 1, 0) = This will output to 1 if dot product of inputs and weights > 0 ,otherwise 0. One of the purpose of activation function is to normalize the data, that’s what we are doing here scaling between 0 and 1.
5. Writing a training function which will calculate difference between predicted and expected output.
eta = learning rate
6.Let’s test it first on the AND function.
As we can see above it’s able to predict correct values in 6th iteration. In remaining 14 iterations it will predict same output. As there is no change in precited and expected after that.
No weights will be updated after. Means it has learnt weights that represents the relationship in data and label.
7. According to tensorboard recommended learning rate is between 0.0001–10.
What if we increase or decrease learning rate?
Increasing learning rate will cause network to reach global minima faster.
Decreasing learning rate will cause network to reach global minima slowly.
The learning rate is perhaps the most important hyperparameter. If you have time to tune only one hyperparameter, tune the learning rate.
— Page 429, Deep Learning, 2016.
Let’s test this out in our train function-
Increasing learning rate-
We can see that we are able to get correct prediction in 1st iteration itself
Decreasing learning rate-
We are not able to get correct output even in 10 iterations. It will reach global minima slowly.
However that’s always not the case.
Using very high learning rate can cause it to jump global minima and using too low learning rate can cause it to stuck in local minima.
To put all these simply, passing input to network with weights and biases which goes through hidden layers containing activation functions. Finally output layer where loss/cost is calculated is called feed forward.
And then passing loss/cost to optimizers that uses gradient descent to update values of weights and biases is called back propagation.
