Lesson 2: Introduction to Neural Networks_I

Udacity: Intro to Deep Learning with PyTorch:

Lesson 2: Introduction to Neural Networks

— Coding the Perceptron Algorithm

Recall that the perceptron step works as follows. For a point with coordinates

(p, q)

, label

y

, and prediction given by the equation

y^= s t e p (w_{1} x_{1} + w_{2} x_{2} + b)

If the point is correctly classified, do nothing.
If the point is classified positive, but it has a negative label, subtract $α p, α q$ and $α$ from $w_{1}, w_{2}$ and respectively.
If the point is classified negative, but it has a positive label, add $α p, α q$ and $α$ to $w_{1}, w_{2}$ and $b$ respectively.

– Error Function

An error function is simply something that tells us how far we are from the solution

– Discrete vs Continuous Predictions

Day 22 of #60daysofudacity:

– The Softmax Function

softmax function, which is the equivalent of the sigmoid activation function, but when the problem has 3 or more classes.

Screen Shot 2019-07-23 at 10.34.43 PM

Notice that they are all negative numbers and that actually makes sense. This is because the logarithm of a number between 0 and 1 is always a negative number since the logarithm of one is zero.

The negative of the logarithm of the probabilities and we’ll get positive numbers.

We’ll take the negative of the logarithm of the probabilities. That sums up negatives of logarithms of the probabilities, we’ll call the cross entropy,

If we calculate the cross entropies, we see that the bad model on left has a cross entropy 4.8 which is high. Whereas the good model on the right has a cross entropy of 1.2 which is low.

A good model will give us a low cross entropy and a bad model will give us a high cross entropy. The reason for this is simply that a good model gives us a high probability and the negative of the logarithm of a large number is a small number and vice versa.

Screen Shot 2019-07-23 at 10.49.45 PM.png

If we calculate the probabilities and pair the points with the corresponding logarithms, we actually get an error for each point.

Now, we take the negative of the logarithms which gives us sum of logarithms and if we pair each logarithm with the point where it came from, we actually get a value for each point. We calculate the values, the points that are mis-classified has like values like 2.3 for this point or 1.6 one for this point, whereas the points that are correctly classified have small values.

A correctly classified point will have a probability that as close to 1, which when we take the negative of the logarithm, we’ll get a small value.

The negatives of these logarithms as errors at each point.
Points that are correctly classified will have small errors and points that are mis-classified will have large errors.
Cross entropy will tell us if a model is good or bad.

Cross-Entropy

There’s definitely a connection between probabilities and error functions, and it’s called Cross-Entropy. This concept is tremendously popular in many fields, including Machine Learning. Screen Shot 2019-07-23 at 11.01.59 PM

Gina Hsu

gina@ginahsu.com

Lesson 2: Introduction to Neural Networks_I

Udacity: Intro to Deep Learning with PyTorch: