What is a KNN classifier

Given a set of labeled data points, we can predict the label of a new data point by just looking at the other examples that are the closest to it.

For example, imagine we have this clustering of dots, some are labeled as solid, and some are labeled as hollow. If there was a new dot at the star’s position, what is the likely label of that new dot?

We could figure out the distance between the new dot and all the other dots and then copy the label of the closest dot, but in this case, it looks like it might get mislabelled because although it’s close to a solid dot, it feels like it’s part of the hollow dot cluster.

Maybe a better solution is to figure out the most common label from say the closest 5 dots, or the closest 10 dots.

The number of other examples to compare with that is the K number in KNN classification.

There is usually an optimum value for K for each data set; you can figure it out by looking at a labeled data set, splitting it into a training and validation data set. Then use increasing values of K and see how well they predict the validation data set.

You may end up with a result like this, which indicates that a value of 7 is suitable for this data set.

Choose the K value with the lowest error.

In our example above, we use a simple 2D graph with data points. It’s easy for the mind to see the distance between two data points in the 2D world. What about 3D, or 5D or 1000D? It gets harder to visualize, but the algorithm is just the same: Euclidian distance^[1].

Another way to look at the data in our dot examples above is a data set with two features: an x value and a y value.

Suppose we wanted to use KNN for other types of problems the data points would need to be able to be mapped onto points in a Euclidean space. That means the features of the points need to be numbers.

One example of a problem you can use KNN with is the Iris Flowers dataset^[2]. This data set involves predicting the specific flower species of the genus Iris from different measurements of Iris flowers.

The data set has 150 data points, where each data point has:

So each data point has 4 features and 1 class. Each of the features is a number (cm), so it can be mapped into Euclidean space, and you can calculate the distance between and data point and any other in this Euclidean space.

This was a quick lecture to cover the concept of the KNN classifier. They are simple machine learning models that are simple to understand, simple to implement; however, their predictive power is limited. However, used in conjunction with a neural network in a transfer learning model, they can become much more powerful. We will be using a KNN classifier as the new machine learning model to add on top of a decapitated neural network in the next lecture.