A supervised non-linear and distanced based ML algorithm capable of solving both classification and regression problem.

KNN assumes that similar things exist in close proximity. In other words, KNN assumes that data points that are close to each other in the feature space are more likely to belong to the same class (for classification) or have similar target values (for regression tasks).

• KNN is easy to understand and implement because it does not involve complex mathematical concepts or require extensive parameter tuning.
• KNN doesn’t require a separate training phase. It simply stores the training data and uses it directly for predictions.

KNN is non-parametric, which means it makes no assumptions about the underlying data distribution, allowing KNN to be used to classify or predict data from any distribution.

KNN can be used for both classification and regression tasks. It can handle various types of data, including numerical and categorical variables.

• Simple and Intuitive.
• Non-Parametric.
• Versatile.
• No Training Period required.

KNN can be computationally expensive, especially for large datasets, as it requires calculating distances to all data points in the training set for each prediction.

KNN is sensitive to outliers because it relies on distance calculations. Outliers can significantly impact the neighbours selected and the resulting predictions.

KNN’s performance can degrade as the dimensionality of the feature space increases. As the number of features increases, the data becomes more sparse, meaning that there are fewer data points in close proximity to each other. This makes it more difficult to identify the k-nearest neighbours of a given query point.

• Computationally expensive.
• Sensitive to Outliers.
• Curse of Dimensionality.

1. Preprocess dataset: Scale the dataset.
2. Select K: The number of nearest neighbours to consider.
3. Compute distance:
   1. Calculate the distance between the data point and all the data points.
   2. Select the K-nearest neighbours based on the smallest distances to the data point.
4. Prediction:
   1. For classification, use mode (majority class based on the k-nearest neighbours) to determine the predicted class for the data point.
   2. For regression, use mean of the target value of the k-nearest neighbours to predict the target value for the data point.

KNN is a distance-based algorithm. Without scaling, variables with higher magnitude may dominate the distance metric, giving those variables greater importance. Therefore, to not have any biases towards variables with higher magnitude, the features needs to be contributed equally to the distance calculations.

K (Number of Neighbours): This hyperparameter determines the number of nearest neighbours to consider when making predictions.

• Euclidean distance
• Manhattan distance

• Choosing Too Many Neighbours (Large K):
  • Underfitting Risk: May experience high bias because the results are based on many data points
  • Increased Computational Complexity: Calculating distances to a large number of neighbours can be computationally expensive, especially for large datasets
• Choosing Too Few Neighbours (Few K):
  • Overfitting Risk: Prone to overfitting because the predictions are relying on a very small K, making the prediction sensitive.
  • Less Stable Predictions: Predictions can be less stable and more subject to changes in the training data.

• The elbow method is a heuristic technique for finding the optimal K based on the trade-off between model complexity and performance.
• Steps:
  • Calculate the model’s performance (e.g., error) for a range of K values.
  • Plot the performance values against the corresponding K values.
  • Look for the “elbow point” on the curve, which represents the K value where the performance improvement starts to diminish. This point is often considered the optimal K.