K-Nearest Neighbors (KNN) Algorithm

Motivation and intorduction

• Both models require an input-output formula or form.
• Do we have something that

K-Nearest Neighbors (KNN) Algorithm

Introduction

K-Nearest Neighbors (KNN) Algorithm

Euclidean Distance

• The core idea of some Non-parametric models is using the outputs of similar data points to predict any query point.
• But what does similar or difference mean?
• In ML, we often use distances to measure how difference the data points are.
• The most common distance is the Euclidean one:
  • For example: \(A = (1,\ 3,\ 4)\) and \(B = (-1,\ 2,\ 5)\) then

\[D(A, B) = \sqrt{(1-(-1))^2 + (3-2)^2 + (4-5)^2} = \sqrt{4} = 2 \text{ (unit)}.\]

• For two input data x = (x₁, x₂, …, x_d) and x’ = (x’₁, x’₂, …, x’_d) then the Euclidean distance between them is given by

\[D(\mathbf{x}, \mathbf{x}') = \sqrt{\sum_{i=1}^{D} (x_i - x'_i)^2}.\]

Smaller distance = Closer the points = More similar the data.

K-Nearest Neighbors (K-NN)

• Given the training data: \(\{(\mathbf{x}_1, y_1), \ldots, (\mathbf{x}_n, y_n)\} \subset \mathbb{R}^d \times \mathcal{Y}\).

• If \(D\) is a distance on \(\mathbb{R}^d\) (e.g. Euclidean distance), \(\mathbf{x}_{(k)}\) is called the \(k\)-th nearest neighbor of \(\mathbf{x} \in \mathbb{R}^d\) if its distance to \(\mathbf{x}\) ranks \(k\)-th among all the input points, i.e.,

• If \(k \geq 1\), then \(k\)-NN predicts the target of an input \(\mathbf{x}\) by

K-Nearest Neighbors (K-NN)

Example

K-Nearest Neighbors (K-NN)

Influence of K

• How to choose a good k?

K-Nearest Neighbors (K-NN)

Fine-Tune K

Data Splitting:Train/Validate/Test

• A good model is the one that can generalize/predict new unseen data.
• The first attempt: splitting the data into 3 parts.

• In this case, the best k is the one achieving the best performance on Validation set.
• The final performance is measured using the Test set.

K-Nearest Neighbors (K-NN)

Fine-Tune K

Data Splitting:K-Fold Cross Validation

K-Nearest Neighbors (K-NN)

Performance Metrics

• Selecting the best k depends not only the splitting scheme, but also the performance metric to define What does the best mean?
• What’s performance metric?
• It’s a value that measures the quality of a model when using to predict new unseen observations.
• They are divided into two main types:

• Score: larger ⇔ better model.
• Error: smaller ⇔ better model.
• ⚠️ Not to confuse:

• Metric: For fine-tuning the key hyperparameters of the model (use validating or testing data when being measured).
  • Example: \(R^2\), Adjusted \(R^2\), Accuracy…
• Loss: For training the model and is computed using the training data.
  • Example: Mean Squared Error (MSE), Mean Absolute Error (MAE)…

K-Nearest Neighbors (K-NN)

Regression Metrics

• These are some common metrics in regression problems.

K-Nearest Neighbors (K-NN)

Classification Metrics

• These are some common metrics in classification problems.

K-Nearest Neighbors (K-NN)

Classification Application

• Let’s work with our Heart Disease Dataset (shape: \(1025 \times 14\)) and choose \(K = 5\).
• K-NN is a distance-based method, it’s essential to

• Scale the inputs
• Watch out for outliers/missing values
• Encode categorical inputs
• Watch out for the effect of imbalanced class…

import numpy as np
import pandas as pd
data = pd.read_csv("../Data/heart.csv")
quan_vars = ['age','trestbps','chol','thalach','oldpeak']
qual_vars = ['sex','cp','fbs','restecg','exang','slope','ca','thal','target']

# Convert to correct types
for i in quan_vars:
  data[i] = data[i].astype('float')
for i in qual_vars:
  data[i] = data[i].astype('category')

# Train test split
from sklearn.model_selection import train_test_split
X, y = data.iloc[:,:-1], data.iloc[:,-1]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)

from sklearn.preprocessing import MinMaxScaler, StandardScaler
scaler = MinMaxScaler()

# OnehotEncoding and Scaling
X_train_cat = pd.get_dummies(X_train.select_dtypes(include="category"), drop_first=True)
X_train_encoded = scaler.fit_transform(np.column_stack([X_train.select_dtypes(include="number").to_numpy(), X_train_cat]))
X_test_cat = pd.get_dummies(X_test.select_dtypes(include="category"), drop_first=True)
X_test_encoded = scaler.transform(np.column_stack([X_test.select_dtypes(include="number").to_numpy(), X_test_cat]))

# KNN
from sklearn.neighbors import KNeighborsClassifier 

knn = KNeighborsClassifier(n_neighbors=5)
knn = knn.fit(X_train_encoded, y_train)
y_pred = knn.predict(X_test_encoded)

from sklearn.metrics import roc_auc_score, accuracy_score, precision_score, recall_score, f1_score, confusion_matrix, ConfusionMatrixDisplay

test_perf = pd.DataFrame(
    data={'Accuracy': accuracy_score(y_test, y_pred),
          'Precision': precision_score(y_test, y_pred),
          'Recall': recall_score(y_test, y_pred),
          'F1-score': f1_score(y_test, y_pred)},
    columns=["Accuracy", "Precision", "Recall", "F1-score"],
    index=["5NN"])
test_perf

• Q3: Can we do better?
• A3: Yes! \(K = 5\) is arbitrary. We should fine-tune it!

K-Nearest Neighbors (K-NN)

Fine-Tuning k: Cross-Validation

• There are many way to perform Cross-validation in python.
• Let’s use GridSearchCV from sklearn.model_selection module.

knn = KNeighborsClassifier()
from sklearn.model_selection import GridSearchCV
from sklearn.metrics import f1_score, make_scorer
scorer = make_scorer(f1_score, pos_label=1)
param_grid = {'n_neighbors': list(range(1,20))}
knn = KNeighborsClassifier()
grid_search = GridSearchCV(knn, param_grid, cv=10, scoring=scorer, return_train_score=True)
grid_search.fit(X_train_encoded, y_train)

knn = KNeighborsClassifier(n_neighbors=grid_search.best_params_['n_neighbors'])
knn = knn.fit(X_train_encoded, y_train)
y_pred = knn.predict(X_test_encoded)

test_perf = pd.concat([test_perf, pd.DataFrame(
    data={'Accuracy': accuracy_score(y_test, y_pred),
          'Precision': precision_score(y_test, y_pred),
          'Recall': recall_score(y_test, y_pred),
          'F1-score': f1_score(y_test, y_pred)},
    columns=["Accuracy", "Precision", "Recall", "F1-score"],
    index=[f"{grid_search.best_params_['n_neighbors']}NN"])], axis=0)
test_perf

K-Nearest Neighbors (K-NN)

What Can Go Wrong?

# Remove duplicates
data_no_dup = data.drop_duplicates()

# Re-split the data
X_no_dup, y_no_dup = data_no_dup.iloc[:,:-1], data_no_dup.iloc[:,-1]
X_train_nd, X_test_nd, y_train_nd, y_test_nd = train_test_split(X_no_dup, y_no_dup, test_size=0.2, stratify=y_no_dup, random_state=42)

# Re-encode and scale
X_train_cat_nd = pd.get_dummies(X_train_nd.select_dtypes(include="category"), drop_first=True)
X_train_encoded_nd = scaler.fit_transform(np.column_stack([X_train_nd.select_dtypes(include="number").to_numpy(), X_train_cat_nd]))
X_test_cat_nd = pd.get_dummies(X_test_nd.select_dtypes(include="category"), drop_first=True)
X_test_encoded_nd = scaler.transform(np.column_stack([X_test_nd.select_dtypes(include="number").to_numpy(), X_test_cat_nd]))

# GridSearch again
grid_search_nd = GridSearchCV(KNeighborsClassifier(), param_grid, cv=10, scoring=scorer, return_train_score=True)
grid_search_nd.fit(X_train_encoded_nd, y_train_nd)

# Train and predict with optimal K
knn_nd = KNeighborsClassifier(n_neighbors=grid_search_nd.best_params_['n_neighbors'])
knn_nd = knn_nd.fit(X_train_encoded_nd, y_train_nd)
y_pred_nd = knn_nd.predict(X_test_encoded_nd)

# Add results to table
test_perf = pd.concat([test_perf, pd.DataFrame(
    data={'Accuracy': accuracy_score(y_test_nd, y_pred_nd),
          'Precision': precision_score(y_test_nd, y_pred_nd),
          'Recall': recall_score(y_test_nd, y_pred_nd),
          'F1-score': f1_score(y_test_nd, y_pred_nd)},
    columns=["Accuracy", "Precision", "Recall", "F1-score"],
    index=[f"{grid_search_nd.best_params_['n_neighbors']}NN_No_Dup"])], axis=0)

K-Nearest Neighbors (\(k\)-NN)

Summary

Questions?

Instinct Institute