Quality Measures

In (projects:forged-banknotes:knn) we saw that k-NN works quite good for detecting forged banknotes from three features. Now we want to compute and interpret several quality measures. To get more wrong predictions (and thus more instructive data to play with) we reduce the number of features to two: variance and skewness.

Predictions

Task: Apply k-NN with \(k=11\) and without weighting to the banknotes classification problem with only two features (variance and skewness). Use test size \(0.3\). Print accuracy on train and test sets.

Solution:

# your solution

Task: Use predict_proba to get a vector of predicted probabilities for class 1 on the test set. For k-NN classification probabilities represent class distribution in a sample’s neighborhood.

Solution:

# your solution

Quality Measures

Confusion Matrix

Task: Generate and print the confusion matrix for the test set. Compare results to Scikit-Learn’s confusion_matrix.

# your solution

Accuracy and Balanced Accuracy

Task: Calculate unbalanced and balanced accuracy on the test set. Compare results to Scikit-Learn’s accuracy_score and balanced_accuracy_score.

Solution:

# your solution

Precision, Recall, F1-score

Task: Calculate and print precision, recall and F1-score on the test set. Compare results to Scikit-Learn’s precision_score, recall_score, f1_score.

Solution:

# your solution

Task: Think about precision and recall from a practical point of view. What do models with low or high values for precision or recall imply for banknote authentication?

Solution:

# your answers

Log Loss

Task: Calculate and print the log loss on the test set. Compare results to Scikit-Learn’s log_loss.

# your solution

Task: What is the log loss for perfect predictions (with your implementation and with Scikit-Learn)?

# your solution

AUC

Task: Calculate and plot false positive rate as well as true positive rate for the test set depending on the threshold \(t\), where a sample is labeled ‘positive’ if the predicted probability is strictly greater than \(t\). Remember that both functions are step functions. Use Matplotlib’s step. Compare results to Scikit-Learn’s roc_curve.

# your solution

Task: Plot the ROC curve for the test set. Compare results to Scikit-Learn’s RocCurveDisplay.from_predictions.

# your solution

Task: Calculate and print AUC for the test set. Compare results to Scikit-Learn’s roc_auc_score.

# your solution