|
15 | 15 | "cell_type": "markdown", |
16 | 16 | "metadata": {}, |
17 | 17 | "source": [ |
18 | | - "\n# Precision-Recall\n\n\nExample of Precision-Recall metric to evaluate classifier output quality.\n\nPrecision-Recall is a useful measure of success of prediction when the\nclasses are very imbalanced. In information retrieval, precision is a\nmeasure of result relevancy, while recall is a measure of how many truly\nrelevant results are returned.\n\nThe precision-recall curve shows the tradeoff between precision and\nrecall for different threshold. A high area under the curve represents\nboth high recall and high precision, where high precision relates to a\nlow false positive rate, and high recall relates to a low false negative\nrate. High scores for both show that the classifier is returning accurate\nresults (high precision), as well as returning a majority of all positive\nresults (high recall).\n\nA system with high recall but low precision returns many results, but most of\nits predicted labels are incorrect when compared to the training labels. A\nsystem with high precision but low recall is just the opposite, returning very\nfew results, but most of its predicted labels are correct when compared to the\ntraining labels. An ideal system with high precision and high recall will\nreturn many results, with all results labeled correctly.\n\nPrecision ($P$) is defined as the number of true positives ($T_p$)\nover the number of true positives plus the number of false positives\n($F_p$).\n\n$P = \\frac{T_p}{T_p+F_p}$\n\nRecall ($R$) is defined as the number of true positives ($T_p$)\nover the number of true positives plus the number of false negatives\n($F_n$).\n\n$R = \\frac{T_p}{T_p + F_n}$\n\nThese quantities are also related to the ($F_1$) score, which is defined\nas the harmonic mean of precision and recall.\n\n$F1 = 2\\frac{P \\times R}{P+R}$\n\nNote that the precision may not decrease with recall. The\ndefinition of precision ($\\frac{T_p}{T_p + F_p}$) shows that lowering\nthe threshold of a classifier may increase the denominator, by increasing the\nnumber of results returned. If the threshold was previously set too high, the\nnew results may all be true positives, which will increase precision. If the\nprevious threshold was about right or too low, further lowering the threshold\nwill introduce false positives, decreasing precision.\n\nRecall is defined as $\\frac{T_p}{T_p+F_n}$, where $T_p+F_n$ does\nnot depend on the classifier threshold. This means that lowering the classifier\nthreshold may increase recall, by increasing the number of true positive\nresults. It is also possible that lowering the threshold may leave recall\nunchanged, while the precision fluctuates.\n\nThe relationship between recall and precision can be observed in the\nstairstep area of the plot - at the edges of these steps a small change\nin the threshold considerably reduces precision, with only a minor gain in\nrecall.\n\n**Average precision** summarizes such a plot as the weighted mean of precisions\nachieved at each threshold, with the increase in recall from the previous\nthreshold used as the weight:\n\n$\\text{AP} = \\sum_n (R_n - R_{n-1}) P_n$\n\nwhere $P_n$ and $R_n$ are the precision and recall at the\nnth threshold. A pair $(R_k, P_k)$ is referred to as an\n*operating point*.\n\nPrecision-recall curves are typically used in binary classification to study\nthe output of a classifier. In order to extend the precision-recall curve and\naverage precision to multi-class or multi-label classification, it is necessary\nto binarize the output. One curve can be drawn per label, but one can also draw\na precision-recall curve by considering each element of the label indicator\nmatrix as a binary prediction (micro-averaging).\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>See also :func:`sklearn.metrics.average_precision_score`,\n :func:`sklearn.metrics.recall_score`,\n :func:`sklearn.metrics.precision_score`,\n :func:`sklearn.metrics.f1_score`</p></div>\n\n" |
| 18 | + "\n# Precision-Recall\n\n\nExample of Precision-Recall metric to evaluate classifier output quality.\n\nPrecision-Recall is a useful measure of success of prediction when the\nclasses are very imbalanced. In information retrieval, precision is a\nmeasure of result relevancy, while recall is a measure of how many truly\nrelevant results are returned.\n\nThe precision-recall curve shows the tradeoff between precision and\nrecall for different threshold. A high area under the curve represents\nboth high recall and high precision, where high precision relates to a\nlow false positive rate, and high recall relates to a low false negative\nrate. High scores for both show that the classifier is returning accurate\nresults (high precision), as well as returning a majority of all positive\nresults (high recall).\n\nA system with high recall but low precision returns many results, but most of\nits predicted labels are incorrect when compared to the training labels. A\nsystem with high precision but low recall is just the opposite, returning very\nfew results, but most of its predicted labels are correct when compared to the\ntraining labels. An ideal system with high precision and high recall will\nreturn many results, with all results labeled correctly.\n\nPrecision ($P$) is defined as the number of true positives ($T_p$)\nover the number of true positives plus the number of false positives\n($F_p$).\n\n$P = \\frac{T_p}{T_p+F_p}$\n\nRecall ($R$) is defined as the number of true positives ($T_p$)\nover the number of true positives plus the number of false negatives\n($F_n$).\n\n$R = \\frac{T_p}{T_p + F_n}$\n\nThese quantities are also related to the ($F_1$) score, which is defined\nas the harmonic mean of precision and recall.\n\n$F1 = 2\\frac{P \\times R}{P+R}$\n\nNote that the precision may not decrease with recall. The\ndefinition of precision ($\\frac{T_p}{T_p + F_p}$) shows that lowering\nthe threshold of a classifier may increase the denominator, by increasing the\nnumber of results returned. If the threshold was previously set too high, the\nnew results may all be true positives, which will increase precision. If the\nprevious threshold was about right or too low, further lowering the threshold\nwill introduce false positives, decreasing precision.\n\nRecall is defined as $\\frac{T_p}{T_p+F_n}$, where $T_p+F_n$ does\nnot depend on the classifier threshold. This means that lowering the classifier\nthreshold may increase recall, by increasing the number of true positive\nresults. It is also possible that lowering the threshold may leave recall\nunchanged, while the precision fluctuates.\n\nThe relationship between recall and precision can be observed in the\nstairstep area of the plot - at the edges of these steps a small change\nin the threshold considerably reduces precision, with only a minor gain in\nrecall.\n\n**Average precision** (AP) summarizes such a plot as the weighted mean of\nprecisions achieved at each threshold, with the increase in recall from the\nprevious threshold used as the weight:\n\n$\\text{AP} = \\sum_n (R_n - R_{n-1}) P_n$\n\nwhere $P_n$ and $R_n$ are the precision and recall at the\nnth threshold. A pair $(R_k, P_k)$ is referred to as an\n*operating point*.\n\nAP and the trapezoidal area under the operating points\n(:func:`sklearn.metrics.auc`) are common ways to summarize a precision-recall\ncurve that lead to different results. Read more in the\n`User Guide <precision_recall_f_measure_metrics>`.\n\nPrecision-recall curves are typically used in binary classification to study\nthe output of a classifier. In order to extend the precision-recall curve and\naverage precision to multi-class or multi-label classification, it is necessary\nto binarize the output. One curve can be drawn per label, but one can also draw\na precision-recall curve by considering each element of the label indicator\nmatrix as a binary prediction (micro-averaging).\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>See also :func:`sklearn.metrics.average_precision_score`,\n :func:`sklearn.metrics.recall_score`,\n :func:`sklearn.metrics.precision_score`,\n :func:`sklearn.metrics.f1_score`</p></div>\n\n" |
19 | 19 | ] |
20 | 20 | }, |
21 | 21 | { |
|
80 | 80 | }, |
81 | 81 | "outputs": [], |
82 | 82 | "source": [ |
83 | | - "from sklearn.metrics import precision_recall_curve\nimport matplotlib.pyplot as plt\n\nprecision, recall, _ = precision_recall_curve(y_test, y_score)\n\nplt.step(recall, precision, color='b', alpha=0.2,\n where='post')\nplt.fill_between(recall, precision, step='post', alpha=0.2,\n color='b')\n\nplt.xlabel('Recall')\nplt.ylabel('Precision')\nplt.ylim([0.0, 1.05])\nplt.xlim([0.0, 1.0])\nplt.title('2-class Precision-Recall curve: AUC={0:0.2f}'.format(\n average_precision))" |
| 83 | + "from sklearn.metrics import precision_recall_curve\nimport matplotlib.pyplot as plt\n\nprecision, recall, _ = precision_recall_curve(y_test, y_score)\n\nplt.step(recall, precision, color='b', alpha=0.2,\n where='post')\nplt.fill_between(recall, precision, step='post', alpha=0.2,\n color='b')\n\nplt.xlabel('Recall')\nplt.ylabel('Precision')\nplt.ylim([0.0, 1.05])\nplt.xlim([0.0, 1.0])\nplt.title('2-class Precision-Recall curve: AP={0:0.2f}'.format(\n average_precision))" |
84 | 84 | ] |
85 | 85 | }, |
86 | 86 | { |
|
134 | 134 | }, |
135 | 135 | "outputs": [], |
136 | 136 | "source": [ |
137 | | - "plt.figure()\nplt.step(recall['micro'], precision['micro'], color='b', alpha=0.2,\n where='post')\nplt.fill_between(recall[\"micro\"], precision[\"micro\"], step='post', alpha=0.2,\n color='b')\n\nplt.xlabel('Recall')\nplt.ylabel('Precision')\nplt.ylim([0.0, 1.05])\nplt.xlim([0.0, 1.0])\nplt.title(\n 'Average precision score, micro-averaged over all classes: AUC={0:0.2f}'\n .format(average_precision[\"micro\"]))" |
| 137 | + "plt.figure()\nplt.step(recall['micro'], precision['micro'], color='b', alpha=0.2,\n where='post')\nplt.fill_between(recall[\"micro\"], precision[\"micro\"], step='post', alpha=0.2,\n color='b')\n\nplt.xlabel('Recall')\nplt.ylabel('Precision')\nplt.ylim([0.0, 1.05])\nplt.xlim([0.0, 1.0])\nplt.title(\n 'Average precision score, micro-averaged over all classes: AP={0:0.2f}'\n .format(average_precision[\"micro\"]))" |
138 | 138 | ] |
139 | 139 | }, |
140 | 140 | { |
|
0 commit comments