|
4 | 4 | "cell_type": "markdown", |
5 | 5 | "metadata": {}, |
6 | 6 | "source": [ |
7 | | - "\n# Linear and Quadratic Discriminant Analysis with covariance ellipsoid\n\nThis example plots the covariance ellipsoids of each class and\ndecision boundary learned by LDA and QDA. The ellipsoids display\nthe double standard deviation for each class. With LDA, the\nstandard deviation is the same for all the classes, while each\nclass has its own standard deviation with QDA.\n" |
| 7 | + "\n# Linear and Quadratic Discriminant Analysis with covariance ellipsoid\n\nThis example plots the covariance ellipsoids of each class and the decision boundary\nlearned by :class:`~sklearn.discriminant_analysis.LinearDiscriminantAnalysis` (LDA) and\n:class:`~sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis` (QDA). The\nellipsoids display the double standard deviation for each class. With LDA, the standard\ndeviation is the same for all the classes, while each class has its own standard\ndeviation with QDA.\n" |
8 | 8 | ] |
9 | 9 | }, |
10 | 10 | { |
11 | 11 | "cell_type": "markdown", |
12 | 12 | "metadata": {}, |
13 | 13 | "source": [ |
14 | | - "## Colormap\n\n" |
| 14 | + "## Data generation\n\nFirst, we define a function to generate synthetic data. It creates two blobs centered\nat `(0, 0)` and `(1, 1)`. Each blob is assigned a specific class. The dispersion of\nthe blob is controlled by the parameters `cov_class_1` and `cov_class_2`, that are the\ncovariance matrices used when generating the samples from the Gaussian distributions.\n\n" |
15 | 15 | ] |
16 | 16 | }, |
17 | 17 | { |
|
22 | 22 | }, |
23 | 23 | "outputs": [], |
24 | 24 | "source": [ |
25 | | - "import matplotlib as mpl\nimport matplotlib.pyplot as plt\nfrom matplotlib import colors\n\ncmap = colors.LinearSegmentedColormap(\n \"red_blue_classes\",\n {\n \"red\": [(0, 1, 1), (1, 0.7, 0.7)],\n \"green\": [(0, 0.7, 0.7), (1, 0.7, 0.7)],\n \"blue\": [(0, 0.7, 0.7), (1, 1, 1)],\n },\n)\nplt.cm.register_cmap(cmap=cmap)" |
| 25 | + "import numpy as np\n\n\ndef make_data(n_samples, n_features, cov_class_1, cov_class_2, seed=0):\n rng = np.random.RandomState(seed)\n X = np.concatenate(\n [\n rng.randn(n_samples, n_features) @ cov_class_1,\n rng.randn(n_samples, n_features) @ cov_class_2 + np.array([1, 1]),\n ]\n )\n y = np.concatenate([np.zeros(n_samples), np.ones(n_samples)])\n return X, y" |
26 | 26 | ] |
27 | 27 | }, |
28 | 28 | { |
29 | 29 | "cell_type": "markdown", |
30 | 30 | "metadata": {}, |
31 | 31 | "source": [ |
32 | | - "## Datasets generation functions\n\n" |
| 32 | + "We generate three datasets. In the first dataset, the two classes share the same\ncovariance matrix, and this covariance matrix has the specificity of being spherical\n(isotropic). The second dataset is similar to the first one but does not enforce the\ncovariance to be spherical. Finally, the third dataset has a non-spherical covariance\nmatrix for each class.\n\n" |
33 | 33 | ] |
34 | 34 | }, |
35 | 35 | { |
|
40 | 40 | }, |
41 | 41 | "outputs": [], |
42 | 42 | "source": [ |
43 | | - "import numpy as np\n\n\ndef dataset_fixed_cov():\n \"\"\"Generate 2 Gaussians samples with the same covariance matrix\"\"\"\n n, dim = 300, 2\n np.random.seed(0)\n C = np.array([[0.0, -0.23], [0.83, 0.23]])\n X = np.r_[\n np.dot(np.random.randn(n, dim), C),\n np.dot(np.random.randn(n, dim), C) + np.array([1, 1]),\n ]\n y = np.hstack((np.zeros(n), np.ones(n)))\n return X, y\n\n\ndef dataset_cov():\n \"\"\"Generate 2 Gaussians samples with different covariance matrices\"\"\"\n n, dim = 300, 2\n np.random.seed(0)\n C = np.array([[0.0, -1.0], [2.5, 0.7]]) * 2.0\n X = np.r_[\n np.dot(np.random.randn(n, dim), C),\n np.dot(np.random.randn(n, dim), C.T) + np.array([1, 4]),\n ]\n y = np.hstack((np.zeros(n), np.ones(n)))\n return X, y" |
| 43 | + "covariance = np.array([[1, 0], [0, 1]])\nX_isotropic_covariance, y_isotropic_covariance = make_data(\n n_samples=1_000,\n n_features=2,\n cov_class_1=covariance,\n cov_class_2=covariance,\n seed=0,\n)\ncovariance = np.array([[0.0, -0.23], [0.83, 0.23]])\nX_shared_covariance, y_shared_covariance = make_data(\n n_samples=300,\n n_features=2,\n cov_class_1=covariance,\n cov_class_2=covariance,\n seed=0,\n)\ncov_class_1 = np.array([[0.0, -1.0], [2.5, 0.7]]) * 2.0\ncov_class_2 = cov_class_1.T\nX_different_covariance, y_different_covariance = make_data(\n n_samples=300,\n n_features=2,\n cov_class_1=cov_class_1,\n cov_class_2=cov_class_2,\n seed=0,\n)" |
44 | 44 | ] |
45 | 45 | }, |
46 | 46 | { |
47 | 47 | "cell_type": "markdown", |
48 | 48 | "metadata": {}, |
49 | 49 | "source": [ |
50 | | - "## Plot functions\n\n" |
| 50 | + "## Plotting Functions\n\nThe code below is used to plot several pieces of information from the estimators used,\ni.e., :class:`~sklearn.discriminant_analysis.LinearDiscriminantAnalysis` (LDA) and\n:class:`~sklearn.discriminant_analysis.QuadraticDiscriminantAnalysis` (QDA). The\ndisplayed information includes:\n\n- the decision boundary based on the probability estimate of the estimator;\n- a scatter plot with circles representing the well-classified samples;\n- a scatter plot with crosses representing the misclassified samples;\n- the mean of each class, estimated by the estimator, marked with a star;\n- the estimated covariance represented by an ellipse at 2 standard deviations from the\n mean.\n\n" |
51 | 51 | ] |
52 | 52 | }, |
53 | 53 | { |
|
58 | 58 | }, |
59 | 59 | "outputs": [], |
60 | 60 | "source": [ |
61 | | - "from scipy import linalg\n\n\ndef plot_data(lda, X, y, y_pred, fig_index):\n splot = plt.subplot(2, 2, fig_index)\n if fig_index == 1:\n plt.title(\"Linear Discriminant Analysis\")\n plt.ylabel(\"Data with\\n fixed covariance\")\n elif fig_index == 2:\n plt.title(\"Quadratic Discriminant Analysis\")\n elif fig_index == 3:\n plt.ylabel(\"Data with\\n varying covariances\")\n\n tp = y == y_pred # True Positive\n tp0, tp1 = tp[y == 0], tp[y == 1]\n X0, X1 = X[y == 0], X[y == 1]\n X0_tp, X0_fp = X0[tp0], X0[~tp0]\n X1_tp, X1_fp = X1[tp1], X1[~tp1]\n\n # class 0: dots\n plt.scatter(X0_tp[:, 0], X0_tp[:, 1], marker=\".\", color=\"red\")\n plt.scatter(X0_fp[:, 0], X0_fp[:, 1], marker=\"x\", s=20, color=\"#990000\") # dark red\n\n # class 1: dots\n plt.scatter(X1_tp[:, 0], X1_tp[:, 1], marker=\".\", color=\"blue\")\n plt.scatter(\n X1_fp[:, 0], X1_fp[:, 1], marker=\"x\", s=20, color=\"#000099\"\n ) # dark blue\n\n # class 0 and 1 : areas\n nx, ny = 200, 100\n x_min, x_max = plt.xlim()\n y_min, y_max = plt.ylim()\n xx, yy = np.meshgrid(np.linspace(x_min, x_max, nx), np.linspace(y_min, y_max, ny))\n Z = lda.predict_proba(np.c_[xx.ravel(), yy.ravel()])\n Z = Z[:, 1].reshape(xx.shape)\n plt.pcolormesh(\n xx, yy, Z, cmap=\"red_blue_classes\", norm=colors.Normalize(0.0, 1.0), zorder=0\n )\n plt.contour(xx, yy, Z, [0.5], linewidths=2.0, colors=\"white\")\n\n # means\n plt.plot(\n lda.means_[0][0],\n lda.means_[0][1],\n \"*\",\n color=\"yellow\",\n markersize=15,\n markeredgecolor=\"grey\",\n )\n plt.plot(\n lda.means_[1][0],\n lda.means_[1][1],\n \"*\",\n color=\"yellow\",\n markersize=15,\n markeredgecolor=\"grey\",\n )\n\n return splot\n\n\ndef plot_ellipse(splot, mean, cov, color):\n v, w = linalg.eigh(cov)\n u = w[0] / linalg.norm(w[0])\n angle = np.arctan(u[1] / u[0])\n angle = 180 * angle / np.pi # convert to degrees\n # filled Gaussian at 2 standard deviation\n ell = mpl.patches.Ellipse(\n mean,\n 2 * v[0] ** 0.5,\n 2 * v[1] ** 0.5,\n angle=180 + angle,\n facecolor=color,\n edgecolor=\"black\",\n linewidth=2,\n )\n ell.set_clip_box(splot.bbox)\n ell.set_alpha(0.2)\n splot.add_artist(ell)\n splot.set_xticks(())\n splot.set_yticks(())\n\n\ndef plot_lda_cov(lda, splot):\n plot_ellipse(splot, lda.means_[0], lda.covariance_, \"red\")\n plot_ellipse(splot, lda.means_[1], lda.covariance_, \"blue\")\n\n\ndef plot_qda_cov(qda, splot):\n plot_ellipse(splot, qda.means_[0], qda.covariance_[0], \"red\")\n plot_ellipse(splot, qda.means_[1], qda.covariance_[1], \"blue\")" |
| 61 | + "import matplotlib as mpl\nfrom matplotlib import colors\n\nfrom sklearn.inspection import DecisionBoundaryDisplay\n\n\ndef plot_ellipse(mean, cov, color, ax):\n v, w = np.linalg.eigh(cov)\n u = w[0] / np.linalg.norm(w[0])\n angle = np.arctan(u[1] / u[0])\n angle = 180 * angle / np.pi # convert to degrees\n # filled Gaussian at 2 standard deviation\n ell = mpl.patches.Ellipse(\n mean,\n 2 * v[0] ** 0.5,\n 2 * v[1] ** 0.5,\n angle=180 + angle,\n facecolor=color,\n edgecolor=\"black\",\n linewidth=2,\n )\n ell.set_clip_box(ax.bbox)\n ell.set_alpha(0.4)\n ax.add_artist(ell)\n\n\ndef plot_result(estimator, X, y, ax):\n cmap = colors.ListedColormap([\"tab:red\", \"tab:blue\"])\n DecisionBoundaryDisplay.from_estimator(\n estimator,\n X,\n response_method=\"predict_proba\",\n plot_method=\"pcolormesh\",\n ax=ax,\n cmap=\"RdBu\",\n alpha=0.3,\n )\n DecisionBoundaryDisplay.from_estimator(\n estimator,\n X,\n response_method=\"predict_proba\",\n plot_method=\"contour\",\n ax=ax,\n alpha=1.0,\n levels=[0.5],\n )\n y_pred = estimator.predict(X)\n X_right, y_right = X[y == y_pred], y[y == y_pred]\n X_wrong, y_wrong = X[y != y_pred], y[y != y_pred]\n ax.scatter(X_right[:, 0], X_right[:, 1], c=y_right, s=20, cmap=cmap, alpha=0.5)\n ax.scatter(\n X_wrong[:, 0],\n X_wrong[:, 1],\n c=y_wrong,\n s=30,\n cmap=cmap,\n alpha=0.9,\n marker=\"x\",\n )\n ax.scatter(\n estimator.means_[:, 0],\n estimator.means_[:, 1],\n c=\"yellow\",\n s=200,\n marker=\"*\",\n edgecolor=\"black\",\n )\n\n if isinstance(estimator, LinearDiscriminantAnalysis):\n covariance = [estimator.covariance_] * 2\n else:\n covariance = estimator.covariance_\n plot_ellipse(estimator.means_[0], covariance[0], \"tab:red\", ax)\n plot_ellipse(estimator.means_[1], covariance[1], \"tab:blue\", ax)\n\n ax.set_box_aspect(1)\n ax.spines[\"top\"].set_visible(False)\n ax.spines[\"bottom\"].set_visible(False)\n ax.spines[\"left\"].set_visible(False)\n ax.spines[\"right\"].set_visible(False)\n ax.set(xticks=[], yticks=[])" |
62 | 62 | ] |
63 | 63 | }, |
64 | 64 | { |
65 | 65 | "cell_type": "markdown", |
66 | 66 | "metadata": {}, |
67 | 67 | "source": [ |
68 | | - "## Plot\n\n" |
| 68 | + "## Comparison of LDA and QDA\n\nWe compare the two estimators LDA and QDA on all three datasets.\n\n" |
69 | 69 | ] |
70 | 70 | }, |
71 | 71 | { |
|
76 | 76 | }, |
77 | 77 | "outputs": [], |
78 | 78 | "source": [ |
79 | | - "plt.figure(figsize=(10, 8), facecolor=\"white\")\nplt.suptitle(\n \"Linear Discriminant Analysis vs Quadratic Discriminant Analysis\",\n y=0.98,\n fontsize=15,\n)\n\nfrom sklearn.discriminant_analysis import (\n LinearDiscriminantAnalysis,\n QuadraticDiscriminantAnalysis,\n)\n\nfor i, (X, y) in enumerate([dataset_fixed_cov(), dataset_cov()]):\n # Linear Discriminant Analysis\n lda = LinearDiscriminantAnalysis(solver=\"svd\", store_covariance=True)\n y_pred = lda.fit(X, y).predict(X)\n splot = plot_data(lda, X, y, y_pred, fig_index=2 * i + 1)\n plot_lda_cov(lda, splot)\n plt.axis(\"tight\")\n\n # Quadratic Discriminant Analysis\n qda = QuadraticDiscriminantAnalysis(store_covariance=True)\n y_pred = qda.fit(X, y).predict(X)\n splot = plot_data(qda, X, y, y_pred, fig_index=2 * i + 2)\n plot_qda_cov(qda, splot)\n plt.axis(\"tight\")\n\nplt.tight_layout()\nplt.subplots_adjust(top=0.92)\nplt.show()" |
| 79 | + "import matplotlib.pyplot as plt\n\nfrom sklearn.discriminant_analysis import (\n LinearDiscriminantAnalysis,\n QuadraticDiscriminantAnalysis,\n)\n\nfig, axs = plt.subplots(nrows=3, ncols=2, sharex=\"row\", sharey=\"row\", figsize=(8, 12))\n\nlda = LinearDiscriminantAnalysis(solver=\"svd\", store_covariance=True)\nqda = QuadraticDiscriminantAnalysis(store_covariance=True)\n\nfor ax_row, X, y in zip(\n axs,\n (X_isotropic_covariance, X_shared_covariance, X_different_covariance),\n (y_isotropic_covariance, y_shared_covariance, y_different_covariance),\n):\n lda.fit(X, y)\n plot_result(lda, X, y, ax_row[0])\n qda.fit(X, y)\n plot_result(qda, X, y, ax_row[1])\n\naxs[0, 0].set_title(\"Linear Discriminant Analysis\")\naxs[0, 0].set_ylabel(\"Data with fixed and spherical covariance\")\naxs[1, 0].set_ylabel(\"Data with fixed covariance\")\naxs[0, 1].set_title(\"Quadratic Discriminant Analysis\")\naxs[2, 0].set_ylabel(\"Data with varying covariances\")\nfig.suptitle(\n \"Linear Discriminant Analysis vs Quadratic Discriminant Analysis\",\n y=0.94,\n fontsize=15,\n)\nplt.show()" |
| 80 | + ] |
| 81 | + }, |
| 82 | + { |
| 83 | + "cell_type": "markdown", |
| 84 | + "metadata": {}, |
| 85 | + "source": [ |
| 86 | + "The first important thing to notice is that LDA and QDA are equivalent for the\nfirst and second datasets. Indeed, the major difference is that LDA assumes\nthat the covariance matrix of each class is equal, while QDA estimates a\ncovariance matrix per class. Since in these cases the data generative process\nhas the same covariance matrix for both classes, QDA estimates two covariance\nmatrices that are (almost) equal and therefore equivalent to the covariance\nmatrix estimated by LDA.\n\nIn the first dataset the covariance matrix used to generate the dataset is\nspherical, which results in a discriminant boundary that aligns with the\nperpendicular bisector between the two means. This is no longer the case for\nthe second dataset. The discriminant boundary only passes through the middle\nof the two means.\n\nFinally, in the third dataset, we observe the real difference between LDA and\nQDA. QDA fits two covariance matrices and provides a non-linear discriminant\nboundary, whereas LDA underfits since it assumes that both classes share a\nsingle covariance matrix.\n\n" |
80 | 87 | ] |
81 | 88 | } |
82 | 89 | ], |
|
0 commit comments