{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"kernelspec": {
 "display_name": "Python (GeoScience)",
 "language": "python",
 "name": "geoscience"
},
"language_info": {
 "codemirror_mode": {
 "name": "ipython",
 "version": 2
 },
 "file_extension": ".py",
 "mimetype": "text/x-python",
 "name": "python",
 "nbconvert_exporter": "python",
 "pygments_lexer": "ipython2",
 "version": "2.7.3"
},
"colab": {
 "name": "",
 "provenance": [],
 "collapsed_sections": []
}
},
"cells": [
{
 "cell_type": "markdown",
 "metadata": {
 "id": "QUNgoL-dl-KB",
 "colab_type": "text"
 },
 "source": [
 ]
},
{
 "cell_type": "code",
 "metadata": {
 "id": "nivITWggl-KF",
 "colab_type": "code",
 "outputId": "e0831f75-1e75-4654-adfa-62fb1270519e",
 "colab": {
 "base_uri": "https://localhost:8080/",
 "height": 275
 }
 },
 "source": [
 "!pip install --user mlxtend"
 ],
 "execution_count": 0,
 "outputs": [
 {
 "output_type": "stream",
 "text": [
 "Requirement already satisfied: mlxtend in /usr/local/lib/python3.6/dist-packages (0.14.0)\n",
 "Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from mlxtend) (41.4.0)\n",
 "Requirement already satisfied: scipy>=0.17 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (1.3.1)\n",
 "Requirement already satisfied: scikit-learn>=0.18 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (0.21.3)\n",
 "Requirement already satisfied: numpy>=1.10.4 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (1.17.3)\n",
 "Requirement already satisfied: pandas>=0.17.1 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (0.25.3)\n",
 "Requirement already satisfied: matplotlib>=1.5.1 in /usr/local/lib/python3.6/dist-packages (from mlxtend) (3.1.1)\n",
 "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.6/dist-packages (from scikit-learn>=0.18->mlxtend) (0.14.0)\n",
 "Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.17.1->mlxtend) (2018.9)\n",
 "Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.17.1->mlxtend) (2.6.1)\n",
 "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (0.10.0)\n",
 "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (2.4.2)\n",
 "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib>=1.5.1->mlxtend) (1.1.0)\n",
 "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil>=2.6.1->pandas>=0.17.1->mlxtend) (1.12.0)\n"
 ],
 "name": "stdout"
 }
 ]
},
{
 "cell_type": "code",
 "metadata": {
 "id": "Bil4WU6ml-KJ",
 "colab_type": "code",
 "colab": {}
 },
 "source": [
 "import numpy as np\n",
 "\n",
 "#Plotting packages\n",
 "import matplotlib.pyplot as plt\n",
 "import seaborn as sns\n",
 "\n",
 "#Classification Algorithms\n",
 "from sklearn.tree import DecisionTreeClassifier\n",
 "from sklearn.neighbors import KNeighborsClassifier\n",
 "from sklearn.naive_bayes import GaussianNB\n",
 "from sklearn.svm import SVC\n",
 "\n",
 "#Ensemble Methods\n",
 "from sklearn.ensemble import BaggingClassifier\n",
 "from sklearn.ensemble import BaggingRegressor\n",
 "from sklearn.model_selection import cross_val_score, train_test_split\n",
 "from sklearn.ensemble import AdaBoostClassifier\n",
 "\n",
 "#Mlxtend for visualizing classification decision boundaries\n",
 "from mlxtend.plotting import plot_decision_regions"
 ],
 "execution_count": 0,
 "outputs": []
},
{
 "cell_type": "code",
 "metadata": {
 "id": "0Vjq9Vpvl-KM",
 "colab_type": "code",
 "colab": {}
 },
 "source": [
 "# Generating Data1\n",
 "\n",
 "np.random.seed(100)\n",
 "\n",
 "a = np.random.multivariate_normal([2,2],[[0.5,0], [0,0.5]], 200)\n",
 "b = np.random.multivariate_normal([4,4],[[0.5,0], [0,0.5]], 200)\n",
 "\n",
 "Data1_X = np.vstack((a,b))\n",
 "Data1_Y = np.hstack((np.ones(200).T,np.zeros(200).T)).astype(int)\n",
 "\n",
 "\n",
 "# Generating Data2\n",
 "\n",
 "np.random.seed(100)\n",
 "\n",
 "a1 = np.random.multivariate_normal([2,2],[[0.25,0], [0,0.25]],200)\n",
 "a2 = np.random.multivariate_normal([2,4],[[0.25,0], [0,0.25]],200)\n",
 "a3 = np.random.multivariate_normal([4,2],[[0.25,0], [0,0.25]],200)\n",
 "a4 = np.random.multivariate_normal([4,4],[[0.25,0], [0,0.25]],200)\n",
 "\n",
 "Data2_X = np.vstack((a1,a4,a2,a3))\n",
 "Data2_Y = np.hstack((np.ones(400).T,np.zeros(400).T)).astype(int)\n",
 "\n",
 "\n",
 "# Generating Data3\n",
 "\n",
 "np.random.seed(100)\n",
 "\n",
 "a1 = np.random.uniform(4,6,[200,2])\n",
 "a2 = np.random.uniform(0,10,[200,2])\n",
 "\n",
 "Data3_X = np.vstack((a1,a2))\n",
 "Data3_Y = np.hstack((np.ones(200).T,np.zeros(200).T)).astype(int)\n",
 "\n",
 "\n",
 "# Generating Data4\n",
 "\n",
 "np.random.seed(100)\n",
 "\n",
 "Data4_X = np.random.uniform(0,12,[500,2])\n",
 "Data4_Y = np.ones([500]).astype(int)\n",
 "Data4_Y[np.multiply(Data4_X[:,0],Data4_X[:,0]) + np.multiply(Data4_X[:,1],Data4_X[:,1]) - 100 < 0 ] = 0"
 ],
 "execution_count": 0,
 "outputs": []
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "8X2ADvEpl-KO",
 "colab_type": "text"
 },
 "source": [
 "### 1. Decision Tree"
 ]
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "1f4TtwrFl-KO",
 "colab_type": "text"
 },
 "source": [
 "Use __Data3__ to answer the following questions."
 ]
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "_FHhp4IBl-KP",
 "colab_type": "text"
 },
 "source": [
 "**Question 1a:** Compute and print the 10-fold cross-validation accuracy using decision tree classifiers with max_depth = 2,4,6,8,10, and 50. "
 ]
},
{
 "cell_type": "code",
 "metadata": {
 "id": "enc8uAOcl-KQ",
 "colab_type": "code",
 "outputId": "64cafbdd-7bf0-4d3b-a79d-7aefd8525748",
 "colab": {
 "base_uri": "https://localhost:8080/",
 "height": 119
 }
 },
 "source": [
 "for depth in [2, 4, 6, 8, 10, 50]:\n",
 " dt = DecisionTreeClassifier(max_depth=depth)\n",
 " dt_scores = cross_val_score(dt, Data3_X, Data3_Y, cv=10, scoring='accuracy')\n",
 " avg_acc = dt_scores.mean()\n",
 " print(\"max_depth: {}, accuracy: {}\".format(depth, avg_acc))"
 ],
 "execution_count": 0,
 "outputs": [
 {
 "output_type": "stream",
 "text": [
 "max_depth: 2, accuracy: 0.8724999999999999\n",
 "max_depth: 4, accuracy: 0.9724999999999999\n",
 "max_depth: 6, accuracy: 0.9674999999999999\n",
 "max_depth: 8, accuracy: 0.9524999999999999\n",
 "max_depth: 10, accuracy: 0.9499999999999998\n",
 "max_depth: 50, accuracy: 0.9374999999999998\n"
 ],
 "name": "stdout"
 }
 ]
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "i_Yu41P_l-KS",
 "colab_type": "text"
 },
 "source": [
 "**Question 1b:** For what values of max_depth did you observe the lowest accuracy? What is this phenomenon called?"
 ]
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "tMsVXk20l-KT",
 "colab_type": "text"
 },
 "source": [
 "**Answer:** \n",
 "\n",
 "For max_depth 2 the lowest accuracy is observed. It is due the phenomenon of under fitted model as depth is too low."
 ]
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "nSnVwx2Sl-KU",
 "colab_type": "text"
 },
 "source": [
 "**Question 1c:** What accuracy did you observe for max depth=50? What is the difference between this accuracy and the highest accuracy? What is this phenomenon called?"
 ]
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "vugxpvkNl-KV",
 "colab_type": "text"
 },
 "source": [
 "**Answer:** \n",
 "At max depth=50, the accuracy decreased as model might be exhibiting over fitting phenomenon. As depth is too high model become complex and overfitting so not able to generalize on the unseeen instances."
 ]
},
{
 "cell_type": "markdown",
 "metadata": {
 "id": "tFShBfmCl-KW",
 "colab_type": "text"
 },
 "source": [
 "**Question 1d:** Plot decision regions for the above decision tree models"
 ]
},
{
 "cell_type": "code",
 "metadata": {
 "id": "gWtQ03g4l-KX",
 "colab_type": "code",
 "outputId": "ef8332d7-01c0-4e0f-ba08-4d109fee736b",
 "colab": {
 "base_uri": "https://localhost:8080/",
 "height": 519
 }
 },
 "source": [
 "fig = plt.figure(figsize=(20, 8))\n",
 "count = 0;\n",
 "for depth in [2, 4, 6, 8, 10, 50]:\n",
 " count = count + 1;\n",
 " dt = DecisionTreeClassifier(max_depth=depth)\n",
 " dt.fit(Data3_X, Data3_Y)\n",
 " ax = plt.subplot(2,3,count)\n",
 " fig = plot_decision_regions(X=Data3_X, y=Data3_Y, clf=dt, legend=2)\n",
 " plt.title('Decision Tree with max_depth:'+str(depth))\n",
 "plt.show()"
 ],
 "execution_count": 0,
 "outputs": [

Hands-on Exercise CLASS Module In [ ]: !pip install --user mlxten...