{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"anaconda-cloud": {},
"kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
},
"language_info": {
   "codemirror_mode": {
   "name": "ipython",
   "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
},
"toc": {
   "base_numbering": 1,
   "nav_menu": {
   "height": "48px",
   "width": "252px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
   "height": "588px",
   "left": "0px",
   "right": "1388px",
   "top": "108px",
   "width": "317.969px"
   },
   "toc_section_display": "block",
   "toc_window_display": true
},
"varInspector": {
   "cols": {
   "lenName": 16,
   "lenType": 16,
   "lenVar": 40
   },
   "kernels_config": {
   "python": {
      "delete_cmd_postfix": "",
      "delete_cmd_prefix": "del ",
      "library": "var_list.py",
      "varRefreshCmd": "print(var_dic_list())"
   },
   "r": {
      "delete_cmd_postfix": ") ",
      "delete_cmd_prefix": "rm(",
      "library": "var_list.r",
      "varRefreshCmd": "cat(var_dic_list()) "
   }
   },
   "types_to_exclude": [
   "module",
   "function",
   "builtin_function_or_method",
   "instance",
   "_Feature"
   ],
   "window_display": false
},
"colab": {
   "name": "HW-07-Classification-NaiveBayes-LogisticRegressionText-CIFAR10-Copy2-2.ipynb",
   "provenance": [],
   "collapsed_sections": [
   "DfGdPJCToxCX",
   "kRY8I-HVoxEE",
   "ceZ4cM_IoxFM",
   "6NaodSg5oxFQ",
   "M-SNWLFGoxF6",
   "V0q-o-eioxF7",
   "C6mKoeAToxGz",
   "_LxCYeImoxHX",
   "lmVUAJGuoxIJ"
   ]
}
},
"cells": [
{
   "cell_type": "markdown",
   "metadata": {
   "id": "4aRHFe9Cow8I",
   "colab_type": "text"
   },
   "source": [
   "# Homework 07\n",
   "\n",
   "`"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "gJOQLWaEow8L",
   "colab_type": "text"
   },
   "source": [
   "# Task: Logistic Regression basics"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "xFWraw2low8M",
   "colab_type": "text"
   },
   "source": [
   "## Task: Unnormalized perpendicular distance"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "fJBiPWa2ow8O",
   "colab_type": "text"
   },
   "source": [
   "Consider a three-class classification problem with the following separating hyperplanes:\n",
   "\n",
   "\\begin{align}\n",
   "1^{st\\:}class\\:\\:\\:\\:\\:\\:4x_1+x_{2\\:\\:}-2=0 \\\\\n",
   "2^{nd\\:}class\\:\\:\\:\\:\\:\\:-2x_1+2x_{2\\:\\:}-11=0 \\\\\n",
   "3^{rd\\:}class\\:\\:\\:\\:\\:-3x_1-3x_{2\\:\\:}-1=0 \n",
   "\\end{align}\n",
   "\n",
   "\n",
   "Calculate the score (unnormalized perpendicular distance) for each class for the test case $\\left(x_1,x_2\\right)=\\:\\left(-1,1\\right) $\n",
   "\n",
   "Complete the code below to get the results"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "id": "R8jWGc0Bow8T",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 34
   },
   "outputId": "dd1c0a6d-4f25-4f48-b99a-425846834729"
   },
   "source": [
   "import numpy as np\n",
   "\n",
   "W = np.transpose(np.array([[4., 1.], \n",
   "          [-2., 2.], \n",
   "          [-3., -3.]])).reshape(2, 3)\n",
   "b = np.array([-2.,-11., -1.])\n",
   "x = np.array([-1., 1.])\n",
   "#==================================================#\n",
   "#             Your code starts here          #\n",
   "#==================================================#\n",
   "scores = np.abs((np.dot(x, W) + b))/np.sqrt(np.sum(W**2, axis=0))\n",
   "print(np.round(scores,3))\n",
   "#==================================================#\n",
   "#             Your code ends here             #\n",
   "#             Please don't add code below here #\n",
   "#==================================================#"
   ],
   "execution_count": 33,
   "outputs": [
   {
      "output_type": "stream",
      "text": [
         "[1.213 2.475 0.236]\n"
      ],
      "name": "stdout"
   }
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "J2uzeA8-ow8Y",
   "colab_type": "text"
   },
   "source": [
   "## Task: Log Loss for logistic regression"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "BHqxW8haow8Z",
   "colab_type": "text"
   },
   "source": [
   "Consider two points $x_1$ and $x_2$. $x_1$ belongs to class 0 and $x_2$ belongs to class 1 ( $y_1=0 $ and $y_2=1$) \n",
   "\n",
   "A logistic regression model predicts the class of $x_1$ with a probability of 0.3 and predicts the class of $x_2$ with a probability of 0.6\n",
   "\n",
   "The log loss formula for the binary case is as follows : $-\\frac{1}{m}\\sum^m_{i=1}\\left(y_i\\cdot\\:\\log\\:\\left(p_i\\right)\\:+\\:\\left(1-y_i\\right)\\cdot\\log\\left(1-p_i\\right)\\right) $\n",
   "\n",
   "where $m$ is the number of data points , log is Natural Logarithm\n",
   "\n",
   "Calculate the log loss for the points $x_1$ and $x_2$\n",
   "\n",
   "Please report your answer to three decimal places (e.g., report .4554 as .455)."
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "7YrQbuS0ow8a",
   "colab_type": "text"
   },
   "source": [
   "| point| Class| Probability|\n",
   "| --- | --- | --- |\n",
   "| $x_1$ | 0 | 0.3|\n",
   "| $x_2$ | 1 | 0.6|\n"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "id": "JSMENy8Wow8b",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 51
   },
   "outputId": "0d1e6048-fd17-4d1a-a2e9-31296a4b34fc"
   },
   "source": [
   "import numpy as np\n",
   "def LogLossBinary(actual, predicted, eps = 1e-15): \n",
   " predicted = np.minimum(np.maximum(predicted, eps), 1-eps) #avoid precision problems at 0, and 1\n",
   " #==================================================#\n",
   " #             Your code starts here          #\n",
   " #==================================================#\n",
   " return( -np.sum(actual*np.log(predicted) + (1-actual)*np.log((1-predicted)))/len(actual))\n",
   " #==================================================#\n",
   " #             Your code ends here             #\n",
   " #             Please don't add code below here #\n",
   " #==================================================#\n",
   "print(f'{np.round(LogLossBinary(np.array([0, 1]),np.array([0.3, 0.6])), 3)}')\n",
   "print(f'{np.round((- np.log(1-0.3) - np.log(0.6))/2, 3)}')"
   ],
   "execution_count": 36,
   "outputs": [
   {
      "output_type": "stream",
      "text": [
         "0.434\n",
         "0.434\n"
      ],
      "name": "stdout"
   }
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "XxjmnI1cow8m",
   "colab_type": "text"
   },
   "source": [
   "## Task: Log Loss for multinomical logistic regression\n",
   "\n",
   "The log loss (aka cross entropy) formula for the multi case is as follows : $$CXE(actual, predicted) = -\\frac{1}{m}\\sum^m_{i=1}\\left(actual_i\\cdot\\:\\log\\:\\left(predicted_i\\right)\\:)\\right) $$\n",
   "\n",
   "where $m$ is the number of data points, and $log$ is Natural Logarithm\n",
   "\n",
   "Complete the code to calculate the CXE for the test cases provided. Verify you response using the sklearn.metrics.log_loss.\n",
   "\n",
   "Please report your answer to three decimal places (e.g., report .4554 as .455)."
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "id": "LYiaJDhzow8n",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 34
   },
   "outputId": "0838b750-bad8-4a38-a441-b21d1040382f"
   },
   "source": [
   "import numpy as np\n",
   "from sklearn.metrics import log_loss\n",
   "# homemade CXE\n",
   "def cross_entropy(predictions, targets):\n",
   " #==================================================#\n",
   " #             Your code starts here          #\n",
   " #==================================================#\n",
   " cxe = -1/len(predictions)*np.sum(targets*np.log(predictions))\n",
   " #==================================================#\n",
   " #             Your code ends here             #\n",
   " #             Please don't add code below here #\n",
   " #==================================================# \n",
   " return cxe\n",
   "\n",
   "# 2 test cases with 4 possible target classes\n",
   "predictions = np.array([[0.25,0.25,0.25,0.25],\n",
   "                      [0.01,0.01,0.01,0.97]])\n",
   "targets = np.array([[1,0,0,0],\n",
   "                [0,0,0,1]])\n",
   "\n",
   "homemadeCXE = cross_entropy(predictions, targets)\n",
   "print(np.round(log_loss(targets, predictions),3), 'homemade CXE:', np.round(homemadeCXE,3))"
   ],
   "execution_count": 37,
   "outputs": [
   {
      "output_type": "stream",
      "text": [
         "0.708 homemade CXE: 0.708\n"
      ],
      "name": "stdout"
   }
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "20yE1NWmow8r",
   "colab_type": "text"
   },
   "source": [
   "## Task: Gradient Descent for logistic regression"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "7PSVPKLIow8s",
   "colab_type": "text"
   },
   "source": [
   "Assume you are learning a logistic regression model with two a training set consisting of two examples:\n",
   "\n",
   "$x_1$ = 1 belongs to class 0 ($y_1=0$ )<br />\n",
   "$x_2$= 2 belongs to class 1 ($y_2=1 $) <br />\n",
   "\n",
   "The current model weight vector is W = [1,1], where W[0] denotes the bias term.<br />\n",
   "\n",
   "Assume a learning rate, $\\alpha= 0.1$ <br />\n",
   "\n",
   "Assume the gradient is defined as follows:\n",
   "\n",
   " $\\frac{\\delta E}{\\delta W} = \\frac{1}{m}\\sum^m_{i=1}\\left(p\\left(x_i\\right)-y_i\\:\\right)\\cdot X\\:\\:\\:where\\:p\\left(x\\right)\\:=\\:\\frac{1}{1+\\:e^{-W^Tx}}\\:\\: $ and $ m $ is the number of data points\n",
   "\n",
   "What is the value of W after applying one iteration of gradient descent?\n",
   "\n",
   "Please report your response to three decimal places."
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "id": "2UyPP_vDow8z",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 102
   },
   "outputId": "5a96d751-5eb5-4ad0-e1ea-c54dc9ea428d"
   },
   "source": [
   "import numpy as np\n",
   "X=np.array([[1,1],[1,2]])\n",
   "w= np.array([1,1])\n",
   "y= np.array([0,1])\n",
   "#==================================================#\n",
   "#             Your code starts here          #\n",
   "#==================================================#\n",
   "perpDist= np.multiply(w.T, x)\n",
   "p =1 / (1 + np.exp(-perpDist)) #sigmoid\n",
   "gradient = (1/len(w))*np.sum(np.dot(p-y,X))\n",
   "print(f'predictions: {p}')\n",
   "print(f'Gradient: {gradient}')\n",
   "print(f'w before: {w}')\n",
   "lr = 0.1\n",
   "print(f'lr * Gradient: {lr *gradient}')\n",
   "w = w - lr*gradient\n",
   "print(f'w after: {np.round(w,3)}')    \n",
   "#==================================================#\n",
   "#             Your code ends here             #\n",
   "#             Please don't add code below here #\n",
   "#==================================================#"
   ],
   "execution_count": 39,
   "outputs": [
   {
      "output_type": "stream",
      "text": [
         "predictions: [0.26894142 0.73105858]\n",
         "Gradient: -0.13447071068499755\n",
         "w before: [1 1]\n",
         "lr * Gradient: -0.013447071068499756\n",
         "w after: [1.013 1.013]\n"
      ],
      "name": "stdout"
   }
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "C9qM3-7sow85",
   "colab_type": "text"
   },
   "source": [
   "# CIFAR10"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "v5SYOvGQow86",
   "colab_type": "text"
   },
   "source": [
   "DOWNLOAD DATA FROM [HERE] AND PUT IT TO THE DATA FOLDER\n",
   "\n",
   "The [CIFAR-10] dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.\n",
   "\n",
   "The classes are completely mutually exclusive. There is no overlap between automobiles and trucks. \"Automobile\" includes sedans, SUVs, things of that sort. \"Truck\" includes only big trucks. Neither includes pickup trucks. Such a situation is called **multiclass** classification in oppose to **multilabel** classification when each example may have multiple label.\n",
   "\n",
   "One can see state-of-the-art results [here](http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html)"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-27T14:22:30.010553Z",
      "start_time": "2018-06-27T14:22:28.286221Z"
   },
   "id": "-qSy_KDRow88",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "import _pickle as cPickle\n",
   "import tarfile\n",
   "\n",
   "import numpy as np\n",
   "import pandas as pd\n",
   "\n",
   "from sklearn.linear_model import LogisticRegression\n",
   "from sklearn.model_selection import train_test_split\n",
   "from sklearn.metrics import accuracy_score\n",
   "\n",
   "import matplotlib\n",
   "import matplotlib.pyplot as plt\n",
   "import seaborn as sns\n",
   "\n",
   "import warnings\n",
   "warnings.filterwarnings('ignore')\n",
   "%matplotlib inline"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "wRS1_lscow9G",
   "colab_type": "text"
   },
   "source": [
   "Set style for plotting"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-27T14:22:30.024577Z",
      "start_time": "2018-06-27T14:22:30.019040Z"
   },
   "id": "ja6aUtnIow9H",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "sns.set(style=\"whitegrid\", font_scale=1.3)\n",
   "matplotlib.rcParams[\"legend.framealpha\"] = 1\n",
   "matplotlib.rcParams[\"legend.frameon\"] = True"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "uTmSh18Bow9N",
   "colab_type": "text"
   },
   "source": [
   "Fix random seed for reproducibility"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "id": "wQLGGo09ow9O",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "np.random.seed(42)"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "0ZVITO1Pow9Y",
   "colab_type": "text"
   },
   "source": [
   "## Data"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "jSEkHTP_ow9Z",
   "colab_type": "text"
   },
   "source": [
   "Unarchive data"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T00:10:50.811438Z",
      "start_time": "2018-06-24T00:10:50.537432Z"
   },
   "id": "0OiVg7PSow9b",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "tar = tarfile.open(\"data/cifar-10-python.tar.gz\", \"r:gz\")\n",
   "tar.extractall(\"data\")\n",
   "tar.close()"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "GZP456k1ow9f",
   "colab_type": "text"
   },
   "source": [
   "## Reading"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "vurQPckBow9g",
   "colab_type": "text"
   },
   "source": [
   "Data are stored as a memory dump with python $\\text{dict}$ object. It was created using **pickle** function. To read one should \"unpickle\" it."
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-25T13:59:58.774950Z",
      "start_time": "2018-06-25T13:59:58.769771Z"
   },
   "id": "rBgsVcCoow9h",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "def unpickle(file):\n",
   " fo = open(file, 'rb')\n",
   " dict = cPickle.load(fo, encoding=\"latin1\")\n",
   " fo.close()\n",
   " return dict"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "2imsoGE7ow9m",
   "colab_type": "text"
   },
   "source": [
   "Data are also splitted in to 5 pieces for conveniece. Let's read them all and concatenate"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:07.700145Z",
      "start_time": "2018-06-24T18:04:07.096937Z"
   },
   "id": "b4_L4CENow9p",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "for b in range(1, 6):\n",
   " data_batch = unpickle(\"data/cifar-10-batches-py/data_batch_\" + str(b))\n",
   " if b == 1:\n",
   "    X_train = data_batch[\"data\"]\n",
   "    y_train = np.array(data_batch[\"labels\"])\n",
   " else:\n",
   "    X_train = np.append(X_train, data_batch[\"data\"], axis=0)\n",
   "    y_train = np.append(y_train, data_batch[\"labels\"], axis=0)"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "49zjRriLow9t",
   "colab_type": "text"
   },
   "source": [
   "Read test data. In this dataset train/test split is provided by authors of the dataset to be able to consistently evaluate solutions"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:09.412708Z",
      "start_time": "2018-06-24T18:04:09.341606Z"
   },
   "id": "y6z4WJi1ow9u",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "data_batch = unpickle(\"data/cifar-10-batches-py/test_batch\")\n",
   "X_test = data_batch[\"data\"]\n",
   "y_test = np.array(data_batch[\"labels\"])"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "GzUCSX3Row9y",
   "colab_type": "text"
   },
   "source": [
   "Read meta-information file with the names of the classes"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:09.788097Z",
      "start_time": "2018-06-24T18:04:09.779669Z"
   },
   "id": "-AMSZ-V6ow93",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "classes = unpickle(\"data/cifar-10-batches-py/batches.meta\")[\"label_names\"]"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "4kK6tZhuow-A",
   "colab_type": "text"
   },
   "source": [
   "## Pre-processing"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "C95Wnx0sow-C",
   "colab_type": "text"
   },
   "source": [
   "We have too many data"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:11.513813Z",
      "start_time": "2018-06-24T18:04:11.506152Z"
   },
   "id": "CSD0xOR7ow-E",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 51
   },
   "outputId": "6593d542-19a7-434b-be6f-d91bf7e589fc"
   },
   "source": [
   "print(\"Train size:\", X_train.shape[0])\n",
   "print(\"Test size:\", X_test.shape[0])"
   ],
   "execution_count": 50,
   "outputs": [
   {
      "output_type": "stream",
      "text": [
         "Train size: 50000\n",
         "Test size: 10000\n"
      ],
      "name": "stdout"
   }
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "5zciuhHSow-L",
   "colab_type": "text"
   },
   "source": [
   "Let's take only 10% of them to train faster"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:12.856320Z",
      "start_time": "2018-06-24T18:04:12.853162Z"
   },
   "id": "A0asleWtow-N",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "subsample_rate = 0.1\n",
   "np.random.seed(42)"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "NpQNW0cjow-Q",
   "colab_type": "text"
   },
   "source": [
   "We want to preserve the same quantity ratio between classes. In python such an option is called **stratification**. Let's randomly (with fixed initial seed for the sake of reproducibility) divide part of train data"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:13.854036Z",
      "start_time": "2018-06-24T18:04:13.672594Z"
   },
   "id": "yMe8lJK7ow-R",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "X_train, _, y_train, _ = train_test_split(X_train, y_train, stratify=y_train, train_size=subsample_rate, random_state=42)"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "BaZc4M2mow-a",
   "colab_type": "text"
   },
   "source": [
   "The same for test"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:14.572748Z",
      "start_time": "2018-06-24T18:04:14.548444Z"
   },
   "id": "Slb0eB7pow-e",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "X_test, _, y_test, _ = train_test_split(X_test, y_test, stratify=y_test, train_size=subsample_rate, random_state=42)"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "Wv4458Zfow-i",
   "colab_type": "text"
   },
   "source": [
   "Indeed, we preserved the number of objects of each class"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:16.945533Z",
      "start_time": "2018-06-24T18:04:16.937049Z"
   },
   "id": "GQBITDaJow-m",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 187
   },
   "outputId": "698799e8-a3cd-4deb-9ed7-3ee89a776363"
   },
   "source": [
   "unique_train = np.unique(y_train, return_counts=True)\n",
   "list(zip(np.array(classes)[unique_train[0]], unique_train[1]))"
   ],
   "execution_count": 54,
   "outputs": [
   {
      "output_type": "execute_result",
      "data": {
         "text/plain": [
         "[('airplane', 500),\n",
         " ('automobile', 500),\n",
         " ('bird', 500),\n",
         " ('cat', 500),\n",
         " ('deer', 500),\n",
         " ('dog', 500),\n",
         " ('frog', 500),\n",
         " ('horse', 500),\n",
         " ('ship', 500),\n",
         " ('truck', 500)]"
         ]
      },
      "metadata": {
         "tags": []
      },
      "execution_count": 54
   }
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "eZtEO9kHow-w",
   "colab_type": "text"
   },
   "source": [
   "## Visualization"
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "QEchd0nDow-y",
   "colab_type": "text"
   },
   "source": [
   "For now each object has the following shape"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:18.527866Z",
      "start_time": "2018-06-24T18:04:18.521986Z"
   },
   "id": "U6NC91wZow-2",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 34
   },
   "outputId": "eb42dc1d-6581-4234-d2db-353d5709d186"
   },
   "source": [
   "X_train[0].shape"
   ],
   "execution_count": 55,
   "outputs": [
   {
      "output_type": "execute_result",
      "data": {
         "text/plain": [
         "(3072,)"
         ]
      },
      "metadata": {
         "tags": []
      },
      "execution_count": 55
   }
   ]
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "orFX6w6now_B",
   "colab_type": "text"
   },
   "source": [
   "$3072 = 32 \\times 32 \\times 3$ where $32 \\times 32$ is the size of the image in pixels and $3$ is the number of channels (RGB)\n",
   "\n",
   "To show this array as an image let's reshape it in the needed from with the shape $(32, 32, 1)$"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-25T20:19:18.128589Z",
      "start_time": "2018-06-25T20:19:18.123999Z"
   },
   "id": "gY_AW0Haow_H",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "def show_pic(x):\n",
   " plt.imshow(x.reshape((3, 32, 32)).transpose(1, 2, 0).astype(\"uint8\"))\n",
   " plt.axis(\"off\")"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "markdown",
   "metadata": {
   "id": "ffxaEJN7ow_N",
   "colab_type": "text"
   },
   "source": [
   "Draw one pic from each class"
   ]
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:19.232922Z",
      "start_time": "2018-06-24T18:04:19.227362Z"
   },
   "id": "KlIZveK0ow_Q",
   "colab_type": "code",
   "colab": {}
   },
   "source": [
   "classes_idx_examples = np.zeros(10, dtype=np.int)\n",
   "for i in range(10):\n",
   " classes_idx_examples[i] = np.where(y_train == i)[0][0]"
   ],
   "execution_count": 0,
   "outputs": []
},
{
   "cell_type": "code",
   "metadata": {
   "ExecuteTime": {
      "end_time": "2018-06-24T18:04:20.090520Z",
      "start_time": "2018-06-24T18:04:19.431438Z"
   },
   "id": "GZUROA09ow_V",
   "colab_type": "code",
   "colab": {
      "base_uri": "https://localhost:8080/",
      "height": 313
   },
   "outputId": "39abe69d-ca6e-487c-b3de-c1eef24a1e2a"
   },
   "source": [
   "plt.figure(figsize=(12, 5))\n",
   "for i in range(10):\n",
   " plt.subplot(2, 5, i + 1)\n",
   " show_pic(X_train[classes_idx_examples[i]])\n",
   " plt.title(classes[i])"
   ],
   "execution_count": 58,
   "outputs": [
   {
      "output_type": "display_data",
      "data": {
         "image/png":

1. The decision boundary in any binary classification problem can b...