{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "8c18734b",
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"np.random.seed(23)"
]
},
{
"cell_type": "markdown",
"id": "ac3d2bf9",
"metadata": {},
"source": [
"# 1.\n",
"\n",
"## Preparation \n",
"\n",
"We firtst form our dataset and, from the given features, determine the condition list, which we will use for bootstraping."
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "802af35d",
"metadata": {},
"outputs": [
{
"data": {
   "text/plain": [
   "array([[ 1. , 0. , 0. , 7. ],\n",
   "    [ 0. , 1. , 0. , 20. ],\n",
   "    [ 1. , 0. , 0. , 1. ],\n",
   "    [ 0. , 1. , 0. , 7.5],\n",
   "    [ 1. , 0. , 0. , 1. ],\n",
   "    [ 1. , 0. , 0. , 0.8]])"
   ]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_vals = np.array([[1, 0, 0, 7], [0, 1, 0, 20], [1, 0, 0, 1], [0, 1, 0, 7.5], [1, 0, 0, 1], [1, 0, 0, .8]])\n",
"data_vals"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "cee9f77f",
"metadata": {},
"outputs": [],
"source": [
"data_labels = ['Apple', 'Watermelon', 'Cherry', 'Apple', 'Strawberry', 'Cherry']"
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "1070aaed",
"metadata": {},
"outputs": [
{
"data": {
   "text/html": [
   "<div>\n",
   "<style scoped>\n",
   " .dataframe tbody tr th:only-of-type {\n",
   "    vertical-align: middle;\n",
   " }\n",
   "\n",
   " .dataframe tbody tr th {\n",
   "    vertical-align: top;\n",
   " }\n",
   "\n",
   " .dataframe thead th {\n",
   "    text-align: right;\n",
   " }\n",
   "</style>\n",
   "<table border=\"1\" class=\"dataframe\">\n",
   " <thead>\n",
   " <tr style=\"text-align: right;\">\n",
   "    <th></th>\n",
   "    <th>Red</th>\n",
   "    <th>Green</th>\n",
   "    <th>Blue</th>\n",
   "    <th>Size (cm)</th>\n",
   "    <th>Fruit (Label)</th>\n",
   " </tr>\n",
   " </thead>\n",
   " <tbody>\n",
   " <tr>\n",
   "    <th>0</th>\n",
   "    <td>1.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>7.0</td>\n",
   "    <td>Apple</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>1</th>\n",
   "    <td>0.0</td>\n",
   "    <td>1.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>20.0</td>\n",
   "    <td>Watermelon</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>2</th>\n",
   "    <td>1.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>1.0</td>\n",
   "    <td>Cherry</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>3</th>\n",
   "    <td>0.0</td>\n",
   "    <td>1.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>7.5</td>\n",
   "    <td>Apple</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>4</th>\n",
   "    <td>1.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>1.0</td>\n",
   "    <td>Strawberry</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>5</th>\n",
   "    <td>1.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>0.0</td>\n",
   "    <td>0.8</td>\n",
   "    <td>Cherry</td>\n",
   " </tr>\n",
   " </tbody>\n",
   "</table>\n",
   "</div>"
   ],
   "text/plain": [
   " Red Green Blue Size (cm) Fruit (Label)\n",
   "0 1.0 0.0 0.0    7.0       Apple\n",
   "1 0.0 1.0 0.0    20.0 Watermelon\n",
   "2 1.0 0.0 0.0    1.0    Cherry\n",
   "3 0.0 1.0 0.0    7.5       Apple\n",
   "4 1.0 0.0 0.0    1.0 Strawberry\n",
   "5 1.0 0.0 0.0    0.8    Cherry"
   ]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data_df = pd.DataFrame(data_vals, columns=['Red', 'Green', 'Blue', 'Size (cm)'])\n",
"data_df['Fruit (Label)'] = data_labels\n",
"data_df"
]
},
{
"cell_type": "markdown",
"id": "a5b95a77",
"metadata": {},
"source": [
"The condition list is as follows (we remove the 'Color == Blue?' condition, as there are no frutis of blue color in our dataset)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "4d85d6c4",
"metadata": {},
"outputs": [
{
"data": {
   "text/html": [
   "<div>\n",
   "<style scoped>\n",
   " .dataframe tbody tr th:only-of-type {\n",
   "    vertical-align: middle;\n",
   " }\n",
   "\n",
   " .dataframe tbody tr th {\n",
   "    vertical-align: top;\n",
   " }\n",
   "\n",
   " .dataframe thead th {\n",
   "    text-align: right;\n",
   " }\n",
   "</style>\n",
   "<table border=\"1\" class=\"dataframe\">\n",
   " <thead>\n",
   " <tr style=\"text-align: right;\">\n",
   "    <th></th>\n",
   "    <th>Condition List</th>\n",
   " </tr>\n",
   " </thead>\n",
   " <tbody>\n",
   " <tr>\n",
   "    <th>0</th>\n",
   "    <td>Color == Red?</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>1</th>\n",
   "    <td>Color == Green?</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>2</th>\n",
   "    <td>Size < 1?</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>3</th>\n",
   "    <td>1 <= Size < 7</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>4</th>\n",
   "    <td>7 <= Size < 20</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>5</th>\n",
   "    <td>Size >= 20?</td>\n",
   " </tr>\n",
   " </tbody>\n",
   "</table>\n",
   "</div>"
   ],
   "text/plain": [
   " Condition List\n",
   "0 Color == Red?\n",
   "1 Color == Green?\n",
   "2    Size < 1?\n",
   "3 1 <= Size < 7\n",
   "4 7 <= Size < 20\n",
   "5    Size >= 20?"
   ]
},
"execution_count": 5,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"conds = np.array(['Color == Red?',\n",
"       'Color == Green?',\n",
"       'Size < 1?',\n",
"       '1 <= Size < 7',\n",
"       '7 <= Size < 20',\n",
"       'Size >= 20?'])\n",
"\n",
"cond_lst = pd.DataFrame(conds, columns=['Condition List'])\n",
"cond_lst"
]
},
{
"cell_type": "markdown",
"id": "5772ce59",
"metadata": {},
"source": [
"We randomly create three bootstraping subsets from the training dataset. We also randomly take $\\sqrt{6}\\approx 3$ features from the given condition list, for each of those subsets. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "34375b53",
"metadata": {},
"outputs": [
{
"data": {
   "text/html": [
   "<div>\n",
   "<style scoped>\n",
   " .dataframe tbody tr th:only-of-type {\n",
   "    vertical-align: middle;\n",
   " }\n",
   "\n",
   " .dataframe tbody tr th {\n",
   "    vertical-align: top;\n",
   " }\n",
   "\n",
   " .dataframe thead th {\n",
   "    text-align: right;\n",
   " }\n",
   "</style>\n",
   "<table border=\"1\" class=\"dataframe\">\n",
   " <thead>\n",
   " <tr style=\"text-align: right;\">\n",
   "    <th></th>\n",
   "    <th>Btstrp1</th>\n",
   "    <th>Btstrp2</th>\n",
   "    <th>Btstrp3</th>\n",
   " </tr>\n",
   " </thead>\n",
   " <tbody>\n",
   " <tr>\n",
   "    <th>0</th>\n",
   "    <td>3</td>\n",
   "    <td>3</td>\n",
   "    <td>4</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>1</th>\n",
   "    <td>0</td>\n",
   "    <td>2</td>\n",
   "    <td>1</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>2</th>\n",
   "    <td>1</td>\n",
   "    <td>1</td>\n",
   "    <td>2</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>3</th>\n",
   "    <td>0</td>\n",
   "    <td>3</td>\n",
   "    <td>3</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>4</th>\n",
   "    <td>5</td>\n",
   "    <td>3</td>\n",
   "    <td>5</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>5</th>\n",
   "    <td>4</td>\n",
   "    <td>5</td>\n",
   "    <td>5</td>\n",
   " </tr>\n",
   " </tbody>\n",
   "</table>\n",
   "</div>"
   ],
   "text/plain": [
   " Btstrp1 Btstrp2 Btstrp3\n",
   "0    3    3    4\n",
   "1    0    2    1\n",
   "2    1    1    2\n",
   "3    0    3    3\n",
   "4    5    3    5\n",
   "5    4    5    5"
   ]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"bootstraps = []\n",
"for _ in range(3):\n",
" bootstraps.append(np.random.choice(6, 6))\n",
"bootstraps = np.array(bootstraps).T\n",
"bootstraps_df = pd.DataFrame(bootstraps, columns=['Btstrp1', 'Btstrp2', 'Btstrp3'])\n",
"bootstraps_df"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "10ef76e9",
"metadata": {},
"outputs": [
{
"data": {
   "text/html": [
   "<div>\n",
   "<style scoped>\n",
   " .dataframe tbody tr th:only-of-type {\n",
   "    vertical-align: middle;\n",
   " }\n",
   "\n",
   " .dataframe tbody tr th {\n",
   "    vertical-align: top;\n",
   " }\n",
   "\n",
   " .dataframe thead th {\n",
   "    text-align: right;\n",
   " }\n",
   "</style>\n",
   "<table border=\"1\" class=\"dataframe\">\n",
   " <thead>\n",
   " <tr style=\"text-align: right;\">\n",
   "    <th></th>\n",
   "    <th>Conds for Btstrp1</th>\n",
   "    <th>Conds for Btstrp2</th>\n",
   "    <th>Conds for Btstrp3</th>\n",
   " </tr>\n",
   " </thead>\n",
   " <tbody>\n",
   " <tr>\n",
   "    <th>0</th>\n",
   "    <td>Size < 1?</td>\n",
   "    <td>Color == Red?</td>\n",
   "    <td>Size < 1?</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>1</th>\n",
   "    <td>1 <= Size < 7</td>\n",
   "    <td>Size >= 20?</td>\n",
   "    <td>1 <= Size < 7</td>\n",
   " </tr>\n",
   " <tr>\n",
   "    <th>2</th>\n",
   "    <td>Size >= 20?</td>\n",
   "    <td>7 <= Size < 20</td>\n",
   "    <td>Color == Red?</td>\n",
   " </tr>\n",
   " </tbody>\n",
   "</table>\n",
   "</div>"
   ],
   "text/plain": [
   " Conds for Btstrp1 Conds for Btstrp2 Conds for Btstrp3\n",
   "0       Size < 1? Color == Red?       Size < 1?\n",
   "1 1 <= Size < 7    Size >= 20? 1 <= Size < 7\n",
   "2    Size >= 20? 7 <= Size < 20 Color == Red?"
   ]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cond_ids = []\n",
"for _ in range(3):\n",
" cond_ids.append(np.random.choice(6, 3, replace=False))\n",
"cond_ids = np.array(cond_ids).T\n",
"conds_df = pd.DataFrame(conds[cond_ids], columns=['Conds for Btstrp1', 'Conds for Btstrp2', 'Conds for Btstrp3'])\n",
"conds_df"
]
},
{
"cell_type": "markdown",
"id": "469dba26",
"metadata": {},
"source": [
"## Calculation\n",
"\n",
"Now we build Decision Trees (based on Gini Impurity Index) for every bootstrap subset. \n",
"\n",
"### Decision Tree 1 for Bootstrap 1 and the 1st condition list\n",
"\n",
"The first bootstrap subset and its corresponding conditions list looks as follows:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "68246056",
"metadata": {},
"outputs": [
{
"data": {
   "text/html": [
   "<div>\n",
   "<style scoped>\n",
   " .dataframe tbody tr th:only-of-type {\n",
   "    vertical-align: middle;\n",
   " }\n",
   "\n",
   " .dataframe tbody tr th {\n",
   "    vertical-align: top;\n",
   " }\n",
   "\n",
   " .dataframe thead th {\n",
   "    text-align: right;\n",
   " }\n",
   "</style>\n",
   "<table border=\"1\" class=\"dataframe\">\n",
   " <thead>\n",
   " <tr style=\"text-align: right;\">\n",
   "    <th></th>\n",
   "    <th>Red</th>\n",
   "    <th>Green</th>\n",
   "    <th>Blue</th>\n",
   "    <th>Size (cm)</th>\n",
   "    <th>Fruit (Label)</th>\n",
   " </tr>\n",
   " </thead>\n",
   " <tbody>\n",
   " <tr>\n",
   "    <th>3</th>\n",
   "    <td>0.0</td>\n",
   "    <td>1.0</td>\n",

1. (10 points) Below, L(w) : Rd R are loss functions. For each L, (...