{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In this project we have implemented the sparse autoencoder, which is one approach to automatically learn features from unlabeled data. The following functions are filled in:\n",
"<ol>\n",
" <li>sampleIMAGES()</li>\n",
" <li>sparseAutoencoderCost()</li>\n",
" <li>computeNumericalGradient()</li>\n",
"</ol>\n",
"The function sampleIMAGES() does not accept arguments. It reads the 'images.npy' file which contains 10 images, each 512 pixels square. The function randomly picks 10000 8x8 patches. The patches are converted to 64 numpy array so an array of shape (10000, 64) is returned. The autoencoder performs better for the normalized data so the samples are normalized using the function normalizeData(). \n",
"\n",
"The function sparseAutoencoderCost() accept following arguments:\n",
"<ol>\n",
" <li>theta - represents unrolled matrices W1, W2, b1 and b2</li>\n",
" <li>visibleSize - the number of columns of input data </li>\n",
" <li> hiddenSize - the size of the hidden layer</li>\n",
" <li>decayWeight -decay parameter that controls the importance of the two decay term in a cost function.</li>\n",
" <li>sparsityParam - ta sparsity parameter,</li>\n",
" <li> beta- controls the importance of the sparsity penalty term in a cost function</li>\n",
" <li> data - the training data. </li>\n",
" \n",
"</ol>\n",
"\n",
"At first we run a forward propagation step in order to compute all the activations throughout the network. Net inputs and activation units are calculated: z_h, a_h, z_out and a_out. As the activation function is used sigmoid function. The total cost function is calculated. The first term of cost function represents (one-half) squared-error cost function, the second term is a regularization term and the third term of the cost function represents sparsity cost function. Using backpropagation algorithm, the W1grad, partial derivatives W2grad, b1grad and b2grad are calculated. The function returns cost function and all partial derivatives.\n",
"\n",
"The purpose of the function computeNumericalGradient() is to test the results obtained by using the function parseAutoencoderCost(). The function computeNumericalGradient() accepts a function J and a vector of parameters. The the central difference method is used to calculate the gradient of the provided function. "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"data": {
   "text/plain": [
   "<Figure size 640x480 with 1 Axes>"
   ]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
   "[[38. 38.]\n",
   " [12. 12.]]\n",
   "The above two columns you get should be very similar.\n",
   "(Left-Your Numerical Gradient, Right-Analytical Gradient)\n",
   "\n",
   "\n",
   "2.1452381569477388e-12\n",
   "Norm of the difference between numerical and analytical gradient (should be < 1e-9)\n",
   "\n",
   "\n",
   "Computing numerical

Answer the following questions. 1. Consider the following set...