Using Back Propagation for Digit Recognition

[kinath_ru]

Hi all,
I'm going to develop an application to recognize handwritten digits. Here I am using the MNIST dataset (http://cis.jhu.edu/~sachin/digit/digit.html). My approach to train the network is like this.
1) create a List<List<Bitmap>> from the dataset
2) convert them to double[] arrays
3) make each of the above array as input( double [][])
4) create corresponding output array ( double [][])
5) Use back propagation for training.

Note : Each bitmap is size of 28*28 pixels and corresponding double array contains 784 elements.
I'm training the digits from 0-9
Therefore each input array consist of 784 elements and each output array consist of 10 elements

Using above information I created the following method to train the network

Code: Select all

/// <summary>
        /// Train using Back Propogation
        /// </summary>
        /// <param name="inputData">The input data.</param>
        /// <param name="outputData">The output data.</param>
        /// <param name="iterations">The iterations.</param>
        /// <param name="neurons">The neurons.</param>
        /// <returns></returns>
        public string[] BackPropogationTrain(double[][] inputData, double[][] outputData, int iterations)
        {
            System.Diagnostics.Debug.WriteLine("Training Started...");
            bool needStopping = false;
            int iterateCount = 0;
            double error = 0;
            string[] results = new string[2];

            // initialize input and output values
            double[][] input = inputData;
            double[][] output = outputData;

            network = new AForge.Neuro.ActivationNetwork(new AForge.Neuro.BipolarSigmoidFunction(2),
                784, // 784 inputs (coz each array corresponding to an image consists of 784 elements )
                784, //784 neurons in the first layer  (corresponding to input)
                10); //10 neurons in the second layer (corresponding to 10 digits )
            network.Randomize();
            bpteacher = new AForge.Neuro.Learning.BackPropagationLearning(network);
            bpteacher.LearningRate = 1;

            while (!needStopping)
            {
                error = bpteacher.RunEpoch(input, output);
                if (error == 0) //If the error rate is 0
                {
                    break;
                }
                else if (Math.Round(error, 2) == 0) // If the error rate is 0 to the second decimal point
                {
                    break;
                }
                else if (iterateCount < iterations) //If the given iteraions are completed
                {
                    iterateCount++;
                }
                else
                {
                    needStopping = true;
                }
                iterateCount++;
                System.Diagnostics.Debug.WriteLine("Iteration  :\t" + iterateCount + " \tError Rate :\t" + error);
            }
            System.Diagnostics.Debug.WriteLine("Error Rate : " + error);
            results[0] = error.ToString();
            System.Diagnostics.Debug.WriteLine("Iterations : " + iterateCount);
            results[1] = iterateCount.ToString();
            System.Diagnostics.Debug.WriteLine("Training Completed...");
            return results;
        }



My problem is this.
* Though I train with a large amout of data, at each iteration the error rate doesn't decrease with a considerable amount. Even for 3 images for each digit, training doesn't complete for hours. I want to know if I'm doing the back propagation in a wrong manner. And whether there are any issues with my training method.
I tried to explain the question at my best, sorry for the lengthy question.
Any help would be appreciated.

Thank you very much.

[andrew.kirillov]

Hello,

1) Make sure values of input/output data are in the [-1,1] range (for output values it is critical);
2) Play with alpha parameter of sigmoid function;
3) Play with other learning parameters, like learning rate.

[kinath_ru]

Hello,

1) Make sure values of input/output data are in the [-1,1] range (for output values it is critical);
2) Play with alpha parameter of sigmoid function;
3) Play with other learning parameters, like learning rate.

Hi Andrew,

Thanks for the quick reply. I'm using [-1,1]range for the outputs because I'm using the BiPolar Sigmoid function. But I use [0,1] in input arrays. I'll work with your instructions. Thanks a lot.

[andrew.kirillov]

Try [-1, 1] range for input as well and you should see the change. Here is the problem ... On the training phase we are searching for optimal weights (those which will solve the task we need). If you have 0s in your inputs, then weights of the first layer will not change for corresponding 0 inputs. See here.