Comparison of search algorithms in two-stage neural network training for optical character recognition of handwritten digits

WCU Author/Contributor (non-WCU co-authors, if there are any, appear on document)
Patrik Wayne Gilley (Creator)
Western Carolina University (WCU )
Web Site:
Yanjun Yan

Abstract: Neural networks are a powerful machine learning technique that give a system the ability to develop multiple levels of abstraction to interpret data. Such networks require training to develop the neural layers and neuron weights in order to form reasonable conclusions from the data being interpreted. The conventional training method is to use backpropagation to feed the error between a neural network’s actual output and desired output back through the neural network to adjust the neural synapses’ weights to minimize such errors on subsequent training data. However, backpropagation has several limitations to its effectiveness in training neural networks. The most relevant limitation of backpropagation is that it tends to become trapped in local optimum solutions. This research studied the effectiveness in using search algorithms to improve upon a feed-forward Multi-Layer Perceptron artificial neural network trained by backpropagation in classifying handwritten digits. The search algorithms used for this research were the Bare Bones Fireworks Algorithm (BBFWA), Canonical Particle Swarm Optimization (PSO), and Cooperative Particle Swarm Optimization (CPSO) algorithms. Two sets of parameters for the BBFWA were tested in this study in order to examine the effects of parameters on the algorithm. The handwritten digit classification data was carried out on the MNIST handwritten character database, a common benchmark for handwritten character recognition. A neural network was trained with backpropagation, and then the search algorithms were seeded with its weights so that they could search for better neural network weight configurations. The complexity of using images of handwritten characters with a feed-forward Multi-Layer Perceptron resulted in a high degree of dimensionality in the problem, which negatively impacted the Particle Swarm Optimization algorithms. To analyze the impact of the problem dimensionality, the neural network was also tested with a PCA compressed MNIST database. It was found that the BBFWA performed the best out of the three algorithms on both datasets, as it was able to consistently improve upon the performance of the original neural networks. Between the two sets of BBFWA parameters, the simulation results indicated that the second parameter set outperformed the first parameter set in terms of both classification accuracy and fitness trends.

Additional Information

Language: English
Date: 2020
classification, fireworks algorithm, neural networks, particle swarm optimization

Email this document to