Neural Network for Recognition of Handwritten Digits

Neural Network for Recognition of Handwritten Digits

Introduction

This article chronicles the development of an artificial neural network designed to recognize handwritten digits. Although some theory of neural networks is given here, it would be better if you already understood some neural network concepts, like neurons, layers, weights, and backpropagation.

The neural network described here is not a general-purpose neural network, and it's not some kind of a neural network workbench. Rather, we will focus on one very specific neural network (a five-layer convolutional neural network) built for one very specific purpose (to recognize handwritten digits).

The idea of using neural networks for the purpose of recognizing handwritten digits is not a new one. The inspiration for the architecture described here comes from articles written by two separate authors. The first is Dr. Yann LeCun, who was an independent discoverer of the basic backpropagation algorithm. Dr. LeCun hosts an excellent site on his research into neural networks. In particular, you should view his "Learning and Visual Perception" section, which uses animated GIFs to show results of his research. The MNIST database (which provides the database of handwritten digits) was developed by him. I used two of his publications as primary source materials for much of my work, and I highly recommend reading his other publications too (they're posted at his site). Unlike many other publications on neural networks, Dr. LeCun's publications are not inordinately theoretical and math-intensive; rather, they are extremely readable, and provide practical insights and explanations. His articles and publications can be found here. Here are the two publications that I relied on:

One of my goals here was to reproduce the accuracy achieved by Dr. LeCun, who was able to train his neural network to achieve 99.18% accuracy (i.e., an error rate of only 0.82%). This error rate served as a type of "benchmark", guiding my work.

As a final introductory note, I'm not overly proud of the source code, which is most definitely an engineering work-in-progress. I started out with good intentions, to make source code that was flexible and easy to understand and to change. As things progressed, the code started to turn ugly. I began to write code simply to get the job done, sometimes at the expense of clean code and comprehensibility. To add to the mix, I was also experimenting with different ideas, some of which worked and some of which did not. As I removed the failed ideas, I did not always back out all the changes and there are therefore some dangling stubs and dead ends. I contemplated the possibility of not releasing the code. But that was one of my criticisms of the articles I read: none of them included code. So, with trepidation and the recognition that the code is easy to criticize and could really use a re-write, here it is.

TABLE OF CONTENT

Title Page———i

Certification——–ii

Dedication———iii

Acknowledgement——-iv

Abstract ———vi

Table of Content——–vii

Chapter One

1.0 Introduction ——-1

1.1 Statement of Problem——4

1.2 Purpose of the Study——5

1.3 Significance of Study——8

1.4 Limitation——–9

1.5 Scope of Study——-11

Chapter Two

2.0 Review of Related Literature —-12

2.6 Summary of Literature Review—- 19

Chapter Three

3.0 Research Methodology and Procedure—22

3.1 Population ——–22

3.2 Sample and Sampling Technique—-22

3.3 Validation of the Instrument —-23

3.4 Reliability of the Instrument —–23

3.5 Data Analysis——-23

Chapter Four

4.0 Presentation and Discussion of Result—24

4.1 Analysis and interpretaion of Data—25

4.2 Discussion of Results——38

Chapter Five

5.0 Summary, Conclusion and Recommendation  –40

5.1 Summary——–40

5.2 Conclusion——–41

5.3 Recommendation——42

References ———45

Appendix 1——–47

Appendix ———50