HAND-WRITTEN POSTCODE RECOGNITION BY FUZZY ARTMAP NEURAL NETWORK Yong Haur Tay, Marzuki Khalid*, Kok Khiang Tan and Rubiyah Yusof Centre for Artificial Intelligence and Robotics (CAIRO) Faculty of Electrical Engineering, Universiti Teknologi Malaysia, Jalan Semarak, 54100 Kuala Lumpur. e-mail: [email protected] Tel: 603-2913710 Fax: 603-2970815 (* all correspondence should be sent to)

Abstract: This paper presents a new method of recognising hand-written postcodes using the Fuzzy ARTMAP neural network, a relatively new architecture of the neural network family. It has the capability to process in parallel as well as the ability to learn and make decisions. The image of the specially formatted envelopes are captured by a frame grabber and processed by an image processing software called WiT. Pre-processing consists of region of interest (ROI) identification, noise reduction, image centring and size normalisation. Feature extraction is used to reduce the size of the input to the neural network and yet retaining the information of the image. The Fuzzy ARTMAP neural network performs recognition of the digit. With proper training of the network, it can be shown that the Fuzzy ARTMAP neural network is capable to recognise the hand-written postcodes for automatic mail sorting. A prototype mail sorting machine has been successfully designed and developed at CAIRO, Universiti Teknologi Malaysia. Keywords: Postcode recognition, Adaptive resonance theory (ART), Fuzzy ARTMAP, Mail sorting.

many samples of different handwritings. The Fuzzy ARTMAP neural network model, can even incrementally learn novel patterns without re-training the network[7]. In this work, we develop a prototype fully automatic hand-written postcode recognition machine based on the Fuzzy ARTMAP neural network. The system uses a CCD camera for capturing the image of the envelope. The specially designed postcode area is identified as the region of interest (ROI). The preprocessing and feature extraction processes normalise and extract the important features of the hand-written digits. The already trained Fuzzy ARTMAP neural network recognises and interprets the captured hand-written image into the appropriate ASCII codes which the computer can understand. The letter is then sorted into the appropriate bin through an automated mail sorter. This paper has been organised as follows. The next section presents a brief background of the Adaptive Resonance Theory (ART) and is followed by the Fuzzy ARTMAP neural network. Next, we describe the postcode recognition and letter sorting machine. The techniques used in the pre-processing process as well as the feature extraction process are also discussed. The experimental results and conclusions are discussed in the sections that follow.

1. INTRODUCTION Optical character recognition (OCR) has been used as one of the techniques implemented for postcode recognition, however, this technique does not learned from example. The system designer has to feed the knowledge to the OCR if they want the system to recognise novel handwritings or new characters. Depending on the level of changes, the system might have to be re-designed in order to cater for new handwritings. This is rather cumbersome and labour intensive and, thus, making this conventional technique not easy to be implemented in practice for hand-written character recognition which have lots of different types of handwritings. Artificial neural networks (ANNs), however, have the advantage over such conventional technique as they are capable to learn from examples. ANNs gain their knowledge from the training patterns which made up of

2. ADAPTIVE RESONANCE THEORY Adaptive Resonance Theory (ART) was developed by Carpenter and Grossberg for pattern recognition purposes [5]. There are several variations of the ART neural network, namely, ART1, ART2, ART3, fuzzy ART, fusion ART, fuzzy ARTMAP, and ART-EMAP. The major distinct advantages of the ART neural network paradigm are:• It retains knowledge of previously learned patterns or patterns categories (STABILITY). • It can also learn new patterns (PLASTICITY). ART is a type of competitive learning network, and is suitable for both pattern formation and recognition. When an input pattern is adequately similar to the one stored in the ART’s long term memory (LTM), ART recognises the patterns as belonging to the category, and

generalises the stored category to accommodate new features of the current input pattern. When an input pattern is not adequately similar to any stored category, pattern formation occurs. ART then selects a new category to store the current input. If no more uncommitted nodes are left, the current input will have no response. Hence, stored categories remain stable to irrelevant inputs and yet are sensitive to novel features of the inputs. In such cases, ART makes a satisfactory trade-off between stability and plasticity.

trial.

2.2 The Fuzzy ARTMAP Dynamics The inputs to ARTa and ARTb are in complement code form which is necessary for the successful operation of Fuzzy ARTMAP[1]. For ARTa, if the input is I o = ( a 1 ,. .. , a Ma ) , then I = ( a , a c ) = ( a 1 ,. .. , a M , a 1c , ... , a cM )

2.1 The Fuzzy ARTMAP Architecture ARTMAP (Fig. 1) is a class of neural networks that performs incremental supervised learning of recognition. It also performs multidimensional mapping in response to binary input vectors presented arbitrarily [1]. The Fuzzy ARTMAP extends the ARTMAP by integrating fuzzy logic into the system, which allows it to accept input vector values between 0 and 1. The generalisation is done by replacing both ART1 modules (ARTa and ARTb) of the binary ARTMAP system with fuzzy ART modules. Fuzzy ARTMAP consists of two fuzzy ART modules (ARTa and ARTb) which are linked together via an inter-ART module, Fab. During the learning phase, the input vector I0, is presented to the ARTa and the desired output vector O0, is presented to the ARTb. The ARTa and ARTb modules classify the input and the desired output vector into categories, then the map field (inter-ART module) makes associations from ARTa category to ARTb category. If I0 predicts an incorrect O0, then a mechanism called match tracking is triggered. This mechanism increases the vigilance parameter of ARTa, ρa, by a minimum value and, hence, force the ARTa module to search for another category suitable to be associated with the desired output vector. ρa is then set back to the baseline vigilance parameter, ρ , for every step of learning

where a ic = 1 − a i

The x = a

F1a

( x 1a , . .. , x 2aM a

1 ≤ i ≥ Ma

(2)

activity vector is denoted by ) and the F2a activity vector with Na

number of category is denoted by y a = ( y1a , .. . , y aN a ) . Ma and Na are arbitrary. For ARTb, Oo = (b1 ,. .. , b M b ) , O = (b , b c ) . The F1b activity vector is denoted by

x b = ( x1b ,... , x b2 Mb ) and the F2b activity vector with Nb number of category is denoted by y b = ( y1b ,..., y bNb ) . Mb and Nb are also arbitrary. For the inter-ART module, Fab output vector is denoted by x ab = ( x1ab ,..., x ab N b ) , and the weight vector from the jth F2a node to Fab is denoted by a a b b ab ab w ab = ( w ab are j 1 , ... , w jNb ) . Vectors x , y , x , y , x initialised to zero between input presentations. Initially, the weight vectors wab are set to 1 (uncommitted). The inter-ART module Fab is activated whenever any of the ARTa or ARTb category is active. Hence,

Inter-ART module ARTa module

ARTb module

F ab ρ

F 2a

F 2b reset

reset F 1b

F 1a

O = ( O 0 , O 0 c)

I = ( I0 , I0 c )

F 0a

I0

ρa

(1)

match tracking

F 0b

O0

Fig. 1 A simplified Fuzzy ARTMAP architecture.

ρb

xab=  y b ∧ wab J  ab  wJ  b y 0 

both the ARTa and ARTb is active ARTa is active, ARTb is inactive ARTa is inactive, and ARTb is active both the ARTa and ARTb is inactive

(3) During the learning phase, ARTa vigilance parameter ρa, equals the baseline vigilance, ρ a at the beginning of every input presentation. When Fuzzy ARTMAP receives an input/output pair (I0/O0), ARTa chooses the Jth node of F2a and ARTb chooses the Kth node of F2b. When both ARTb and ARTb are active and xab ≠ 0, then the input/output pairs are associated with the equation w ab jk

1 = 0

j = J and k = K otherwise

(4)

If xab = 0, then there is a mismatch. The InterART module triggers a match tracking mechanism which increases ρa, by a minimum value and hence, forces the ARTa module to search for another category suitable to be associated with the desired output vector.

3. APPLICATION TO POST-CODE RECOGNITION PROBLEM The Fuzzy ARTMAP algorithm is applied to solve the post-code recognition problem for letter sorting in Malaysia. The postcode system in Malaysia consists of five digits which represents the regional post-offices in the country. In this application the Fuzzy ARTMAP neural network is trained to recognise different sets of handwritten digits from 0 to 9. Training of the neural network is done off-line with about 100 samples of each digit taken from hand-written images using a Sony CCD camera. After the training phase the neural network is tested with images taken from envelopes through the CCD camera. The camera is used to capture the image from envelopes which are pre-designed to have the post-codes written in the appropriate boxes. A rectangle which contains five small boxes is located at the bottom of the envelope’s front as shown in Fig. 2, where in each box a digit is inserted. The rectangle is surrounded by four black dots at each corner for the computer to locate the location of the box. The sender has to explicitly write the postcode inside the boxes of the rectangle. Anything outside the rectangle is ignored by the system.

Fig. 2

Example of envelopes with the hand-written post-codes in the post-code box.

4. IMAGE PRE-PROCESSING The envelope's region of interest which is the post-code is captured by the CCD camera which is then converted into a digital image by the frame grabber. This digital image which carries the post-code information is then fed into the pre-processing process. A digitised image refers to a two-dimensional light intensity function f ( x , y ) , where x and y are spatial coordinates and the value of f at any point (x,y) is proportional to the brightness or grey level of the image at that point. A digitised image can be considered as a matrix in which each row and column identify a point in the image and the corresponding matrix element value represents the grey level at that point. The elements of such a digitised array are called picture elements or pixels. The first stage in this recognition system is the pre-processing process, also known as low-level process, which deals with early vision processing. This is a necessary process to eliminate the noise and enhance the image so that the later process can have a better visibility of the image. The goal is to transform the input image into a standard form with minimum noise. Image processing in this system is performed in the spatial domain (the array of pixels comprising the conventional view of the image). In the spatial domain, pixel values may be modified according to rules that depend on the original pixel value. Alternatively, pixel values may be combined with or compared to others in their immediate neighbourhood in a variety ways.

4.1 ROI Identification The Region Of Interest (ROI) identification process scans through the whole image captured by the CCD camera to find the region which is only useful for the recognition process, i.e. the postcode area. The four black dots at each corner of the postcode rectangle are used as the indicators of the ROI. The Euclidean Distance Map (EDM) technique is used to identify the four dots. Once the ROI is found, the rectangle is segmented

proportionately into five smaller units as each unit represents an image of the consecutive Arabic digits.

4.2 Image Centering To recognise a hand-written character regardless of position and size, some sort of transformation has to be applied. Alternatively, the position and size of the original character can be adjusted to standard values, and apply the classification to the adjusted image. This process of adjustment is called normalisation. x_low

x_centre

x_high

x profile

y_low y_centre

an adequate feature extraction method. The Kirsch edge detector is chosen to be the feature extractor in this postcode recognition problem as it has been known to detect four-directional edges more accurately than other edge detectors. The Kirsch edge detector actually considers all eight neighbours [3].

5.1 Kirsch Edge Detection The Kirsch edge detection [3] technique is a rather simple algorithm to be used to detect the edges of an image. Despite its simplicity, it is still capable to capture the features effectively. The simplicity of the process makes it rather fast in the computation process. Figure 4 shows the algorithm of the Kirsch edge detection, which can be implemented in a mask operation. Kirsch masks reduce the time for computing significantly. It can be implemented in hardware such as on a specialpurpose chip or a digital signal processor (DSP).

5.1 Image Compression y_high y profile

Fig. 3

The image is trimmed to the boundary of the hand-written character.

The normalisation process trims the image to the area of the writing only. From Fig. 3, the process first obtains the column and row summation of the image, which are the x and y profiles. The point with the maximum value of each profile is set as the centre point. Next, from the centre point, it searches left/up until a point in which its value is lower than a specified value and then sets it as the lower point of the profile. Again from the centre point, it searches right/down for the higher point of the profile. The new rectangle is formed as (x_low, y_low, x_high, y_high).

Once we have got the local features of the image, we need to compress them so that the number of input node of the classifier can be reduced. Compression had reduced the amount of input 16 times less. The compression algorithm is given below :The original image has a resolution of 32 x 32 pixels with grey-scale colour (Fig. 4a). It is segmented into an 8 x 8 regions (Fig. 4b) with each region taking up 4 x 4 pixels. Each region has a floating-point value between 0 and 1. The value of the region, v obeys the equation v=

1 M×N

M

N

∑ ∑ f ( x, y)

(5)

x =1 y =1

where M denotes the number of pixels in one row = 4 N denotes the number of pixels in one column = 4 f(x,y) denotes the value of pixel at position (x, y)

5. FEATURE EXTRACTION The main purpose of the feature extraction process is to reduce the number of inputs into the classifier while maintains the important features or properties of the image. The more meaningful of the features that are extracted, the better this feature extractor is. Additionally, the features must satisfy other desirable requirements such as fast processing speed, low computational cost and low complexity of the feature extraction techniques. Thus, a simpler and more powerful features cannot be easily found. As in the case of English characters, either handwritten or typed, which are basically line drawings, i.e., one-dimensional structure in a two-dimensional space. Therefore, local detection of line segments appears to be

a

Fig. 4

b

(a) A 32x32 bitmap segmented into 8 x 8 region. (b) 8 x 8 bitmap after the compression.

6. SYSTEM DESIGN Windows NT 4.0 Workstation Postcode Recognition Module WiT

DDE

Main Module

postcode digits extracted from the image (top) and the user interface captured by the PRM through the CCD camera. The ROI image is then segmented as shown at the top right. The Fuzzy ARTMAP algorithm is then invoked for the recognition process and the recognition results are given by displaying the digits again as shown below. With the postcode identified the appropriate bin is selected and highlighted in the GUI.

Parallel port Image Capturing Device

Mail Sorting Machine

Fig. 5 Components of the post-code recognition system. The system is developed under Microsoft Windows NT 4.0 environment and the hardware platform is an Intel’s Pentium 133 MHz with 64 Megabytes (MB) of Random Access Memory (RAM). The overall configuration of the post-code recognition and letter sorting system is shown in Fig. 5. The system can be divided into four separated components or modules: • Image Capturing Device (ICD) - A Sony CCD camera with a frame grabber is used to capture the postcode image from the envelope. • Postcode Recognition Module (PRM) - processes the raw images and recognised the hand-written postcodes through the trained fuzzy ARTMAP algorithm. • Main Module (MM) - consists of the graphical user interface. It also consists of the decision-making functions where the mails are to be sorted into the appropriate bins. • Mail Sorting Machine (MSM) - responsible for transporting the mails to the ICD and to push them into the appropriate bins. When a mail reaches the ICD, its ROI image is automatically captured by the camera. The MSM acknowledges the MM, which always waits for signals from the MSM via the parallel port. The MM then informs the PRM through the Windows’ inter-process communication called Dynamic Data Exchange (DDE) to start its processing. The PRM instructs the IMD to capture the image of the envelope and starts the recognition process. The results of the recognition is then passed back to the MM via the DDE. The MM then decides the appropriate bin that the mail should be sent to, based on the pre-defined database. It fires the appropriate solenoid to push the mail to the designated bin when the mail reaches the bin. Fig. 6 shows the Graphical User Interface (GUI) of the system. At the top-left side of the screen is the envelope image captured by the PRM, five consecutive

Fig. 6 The Graphical User Interface of the system.

6.1 Hardware Design The Mail Sorting Machine (MSM), as shown in Fig. 7, has five bins for mail storing, labelled ‘A’ to ‘E’. At the top, there is a movable octagon plate which has eight trays mounted at each side of the plate. The trays are used to place mails. There are five solenoids in front of the bins that acted as pusher. They are used to push the mails to the desired bin when it is fired. A proximity sensor is used to detect the presence of an envelope at the first tray. A CCD camera is mounted on a designated arm for capturing the post-code written on the envelope. A light bulb is located near to the CCD camera providing a constant light source. Fig. 8 illustrates a view of the MSM from the top. The machine is connected to the PC via the parallel port.

7. RESULTS AND CONCLUSION A prototype of the post-code recognition and letter sorting system has been developed as shown in Fig. 9. Several experiments were conducted on the machine and it was found that the system successfully sorted almost all the mails into the appropriate bin. The recognition ability is about 90% for both trained and untrained digits. It can also recognise postcodes written on envelopes in different colour. Trivial changes to the size, position and stroke width of the postcode digits are irrelevant to the recognition ability of the Fuzzy ARTMAP. The recognition speed of the system is very much dependent on the image processing software, the frame grabber, and the CPU of the computer. However,

this problem can be overcome by replacing with a faster computer and a more efficient imaging library. The system retains its advantage of a neural network-based system which learns or gains its knowledge by training. The recognition capability of the Fuzzy ARTMAP algorithm can be further improved through training of new styles of hand-written digits which it had not learnt before. The primary advantage of the Fuzzy ARTMAP algorithm is that it can incrementally learn new patterns without retraining the existing patterns. Light source CCD Camera

Tray

Bin D

[5] Freeman, J.A., Simulating Neural Networks with Mathematica, Addison-Wesley, USA.

[6] Kulkarni, A.D, Artificial neural networks for image understanding, ITP, USA.

[7] Tay, Y.H. and Khalid, M., “Comparison of Fuzzy ARTMAP and MLP Neural Networks in Hand-written Character Recognition,” Pre-Prints of the IFAC Symposium on AI in Real-time Control 97, Kuala Lumpur, 22-25 Sept., 1997.

Envelope D

Tray CCD Camera Pusher

C

Bin B

Bin

E

Sensor B

Movable plate Light source A

Fig. 7 The Mail Sorting Machine.

Fig. 8

Top view of the Mail Sorting Machine.

REFERENCES [1] Carpenter, G.A., Grossberg, S., et al., “Fuzzy ARTMAP: A Neural Network Architecture for Incremental Supervised Learning of Analog Multidimensional Map,” Neural Networks, vol. 3, 1992, pp. 698-713.

[2] Carpenter, G.A. and Grossberg, S., “A Massively Parallel Architecture for a Self-Organizing Neural Pattern Recognition Machine,” Computer Vision, Graphs, and Image Processing, vol. 37, 1987, pp. 54115. PC

[3] Lee, S.W., “Multlayer Cluster Neural Network for Totally Unconstrained Handwritten Numeral Recognition,” Neural Networks, vol. 8, no. 5, pp. 783792, 1995.

[4] Jabatan Perkhidmatan Pos, Buku Panduan Sistem Poskod Kebangsaan, Dany Press, KL, 1990.

Fig. 9

Control Box

Prototype of the recognition system.

Mail Sorting Machine

hand-written

postcode