Artificial Speciation of Neural Network Ensembles Vineet Khare & Xin Yao School Of Computer Science
Overview • • • • • • •
Introduction Benchmark problems Overview of the methodology Encoding of neural networks Evolving ANNs with fitness sharing Results and discussion Conclusion and suggestions
Introduction • Modular approach for complex problems fitness sharing in evolving ANNs • Making use of population information
Introduction • Background – Automatic modularization using speciation (Darwen and Yao, 1996) – Use of population information in evolution (Yao and Liu, 1998) – Speciation in neural networks (Ahn and Cho, 2001)
Benchmark Problems • Problems taken from UCI machine learning benchmark repository – Wisconsin breast cancer dataset • 699 instances • 2 classes - malignant or benign • 9 integer valued attributes
– Heart disease dataset • 270 instances • 2 classes - presence or absence • 13 real valued attributes
• Datasets divided into - training (1/2th), validation (1/4th) and testing (1/4th)
Overview of Methodology 1. Encoding ANNs 2. Initialize ANNs 3. Train each ANN partially (on training set) 4. Fitness evaluation (on validation set) with sharing 5. Copy elites, apply variational operators 6. Termination criteria satisfied ? No => go to step 3 Yes => train ANNs fully, combine the outputs
Encoding of Neural Networks
Evolving ANNs • Partial training - Lamarckian evolution • Fitness evaluation and fitness sharing at phenotypic level 1 f raw = MSE
• Distance between two individuals modified Kullback Leibler entropy pj qj 1 n D( p, q ) = ∑ p j log + q j log 2 j =1 qj pj
• Sharing radius (σshare = 0.5, 1 and 2)
• Elitism
Evolving ANNs
– Preserving best individual with raw fitness – Preserving best individual with shared fitness
• Genetic operators - mutation – Connection creation – Connection deletion
• Genetic operators - crossover – Sub-graph exchange
Evolving ANNs
• Genetic operators - crossover (contd.)
Evolving ANNs
• Full training • Combination of outputs
– Majority voting – Averaging – Recursive least square (RLS) (Yao and Liu, 1998)
• Parameter settings for experiments
Evolving ANNs • Parameter settings for experiments (contd.) PARAMETER # of input nodes Maximum # of hidden nodes # of output nodes Population size Learning rate for training Data points in training set Data points in validation set Data points in testing set SEED Crossover probability Mutation Probability Sharing Radius # of Generations # of Runs
BREAST CANCER DATASET 9 5 1 20 0.1 349 175 175 System time 0.3 0.1 1 200 30
HEART DISEASE DATASET 13 6 1 20 0.1 135 67 68 System time 0.3 0.1 1 350 24
Results and Discussion
• Results for breast cancer problem Mean SD Min Max
Voting Validatio Training n 0.0378 0.0189 0.0100 0.0153 0.0074 0.0000 0.0544 0.0514
Averaging Testing 0.0231 0.0176 0.0000 0.0514
Training Validation 0.0374 0.0235 0.0102 0.0151 0.0078 0.0114 0.0544 0.0571
RLS Training+Validat Testing Testing ion 0.0237 0.0229 0.0167 0.0137 0.0152 0.0122 0.0000 0.0016 0.0000 0.0514 0.0267 0.0343
Table 1: Final E rror Rates (averaged over 30 runs) for B reast Cancer P roblem
• Results for heart disease problem Voting
Averaging
RLS Training+Validati Training Validation Testing Training Validation Testing Testing on 0.1960 0.1623 0.1733 0.1828 0.1462 Mean 0.1642 0.1612 0.1612 0.0282 0.0265 0.0404 0.0231 0.0293 0.0323 0.0243 0.0337 SD 0.1333 0.1194 0.1176 0.1333 0.1194 0.1029 0.1188 0.1176 Min 0.2667 0.2388 0.2794 0.2370 0.2388 0.2353 0.2129 0.2500 Max Table 2: Final Error Rates (averaged over 24 runs) for Heart Disease Problem
Results and Discussion • Best individual performance vs different combination methods 0.25
Error Rates on Training Set for Breast Cancer problem (Comparing Combination methods with best individual in the population)
Best Individual Voting Averaging
0.15
RLS
0.1
0.05
# of generations
199
190
181
172
163
154
145
136
127
118
109
100
91
82
73
64
55
46
37
28
19
10
0 1
Error Rates
0.2
Results and Discussion • Best individual performance vs different combination methods 0.49 0.44
Error Rates for Heart Disease Problem (Comparing Combination Methods with the Best Individual in the Population)
Error Rates
0.39 0.34
Best Individual Voting Averaging RLS
0.29 0.24 0.19 0.14 0.09 1
21
41
61
81
101 121 141 161 181 201 221 241 261 281 301 321 341
# of Generations
Conclusion and Suggestions • Comparison with known results C lassification A ccuracy on T est D ata K now n B est O ur R esult B reast C ancer H eart D isease
98.29% (Ahn and C ho , 2001)
98.33%
84.9% (Y ao and Liu, 1998)
83.88%
• Combination of outputs produced much better results than individual best. • RLS combination method produced best results. • Shortcomings
Conclusion and Suggestions
• Shortcomings
– Too expensive – Choice of sharing radius
• Suggestions – Less expensive schemes for speciation – Simple Sub-population Scheme (Spears, 1994) – Other schemes that require less problem knowledge • Multi-national EA (Ursem, 1999) • DNC (Gan and Warwick, 2001)
References Ahn, J. H. & Cho, S. B., 2001, Speciated Neural Networks Evolved with Fitness Sharing Technique. Proceedings of the 2001 Congress on Evolutionary Computation, 390-396, Seoul, Korea. Darwen, P., & Yao, X., 1996, Automatic Modularization by Speciation. Proceedings of the 1996 IEEE International Conference on Evolutionary Computation (ICEC '96), 88-93, Nagoya, Japan: IEEE Computer Society Press. Gan, J. & Warwick, K, 1999, A Genetic Algorithm with Dynamic Niche Clustering for Multimodal Function Optimisation. In proc. of the 4th Inter. Conf. on Artificial Neural Networks and Genetic Algorithms, 248-255, Springer Wien New York. Spears, W. M., 1994, Simple Subpopulation Schemes. Proceedings of 3rd Annual conf. on Evolutionary Programming, 296-307, World Scientific. Ursem, R., 1999, Multinational Evolutionary Algorithms. Proceedings of Congress of Evolutionary Computation, 3, 1633-1640 Yao, X. & Liu, Y., 1998b, Making Use of Population Information in Evolutionary Artificial Neural Networks. IEEE Transactions on Systems, Man and Cybernetics, Part B: Cybernetics, 28(3), 417-425.
