UA@SRVC: Qualification Paper M. Antunes, R. Pereira, A. Chauhan, L. Seabra Lopes Transverse Activity on Intelligent Robotics IEETA/DETI – Universidade de Aveiro – Portugal

Abstract This document outlines the strategy of the Universidade de Aveiro entry for the Software League of the Semantic Robot Vision Challenge’2009. We have chosen to use a variety of classifiers that will rank the images from our workset according to the probability of containing a certain object. Afterwards a meta-level classier will, based on those various rankings, decide whether an image has a requested object. So far we have three classifiers: a classier based on SIFT for specific (narrow) categories, and a voting combination of two shape-based classifiers for more general (broad) categories.

Learning Stage In the first stage of the challenge, lasting 2 hours, classification models for the requested objects are created. For that purpose we have access to any public web-based database.

Data acquisition In the data acquisition stage the images that will be used to create the classification models are collected using public search engines such as Google images. Clustering techniques will be used to determine relevant subsets of training images.

Model creation From the data gathered from the internet, each classifier will build a model of the object. That model will then be used during the second stage to rank the images from the workset. One of the classifiers is based on SIFT models. In this case, a category model is simply a concatenation of all keypoints extracted from the gathered images using the SIFT algorithm (Lowe 2004). For broad categories, two shape representations are used. One is the Global Shape Context, or GSC, a polar histogram of edge pixels (Pereira et al., 2009). To build a GSC, a frame of reference is located at the geometric center of the object. Then, the space around the centre up to the most eccentric pixel of the object is divided into a slices and d layers. The intersection of slices and layers results in a polar matrix that will be mapped to a 2D histogram counting the number of pixels in each cell. This histogram

is finally normalized by dividing the counts for each cell by the total count. The other representation is the 2D histogram proposed by Roy (2000), here called Roy’s Shape Representation, or RSR. For all pairs of edge pixels, the distances between them, D, and the angles between the respective tangents, δ, are computed. All this information is also summarized through a two-dimensional histogram with a angle bins on one dimension and d distance bins on the other. In each cell, the histogram counts the number of edge pixel pairs in the corresponding distance and angle bins. A category model is a collection of GSCs and RSRs for each object.

Performance Stage On the second stage, the supplied image set is analyzed and object detection and classification attempted.

Image pre-processing and object detection A resize operation is carried out, so that the remaining operations become faster. Then, based on color saliencies, areas of interest (rectangular sub-windows) are extracted. Next, edges are detected in these sub-windows. Through clustering, some of these edges will be presumed as part of the detected object, and the rest will be presumed to be clutter. Finally, models (SIFT, GSC and RSR) will be created for each image in the workset.

Classification Classification is done on two stages. Basic classifiers are executed first. Each of these classifiers will return a list per category with the rankings of the sub-images on the workset. The higher the ranking, the more likely the classifier thinks the sub-image is representative of the requested category. Afterwards, another classifier will take all those results and determine the final object classification. The SIFT classifier will match the keypoints extracted from the workset image with the concatenation of keypoints gathered from the images obtained from internet. The higher the number of keypoints matched the higher the final classification. Classification uses the nearest-neighbor rule. In the case of shape-based classification (with GSCs and RSRs), the metric is the χ2 distance between histograms:

D pq

2 1 a d ( h p ( i , j )  h q ( i , j ))   2 i 1 j 1 h p ( i , j )  h q ( i , j )

where p and q are two objects, hp and hq are the respective histograms, a is the number of angle bins, d is the number of distance bins, i is an angle bin and j is a distance bin. The higher the distance, the lower the probability that both histograms refer to the same category. The final classifier will, given the type of category (either broad or narrow), balance the weight given to the different basic classifiers and proceed to issue the final classification.

Bounding Boxes The bounding boxes are drawn with the dimensions of the calculated interest area.

Acknowledgements Our current entry is built over UA@SRVC’2008. It integrates a SIFT keypoint detector by Rob Hess.

References Arthur, D. and Vassilvitskii, S. 2007. k-means++: the advantages of careful seeding. Proceedings of the eighteenth annual ACM-SIAM Symposium on Discrete algorithms. p. 1027-1035. Lowe, D. G. 2004. Distinctive image features from scaleinvariant keypoints. International Journal of Computer Vision 60:91-110. Pereira, R., and Seabra Lopes, L. 2009. Learning Visual Object Categories with Global Descriptors and Local Features, Progress in Artificial Intelligence: 14th Portuguese Conference on Artificial Intelligence, EPIA 2009: Aveiro, Portugal, October 2009, Proceedings, LNCS 5816, Springer, 225-236. Pereira, R., Seabra Lopes, L., and Silva, A. 2009. Semantic Image Search and Subset Selection for Classifier Training in Object Recognition, Progress in Artificial Intelligence: 14th Portuguese Conference on Artificial Intelligence, EPIA 2009: Aveiro, Portugal, October 2009, Proceedings, LNCS 5816, Springer, 338-349. Roy, D. K. Learning Words from Sights and Sounds: A Computational Model. PhD thesis, MIT, 2000. Seabra Lopes, L., Valente, F., Pereira, R., Ribeiro, L. 2008. Universidade de Aveiro - SRVC Agent. SRVC’2008 Qualification Paper.

Fig. 1 – Flow diagram of UA@SRVC