Fuzzy Correspondences and Kernel Density Estimation for Contaminated Point Set Registration Gang Wang, Zhicheng Wang, Yufei Chen, Weidong Zhao, and Xianhui Liu CAD Research Center, School of Electronics and Information Engineering Tongji University Shanghai, P. R. China [email protected], {zhichengwang, yufeichen, wd, lxh}@tongji.edu.cn Abstract—Point set registration problem is challenging to solve in the presence of outliers. In this paper, we proposed a registration method based on fuzzy correspondences and kernel density estimation. The main idea of our method is that the moving point set consists of inliers represented using a mixture of Gaussian, and outliers represented via an additional uniform distribution, then we use the fuzzy correspondences to estimate the Gaussian elements in the mixture model. There are four parts of the paper: we formulate the contaminated point set registration problem as a mixture model according to the well known Gaussian mixture model (GMM) based method firstly. Secondly, Gaussian elements are estimated by fuzzy correspondences to increase the registration accuracy efficiently. Thirdly, the optimal transformation between two contaminated point sets is expressed by representation theorem, and solved by EM algorithm iteratively. Finally, we compare our proposed method with several state-of-the-art methods, and the results show that our method gets better performances than the other methods in most tested scenarios. Index Terms—Point set registration, Fuzzy correspondence, Kernel density estimation, Mixture model, Regularization

I. I NTRODUCTION Point set registration has been widely applied in a variety of computer vision tasks, such as image registration, stereo matching, image retrieval, and biomedical image analysis [21]. The essence of point set registration is to align correspondences between both point sets via a certain transformation. From an optimization viewpoint, estimating correspondences is a linear assignment problem which can be solved with the Hungarian algorithm [3] efficiently, and updating transformation is a similarity problem which can be solved with the regularized least squares. Although extensive research has been well done on point set registration for decades, there still exists some challenges: (1) Point sets are usually contaminated by outliers, as shown in Fig. 1, which have no corresponding points in the other point set. Then the underlying correspondences and transformation are challenging to estimate and recover. (2) Non-rigid transformation usually has a large number of parameters which are hard to express and sensitive to outliers. (3) Contaminated point set registration needs a complex model to model inliers and outliers precisely. In order to solve those problems, in this paper, we present a robust method via fuzzy correspondences and kernel density c 978-1-4799-8697-2/15/$31.00 2015 IEEE

?

Fig. 1. An example of the contaminated point set. Both Fish shape point sets are contaminated by outliers, and it is hard to estimate their correspondences, and recover the transformation.

estimation for point set registration in the presence of outliers. Let the fixed point set be the scene set, and the moving point set be the model set, then the model set is transformed to align onto the scene set. Then we model the model set as a Gaussian mixture model, and the Gaussian kernel elements are estimated by fuzzy correspondences. Regularized least squares method is used to recover non-rigid transformation parameters in a reproducing kernel Hilbert space (RKHS) with the representation theorem. Experimental results demonstrate that our method outperforms several comparison methods in most tested scenarios. The rest of our paper is organized as follows: In Section 2, we overview the related work for point set registration. In Section 3, we present our proposed method. In Section 4, we describe the experimental results on different test data sets. In Section 5, we give a conclusion with some discussions. II. R ELATED W ORK Many related methods exist for point set registration. In this section, we briefly overview the point set registration methods. From the estimating correspondences viewpoint, many well known methods have been proposed for registration, such as soft assignment [5], shape context [1], Gaussian mixture model [8], spectral context [18], and graph-based methods [25], [26]. From the perspective of recovering transformation, there are two categories: rigid and non-rigid transformation. The most well known method for rigid point set registration is the iterative closest point (ICP) [2], while it is hard to solve the complex non-rigid transformation. Usually, non-rigid transformation is modeled by Gaussian radial basis functions (GRBF)

[16], [15], and thin plate splines (TPS) [5]. Chui and Rangarajan [5] proposed a robust point matching method based on soft assignment and TPS non-rigid transformation model. In the proposed registration framework, they use the annealing scheme to estimate correspondences and recover transformation iteratively. Tsin and Kanade [19] proposed a kernel correlation based method to align point sets by minimizing the kernel density estimation. Zheng and Doermann [24] proposed a robust method to match point set by preserving local structures. Ma et al. [9], [12] introduced the L2-minimizing estimator (L2E), and then proposed a robust method to estimate non-rigid transformation. Gaussian mixture model based methods have been proposed as follows: Jian and Vemuri [8] represented both point sets as Gaussian mixture models, then they used L2 distance to estimate the transformations. Myronenko and Song [15] just modeled the moving model set as the Gaussian mixture model, and the other set is considered as the data points. However, when both point sets are contaminated by outliers, the mixture model needs to be modified for outliers. Moreover, Ma et al. proposed an interesting point set registration method via vector field consensus [11], [13], and Wang et al. [20], [22] used mixture of asymmetric Gaussian to model point sets, and the non-rigid transformation is recovered robustly by L2E. Ma et al. proposed a regularized Gaussian fields criterion [10] for non-rigid image registration.

Considering the ill-posed problem of f , then the smoothness constraint is used to deal with the problem. Thus, a regularization term is added to the negative log-likelihood function: E(θ) = −

n=1

Motivated by the limitation of the coherent point drifting (CPD) method [15] when both point sets are contaminated by outliers, we use the fuzzy correspondences to estimate the number of Gaussian kernel elements. Without loss of generality, we will assume here that the model set as a mixture model, and the scene set as the data set. Note that the model set is transformed to align onto the scene set. T Formally, given the scene set XN ×D = (x1 , . . . , xN ) , and T the model set YM ×D = (y1 , . . . , yM ) , where M denotes the number of model point set, N denotes the number of scene point set, and D denotes the dimension of the point sets. Note that X ∈ RD and Y ∈ RD , where D = 2 or 3. For inliers, we define the Gaussian kernel density kG (x|m) = x−ym 2 1 ) with zero mean and uniform stanD exp(− 2σ 2 2 (2πσ )

2

dard deviation σ, and for additional outliers, the distribution is assumed to be uniform distribution kG (x|M + 1) = N1 . Thus, the mixture model of distributions for model point set can be written as follows: kG = ω kG (x) + (1 − ω) kG (x|M + 1) ,

(1)

where ω, 0 ≤ ω ≤  1 denotes the weight of the mixture of M Gaussian, kG (x) = m=1 ξ kG (x|m), and ξ = kG (m) denotes the coefficient for Gaussian elements which determined by the fuzzy correspondences. Given a non-rigid transformation f , the newly transformed model set Y = f (Y ), and the parameter family θ = {f, σ 2 , ω}.

log

M 

ξ kG +

m=1

λ f 2H , 2

(2)

where H is an RKHS, and λ > 0 is a trade-off parameter. In the Bayesian framework, the regularization term is formulated by the prior on transformation: p(f ) = exp (− λ2 f 2H ). Then the parameter θ can be estimated by minimizing the Eq. 2 B. EM Algorithm The Expectation-Maximization (EM) algorithm [7] is well used to solve mixture model, and it consists of two steps: E-step, and M-step. Specifically, the EM algorithm uses the old parameter values to compute the responsibility kG (m|x) of mixture Gaussian elements in the E-step, and finds the new parameter values σ 2 , ω via maximizing the expectation of the complete log-likelihood function, or minimizing the negative log-likelihood function in the M-step. Starting from some initial estimate of θ, and iteratively updating θ until convergence. Thus the objective function can be written as: Q(θ, σ 2 , ω) =

N M 1   old k (m|xn , θ, σ 2 , ω) xn − f (ym , θ)2 2σ 2 n=1 m=1 G

+

N  M  D old kG (m|xn , θ, σ 2 , ω) log σ 2 2 n=1 m=1

III. M ETHOD A. Problem Formulation

N 

− log ω

N  M 

n=1 m=1 N 

− log (1 − ω) λ + f 2H , 2

(3)

old kG (m|xn , θ, σ 2 , ω)

old kG (M + 1|xn , θ, σ 2 , ω)

n=1

where we omit terms which are independent of parameter θ. E-Step: Evaluate responsibilities. Based on the Bayes theorem: kG (m|x) = ξkkGG(x|m) (x) , we can obtain the expectation of the negative log-likelihood function as follows: kG (m|xn , θ, σ 2 , ω) 2

(ym ,θ) ξ exp(− xn −f2σ ) 2

= M

xn −f (yk ,θ)2 ) k=1 exp(− 2σ 2

+ (2πσ 2 )

D 2

1−ω M ω N

.

(4)

M-Step: Update parameters by minimizing negative loglikelihood results. 2 ,ω) Let ∂Q(θ,σ = 0, then we can obtain the new value of ∂σ 2 the standard deviation: σ2 =

N n=1

M

old 2 2 m=1 kG (m|xn , θ, σ , ω) xn − f (ym , θ) N M old 2 n=1 m=1 kG (m|xn , θ, σ , ω)

. (5)

2

,ω) Let ∂Q(θ,σ = 0, then we can obtain the new weight of ∂ω the Gaussian elements: N M k old (m|xn , θ, σ 2 , ω) ω = n=1 m=1 G . (6) N We define an RKHS with a kernel matrix GM ×M , where y −y 2 the kernel element gij = G(yi , yj ) = exp(− i2β 2j ). Fortunately, the optimal non-rigid transformation f can be M written as f (y) = GO = m=1 G(y, ym )om based on the representation theorem [14]. 2 ,ω) = 0, then we can obtain the new parameter Let ∂Q(θ,σ ∂O values of the non-rigid transformation:   d(K1N ×1 )G + λσ 2 1M ×M O = KX − d(K1N ×1 )Y , (7)

where KM ×N denotes the poster probability density matrix with elements kG , d(·) denotes the diagonal matrix, and OM ×D is the main coefficients to represent the non-rigid  0. transformation. Note that |d(K1N ×1 )G + λσ 2 1M ×M | = Then the transformed model set can be expressed as f (Y, θ) = Y + GO. C. Fuzzy Correspondences It is very hard to estimate accurate correspondences from the point set which is contaminated by outliers. Motivated by the idea of fuzzy correspondences, we use the shape context descriptor [1] to construct a weight-assignment for Gaussian elements to reduce the sensitivity of Gaussian kernel density estimation to outliers. Let C(xn , ym ) denote the similarity between these point pairs, then the χ2 test statistic: C(xn , ym ) = K hn (k)−hm (k)2 1 k=1 hn (k)+hm (k) , where h(k) denotes the K-bin nor2 malized histogram at each points. Then we obtain the affinity matrix CN ×M . Thus, we can estimate the Gaussian kernel elements by the results: exp(−αCnm ) ξnm = N , t=1 exp(−αCtm )

(8)

where α denotes a constant which is used to tune the scale of the probability density distribution. Updating fuzzy correspondences ξ iteratively for mixture of Gaussian model density estimation. D. Implementation Details Our proposed method is an iteration method, starting with some initial parameter values until reach some termination conditions. The parameter β of the Gaussian radial basis function is set to 2.0, the regularization parameter λ is set to 3.0, weight of Gaussian mixture model  ω is initialized as 0.7, and M N 1 2 x the initial scale σ 2 is set to N M n − ym  n=1 m=1 D following the suggestion as discussed in [15]. Note that we set the termination conditions: the maximization iteration number maxiter = 100, the final scale σf2 inal = 10−8 , and the energy value of the negative log-likelihood tol = 10−15 . Note that both point sets are normalized as zero mean and unit variance first. The pseudo code of our proposed method (FCKDE for short) is shown in Algorithm 1.

Algorithm 1 Solving contaminated point set registration by the FCKDE algorithm Input: The scene set: X, and the model set: Y . Initialize: σ 2 , β, λ, ω, σf2 inal , and O = 0M ×D . Output: The optimal transformation parameter θ, and the optimal transformed point set Y . Begin Construct kernel matrix G based on a Gaussian kernel with parameter β. while not converge do 1) Estimate the fuzzy correspondences ξ using Eq. 8. 2) E-step: compute the responsibility using Eq. 4. 3) M-step: Update the parameters σ 2 , ω, and O using Eqs. 5, 6, 7 respectively. 4) Update the model set: Y ← Y + GO. 5) Until converge (i.e. reach a termination condition: σ 2 > σf2 inal , maxiter > 100, or tol < 10−15 ). end while End IV. E XPERIMENTS In this section, we will show the experimental setup and results on some data sets. Our proposed method is implemented in Matlab 2012b and tested on a laptop with Pentium Core I5 2.45GHz and 8GB RAM. A. Experimental Setting We test our method on three different data sets: Chinese character point sets, Fish shape point sets, and CMU house images. The first two data sets are selected from the ChuiRangarajan data set [5]. There are four groups of outlier point sets, and the outlier to data ratio is from 0.5 to 2.0 (i.e. ratio = {0.5, 1.0, 1.5, 2.0}), and each group consists of a hundred of different shape poses. We select the first point set of each group as the scene set, and then we just select the other ninety point sets as the model sets. Note that the outliers satisfy uniform distribution. The last data set is CMU house image sequences [6]. The data set consists of 111 different views of a toy house, then we choose the frame 1 as our scene set, and the frame 111 as our model set. There are total 30 corner points in each image. In order to test our method on the contaminated point sets, we randomly sample different corners from each set to construct some point set pairs with a certain degree of outliers. Note that we run the method over 20 times on each test group. In order to verify the performance of the method, several state-of-the-art methods are selected to be the comparison methods. Two point set registration methods: L2-TPS [8], and CPD [15]. Moreover, we also test two graph-based methods: reweighted random walks for graph matching (RRWM) [4], and factorized graph matching (FGM) [25], [26], where FGMD for directed graphs and FGMU for undirected graphs. In the experiments, we use root-mean-square error (RMSE), accuracy, and recall to evaluate the registration results quanti-

Initial pose

L2-TPS

CPD

Ours

Fig. 2. An example of experimental results on Chinese character point sets. The group figures denote an example of registration results (outlier to data ratio: 0.5). The red ◦ denotes the scene point set, and the blue + denotes the moving model point set.

Fig. 4. Performances of registration methods on Chinese character point sets under outlier (outlier to data ratio from 0.5 to 2.0). Recall-accuracy curves are used to evaluate L2-TPS, CPD and FCKDE.

where L denotes the number of the ground truth correspondences, T P denotes true positive, F P denotes false positive, T N denotes true negative, and F N denotes false negative. B. Results on Chinese Character Point Sets

Fig. 3. The error bars are used to evaluate L2-TPS, CPD and FCKDE on Chinese character point sets, which indicate the registration error means and standard deviations over 90 random trials.

tatively. More precisely, RMSE, precision, recall, and accuracy evaluation metrics are defined as follows.   L 1  (xl − yˆl )2 , RMSE = L l=1

precision = recall =

TP , TP + FP TP , TP + FN

TP + TN accuracy = TP + TN + FP + FN

Chinese character point set consists of 105 true correspondences, and each group is added on 52, 105, 157, 210 outliers, respectively. An example of the registration results is shown in Fig. 2. We compare our FCKDE method with two state-ofthe-art registration methods, L2-TPS [8], and CPD [15]. The L2-TPS method uses L2E to compute the distance between two point sets, which are represented by Giaussian mixture models (GMM). The CPD method considers the points in the model set as the GMM centroids, and the other points in the scene set as the data points, which are generated by the GMM [15]. Both methods are implemented in Matlab, and the parameters are set by default (according to their original papers [8], [15]). Note that we fixed the parameters of each method in all experiments. The error bars show the performance of FCKDE compared to other two methods, as shown in Fig. 3. The FCKDE gives more accurate registration results than the other methods on four groups of test point sets. The recall-accuracy curves are shown in Fig. 4, as well used in [17], which are used to evaluate the true positive rate within a given accuracy threshold [8]. Observe that all recall curves of FCKDE are higher than the other methods. But recall curves of FCKDE become slow down when increasing the outlier to data ratio, because more outliers disturb the estimation accuracy of fuzzy correspondences. C. Results on Fish Point Sets Fish shape point sets are different from Chinese character point sets, because the shape of fish is well clustered. An example of registration results on the fish point sets is shown in

Initial pose

L2-TPS

CPD

Ours

Fig. 5. An example of experimental results on Fish point sets. The group figures denote an example of registration results (outlier to data ratio: 0.5). The red ◦ denotes the scene point set, and the blue + denotes the moving model point set.

Fig. 7. Performances of registration methods on Fish point sets under outlier (outlier to data ratio from 0.5 to 2.0). Recall-accuracy curves are used to evaluate L2-TPS, CPD and FCKDE.

our FCKDE with four state-of-the-art methods, RRWM [4], FGM [25], [26], L2-TPS [8], and CPD [15]. The RRWM and FGM are graph based methods, they use triangulation to construct graphs with the given corners. The matching results used accuracy error bars are shown in Fig. 9. Observe that the traditional point set registration methods outperform those graph-based method when point sets are contaminated by outliers in these experiments. While our FCKDE still has better accuracy than the other comparison methods in most test scenarios. V. D ISCUSSION AND C ONCLUSION

Fig. 6. The error bars are used to evaluate L2-TPS, CPD and FCKDE on Fish point sets, which indicate the registration error means and standard deviations over 90 random trials.

Fig. 5. Our proposed FCKDE gives the best alignment of test point sets, and the results are similar to the above experiment. The error bars are shown in Fig. 6, and the recall-accuracy curves are shown in Fig. 7. The results show that our proposed FCKDE method aligns correspondences in the presence of outliers very well, and has better performance than L2-TPS, and CPD. D. Results on CMU House Images In this experiment, following the experimental setup contributed by Zhou and De la Torre [26], we randomly select 20, 22, 24, 26, 28, and 30 corners to test the matching performances of methods. Fig. 8 shows an example of point matching results with 25 corners. In our proposed method, the final updated correspondences are shown by the responsibility KM ×N . Here, we compare

In computer vision and pattern recognition, point set registration, or point matching, is still an important problem and attracts researchers’ interests [8], [16], [15], [12], [23]. It is challenging to solve the registration problem when facing outliers, because the correspondences are very hard to represent and estimate. It is interesting to note that the iteration idea is helpful in estimating correspondences effectively. Shape context descriptor and Gaussian mixture model based methods are well used in point set registration, while some limitations still exist, especially for contaminated point set registration. In this paper, we mainly focus on the limitation of the Gaussian mixture model based method, then we proposed a novel method based on fuzzy correspondences and kernel density estimation to solve the contaminated point set registration problem. Firstly, we consider the inliers in the model set satisfy mixture of Gaussian, and the outliers in model set satisfy an additional uniform distribution. We also model the contaminated point set registration problem according to the GMM based method. Secondly, we use fuzzy correspondences to estimate the Gaussian elements, where shape context is used to represent the model point set. Thirdly, EM algorithm and regularization framework in RKHS to solve the objective

function, and find optimal parameters of the transformation f . Finally, experimental results demonstrate that our proposed FCKDE gets more accurate registration results, and outperforms several tested state-of-the-art methods in most test scenarios.

Fig. 8. An example of point matching on CMU house images. We randomly select 25 points out of 30 correspondences, and the unmatched points are outliers.

The Number of Corners Fig. 9. Performances on CMU house images by accuracy curves within different sample points. We compute means and standard deviations of matching accuracy over 20 random trials.

ACKNOWLEDGMENTS The authors wish to acknowledge Drs. Haili Chui, Andriy Myronenko, Bin Jian, Feng Zhou, and Jiayi Ma for providing their implemented source codes and test data set. This work was supported by National Natural Science Foundation of China (NSFC, No. 61103070), and Program for Young Excellent Talents in Tongji University (2013KJ008). R EFERENCES [1] S. Belongie, J. Malik, and J. Puzicha, “Shape matching and object recognition using shape contexts,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 4, pp. 509–522, 2002. [2] P. J. Besl and N. D. McKay, “A method for registration of 3-d shapes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, no. 2, pp. 239–256, 1992. [3] R. E. Burkard, M. DellAmico, and S. Martello, Assignment Problems. Siam, 2009. [4] M. Cho, J. Lee, and K. M. Lee, “Reweighted random walks for graph matching,” in Computer Vision–ECCV 2010. Springer, 2010, pp. 492– 505. [5] H. Chui and A. Rangarajan, “A new point matching algorithm for nonrigid registration,” Computer Vision and Image Understanding, vol. 89, no. 2, pp. 114–141, 2003.

[6] CMU, “Cmu house image data set,” http://vasc.ri.cmu.edu/idb/html/ motion/house/. [7] A. P. Dempster, N. M. Laird, and D. B. Rubin, “Maximum likelihood from incomplete data via the em algorithm,” Journal of the royal statistical society. Series B (methodological), pp. 1–38, 1977. [8] B. Jian and B. C. Vemuri, “Robust point set registration using gaussian mixture models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 33, no. 8, pp. 1633–1645, 2011. [9] J. Ma, W. Qiu, J. Zhao, Y. Ma, A. L. Yuille, and Z. Tu, “Robust l2e estimation of transformation for non-rigid registration,” IEEE Transactions on Signal Processing, vol. 63, no. 5, pp. 1115–1129, 2015. [10] J. Ma, J. Zhao, Y. Ma, and J. Tian, “Non-rigid visible and infrared face registration via regularized gaussian fields criterion,” Pattern Recognition, vol. 48, no. 3, pp. 772–784, 2015. [11] J. Ma, J. Zhao, J. Tian, X. Bai, and Z. Tu, “Regularized vector field learning with sparse approximation for mismatch removal,” Pattern Recognition, vol. 46, no. 12, pp. 3519–3532, 2013. [12] J. Ma, J. Zhao, J. Tian, Z. Tu, and A. L. Yuille, “Robust estimation of nonrigid transformation for point set registration,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2013, pp. 2147–2154. [13] J. Ma, J. Zhao, J. Tian, A. L. Yuille, and Z. Tu, “Robust point matching via vector field consensus,” IEEE Transactions Image Processing, vol. 23, no. 4, pp. 1706 – 1721, 2014. [14] C. A. Micchelli and M. Pontil, “On learning vector-valued functions,” Neural Computation, vol. 17, no. 1, pp. 177–204, 2005. [15] A. Myronenko and X. Song, “Point set registration: Coherent point drift,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 32, no. 12, pp. 2262–2275, 2010. [16] A. Myronenko, X. Song, and M. A. Carreira-Perpin´an, “Non-rigid point set registration: Coherent point drift,” in Advances in Neural Information Processing Systems, 2006, pp. 1009–1016. [17] J. Starck and A. Hilton, “Correspondence labelling for wide-timeframe free-form surface matching,” in IEEE 11th International Conference on Computer Vision (ICCV). IEEE, 2007, pp. 1–8. [18] J. Tang, L. Shao, and X. Zhen, “Robust point pattern matching based on spectral context,” Pattern Recognition, vol. 47, no. 3, pp. 1469–1484, 2014. [19] Y. Tsin and T. Kanade, “A correlation-based approach to robust point set registration,” in Computer Vision-ECCV 2004. Springer, 2004, pp. 558–569. [20] G. Wang, Z. Wang, Y. Chen, and W. Zhao, “A robust non-rigid point set registration method based on asymmetric gaussian representation,” Computer vision and image understanding, 2015. [21] ——, “Robust point matching method for multimodal retinal image registration,” Biomedical Signal Processing and Control, vol. 19, pp. 68–76, 2015. [22] G. Wang, Z. Wang, W. Zhao, and Q. Zhou, “Robust point matching using mixture of asymmetric gaussians for nonrigid transformation,” in Computer Vision-ACCV 2014. Springer, 2015, pp. 433–444. [23] Y. Yang, S. H. Ong, and K. W. C. Foong, “A robust global and local mixture distance based non-rigid point set registration,” Pattern Recognition, vol. 48, no. 1, pp. 156–173, 2015. [24] Y. Zheng and D. Doermann, “Robust point matching for nonrigid shapes by preserving local neighborhood structures,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, no. 4, pp. 643–649, 2006. [25] F. Zhou and F. De la Torre, “Factorized graph matching,” in Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012, pp. 127–134. [26] ——, “Deformable graph matching,” in Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013, pp. 2922– 2929.