Incremental Calibration for Multi-Camera Systems Anil K. Nelakanti Advised by Edmond Boyer

1 / 18

Introduction Calibration: Retrieving rotation and translation of a camera w.r.t. a global coordinate system. Objective: Calibration of multi-camera systems. • We wish to efficiently calibrate multiple cameras looking at a

common scene using image correspondences alone. Motivation: • Systems with more than two cameras are becoming more

common. • Any vision algorithm that processes such data in 3D requires

prior calibration. • Calibrating methods are time consuming and use calibration

patterns in common volume. Vicon systems use L-bar and a wand. 2 / 18

Current Literature

• Two camera systems do the following: a. estimate Essential matrix1 b. decompose into Rotation and Translation • Multi-camera systems do either : a. Full reconstruction + Bundle Adjustment Requires initialization. Extremely slow. b. Averaging of rotations and translations Naive extension.

1

studied extensively

3 / 18

Naive Extension Two camera systems do the following, a. estimate Essential matrix b. decompose into Rotation and Translation Naive extension to multi-camera case • average in pairs for consistent rotation and translation.

Averaging on translation is simple • Space of translations is linear

Averaging on rotations is not easy • Rotations belong to Lie Algebraic space

4 / 18

Rotation Space Representations for rotation: Orthonormal matrices SO(3): A linear combination of elements in SO(3) does not belong to SO(3). Unit Quaternions: They tend to fail in attempt to stay on the manifold [Martinec 07]. Euler Angles: Marred by singularities and accumulated error. Double Rotation: Requires an approximate initialization. Here lies the motivation for a closed form solution that avoids rotations. 5 / 18

Incremental Setting Consider a simpler problem: introduce an uncalibrated camera to a system of n calibrated cameras looking at a scene. Uncalibrated camera

Scene

R3… R1

R2 t2

t1 Calibrated cameras

Figure: Multi-camera system looking at a common scene.

Solution to this problem can be used to incrementally calibrate multi-camera systems. 6 / 18

Properties of Formulation Requirements, • Use only correspondences.

Desirables, • Use minimal number of correspondences. • Use computations not involving rotations. • Solve in closed form. • Make use of multiple views.

7 / 18

Properties of Formulation Requirements, • Use only correspondences.

Desirables, • Use minimal number of correspondences. • Use computations not involving rotations. • Solve in closed form. • Make use of multiple views.

Exploit a simple fact; Dual

Structure ←→ Motion Solve for structure and use that to deduce motion.

7 / 18

Structure Formulation Two 3D points and a camera make a triangle.

d Scene points

|v1|=l1

|v2|=l2

= d2 C1

Camera center

Figure: Constraint from the angles subtended.

Cosine rule for triangles : a constraint in 3 sides + angle between light rays. 8 / 18

Structure Formulation given 3D-2D matchings in single camera: use simple elimination [Ameller 00]

Uncalibrated camera

v1

|vi| = li |wi| = mi

v2

Scene

w1 w2

Calibrated cameras

Figure: Two triangles from two cameras.

given only 2D-2D matchings use two triangles to eliminate scene variables. 8 / 18

Structure Formulation for two cameras with 2D-2D correspondences we get,

d

l1 m2

l2 m1 C1

C2

= d2 d2 =

Figure: Constraint from pair of points. _

_

l12 + l22 − 2 cos(v1 , v2 )l1 l2 = m12 + m22 − 2 cos(w1 , w2 )m1 m2 8 / 18

Counting Argument Considering the two camera case, _

_

l12 + l22 − 2 cos(v1 , v2 )l1 l2 = m12 + m22 − 2 cos(w1 , w2 )m1 m2

• Each pair of 3D points lead to a polynomial.


k points lead to

k 2

polynomials.

• Each 3D point leaves two unknowns, one for each camera.

k points leave 2k unknowns. • The system is solvable only if

  k ≥ 2k ⇒ k ≥ 5 2 We need a minimum of 5 scene points to make a solvable system. 9 / 18

Resulting Polynomials

F =

                                    

_

_

l12 + l22 − 2 cos(v1 , v2 )l1 l2 = m12 + m22 − 2 cos(w1 , w2 )m1 m2 _ _ l12 + l32 − 2 cos(v1 , v3 )l1 l3 = m12 + m32 − 2 cos(w1 , w3 )m1 m3 _ _ l12 + l42 − 2 cos(v1 , v4 )l1 l4 = m12 + m42 − 2 cos(w1 , w4 )m1 m4 _ _ l12 + l52 − 2 cos(v1 , v5 )l1 l5 = m12 + 1 − 2 cos(w1 , w5 )m1 m5 _ _ l22 + l32 − 2 cos(v2 , v3 )l2 l3 = m22 + m32 − 2 cos(w2 , w3 )m2 m3 _ _ l22 + l42 − 2 cos(v2 , v4 )l2 l4 = m22 + m42 − 2 cos(w2 , w4 )m2 m4 _ _ l22 + l52 − 2 cos(v2 , v5 )l2 l5 = m22 + 1 − 2 cos(w2 , w5 )m2 m5 _ _ l32 + l42 − 2 cos(v3 , v4 )l3 l4 = m32 + m42 − 2 cos(w3 , w4 )m3 m4 _ _ l32 + l52 − 2 cos(v3 , v5 )l3 l5 = m32 + 12 − 2 cos(w3 , w5 )m3 m5 _ _ l42 + l52 − 2 cos(v4 , v5 )l4 l5 = m42 + 1 − 2 cos(w4 , w5 )m4 m5

                                    

Solve F. 10 / 18

Available Methods

1. Analytical Methods. Newton like methods [initialization with gradient descent].

2. Subdivision Methods. bound the roots [ Descarte’s rule or B´ezout bound] and isolate them [ Sturm sequences or interval arithmetic2 ].

2

interval splitting and Taylor series

11 / 18

Available Methods

1. Analytical Methods. Newton like methods [initialization with gradient descent]. * but like non-convex optimization problems require a good initialization 2. Subdivision Methods. bound the roots [ Descarte’s rule or B´ezout bound] and isolate them [ Sturm sequences or interval arithmetic2 ]. * but are efficient only with low number of variables

2

interval splitting and Taylor series

11 / 18

More Methods

3. Algebraic Methods. Reduce to univariate or eigenvalue problem. 4. Homotopic Methods. Start with known system and deform to given system. 5. Geometric Methods. Project onto smaller subspace and reduce to univariate or eigenvalue problem.

12 / 18

Algebraic Methods

Solve F but since F is hard to solve, transform

F −−−−−−→ F 0 transform such that the roots remain the same and F0 is easier to solve.

13 / 18

Algebraic Methods

Solve F but since F is hard to solve, trans.

hF i −−−→ hF 0 i ideal gen.

transform such that the variety remains the same and F0 is easier to solve.

13 / 18

Algebraic Methods

Solve F but since F is hard to solve, for example   2  x +y +z −1  trans. x + y2 + z − 1 −−−→ hF 0 i = hF i =   ideal gen. x + y + z2 − 1

x + y + z2 − 1 y2 − y − z2 − z 2yz 2 + z 4 − z 2    6 z − 4z 4 + 4z 3 − z 2    

      

transform such that the variety remains the same and F0 is easier to solve. Note: F0 [last] is univariate

13 / 18

Algebraic Methods Solve F but since F is hard to solve, for example  2   x +y +z −1  trans. x + y2 + z − 1 hF i = −−−→ hF 0 i =   ideal gen. x + y + z2 − 1

 lex Gr¨obner2 basis x +y +z −1    y2 − y − z2 − z 2yz 2 + z 4 − z 2    6 z − 4z 4 + 4z 3 − z 2

      

transform such that the variety remains the same and F0 is easier to solve. Note: F0 [last] is univariate , F0 [last but one] is bivariate, and so on.

13 / 18

Verifying Counting Argument

Consider minimal case of 5 scene points in two cameras. Number of unknowns: 9 [fixing one variable for scale ambiguity]. Number of polynomials: 10. Do the equations carry sufficient information to solve for unknowns [linear independence] ? Yes if remainder space is finite dimensional. Quotient Algebra: A = k[x1 , · · · , xn ]/I Using Macaulay2: dimZ (A) = 40 So, there exist 40 point solutions [counting multiplicities].

14 / 18

Lex. Gr¨obner Basis Using Maple’s F4 : 24 polynomials in lexicographic Gr¨obner basis with last polynomial 1 − 4m32 + 6m34 − 4m36 + m38 = 0 solving which, m3 = ±1. Substituting m3 in above poly. leads to −m2 + 3m2 m32 − 3m2 m34 + m2 m36 = 0 Finally, l1 = l2 = l3 = l4 = l5 = m1 = m2 = 0, m3 = ±1.

15 / 18

Analyzing Result

The cause for polynomials to vanish is not theoretical: • is not a case of failure for Extension Theorem. • symbolically verifies the existence of two valid solutions.

Numerical issues seem to be the possible cause: • a small perturbation in coefficients leaves considerable residue

with the solution. • a slightly perturbed solution values leave a small residue.

Surface of error function seems rugged without deep valleys.

16 / 18

Current Status

Symbolic method inappropriate for problem at hand. Three interesting directions being looked into: • Numerical Methods. Homotopic continuation

methods [Morgan 87, Morgan 89] provide excellent numerical alternative. • Convex-concave bounds. Bilinear terms can be

relaxed [McCormick 76, Sherali 02] and the system optimized. • LMI Relaxation. Rewrite the problem as a matrix inequality and

use SDP relaxation to solve it [Li 10].

17 / 18

Thank you.

18 / 18

Marc-Andr´e Ameller, Bill Triggs & Long Quan. Camera Pose Revisited - New Linear Algorithms, 2000. Hongdong Li. Multi-View Structure Computation without Explicitly Estimating Motion. CVPR, 2010. D. Martinec & T. Pajdla. Robust Rotation and Translation Estimation in Multiview Reconstruction. CVPR, 2007. G. McCormick. Computability of global solutions to factorable non-convec programs-part i-convex underestimating problems. Math. Prog., 1976. A. P. Morgan & A. J. Sommese. A homotopy for solving general polynomial systems that respects m-homogeneous structures. 18 / 18

Appl. Math. Comp., 1987. A. P. Morgan & A. J. Sommese. Coefficient-parameter polynomial continuation. Appl. Math. Comp., 1989. H. Sherali & A. Alameddine. A new reformulation-linearization technique for bilinear programming problems. Journal of Global Optimization, 2002.

18 / 18