I spy with my little eye: Learning Optimal Filters for Cross-Modal Stereo under Projected Patterns

Motivation

Wei-Chen Chiu, Ulf Blanke, Mario Fritz http://www.mpi-inf.mpg.de/~walon/files/cross_modal_kinect/ Kinect: active sensing failed on specular, transparent or reflective surface ► compliment with cross-modal stereo between Kinect's RGB and IR camera

Problem

RGB image

IR image

Depth from Kinect

► Previous work from Chiu et al.@BMVC: simple pixel-wise linear combination Pixel-wise re-weighting of RGB channels to be an IR-like image Cross-modal stereo ► under assumption that linear combination of RGB channels can recover sensor response characterisitic of IR camera lambertian surface

RGB IR Cross-modal adaptation

IR-like image

Depth

two slits & grating

light source

Kinect

Fusion

w_r w_g w_b

closed box

Pointcloud

Disparity map

Stereo

w_ir

raw / f tted

• simple pixel-wise re-weighting of IR and RGB seems to be to limited • The projected pattern on IR image required for active depth sensing make interference to compute the cross-modal stereo

Method

Disparity from stereo

rgb

blue

red green blue ir1 ir2 ir3 ir4 rgb mimics ir rgb mimcs ir amplified

ir

green

red

ir

Extend transformation learning from pixel-wise to patch-wise ► linear filters that leverage a pixel neighborhood in all color channels to preserve matches across modalities

Result

► learned filters encompass smoothing, sharpening and edge detection: helpful as prefilters in stereo matching and alleviate problems with IR pattern

Disparity map on unfiltered pairs

Disparity map on patch-filtered pairs

Perform evaluation on object segmentation task from 3D point clouds. ► comparing to avg precision 46% of Kinect depth estimate, our method improve over 20% and is on par with previous work under IR-projector-off which has practical issues.

kinect only (AP = 46.0)

0.9

spectral (AP = 64.6) best for IR-on (AP = 69.0)

0.8

proposed method (AP = 71.5) best for IR-off (AP = 72.5)

0.7 0.6 0.5 0.4 0.3 0.2 0.1 0

0

0.1

0.2

0.3

0.4

0.5

0.6

1 - precision

0.7

0.8

0.9

1