INFORMATIK

Hardware-Accelerated Rendering of Photo Hulls

Ming -Peter Seidel Ming Li, Li, Marcus Marcus Magnor Magnor,, Hans Hans-Peter Seidel MPI MPI Informatik Informatik Saarbr ücken, Germany Saarbrücken, Germany Sept. 2004

Ming Li

Motivation

INFORMATIK

Research goal •• Fast Fast and and accurate accurate 3D 3D reconstruction reconstruction and and rendering rendering from from multiple multiple views views of of real real dynamic dynamic objects objects

One possible approach •• Visual VH) representation Visual hull hull ((VH) representation •• Fast Fast but but limited limited reconstruction reconstruction –– Only Only an an approximation approximation –– Unable Unable to to reconstruct reconstruct certain certain concave concave regions regions Lead Lead to to rendering rendering artifacts artifacts Eurographics‘ Eurographics‘04

Ming Li

Motivation

INFORMATIK

Alternative approach •• Photo PH) representation Photo hull hull ((PH) representation

Visual Hull ⊇ Photo Hull ⊇ True Object –– More More accurate accurate reconstruction, reconstruction, but but expensive expensive –– Exploit Exploit graphics graphics hardware hardware to to accelerate accelerate PH PH reconstruction reconstruction as as well well as as rendering rendering Eurographics‘ Eurographics‘04

Ming Li

Talk Outline

INFORMATIK

The Photo Hull Previous Work HW -Accelerated Photo Hull Rendering HW-Accelerated

Results and Performance Conclusions Eurographics‘ Eurographics‘04

Ming Li

The Photo Hull

INFORMATIK

Basic principle of PH reconstruction •• •• ••

Discretize Discretize aa confined confined 3D 3D space space into into voxels voxels Check -consistency of Check photo photo-consistency of each each voxel voxel Discard -inconsistent voxels Discard photo photo-inconsistent voxels

Visibility issue during reconstruction ?

Eurographics‘ Eurographics‘04

Ming Li

The Photo Hull

INFORMATIK

Photo hull definition •• The -consistent objects The union union of of all all photo photo-consistent objects w.r.t w.r.t aa given given set set of of color color images images –– Photo -consistency for Photo-consistency for Lambertian Lambertian surfaces surfaces

Two ways to render a photo hull •• Reconstruct Reconstruct and and then then render render –– Explicit Explicit reconstruction reconstruction •• Reconstruct Reconstruct while while rendering rendering –– Implicit Implicit reconstruction reconstruction –– More More efficient efficient for for novel novel view view synthesis synthesis Eurographics‘ Eurographics‘04

Ming Li

Previous Work

INFORMATIK

Purely software -based software-based •• Explicit Explicit reconstruction reconstruction Voxel Coloring [Seitz et al. 1997], Space Carving [Kutulakos et al. 1999]

•• Implicit Implicit reconstruction reconstruction Image-based Photo Hulls [Slabaugh et al. 2002] (IBPH)

Partially hardware -accelerated hardware-accelerated •• Leverage Leverage GH GH to to accelerate accelerate partial partial computation computation –– Need -back from Need copy copy-back from GPU GPU to to main main memory memory –– Explicit Explicit reconstruction reconstruction Towards Real-time Voxel Coloring [Prock et al. 1998] Hardware Accelerated Voxel Carving [Sainz et al. 2002] Eurographics‘ Eurographics‘04

Ming Li

Previous Work

INFORMATIK

Purely hardware -accelerated hardware-accelerated Real-Time Consensus-Based Scene Reconstruction Using Commodity Graphics Hardware [Yang et al. 2002]

•• Implicit Implicit reconstruction reconstruction •• Photo -consistency check Photo-consistency check without without considering considering visibility visibility –– Reference Reference viewpoints viewpoints close close to to each each other other Constrained Constrained range range of of novel novel viewpoints viewpoints Use Use aa large large number number of of reference reference views views

Eurographics‘ Eurographics‘04

Ming Li

HW-Accelerated Photo Hull Rendering

INFORMATIK

Features •• Implicit -accelerated Implicit reconstruction, reconstruction, fully fully hardware hardware-accelerated •• Consider -consistency check Consider visibilities visibilities during during photo photo-consistency check •• Employ Employ VH VH to to accelerate accelerate PH PH rendering rendering

Key idea •• Project Project reference reference views views onto onto the the slicing slicing planes planes novel

•• Maintain Maintain and and update update visibility visibility viewpoint information information on on graphics graphics hardware hardware Eurographics‘ Eurographics‘04

Ming Li

HW-Accelerated Photo Hull Rendering

INFORMATIK

Input A -calibrated reference A set set of of pre pre-calibrated reference views views –– Fg/Bg Fg/Bg segmentation segmentation information information

Pseudo -code Pseudo-code Generate Generate slicing slicing planes planes Clear Clear all all visibility visibility masks masks to to 00 foreach foreach slicing slicing plane plane H H (front (front to to back back order) order) Render -consistency check Render H H with with photo photo-consistency check Update Update visibility visibility masks masks End End foreach foreach Eurographics‘ Eurographics‘04

Ming Li

Slicing Plane Generation

INFORMATIK

Naïve method •• A A set set of of slicing slicing planes planes –– Projections Projections cover cover the the whole whole image image plane plane –– Depths Depths span span aa large large range range

Optimized method •• Visual Visual hull hull ⊇ Photo Photo hull hull –– Compute Compute VH’s VH’s bounding bounding volume volume Constrain Constrain the the area area of of each each slicing slicing plane plane Reduce Reduce the the number number of of slicing slicing planes planes Eurographics‘ Eurographics‘04

Ming Li

Slicing Plane Generation

INFORMATIK

Optimized method (cont.) •• Generate Generate the the depth depth map map of of VH VH –– Exploit Exploit aa fast fast VH VH rendering rendering algorithm algorithm Hardware-accelerated VH Reconstruction and Rendering [Li et al. ‘03]

•• Compute Compute aa bounding bounding box box and and aa depth depth range range

x

•• Generate Generate aa slicing slicing plane plane by -projecting the by back back-projecting the bounding bounding box box Eurographics‘ Eurographics‘04

Ming Li

Slicing Plane Rendering

INFORMATIK

Render from a novel viewpoint •• Project Project reference reference images images & & visibility visibility masks masks

Visibility masks •• Stored -screen buffer Stored in in the the alpha alpha channel channel of of an an off off-screen buffer

Active reference view •• The The viewpoint viewpoint is is in in front front of of all all slicing slicing planes planes •• A A 3D 3D point point in in the the slicing slicing plane plane most most likely likely visible visible

Eurographics‘ Eurographics‘04

Ming Li

Slicing Plane Rendering

INFORMATIK

Photo -consistency check Photo-consistency •• Compute Compute the the color color variance variance of of visible visible samples samples –– Silhouette -consistency: improve Silhouette-consistency: improve efficiency efficiency –– Visibility Visibility masks: masks: suppress suppress occluded occluded samples samples N N N  2 2 σ = ∑ ( Ri − R ) + ∑ (Gi − G ) + ∑ ( Bi − B ) 2  i =1 i =1  i =1  2

1, σ 2 > T photo-consistenc y =  0, otherwise

Eurographics‘ Eurographics‘04

Ming Li

Slicing Plane Rendering

INFORMATIK

Fragment color computation •• RGB RGB values values –– Blending Blending texel texel values values of of visible visible samples samples Weight Weight W Wkk :: eliminate eliminate seams seams  N  Color =  ∑ W k * Tk   k =1 

N

∑W k =1

k

•• Alpha Alpha value value –– Assign -consistency Assign binary binary photo photo-consistency –– For For updating updating visibility visibility masks masks Eurographics‘ Eurographics‘04

Ming Li

Visibility Mask Updating

INFORMATIK

Prepare for the next slicing plane •• Set Set the the alpha alpha map map in in the the novel novel view view as as aa texture texture •• For For each each active active reference reference view view –– Render Render slicing slicing plane plane in in the the corresponding corresponding tile tile

Eurographics‘ Eurographics‘04

Ming Li

Comparison of Rendering Quality – VH vs. PH

INFORMATIK

Textured Textured visual visual hull hull

Depth Depth map map of of the the visual visual hull hull

Textured Textured photo photo hull hull

Depth Depth map map of of the the photo photo hull hull

One One of of eight eight reference reference images images

Eurographics‘ Eurographics‘04

Ming Li

Comparison of Rendering Quality - Visibility Issue

Consistency Consistency check check without without consideration consideration of of visibility visibility

Eurographics‘ Eurographics‘04

INFORMATIK

Consistency Consistency check check with with consideration consideration of of visibility visibility

Ming Li

Rendering Performance

INFORMATIK

Hardware setup •• P4 P4 1.7GHz 1.7GHz machine machine •• GeForce GeForce FX FX 5800 5800 graphics graphics card card

Rendering configuration •• Reference Reference view view resolution resolution:: 320x240 320x240 pixels pixels •• Novel Novel view view resolution: resolution: 320x240 320x240 pixels pixels •• Number Number of of slicing slicing planes: planes: 30 30 ~~ 60 60 •• Area Area occupied occupied by by PH: PH: approx. approx. 1/3 1/3 of of aa novel novel view view Eurographics‘ Eurographics‘04

Ming Li

Rendering Performance

INFORMATIK

Implementation •• OpenGL, OpenGL, Linux Linux Platform Platform •• Cg Cg code code ! ! Fragment Fragment program program

Frame rates •• Matrix -4 active) Matrix parody: parody: 88 reference reference views views (3 (3-4 active) –– About About 2.2 2.2 fps fps •• Puppy: Puppy: 55 reference reference views views (all (all active) active) –– About About 2.7 2.7 fps, fps, approx. approx. 77 times times faster faster than than IBPH IBPH •• Conditional Conditional instructions instructions –– Expected Expected 22 or or 33 times times faster faster Eurographics‘ Eurographics‘04

Ming Li

Demo Video

INFORMATIK

Parody of the bullet time scene in “Matrix”

88 reference GeForce FX reference views, views, 2.2 2.2 fps fps ((GeForce FX 5800) 5800) Eurographics‘ Eurographics‘04

Ming Li

Conclusions

INFORMATIK

•• A -accelerated algorithm A HW HW-accelerated algorithm to to render render PHs PHs from from multiple multiple views views •• Visibility Visibility consideration consideration improves improves rendering rendering quality quality •• VH VH helpful helpful to to accelerate accelerate PH PH rendering rendering

Future work •• More -consistency metrics More robust robust photo photo-consistency metrics •• More More general general reflection reflection models models for for photo photo-consistency consistency check check

Eurographics‘ Eurographics‘04

Ming Li

INFORMATIK

Thanks for your attention!

Refer to more information on http://www.mpi -sb.mpg.de/~ming http://www.mpi-sb.mpg.de/~ming

Sept. 2004

Ming Li