Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Video Stabilization of Atmospheric Turbulence Distortion Yifei Lou1 4 Sung Ha Kang2 Stefano Soatto3 Andrea Bertozzi4 1 Department

of Mathematics, UCSD of Mathematics, Georgia Tech 3 Computer Science Department, UCLA 4 Department of Mathematics, UCLA 2 School

May 20, 2012

1/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Examples of turbulence videos

2/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Two effects of turbulence blurry image frames temporal oscillations

3/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction

3/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction

We propose the following PDE model for video stabilization: ut (x, y , k ) = S[u(x, y , k )] + µ4k u where S[·] denotes the Sobolev sharpening method on spatial domain, and 4k is the Laplacian operator acting on time direction k , 4k u = u(x, y , k + 1) − 2u(x, y , k ) + u(x, y , k − 1).

3/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Outline

1

Sobolev sharpening method

2

Video stabilization

3

Reconstruct a latent image

4

Discussion

4/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Sobolev gradient flow The heat equation is the gradient descent on the functional Z 1 E(u) = k∇uk2 , 2 Ω with respect to the L2 metric. Calder-Mansouri-Yezzi consider an inner product on the Sobolev space H01 (Ω) hv , wi −→ gλ (v , w) = (1 − λ)hv , wiL2 + λhv , wiH 1 , for any λ > 0. Then the Sobolev metric gλ on H01 (Ω) is given by ∇gλ E|u = −4(Id − λ4)−1 u .

5/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Sobolev gradient flow

The authors proved the well- posedness of the Sobolev gradient flow (SOB), both in the forward and backward directions. The backward direction can be used for image sharpening. In particular, 1 Es (u) = 4

Z

2

k∇u0 k Ω



!2 R k∇uk2 Ω R −α , 2 Ω k∇u0 k

where u0 is the initial value and α is a scale. For α < 1, we get blurring while we get sharpening for α > 1.

6/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Sobolev gradient flow

The gradient descent to minimize such energy is

ut =

! R k∇uk2 Ω R − α 4(Id − 4)−1 u . 2 k∇u k 0 Ω

(1)

It is a nonlinear PDE. We prove the local and global existence and uniqueness of the solutions to (1).

7/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

(a) original

(b) PM

(c) AM

(d) SOB

Discussion

8/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Our model: ukn+1 − ukn dt

! k∇uk k2 = − α 4(Id − λ4)−1 ukn k∇uk0 k2   +µ ukn+1 + ukn−1 − 2ukn+1 ,

with uk (x, y ) = u(x, y , k ), u 0 be the original video sequence. α > 1 for deblurring. µ is a weighting parameter.

9/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Computations – in spatial domain

Calder et. al. derive an explicit expression for the operator (Id − λ4)−1 on Ω = R2 , i.e., (Id − λ4)−1 f (x) = Sλ ∗ f (x), with ∗ be the convolution operator and 1 Sλ (x) = 4λπ

Z 0

+∞

|x|2

e−t− 4tλ dt . t

10/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Computations – in frequency domain ˆkn (m1 , m2 ) be the discrete Fourier transform of ukn (x, y ). Let u ˆkn+1 − u ˆkn u −4D(m1 , m2 ) ˆ n + µ(u ˆkn+1 + u ˆkn−1 − 2u ˆkn+1 ) , = Ckn u dt 1 + 4λD(m1 , m2 ) k where P Ckn

m1 ,m2

=P

ˆkn (m1 , m2 )|2 D(m1 , m2 )|u

ˆk0 (m1 , m2 )|2 D(m1 , m2 )|u m1 π 2 m2 π 2 D(m1 , m2 ) = sin( ) + sin( ) M1 M2

−α

m1 ,m2

for discrete coordinates m1 = 1, · · · , M1 and m2 = 1, · · · , M2 .

11/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: Raw data.

12/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB.

13/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB+LAP.

14/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: Raw data.

15/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB.

16/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB+LAP.

17/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Finding a latent image

(a)

(b)

(c)

Figure: Latent Images: (a) applying SOB on the temporal mean. (b) The temporal mean of the video sequence after SOB+LAP. (c) Further improvement using our image fusion technique from the video reconstruction by SOB+LAP.

18/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Partition of image domain

Figure: Partition of the image domain Ω. There is one row or one column overlap between two adjacent sub-regions Ωi and Ωj .

19/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Our “lucky” fusion

From these partitions, Select the best patch u(Ωj , kˆ ) from all the frames for each Ωj for 1 ≤ k ≤ N. The best patch is selected by measuring two terms: the similarity to the mean and the sharpness. Why? Similarity is to enforce correct pixel location, while sharpness is to produce sharp edges.

20/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

original

Lucky-region

Mao-Gilles

one of SOB+LAP

mean of SOB+LAP

our method

Discussion

21/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Conclusions

We proposed SOB+LAP, which stabilizes the temporal oscillation and sharpens the video frame at the same time. We use the lucky-region image fusion technique to construct a latent image from results of SOB+LAP. Future direction is to analyze the turbulence behavior in order to resolve fine details in the case of destructive turbulent degradation.

22/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Turbulence motion analysis

(a)

(b)

Figure: The positioning of the key points along a line. The key points are displayed as the blue dots on (a). (b) shows how these points are oscillating can time t changes. The wave movement of the turbulence happens in groups.

23/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Reference: Yifei Lou, Sung Ha Kang, Stefano Soatto, and Andrea L. Bertozzi. Video Stabilization of Atmospheric Turbulence Distortion. (CAM report 12-30) Web: sites.google.com/site/louyifei/research/turbulence

Thank You!

24/24