Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Video Stabilization of Atmospheric Turbulence Distortion Yifei Lou1 4 Sung Ha Kang2 Stefano Soatto3 Andrea Bertozzi4 1 Department

of Mathematics, UCSD of Mathematics, Georgia Tech 3 Computer Science Department, UCLA 4 Department of Mathematics, UCLA 2 School

May 20, 2012

1/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Examples of turbulence videos

2/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Two effects of turbulence blurry image frames temporal oscillations

3/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction

3/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Two effects of turbulence ⇒ our approach blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction

We propose the following PDE model for video stabilization: ut (x, y , k ) = S[u(x, y , k )] + µ4k u where S[·] denotes the Sobolev sharpening method on spatial domain, and 4k is the Laplacian operator acting on time direction k , 4k u = u(x, y , k + 1) − 2u(x, y , k ) + u(x, y , k − 1).

3/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Outline

1

Sobolev sharpening method

2

Video stabilization

3

Reconstruct a latent image

4

Discussion

4/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Sobolev gradient flow The heat equation is the gradient descent on the functional Z 1 E(u) = k∇uk2 , 2 Ω with respect to the L2 metric. Calder-Mansouri-Yezzi consider an inner product on the Sobolev space H01 (Ω) hv , wi −→ gλ (v , w) = (1 − λ)hv , wiL2 + λhv , wiH 1 , for any λ > 0. Then the Sobolev metric gλ on H01 (Ω) is given by ∇gλ E|u = −4(Id − λ4)−1 u .

5/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Sobolev gradient flow

The authors proved the well- posedness of the Sobolev gradient flow (SOB), both in the forward and backward directions. The backward direction can be used for image sharpening. In particular, 1 Es (u) = 4

Z

2

k∇u0 k Ω



!2 R k∇uk2 Ω R −α , 2 Ω k∇u0 k

where u0 is the initial value and α is a scale. For α < 1, we get blurring while we get sharpening for α > 1.

6/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Sobolev gradient flow

The gradient descent to minimize such energy is

ut =

! R k∇uk2 Ω R − α 4(Id − 4)−1 u . 2 k∇u k 0 Ω

(1)

It is a nonlinear PDE. We prove the local and global existence and uniqueness of the solutions to (1).

7/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

(a) original

(b) PM

(c) AM

(d) SOB

Discussion

8/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Our model: ukn+1 − ukn dt

! k∇uk k2 = − α 4(Id − λ4)−1 ukn k∇uk0 k2   +µ ukn+1 + ukn−1 − 2ukn+1 ,

with uk (x, y ) = u(x, y , k ), u 0 be the original video sequence. α > 1 for deblurring. µ is a weighting parameter.

9/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Computations – in spatial domain

Calder et. al. derive an explicit expression for the operator (Id − λ4)−1 on Ω = R2 , i.e., (Id − λ4)−1 f (x) = Sλ ∗ f (x), with ∗ be the convolution operator and 1 Sλ (x) = 4λπ

Z 0

+∞

|x|2

e−t− 4tλ dt . t

10/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Computations – in frequency domain ˆkn (m1 , m2 ) be the discrete Fourier transform of ukn (x, y ). Let u ˆkn+1 − u ˆkn u −4D(m1 , m2 ) ˆ n + µ(u ˆkn+1 + u ˆkn−1 − 2u ˆkn+1 ) , = Ckn u dt 1 + 4λD(m1 , m2 ) k where P Ckn

m1 ,m2

=P

ˆkn (m1 , m2 )|2 D(m1 , m2 )|u

ˆk0 (m1 , m2 )|2 D(m1 , m2 )|u m1 π 2 m2 π 2 D(m1 , m2 ) = sin( ) + sin( ) M1 M2

−α

m1 ,m2

for discrete coordinates m1 = 1, · · · , M1 and m2 = 1, · · · , M2 .

11/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: Raw data.

12/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB.

13/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB+LAP.

14/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: Raw data.

15/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB.

16/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Figure: SOB+LAP.

17/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Finding a latent image

(a)

(b)

(c)

Figure: Latent Images: (a) applying SOB on the temporal mean. (b) The temporal mean of the video sequence after SOB+LAP. (c) Further improvement using our image fusion technique from the video reconstruction by SOB+LAP.

18/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Partition of image domain

Figure: Partition of the image domain Ω. There is one row or one column overlap between two adjacent sub-regions Ωi and Ωj .

19/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Our “lucky” fusion

From these partitions, Select the best patch u(Ωj , kˆ ) from all the frames for each Ωj for 1 ≤ k ≤ N. The best patch is selected by measuring two terms: the similarity to the mean and the sharpness. Why? Similarity is to enforce correct pixel location, while sharpness is to produce sharp edges.

20/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

original

Lucky-region

Mao-Gilles

one of SOB+LAP

mean of SOB+LAP

our method

Discussion

21/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Conclusions

We proposed SOB+LAP, which stabilizes the temporal oscillation and sharpens the video frame at the same time. We use the lucky-region image fusion technique to construct a latent image from results of SOB+LAP. Future direction is to analyze the turbulence behavior in order to resolve fine details in the case of destructive turbulent degradation.

22/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Turbulence motion analysis

(a)

(b)

Figure: The positioning of the key points along a line. The key points are displayed as the blue dots on (a). (b) shows how these points are oscillating can time t changes. The wave movement of the turbulence happens in groups.

23/24

Sobolev sharpening method

Video stabilization

Reconstruct a latent image

Discussion

Reference: Yifei Lou, Sung Ha Kang, Stefano Soatto, and Andrea L. Bertozzi. Video Stabilization of Atmospheric Turbulence Distortion. (CAM report 12-30) Web: sites.google.com/site/louyifei/research/turbulence

Thank You!

24/24

Video Stabilization of Atmospheric Turbulence Distortion

May 20, 2012 - blurry image frames ⇒ sharpen individual frame temporal oscillations ⇒ stabilize temporal direction. We propose the following PDE model for video stabilization: ut (x,y,k) = S[u(x,y,k)] + ต△k u where S[·] denotes the Sobolev sharpening method on spatial domain, and △k is the Laplacian operator acting on ...

1MB Sizes 1 Downloads 40 Views

Recommend Documents

Robust Video Stabilization to Outlier Motion using ...
have large depth variation and motion of multiple planes. Lee et al ... are used as data to estimate paramters of the assigned ..... Flow Chart of Video Stabilization.

Rate-Distortion based Video Watermarking for storing ...
to monitor the improper activities in an environment. At the same time, ... But the security flaw overlooked in this system is that once the modifications are done.

Video Stabilization with Reinitialization on Sudden ...
The 6th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI 2009). Video Stabilization with Reinitialization on Sudden Scene Change. Sunglok Choi ... Service System Technology Development for Ubiquitous City).

Auto-Directed Video Stabilization with Robust ... - Research at Google
in this paper. Given the ..... frame motion transforms Ft, (2) computing the optimal. 229 .... backward (green) from next key-frame, and the resulting locations.

Discharge characteristics of atmospheric-pressure ...
School of Public Health and Family Medicine, Capital University of Medical Sciences, Beijing 100069,. People's ... (Received 5 July 2006; accepted 29 August 2006; published online 19 October 2006) ..... D. Shim, and C. S. Chang, Appl. Phys.

12 Turbulence modeling.pdf
Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. 12 Turbulence modeling.pdf. 12 Turbulence modeling.pdf. Open.