Efficient neural models for visual attention Sylvain Chevallier, Nicolas Cuperlier and Philippe Gaussier ETIS - Neurocybernetic team Univ. Cergy-Pontoise – ENSEA – CNRS Cergy, France [email protected]

September, 22th. 2010

Framework

Outline

1

Framework Visual attention Neural models

2

Models and implementation Attentional architecture Implementations

3

Experimental results

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

2 / 20

Framework

Visual attention

Change blindness

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

3 / 20

Framework

Visual attention

Change blindness

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

3 / 20

Framework

Visual attention

Change blindness

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

3 / 20

Framework

Visual attention

Bio-inspired attentional vision systems Attentional spotlight metaphor Reduce the search space [Tsotsos, 90] Attentional architecture Feature extraction Combination on saliency map Focus selection through Winner-Take-All

[Itti & Koch, 98]

Applications Driver assistance [Michalke, 08] Retinal prostheses [Parikh, 10] Robotics [Frintrop, 06] S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

4 / 20

Framework

Neural models

Bio-inspired information coding Neurons exchange information through spikes

Spikes have little variations in amplitude and duration Spikes are fully characterized by their emission dates Level of description for neural models: Neuron level Temporal coding, precise spike timing Population level Rate coding, mean firing rate S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

5 / 20

Framework

Neural models

Neural models

Spiking Neuron Network Network of [1, . . . , i, . . . , N] spiking neurons: ( P P (s) dVi j∈Pre wij s∈Trainj δ(t − tj ) + I(t), if Vi < ϑ dt = −λi Vi (t) + else trigger a spike and Vi ← Vreset Frequency-based Neural Network Continuum neural field τ

∂u (x, t) = −u(x, t) + ∂t

S. Chevallier (ETIS)

Z

w(x − x0 )f [u(x0 , t)]dx0 + I(x, t) + h

Efficient neural models

September, 22th. 2010

6 / 20

Framework

Neural models

Goal of this paper

Question What is the most suited neural coding scheme for an efficient bio-inspired attentional architecture ?

Compare SNN and FNN Complexity analysis Quality of results Simple artificial images Natural images

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

7 / 20

Models and implementation

Outline

1

Framework Visual attention Neural models

2

Models and implementation Attentional architecture Implementations

3

Experimental results

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

8 / 20

Models and implementation

Attentional architecture

Preattentive visual architecture IOR

Input image

Low spatial frequencies

WTA

Saliency Input maps

High spatial frequencies

Multi-scale Features

FNN needs WTA to sort saliencies

Contrast of luminance (DOG) Orientations (Gabor) Color opponencies (DOG) S. Chevallier (ETIS)

Efficient neural models

SNN is an anytime process September, 22th. 2010

9 / 20

Models and implementation

Implementations

SNN implementation

DOG filter

details

Neural filter S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

10 / 20

Models and implementation

Implementations

Complexity analysis FNN Filtering cost: for f features, s spatial scales, filters of size M and N input image pixels WTA cost: O(N) with ARGMAX O(f × s × M × N) SNN Hybrid synchronous simulator, with time step ∆t Total cost = Spike propagation cost + neuron update cost cp × F × M × N + cu ×

A ∆t

F is mean firing rate, A is number of active neurons. cu is 10 FLOP. S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

11 / 20

Models and implementation

Implementations

Complexity analysis SNN computational cost depends on emitted spikes Is the number of spikes constant for processing different images ?

CPU cycles (106 )

2.5

1 patch 10 patchs 50 patchs 100 patchs

2 1.5 1 0.5 0

0

10

20 30 40 Simulated time (t)

50

60

For SNN, computational cost depends on the input image Rich images (w.r.t chosen filters) induce large number of spikes S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

12 / 20

Experimental results

Outline

1

Framework Visual attention Neural models

2

Models and implementation Attentional architecture Implementations

3

Experimental results

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

13 / 20

Experimental results

Comparison on artificial images Pop-out artificial images

FNN Circle shows most salient region, winner of WTA (FNN) SNN Dots indicate the most salient pixels (SNN) Same salient items are found for FNN and SNN (20 images) S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

14 / 20

Experimental results

Natural images 19 webcam images of 160x120 pixels

Salient regions might not be extracted in the same order Measured computational cost (as CPU cycle): Constant for FNN SNN can find salient regions before FNN (1/4 of the images) S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

15 / 20

Experimental results

Conclusion and perspective Comparison of two neural models for an attentional system Frequency-based Neural Network: have a constant and lower computational cost, needs a WTA to sort saliencies Spiking Neuron Network: have a variable computational cost have anytime capabilities Perspective Formal analysis of spiking neuron processing Learning capability of neural network Attentional bias modulating salient regions Long term adaptation of input signal (slow variation of illumination) S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

16 / 20

Experimental results

Annex

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

17 / 20

Experimental results

Input maps



dVi dt

= −λi Vi (t) + KLi , if Vi < ϑ else trigger a spike and Vi ← Vreset

with Li the considered pixel value Φi =

  1 λϑ ˆti = − ln 1 − i λi KLi

λi

= −



ln 1 −

back

≈

S. Chevallier (ETIS)

1 ˆti

Efficient neural models

λi ϑ KLi



K Li ϑ

September, 22th. 2010

18 / 20

Experimental results

Input maps

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

18 / 20

Experimental results

Integration maps



PPj = −λj Vj (t) + i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset dVj dt

Si (t) =

Ni X

δ(t − tif )

f =1

back

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

19 / 20

Experimental results

Integration maps



PPj = −λj Vj (t) + i=1 wij Si (t), if Vj < ϑ else trigger a spike and Vj ← Vreset dVj dt

Vj (t) =

Pj X

wij

Ni X

e−λj (t−fˆti ) H(t, f ˆti )

f =1

i=1

Vj (Tj ) ≈

Pj X i=1

wij

1 − e−QNi /Li 1 − e−Q/Li

with Q =

λj ϑ K

back

S. Chevallier (ETIS)

Efficient neural models

September, 22th. 2010

19 / 20

Experimental results

Frequency coding

P1 P2 P3 P4

V

ϑ S t ISI

S. Chevallier (ETIS)

6 ms

4 ms

5 ms

Efficient neural models

5 ms

4 ms

September, 22th. 2010

20 / 20