Parallels between sensory and motor information processing Emanuel Todorov Cognitive Science Department, UCSD The computational problems solved by the sensory and motor systems appear very di¤erent: one has to do with inferring the state of the world given sensory data, the other with generating motor commands appropriate for given task goals. However recent mathematical developments summarized here show that these two problems are closely related; in particular, Bayesian inference is related to stochastic optimal control. Therefore information processing in the sensory and motor systems may be more similar than previously thought. We …rst introduce the ideas intuitively. Consider the control problem of grasping a co¤ee cup within 1 sec. How can this be interpreted as an inference problem? Instead of aiming for a goal in the future, imagine that the future is now and the goal has been achieved. More precisely, shift the time axis by 1 sec and create a …ctive sensory measurement corresponding to the hand grasping the cup. The inference problem then is to compute the muscle activations which caused the observed state transition. Of course the solution is not unique because there are many sequences of muscle activations that could have caused the hand to grasp the cup. The same ill-posedness is present in the control domain and is known as motor redundancy. Conversely, inference problems can be interpreted as control problems. Consider estimating the state of the world given the previous estimate and a new sensory measurement. This can be done in two steps: (1) predict the state using the previous state estimate and a model of one-step dynamics; (2) correct the prediction so as to make it more compatible with the measurement. The corresponding control formulation is as follows: the entity being controlled is the state estimate and the control signal is the measurementbased correction. If the control is chosen so as minimize the sum of an energy cost (favoring no correction) and an accuracy cost (favoring full correction), the optimal control will achieve a trade-o¤ similar to the Bayesian trade-o¤ between priors and likelihoods. The above intuitions can be formalized as follows. Let r (x; t) = p yt ytf jxt = x denote the backward …ltering density, i.e. the probability of the future measurements given the current state. Control corresponds to inference about the future, thus we work backwards in time. r (x; t) satis…es X r (x; t) = py (yt jx) p x0 jx r x0 ; t + 1 (1) x0

Here p is the state transition probability in the absence of controls (passive dynamics) and py (yt jx) is the emission probability or generative model (model of how the measurements are generated). On the control side let v (x; t) denote the optimal cost-to-go, i.e. the cost expected to accumulate if we initialize the system in state x at time t and control it optimally until the …nal time tf . The function v (x; t) satis…es the Bellman equation ( ) X 0 0 v (x; t) = min ` (x; u; t) + p x jx; u v x ; t + 1 (2) u

x0

Here ` is the cost and p is the probability of a transition from state x to state x0 under control u. We focus on cost functions that include an accuracy cost q (x; t) 0 and an energy cost as follows: ` (x; u; t) = q (x; t) + KL (p ( jx; u) jjp ( jx))

(3)

The energy cost (Kullback-Liebler divergence between the controlled and passive dynamics) encourages the controller to let the system evolve according to its passive dynamics. 1

Let p (x0 jx; t) denote the state transition probability under the optimal controls, and let z (x; t) = exp ( v (x; t)) denote the exponentiated optimal cost-to-go. z is large when v is small and vice versa. Thus we will call z the desirability function. With some additional assumptions, it can be shown that the optimal transition probabilities are proportional to the passive transition probabilities scaled by the desirability of the next state: p

x0 jx; t / p x0 jx z x0 ; t + 1

(4)

This form of control is illustrated in Fig (d). Note that multiplication by z shifts the passive dynamics towards more desirable states. Substituting (4) in (2) and exponentiating yields X z (x; t) = exp ( q (x; t)) p x0 jx z x0 ; t + 1 (5) x0

But this is identical to (1) when q (x; t) = log py (yt jx). Thus, for the family of problems we considered, stochastic optimal control is equivalent to Bayesian estimation. Although we focused on discrete systems for simplicity, similar results can be obtained for continuous systems by taking a certain limit. In that case the KL divergence energy cost in (3) reduces to the familiar quadratic cost. Fig (c,e) illustrate a stochastic car-on-a-hill problem which …ts in our framework and can therefore be solved using estimation methods. The traces in Fig (e) are noisy trajectories. Bayesian estimation problems are often expressed as graphical models (Fig a) which expose structure that can be taken advantage of. Our results make it possible to express optimal control problems as graphical models (Fig b), and apply a variety of e¢ cient estimation algorithms to control. One very appealing class of algorithms are particle …lters – which represent probability distributions with samples and avoid the need for function approximation. We are currently working on adapting such algorithms to the control domain, in particular to problems like locomotion and manipulation which involve contact phenomena. Details can be found on our website in: E. Todorov (2008) General duality between optimal control and estimation. To appear in proceedings of the 47th IEEE Conference on Decision and Control. E. Todorov (2008) Parallels between sensory and motor information procession. To appear in The Cognitive Neurosciences, 4th edition, M. Gazzaniga (ed), MIT Press. 2