Cognitive Brain Research 14 (2002) 177–186 www.elsevier.com / locate / bres

Research report

Influence of a sensorimotor conflict on the memorization of a path traveled in virtual reality a, b a Simon Lambrey *, Isabelle Viaud-Delmon , Alain Berthoz a

` de France-CNRS, 11 Place Marcelin Berthelot, 75005 Paris, France LPPA, College b ˆ ˆ ` , Paris, France CNRS-UMR7593, Hopital de la Salpetriere Accepted 27 October 2001

Abstract Studies of visual–vestibular and vestibular–proprioceptive interactions suggest that prolonged exposure to sensory conflicts induces a modification of the relation between sensory modalities for self-motion perception. With most models conflicts are solved by a weighting process. However, the brain could also switch between conflicting cues. The present study focused on the effect of mismatched visual and non-visual information on the reproduction of actively performed turns. Standing subjects viewed a virtual corridor in which forward movements were simulated at a constant linear velocity, and rotations were actually performed. They were asked to learn the trajectory and then to reproduce it from memory in total darkness. In the baseline condition, the relative amplitudes of visual and non-visual information for the rotations performed were the same, but were manipulated in the two ‘sensory conflict’ conditions. The results show that even when subjects did not notice the sensory conflict, the discrepancy between visual and non-visual information affected their ability to reproduce the angular displacements. In one conflict condition, subjects relied on visual information when asked to draw the trajectory traveled, yet reproduced rotations on the basis of non-visual information during active blindfolded movements. This dissociation suggests that for mental simulation of the same path, there are at least two cognitive strategies of memory storage and retrieval, using either visual or non-visual information, according to the task and the sensory context.  2002 Elsevier Science B.V. All rights reserved. Theme: Neural basis of behavior Topic: Learning and memory: systems and functions Keywords: Spatial memory; Sensory conflict; Human; Multisensory interaction; Virtual reality

1. Introduction Self-motion perception involves multiple sensory systems (vision, vestibular system, proprioception) that provide information on the direction and the speed of displacements. The movement of the image of the external world across the retina can create the illusion of selfmotion (vection) while the observer remains stationary [4,8,11,12,29,30,38]. The heading direction can be extracted from the optic flow [7,9,15,28,39,44]. Vestibular and somatosensory information is sufficient to estimate a body’s displacement during translations [2,20,21] as well as during rotations [22,23] and the amplitude of a passive rotation in total darkness can be estimated and stored in the *Corresponding author. Tel.: 133-1-44-27-13-86; fax: 133-1-44-2713-82. E-mail address: simon.lambrey@college de france.fr (S. Lambrey). ] ]

memory [19,22,24]. Other results suggest that the availability of proprioceptive information significantly improves the accuracy of subjects in path integration tasks [40] as well as in the estimation of angular displacements [1]. Proprioceptive information on trunk rotation relative to the feet can be used as a means of controlling heading movement and trajectory curvature during locomotion [16,45]. These findings suggest that each sensory system can generate an estimate of self-motion, but the mechanisms of high-level interaction between these different systems remains unclear despite the variety of models of sensory interaction in the literature [6,13,33–35]. Under normal conditions, visual, proprioceptive and vestibular systems provide congruent information. Numerous studies have used paradigms in which intersensory conflicts were generated in order to investigate multisensory interactions (see Ref. [47] for a review of this literature). One way to

0926-6410 / 02 / $ – see front matter  2002 Elsevier Science B.V. All rights reserved. PII: S0926-6410( 02 )00072-1

178

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

measure the effect of exposure to an intersensory conflict is to study post-exposure after-effects, e.g. the mechanisms of changes to the vestibulo-ocular reflex after prism adaptation, which have been extensively investigated [3]. Recently, virtual reality has been used to study visualvestibular interaction during whole-body rotations in humans [26,42,43]: subjective estimates of passive wholebody rotations in darkness were recorded before and after prolonged exposure to conflicting visual–vestibular stimulation. The findings suggest that subjects may have undergone plastic changes induced by exposure to visual–vestibular conflict as vestibular perception of self-rotations was recalibrated after the conflict [26,43]. In other studies investigating vestibular–proprioceptive interaction, subjects walked around the edge of a horizontally rotating disc holding the trunk and head steady. Before and after stimulation, the ability of a blindfolded person to move in a straight line, either walking or in a wheel chair, was tested. All subjects generated curved trajectories after stimulation, without being conscious of turning [16]. According to the authors, these findings suggest a remodeling of the relation between somatosensory / motor elements of gait and the perception of trunk rotation. Similar results were obtained when the subjects were asked to step in place time on the center of a rotating disc, instead of walking [45]. These observations of visual–vestibular and vestibular–proprioceptive interactions suggest that prolonged exposure to sensory conflicts may induce an adaptive modification of the relation between sensory modalities for self-motion perception. They do not, however, provide information on the perceptual experience of the subject during exposure to the conflict. In the present study, these questions were extended to the mechanisms of multisensory interaction during navigation. To study the influence of a sensory conflict on spatial orientation, a paradigm in virtual reality was designed so as (a) to manipulate the relative amplitude of the visual and non-visual signals of rotation in a navigation task and (b) to investigate the interaction between conflicting visual and non-visual information in a reproduction task. Standing subjects were shown a virtual corridor in which their forward motion was simulated at a constant linear velocity while they actively controlled their direction (i.e. orientation in the corridor) by rotating in place. They were asked to learn the path traveled and actively reproduce it in total darkness without any visual cues. Three experimental conditions were used. In the baseline condition, the relative amplitudes of visual and non-visual information on the rotations to be performed were the same, while they were manipulated in two conflict conditions. This paradigm examined three questions: (i) are subjects able to navigate accurately in the virtual world despite sensory conflicts? (ii) are they able to memorize the path traveled? and if so (iii) how do conflicting visual and non-visual signals interact in the representation of the angular displacements stored in their memory? The ability of subjects to faithfully reproduce three key features of the

trajectory was analyzed and produced two groups of subjects (Good reproducers vs. Poor reproducers). For the third question, two different hypotheses were examined. One is that conflicting visual and non-visual signals are combined through weighted multisensory integration and produce a single ‘compromise’ estimate of the angular displacements, an estimate by definition different from both visual and non-visual estimates. The second hypothesis is that one of the two conflicting inputs is selected, thus ‘turning off’ one sensory system, disregarding any information provided by that sensory system even though input is still available; this selection may occur during encoding or recall.

2. Materials and methods

2.1. Subjects Twenty-one healthy volunteers (10 female, 11 male; mean age524 years, S.D.52.9) took part in the study. All but one were right-handed. No subject had a history of neurological disease, head injury, or psychiatric disorders; none had complained of memory difficulties. All subjects had normal, or corrected to normal, vision. Informed consent was obtained after the nature and possible consequences of the study were explained. The experiment was approved by the local ethics committee (CCPPRB 120-98).

2.2. Experimental set-up Subjects stood at the central point of a circular safety rail and were fitted with a virtual reality helmet (Fig. 1a). They were shown virtual corridors with simulated forward motion generated by the computer at a constant translation velocity. The linear speed in the corridor was within the locomotor range of the human walking gait (approximately 1 m / s). When a corner appeared in the corridor, subjects had to actually rotate their body to turn in the virtual environment. A gain g was included in the computation of virtual image updated on the basis of body rotations, so that visual rotations in the helmet were equal to g times the body rotations. The software did not allow subjects to cross walls, and in the event of collisions, the trajectory ran along the wall. The virtual reality apparatus has been used in other investigations [17,42,43]. It included an LCD display (VR4, Virtual Research Systems, Santa Clara, CA), which had a monocular field of view of 488 by 368 (total resolution 7423230) and was refreshed at 30 Hz (NTSC interlaced standard). The refresh rate did not induce any noticeable flicker or eyestrain. The orientation of the subject’s head on the horizontal plane was measured by an ultra-sound system (Logitech, Fremont, CA), with a report rate of 50 Hz. The image generator (Indigo Extreme, Silicon Graphics) recorded the head angular position, the data being sent from the tracker, transmitting the corre-

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

179

walls (Fig. 1b). Different colors were used to distinguish each corridor (C1, C2, C3). Each of the three corridors had four straight segments (all the same length) connected by three angles (45, 90 and 135 8).

2.4. Procedure

Fig. 1. Experimental set-up. (a) The standing subject was immersed in a virtual environment by means of a head-mounted display. Translation in the corridor was imposed by the computer. The direction of the subject’s head was recorded using an ultrasound tracking system. Custom-written software processed the data on the head angular position sent by the tracker and updated the corresponding visual image, providing the visual rotation components in the virtual corridor. (b) Graphic model of the virtual environment (example: view of the entrance to the corridor C3).

sponding image to the display, with a 30 Hz update rate (the total latency between sensor motion and complete image display being estimated at 150 ms).

2.3. Virtual environment Three corridors were used. For each the graphic model consisted of a checkerboard pattern on the floor, a plain colored ceiling, and black stripes evenly spaced on the

Before the experiment, subjects were instructed not to move their head in relation to their trunk when turning. They were asked to control their direction when trying to go around the corners as if they were in a real corridor and to stay as close as possible to the middle of the corridor. The experiment began with a training period during which subjects navigated along a single corridor with seven 908 angles. In the event of a collision along the path, subjects had to travel through the corridor once more. The experiment was comprised of two tasks. Subjects were first asked to navigate along a virtual corridor twice, trying to memorize the trajectory traveled (navigation task). They were then asked to reproduce the path traveled without any visual cues, by mentally visualizing the route (reproduction task). During the reproduction task, they had to remain immobile during the imagined straight segments, but when they remembered a turn, had to rotate their body by the amount needed to fit the real image from the previous navigation. The tasks described above were successively performed by the subjects under three distinct experimental conditions (Fig. 2): (1) baseline; (2) high-gain; and (3) low-gain. High-gain and low-gain conditions provided sensory conflicts as visual and non-visual cues did not provide the same information on angular amplitudes during navigation. In the baseline condition, when subjects navigated along corridor C1, the gain g (see Section 2.2) was equal to 1, i.e. visual and body rotations had the same amplitude. In the high-gain condition, subjects navigated along corridor C2 and the gain was 1.5, i.e. visual rotations were 3 / 2 the size of body rotations. In the low-gain condition, subjects navigated along corridor C3 and the gain was 0.66, i.e. visual rotations were 2 / 3 the size of body rotations. To assess the subjective representation of the trajectory in the low-gain condition, subjects also performed a drawing task, sketching corridor C3 on a sheet of paper. Subjects were not informed of this task until they had completed all stages of the experiment, so as to avoid any deliberate preparation of a mental map. The full length of the experimental session was approximately 45 min.

2.5. Data analysis 2.5.1. Qualitative analysis of trajectories The computer recorded the number of collisions during navigation and the instantaneous angular direction of subjects during both navigation and reproduction. The virtual trajectories of the path followed by subjects when navigating in the corridor and when reproducing the path were plotted as a function of the angular direction of the

180

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

the image generator recorded information on the angular direction of the head from the tracker and transmitted the corresponding visual image to the head-mounted display, the amplitudes of visual rotations of navigation were computed by multiplying those of body rotations by the gain, g.

2.5.3. Translation duration analysis On the direction profiles, one segment with constant angular direction corresponds to one translation along one segment of the corridor. The analysis of direction profiles could therefore measure the duration (in seconds) of each translation portion of the paths traveled.

2.5.4. Drawings analysis The amplitude of each turn marked in the drawings was measured using a protractor. The measurement was made by two different experimenters.

2.5.5. Statistical analysis

Fig. 2. Synopsis of the experiment. T1 and T2 were, respectively, the first and the second trials for the navigation task. R was to the reproduction task.

head. A qualitative assessment of the ability to reproduce the trajectories was made on the basis of three key features: (a) the hierarchy of the turn amplitudes (e.g. in C1, the first turn has to be greater than the second and smaller than the third); (b) the directions of the turns (e.g. right, then left, then right in C1); (c) the total number of turns (three turns in each corridor). Subjects were divided into two groups for further analysis: those who reproduced all features without any error for each of the three corridors—the ‘good reproducers’—and subjects who made one or more mistakes—the ‘poor reproducers’.

2.5.2. Angular amplitude analysis Direction profiles were plotted, producing graphs showing the angular direction of the head (in degrees) as a function of time. Analysis of the profiles gave measurements of the amplitudes of the rotations performed (body rotations of navigation and reproduced body rotations). As

2.5.5.1. Angular amplitude analysis. To compare the amplitudes of reproduced body rotations with those of both visual and body rotations of navigation, two distinct and successive analyses of variance were conducted. The first analysis examined the amplitudes of reproduced body rotations and visual rotations of navigation. The second examined the amplitudes of reproduced body rotations and body rotations of navigation. A 2333233 ANOVA design was used, the GROUP factor (Good vs. Poor) being considered as a between-subject factor, while CONDITION (Baseline vs. High-Gain vs. Low-Gain), TASK (Navigation vs. Reproduction) and ANGLE (458 vs. 908 vs. 1358) were manipulated within-subjects. Post-hoc analyses were performed using the Tukey test. 2.5.5.2. Translation duration analysis. A 2333234 ANOVA design was used: the GROUP factor (Good vs. Poor) was considered as a between-subject factor, while CONDITION (Baseline vs. High-Gain vs. Low-Gain), TASK (Navigation vs. Reproduction) and SEGMENT (S1 vs. S2 vs. S3 vs. S4) were manipulated within-subjects. The dependent variable was the duration of the translation. Post-hoc analyses were performed using the Tukey test. 2.5.5.3. Drawings analysis. A 23433 ANOVA design was used: the GROUP factor (Good vs. Poor) was considered as a between-subject factor, while the ROTATION TYPE (Visual rotation of navigation vs. Body rotation of navigation vs. Reproduced body rotation vs. Drawing turn) and ANGLE (458 vs. 908 vs. 1358) were manipulated within-subjects. The dependent variable was

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

angular amplitude. Post-hoc analyses were performed using the Tukey test.

3. Results

3.1. Qualitative analysis of trajectories 3.1.1. Navigation Under all baseline and conflict conditions, subjects were able to perform the navigation task easily (Fig. 3a), although most subjects collided with a wall at least once during the experiment. The number of collisions was greater for the first experience of the path than for the second. In both conflict conditions, subjects managed to navigate accurately along the corridors in spite of the incongruent gain between visual and non-visual information on angular displacements. Surprisingly, of the 21 subjects only one noticed the difference between body and visual rotations; the others did not mention any difference between the baseline and conflict conditions. 3.1.2. Reproduction The ability of subjects to reproduce the three key features of the trajectories (see Section 2.5.1) did not clearly differ from one condition to another. Of the 21 subjects taking part in the experiment, 12 (five females and seven males) did not make any errors in reproducing the three features, in any of the conditions. Subjects were divided into two groups. Those who reproduced all the features without any errors in any of the three corridors comprised the group of good reproducers. Subjects who made one or more mistakes were called poor reproducers, but even in the group of good reproducers, there was noticeable inter-subject variability in the reproduction of the trajectories (Fig. 3b). It should be noted that no significant Group effects were found in any of the statistical analyses. To avoid repetition, this finding will not be restated below.

181

low-gain (P,0.001) conditions. Furthermore, reproduced body rotations were greater in the low-gain condition than in the baseline condition (P,0.01). There was one main effect of task (F1,19 553.76, P,0.0001) with reproduced body rotations being, overall, greater than visual rotations of navigation. Interestingly, there was a significant interaction between condition and task (F2,38 514.54, P,0.0001). The Tukey test showed that reproduced body rotations were significantly greater than visual rotations of navigation in both baseline (P,0.001) and low-gain (P,0.001) conditions, but not in the high-gain condition (no significant difference).

3.2.2. Comparison of body rotations of navigation and reproduced body rotations The ANOVA showed one main effect of condition (F2,38 5148.58, P,0.0001). There was one main effect of the task (F1,19 534.13, P,0.0001) with reproduced body rotations being, overall, greater than body rotations of navigation. It is important to note that there was a significant interaction between condition and task (F2,38 522.87, P, 0.0001). A Tukey test showed that reproduced body rotations were significantly greater than body rotations of navigation in both baseline (P,0.001) and high-gain (P, 0.001) conditions, but not in the low-gain condition (no significant difference). All data are reported in Fig. 4.

3.3. Translation duration analysis Translation durations were shorter for reproduction than for navigation (Fig. 5). The ANOVA showed one main effect of task (F1,19 511.31, P,0.004), but no conditionrelated effect was observed.

3.4. Drawings analysis 3.2. Angular amplitude analysis 3.2.1. Comparison of visual rotations of navigation and reproduced body rotations Three experimental conditions were distinguished: one baseline and two conflict conditions. In the high-gain conflict condition, body rotations were smaller than visual rotations during navigation. In the low-gain conflict condition, body rotations were greater than visual rotations during navigation. Sensory conflict influenced the ability to reproduce angular displacements (Fig. 4.). In fact, the ANOVA showed one main effect of condition (F2,38 513.60, P, 0.0001) on reproduced body rotations (Fig. 4). The Tukey test showed that reproduced body rotations were smaller in the high-gain condition than in both baseline (P,0.01) and

Of the nine poor reproducers, one subject drew the map of corridor C2 instead of C3 and another subject made a drawing, which could not be interpreted (not tallying with the map of any of the three corridors or the verbal description of the path by the subject). Data from these two subjects were excluded from further analysis of the drawings. The other seven poor reproducers seemed to draw the map of corridor C3 in the reference frame of the virtual world (i.e. on the basis of visual information). Of the 12 good reproducers, 10 drew angles of each turn of corridor C3 in the reference frame of the virtual world, although one subject put the wrong direction for the 458 turn (Fig. 6). The ANOVA showed one main effect related to the type of rotation (F3,51 587.93, P,0.0001), which increased with the size of the angle (F6,102 518.52, P,

182

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

Fig. 3. Trajectories followed by the 12 good reproducers (plotted together) in navigation (A) and reproduction (B) under the three experimental conditions (baseline, high-gain and low-gain). All the trajectories are shown in the reference frame of the virtual world, i.e. the amplitudes of turns in the navigation trajectories are the amplitudes of visual rotations and the amplitudes of turns in the reproduction trajectories are the amplitudes of body rotations corrected with the gain g. The arrows show the direction of the displacement. As subjects tended to reproduce rotations on the basis of visual information in the high-gain condition, the amplitudes of turns in the reproduction trajectories seem to be very wide in the virtual world reference frame. In the low-gain condition, subjects reproduced rotations on the basis of non-visual information and the amplitudes of turns in the trajectories of reproduction do not appear to differ from the turn amplitudes in navigation.

Fig. 5. Mean duration of translation under the three experimental conditions (baseline, high-gain and low-gain). Translation durations have been averaged across the four linear segments of the corridor and the 21 subjects. The error bars show the standard deviations. Fig. 4. Mean angular amplitudes of visual rotations of navigation, body rotations of navigation and reproduced body rotations under the three experimental conditions (baseline, high-gain and low-gain). Angular amplitudes have been averaged across the three corridor turns (458, 908 and 1358) and the 21 subjects. The error bars show the standard deviations.

0.0001). A Tukey test showed the turns in the drawings to be smaller than body rotations of navigation (P,0.001) and greater than visual rotations of navigation (P,0.05). Turns in drawings were also smaller than reproduced body rotations (P,0.001). Data are reported in Fig. 7.

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

183

4. Discussion In the present study, subjects were asked to reproduce a path through a virtual environment in one baseline and two conflict conditions (high-gain and low-gain) in which the normal relationship between actual motion and visual consequences was manipulated. In the high-gain condition, actual body rotation led to oversized changes in the visual angle. In the low-gain condition, the same rotation led to undersized changes in the visual angle. The key question was whether the imposed sensorimotor mismatch would have an influence on the reproduction of the turn angles memorized. The ability of subjects to faithfully reproduce the trajectory (order, direction and number of turns) was analyzed and divided subjects into two groups (Good vs. Poor). Both groups of subjects overestimated angles in the baseline condition. In the high-gain condition, amplitude estimation appeared to be dominated by visual cues whilst in the low-gain condition, amplitude estimation was dominated by non-visual cues. The drawings made by subjects at the end of the experiment suggested that, in parallel, the brain may also store a map based on visual information. Fig. 6. Drawings of the corridor C3 (by the 12 ‘good reproducers’). (a) Theoretical map for corridor C3 where the angular amplitudes of turns are equal to the amplitude a of the visual rotations (in the virtual world reference frame). (b) Theoretical map for corridor C3 where the angular amplitudes of turns are equal to the theoretical amplitude 1.53a of the body rotations of navigation (in the real world reference frame). (c) Superimposition of the sketch-maps made by the 12 good reproducers for corridor C3. For each subject, the sketch-map of trajectory C3 was re-drawn by considering the angular amplitudes of turns in the primary drawing and by taking one single arbitrary length for all linear segments (the same length was considered for all 12 subjects). The reconstructed drawings of all subjects were plotted together.

4.1. Subjective reports and qualitative reproduction of the trajectories It is quite remarkable that only one of the 21 subjects was consciously aware of the difference between body and visual rotations in conflict conditions, yet they all managed to navigate accurately in the corridors in all three baseline and conflict conditions. No subject reported that the memorization task was more difficult in one condition compared to the others. Furthermore, the number of errors in reproducing the three key features of the trajectories (hierarchy of angles, directions and number of turns, see Section 2.5.1) was similar in all conditions. None of the subjects reported any feeling of motion sickness. All these findings suggest that the conflict between visual and non-visual information did not disturb the subjects’ performance in the navigation or memorization tasks. This may be because the navigation task was performed under visual guidance and therefore, the sensorimotor conflict was not noticed by the subjects.

4.2. Translation duration analysis

Fig. 7. Mean angular amplitudes of visual rotations of navigation, body rotations of navigation, reproduced body rotations and drawing turns in the low-gain condition. Angular amplitudes have been averaged across the three corridor turns (458, 908 and 1358) and the 21 subjects. The error bars show the standard deviations.

The conflict was not found to have any influence on the reproduction of translation durations. This may be related to the fact that simulated forward motion was set by the computer at the same constant translation velocity for all conditions. Furthermore, the discrepancy between visual and non-visual signals in conflict conditions only concerned angular displacements. Subjects were quite accurate in reproducing the translation durations of the trajectories. This finding is consistent

184

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

with previous studies on the timing of ‘mental walking’ by humans [10,14,32], although data from the present study also indicate that subjects systematically shortened translation durations during reproduction, with a reduction of approximately 2 s (13%). This may be because of the lack of environmental constraints in mentally imagined actions.

4.3. Quantitative reproduction of angular amplitudes in baseline and conflict conditions

4.3.1. Reproduction of angular amplitudes in the baseline condition In the baseline condition, reproduced body rotations were significantly greater than rotations of navigation. This overshooting may have been caused by either an overestimation of the angular displacements during navigation or an underestimation of the angular displacements during reproduction. This is consistent with the findings of Siegler [41], who observed that subjects required to carry out 3608 rotations of a robotic chair on which they were seated tended to overestimate self-performed whole-body rotations in darkness. Other experiments on self-rotation estimation resulted in the same observation [5,23,25]. In a recent study [1], subjects were verbally instructed to turn their body specific angles (458, 908, 1808 and 2708) under different combinations of visual, vestibular and kinesthetic feedback: they all stopped their movements before reaching the required angle, suggesting they had overestimated the angle actually turned. This overestimation was most marked when only visual information was available to estimate the rotation, but was dramatically reduced when proprioceptive information was available with or without vision. Consistent with this observation, the subjects of the present study would be expected to accurately reproduce the rotations they had memorized during navigation as proprioceptive information was available to them to control their angular displacements during reproduction. The overshoot observed in the reproduction of angular amplitudes would therefore result from overestimating rotations performed during navigation, but as both non-visual (including proprioceptive) and visual information were available during navigation, the estimation of rotations should also have been accurate [1]. The hypothesis is then that in the baseline condition subjects gave preference to visual information to memorize their angular displacements, leading to an overestimation of rotations during navigation and to an overshoot in the reproduction of angular amplitudes. Such a hypothesis is consistent with the fact that navigation was performed under visual guidance and that vision predominates in self-motion perception in stationary environments. But an alternative hypothesis for the memory process can be proposed: the overshoot observed in the present experiment may result from an inaccuracy in encoding and not from the use of

visual information in the estimation of angular displacements. Further experiments need to be devised to distinguish between these two hypotheses.

4.3.2. Influence of the sensory conflict on the reproduction of angular amplitudes The present findings clearly show that even when subjects did not perceive the conflict, the imposed sensorimotor mismatch did have an influence on the reproduction of memorized turn angles. But how did conflicting visual and non-visual signals interact in the stored representation of angular displacements? One hypothesis is that conflicting signals are combined through a weighted multisensory integration to produce a single ‘compromise’ estimate of displacements (e.g. Refs. [18,36,37,46]). Working on this hypothesis, which implies some kind of linear addition of different cues, Warren et al. asked subjects to actually walk to visual targets in an immersive virtual environment [44]. The virtual optic flow was artificially displaced from the actual direction of walking. The authors found that the visual control law for steering to the target was a linear combination of both optic flow and walking direction. However, according to that hypothesis, the estimate of angular displacements (measured in the reproduction task) would be by definition different from both conflicting visual and non-visual estimates, which is not the case in the present experiment. In the high-gain condition, reproduced body rotations were greater than body rotations of navigation, but did not differ from the visual rotations of navigation. Conversely, in the low-gain condition, reproduced body rotations were greater than visual rotations of navigation, but did not differ from body rotations of navigation. These findings tend to suggest that subjects reproduced rotations on the basis of visual information in the high-gain condition and on the basis of non-visual information in the low-gain condition, therefore supporting the hypothesis that one of the two conflicting inputs is selected. One sensory modality may be dominant, with either visual or non-visual information being selected to estimate angular displacements, depending on the sensory context (i.e. on the gain value). This process of selection would mean that any information provided by the non-dominant modality would be completely disregarded, even though the input is still available. Supporting this selection hypothesis, recent studies have suggested that sensory fusion has non-linear aspects [9,27]. Jeka et al. [27] conducted tests to see whether a linear additive model could account for the combination of touch and vision in postural control in humans. While certain aspects of their data were generally consistent with qualitative predictions using an additive model of multisensory integration, quantitative comparisons between the empirical data and model predictions clearly indicate the presence of nonlinearities, inconsistent with any linear additive model. The authors have suggested that an experimental paradigm

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

testing small sensory discrepancies might be insufficient to point out non-linear integration [27]. In the case of major discrepancies between two or more sensory modalities, a compromise between inputs may not allow accurate actions as such a compromise would be necessarily different from all incoming sensory signals. Conversely, the selection of one sensory modality would allow an immediate and correct (at least, partially) perceptual response to intersensory discrepancies. Such selection can be seen as nothing more than an extreme case for low-level sensory re-weighting, entailing the ‘turning off’ of one or more sensory systems. The present study, however, suggests that a higher-level mechanism affecting perceptual decision may also underlie the selection process. Sensory re-weighting, involving adaptation after prolonged exposure to sensory conflicts, is not an instantaneous process and would not operate during exposure.

4.4. Analysis of memorization of the path using drawings When required to draw the trajectory of corridor C3 on completion of the last stage of the experiment, subjects relied on visual information as seen in Fig. 5. Turns in the drawings were greater than visual rotations but significantly smaller than reproduced body rotations. As discussed in Section 4.3.2, for the same corridor, C3, subjects reproduced rotations on the basis of non-visual information during active blindfolded reproduction. Such dissociation suggests that there are at least two kinds of representation of the same path, constructed on the basis of either visual or non-visual information, which supports the hypothesis that the brain stores both forms of information in parallel. According to the present findings, the selection of relevant information occurs when reproducing the path, involving action-driven memory mechanisms. This point of view is consistent with recent findings suggesting the influence of top-down processes on spatial orientation [31]. In conclusion, the following hypothesis is proposed: all of the sensory systems involved in self-motion perception generate a representation of displacements in the memory and these modality-dependent representations may be either combined or selected at the time of recall, according to the task and the sensory context.

Acknowledgements This work was supported by SmithKline Beecham, ‘GIS-Sciences de la Cognition’ and HFSP: RG71 / 96B. The equipment was provided in part by the CNES. The authors wish to thank Sidney Wiener, Michel-Ange Amorim and Isabelle Siegler for their helpful comments on the text, as well as France Maloumian for the graphs.

185

References [1] N.H. Bakker, P.J. Werkhoven, P.O. Passenier, The effects of proprioceptive and visual feedback in geographical orientation in virtual environments, Presence 8 (1999) 36–53. ¨ P. George-Franc¸ois, R. Grasso, T. Tsuzuku, [2] A. Berthoz, I. Israel, Spatial memory of body linear displacement: What is being stored, Science 269 (1995) 95–98. [3] A. Berthoz, G. Melvill Jones (Eds.), Adaptative Mechanisms in Gaze Control, Elsevier, Amsterdam, 1985. [4] A. Berthoz, B. Pavard, L. Young, Perception of linear horizontal self motion induced by peripheral vision (linear-vection), Exp. Brain Res. 23 (1975) 471–489. [5] W. Bles, J.M.B.V. Dejong, G. De Wit, Somatosensory compensation for loss of labyrinthine function, Acta Otolaryngol. 97 (1984) 213–221. [6] J. Borah, L.R. Young, R.E. Curry, Optimal estimator model for human spatial orientation, Ann. NY Acad. Sci. 545 (1988) 51–73. [7] D.C. Bradley, M. Maxwell, R.A. Andersen, M.S. Banks, K.V. Shenoy, Science 273 (1996) 1544–1547. [8] T. Brandt, J. Dichgans, E. Koenig, Differential effects of central versus peripheral vision on egocentric and exocentric motion perception, Exp. Brain Res. 16 (1973) 476–491. [9] J.A. Crowell, M.S. Banks, K.V. Shenoy, R.A. Andersen, Visual self-motion perception during head turns, Nat. Neurosci. 1 (1998) 732–737. [10] J. Decety, M. Jeannerod, C. Prablanc, The timing of mentally represented actions, Behav. Brain Res. 34 (1989) 35–42. [11] J. Dichgans, T. Brandt, Visual–vestibular interaction and motion and perception, in: J. Dichgans, E. Bizzi (Eds.), Cerebral Control of Eye Movements and Motion Perception, Karger, New York, 1972, pp. 327–338. [12] J. Dichgans, T. Brandt, Visual–vestibular interactions: effects on self-motion perception and postural control, in: H. Leibowitz, H.L. Teuber (Eds.), Handbook of Sensory Physiology, Springer, Berlin, 1978, pp. 755–804. [13] J. Droulez, C. Darlot, The geometric and dynamic implications of the coherence constraints in three dimensional sensorimotor coordinates, in: M. Jeannerod (Ed.), Attention and Performance XIII, Laurence Erlbaum Associates, NJ, 1989, pp. 495–526. [14] O. Ghaem, E. Mellet, F. Crivello, N. Tzourio, B. Mazoyer, A. Berthoz, M. Denis, Mental navigation along memorized routes activates the hippocampus, precuneus and insula, NeuroReport 8 (1997) 739–744. [15] J.J. Gibson (Ed.), The Senses Considered as Perceptual Systems, Houghton, Boston, MA, 1966. [16] C.R. Gordon, W.A. Fletcher, G. Melvill Jones, E.W. Block, Adaptative plasticity in the control of locomotor trajectory, Exp. Brain Res. 102 (1995) 540–545. [17] R. Grasso, Y.P. Ivanenko, J. McIntyre, I. Viaud-Delmon, A. Berthoz, Spatial, not temporal cues drive predictive orienting movements during navigation: a virtual reality study, NeuroReport 11 (2000) 775–778. [18] I.P. Howard (Ed.), Human Visual Orientation, Wiley, New York, 1982. ¨ C. Andre-Deshays, ´ [19] I. Israel, O. Charade, A. Berthoz, C. Popov, M. Lipsitz, Gaze control in microgravity. II. Sequences of saccades towards memorized visual targets, J. Vest. Res. 3 (1993) 345–360. ¨ A. Berthoz, Contribution of the otoliths to the calculation [20] I. Israel, of linear displacement, J. Neurophysiol. 62 (1989) 247–263. ¨ N. Chapuis, S. Glasauer, O. Charade, A. Berthoz, Estima[21] I. Israel, tion of passive horizontal linear whole-body displacement in humans, J. Neurophysiol. 70 (1993) 1270–1273. ¨ M. Fetter, E. Koenig, Vestibular perception of passive [22] I. Israel, whole-body rotation about horizontal and vertical axes in humans: goal-directed vestibulo-ocular reflex and vestibular memory-contingent saccades, Exp. Brain Res. 96 (1993) 335–346.

186

S. Lambrey et al. / Cognitive Brain Research 14 (2002) 177 – 186

¨ S. Rivaud, A. Berthoz, C. Pierrot-Desseiligny, Cortical [23] I. Israel, control of vestibular-guided saccades in man, Brain 118 (1995) 1169–1183. ¨ S. Rivaud, C. Pierrot-Desseiligny, A. Berthoz, ‘Delayed [24] I. Israel, VOR’: an assessment of vestibular memory for self motion, in: J. Requin, G.E. Stelmach (Eds.), Tutorials in Motor Neuroscience, Kluwer, Dordrecht, 1991, pp. 599–607. ¨ A. Berthoz, Spatial orientation in [25] Y.P. Ivanenko, R. Grasso, I. Israel, humans: perception of angular whole-body displacements in twodimensional trajectories, Exp. Brain Res. 117 (1997) 419–427. ¨ A. Berthoz, The [26] Y.P. Ivanenko, I. Viaud-Delmon, I. Siegler, I. Israel, vestibulo-ocular reflex and angular displacement perception in darkness in humans: adaptation to a virtual environment, Neurosci. Lett. 241 (1998) 167–170. [27] J. Jeka, K.S. Oie, T. Kiemel, Multisensory information for human postural control: integrating touch and vision, Exp. Brain Res. 134 (2000) 107–125. [28] M. Lappe, F. Bremmer, A.V. van den Berg, Perception of selfmotion from visual flow, Trends Cogn. Sci. 3 (1999) 329–336. [29] D.N. Lee, E. Aronson, Visual proprioceptive control of standing in human infants, Percept. Psychophys. 15 (1974) 529–532. [30] D.N. Lee, J.R. Lishman, Visual proprioceptive control of stance, J. Human Movement Studies 1 (1975) 87–95. [31] F. Mast, S.M. Kosslyn, A. Berthoz, Visual mental imagery interferes with allocentric orientation judgements, NeuroReport 10 (1999) 3549–3553. [32] E. Mellet, S. Briscogne, N. Tzourio-Mazoyer, O. Ghaem, L. Petit, L. Zago, O. Etard, A. Berthoz, B. Mazoyer, M. Denis, Neural correlates of topographic mental exploration: the impact of route versus survey perspective learning, Neuroimage 12 (2000) 588–600. [33] D.M. Merfeld, L. Zupan, R.J. Peterka, Humans use internal models to estimate gravity and linear acceleration, Nature 398 (1999) 615–618. [34] T. Mergner, T. Rosemeier, Interaction of vestibular, somatosensory and visual signals for postural control and motion perception under terrestrial and microgravity conditions—a conceptual model, Brain Res. Rev. 28 (1998) 118–135.

¨ [35] T. Mergner, G. Schweigart, M. Muller, F. Hlavacka, W. Becker, Visual contributions to human self-motion perception during horizontal body rotation, Arch. Ital. Biol. 138 (2000) 139–167. [36] C.M. Oman, A heuristic mathematical model for the dynamics of the sensory conflict and motion sickness, Acta Otolaryngol. 44 (1982) 1–44. [37] D.E. Parker, R.L. Poston, Tilt from head-inverted position produces displacement of visual subjective vertical in the opposite direction, Percept. Psychophys. 36 (1984) 461–465. [38] B. Pavard, A. Berthoz, Linear acceleration modifies the perceived velocity of a moving scene, Percept. Psychophys. 6 (1977) 529– 540. [39] C.S. Royden, M.S. Banks, J.A. Crowell, The perception of heading during eye movements, Nature 360 (1992) 583–585. [40] M.J. Sholl, The relation between horizontality and rod-and-frame and vestibular navigational performance, J. Exp. Psychol.: Learning, Memory, Cognition 15 (1989) 110–125. [41] I. Siegler, Idiosyncratic orientation strategies influences self-controlled whole-body rotations in the dark, Cognitive Brain Res. 134 (2000) 66–73. [42] I. Viaud-Delmon, Y.P. Ivanenko, A. Berthoz, R. Jouvent, Sex, lies and virtual reality, Nat. Neurosci. 1 (1998) 15–16. ¨ Non-specific [43] I. Viaud-Delmon, Y.P. Ivanenko, R. Grasso, I. Israel, directional adaptation to asymetrical visual-vestibular stimulation, Cognitive Brain Res. 7 (1999) 507–510. [44] W.H. Warren, B.A. Kay, W.D. Zosh, A.P. Duchon, S. Sahuc, Optic flow is used to control human walking, Nat. Neurosci. 4 (2001) 213–216. [45] K.D. Weber, W.A. Fletcher, C.R. Gordon, G. Melvill Jones, E.W. Block, Motor learning in the ‘podokinetic’ system and its role in spatial orientation during locomotion, Exp. Brain Res. 120 (1998) 377–385. [46] R.B. Welch, R.B. Post, Accuracy and adaptation of reaching and pointing in pitched visual environments, Percept. Psychophys. 58 (1996) 389. [47] R.B. Welch, D.H. Warren, Immediate perceptual response to intersensory discrepancy, Psychol. Bull. 88 (1980) 638–667.