USO0RE42289E
(19) United States (12) Reissued Patent Vincent (54)
(10) Patent Number: US (45) Date of Reissued Patent:
SPATIAL REFERENCED PHOTOGRAPHIC
5,262,867 A 5,335,072 A 5,490,075 A
SYSTEM WITH NAVIGATION ARRANGEMENT
(75) Inventor:
* 11/1993
RE42,289 E Apr. 12, 2011
Kojima ...................... ..
348/39
8/1994 Tanaka et al. 2/ 1996 Howard et a1.
(Continued)
Robert Vincent, Wake?eld, MA (US)
OTHER PUBLICATIONS
(73) Assignee: Transcenic, Inc., Lake Charles, LA (US)
Mohl, Robert, “Development and Implementation of the
(21) Appl. No.: 12/126,664 May 23, 2008 (22) Filed:
Movie Map,” Chapters 1 and 2 (pp. 1 to 86, particularly the overview description presented on p. 9 and the system con
?guration description presented on pp. 56481) of “Cognitive
Related US. Patent Documents
Reissue of:
(64)
Patent No.:
Appl. No.:
7,050,102 May 23, 2006 09/723,767
Filed:
Nov. 28, 2000
Issued:
space in the interactive movie map: an investigation of spa tial learning in virtual environments.” M.I.T. Ph. D. Thesis
(Department of Architecture, Massachusetts Institute of
Technology, Cambridge, Massachusetts, 1982)i(Down loadable from MIT Libraries DSpace Citable URI: http://
hdl.handle.net/1721.1/15702). (Continued)
US. Applications: (63)
Continuation of application No. 08/894,206, ?led as applica tion N0. PCT/US96/01434 on Jan. 31, 1996, now Pat. No.
6,195,122.
(51)
Int. Cl. H04N 5/222
Primary ExamineriDavid L OmetZ Assistant ExaminerACarramah J Quiett (74) Attorney, Agent, or FirmiFitch, Even, Tabin &
Flannery (2006.01)
(57)
ABSTRACT
H04N 5/228
(52)
US. Cl. ................. .. 348/333.02; 348/141; 348/143;
348/208.2; 348/208.14
(58)
Field of Classi?cation Search ................ .. 348/135,
348/1414143, 207.99, 207.1, 208.2, 208.12, 348/20814, 222, 3734375 See application ?le for complete search history. (56)
References Cited U.S. PATENT DOCUMENTS 4,084,184 4,169,666 4,488,249 4,561,063 4,855,822 5,075,861
A A A A A A
4/1978 Crain 10/1979 Slater et a1. 12/1984 Baker
12/1985 Craig et a1. 8/1989 Narendraetal. 12/1991 Hasson et a1.
5,124,938 A
6/1992 Algrain
5,189,402 A
2/1993 Naimark et a1.
An image system which captures, along with the images, information de?ning both the position and the orientation of the camera along with the distance to the subject. A video camera is attached to three accelerometers, two gyroscopes,
and a range?nder. Data gathered from these devices and de?ning the pitch, yaw, and roll of the camera, the camera’s acceleration, and the distance to the subject is captured and recorded along with video images. The video images are later stored within a computer’s data base along with data de?ning the position and orientation of the camera and the distance to the subject for each image, this latter data being
computed from the captured data. The images may then be presented to the user in a three-dimensional display in which
the user can navigate through the images using a joystick
device, with the images located in positions corresponding to the positions in space of the objects that were imaged.
77 Claims, 30 Drawing Sheets 10100
10188 10186
US RE42,289 E Page 2
US. PATENT DOCUMENTS 5,523,783 A 5,559,707 A
6/1996 Cho *
5,645,077 A 5,689,611 A 5,729,471 A
,
9/1996
*
5/2006 Vincent
OTHER PUBLICATIONS
DeLorme et a1. .......... .. 701/200
7/ 1997 Foxlin 11/1997 Ohta et a1.
5,768,640 A 5,793,367 A ,
7,050,102 B1
RadatZ, J.; “gimbal”, The IEEE Standard Dictionary of Elec trical and Electronic Terms; 1997, NeW York, NY; IEEE
3/1998 Jain et a1. ................. .. 725/131
Standards O?'ice, Sixth Edition, p, 454,
6/1998 TakAha§hi et a1~ 8/1998 Tagucl"
Mohl, R.; “Development and Implementation of the Movie Map,” Chapters 3A6, pp. 87*227. 1981.
i
£1.0ka 7t 31' 1v010w1tZ
Fisher, S.S. et a1.; “Virtual Environment Display System”, .
.
5,897,223 A
4/1999 Tritchew et al‘
Interact1ve 3D Graphics, Oct. 23*24, pp. 77*87, 1986.
5,913,073 A
6/1999 Kimura et a1,
Fisher, S. S.; “Viewpoint Dependent Imaging: An Interactive
5,937,096 6,037,936 6,195,122 6,292,215
8/1999 3/2000 2/2001 9/2001
Stereoscopic Display”; Department of Architecture, Oct. 8, 1981_ pp_ 1i77_
A * A B1 B1
Kawai ...................... .. 382/236 Ellenby et a1. Vincent Vincent
* cited by examiner
US. Patent
Apr. 12, 2011
1
Sheet 1 of 30
US RE42,289 E
SPATIALLY REFERENCED CAMERA
°
100
125 Audio In /
r]
120
130
°
115
Modulated
[I
Tracking
U
Data
Audio out
L'
V0
135
H /
140
CAMERA
Signal
Video out
m
D
Tracking Data
110
Acquisition Unit2 \
0111le
Modulator /
107
105 1-45
Demodulator
Modu|ated
755
tracking
150
data
Video
signal
signal 160 165
185 /
I
180
170
175
Serial data
US. Patent
Apr. 12, 2011
Fig. 3
SENSORS & TRACKING DATA AQUISITION UNIT HARDWARE BLOCK DIAGRAM
E Range?nder \I\
RANGE
_ Verlrcal
W105 470
450
M
485
400 \
US RE42,289 E
Sheet 3 0f 30
//
'
-—>
3:,er , Pitch
Gyro
VIA
e°°der
VlB
Data . . .
.._._.
‘_> Aqursrtron
VOA Roll
Tracking
D
U?it
VOB
Roll Quad
410“
D0437
Control CirCU
Decoder
Directional Gyro
YAW DOA
Yaw
(FlG.15) ” 460
Tracking
Quad Decoder r
Signal Data
.
Modulator
3’35 “\ U §'-Q-°-—-> 4 .__SQAT_A__ Accelerometer 445
Y
//
cram
CH.
Sig 1|
ND “465
H
1 15 /
Accelerometer
RSIGs Si 2
440
Modulated
2 Acce'emmeter
Thermal Correction Signals MPXR.
|:‘ ‘
Tracking Data
Video Si nal
FL
130
Video Camera
‘L 180 Vd
I
; Video
/
\
:_|r_u_.____ 175 _____ _ _
;, Serial [/1 Port 185
S_
l 80
I
'9—
150 I
145
VCR
Modulated
Tracking Demodulator
Audio in
Video out Audio out
Trackrnglea'ral lgna
170 / 155
Video in
"a
,/
Data Signal
US. Patent
Apr. 12, 2011
Sheet 5 0130
Fig. 5 )052900
TRACKING
&RAN
MICROCONTROLER TDA,
DATA AQUISITION
DATA GE
US RE42,289 E
,............... __
;
:
\T/FDAECJE'IS:
-———-'——> Emma
“- 8‘ M- PROGRAM
STORAGE
(FIG' 16) l/VIDEO 120 CAMERA
2
,l _________________________ __; i
E i TRACKING
l
; DATABASE TO ; POSITIONAL ; DATABASE
MOV'E
.
MAPPER
. CONVERSION 1 PROGRAM
PROGRAM ,
f (FIG. 9) :
i
A——-—
325 310
_
i
POSITIONAL
; ;
FRAME HETREIVAL
:
PROGRAM
:
E5 ;
(Fig. 7) /l ,v
VIDEO DATABASE
i
E
PROGRAM
4—— '
/
PC
330
_SPATIAL
335/ DATABASE *—-——————-PROGRAM _--_-------—-_T----------—-—---—----__-_-_ - -
- -
- ---,----------qp
US. Patent
Apr. 12, 2011
US RE42,289 E
Sheet 6 0f 30
Fig. 6
TRACKING DATA AQUISITION UNIT DATA FLOW DIAGRAM
CONCATENATE FRAME ID PATTERN, THREE ACCELERATION DATA PACKETS (ONE FROM EACH ORDINAL DIRECTION), AND THREE
———t_ 600
ATTITUDE DATA PACKETS (ONE REPRESENTING ROTATION ABOUT EACH OF THE THREE CARTESIAN AXIS) TO FORM A TRACKING DATA FRAME.
ATTITUDE DATA TRANSLATIONAL ACCELERATION
IN 16 BIT PACKETS FROM
500
DATA IN 24 BIT PACKETS FROM ACCELEROMETERS.
BREAK FRAME INTO NIBBLES.
520
RANGE DATA IN 32 BIT PACKET FROM RANGEFINDER.
GYROSCOPES.
505
i
605
SHIFT EACH NIBBLE
LEFT 3 BITS.
\ 610
ADD RESULTING
BYTE TO HEX so. \
l
615
MANCHESTER ENCODE RESULTING
BYIES.
\620
SERIALLY TRANSMIT ENCODED BYTES WITH
MSBs LEADING. I
O
Z?ogM gzaggi
\ 625
l
MODULATE SERIAL BIT STREAM TO RCA
\630
LINE LEVEL AUDIO SIGNAL.
lMODULATED TRACKING DATA 570
RECORD VIDEO SIGNAL AND MODULATED TRACKING DATA TO VIDEO TAPE. \ 635
US. Patent
Apr. 12, 2011
13'
Sheet 7 0130
US RE42,289 E
TRACKING AND VIDEO DATA ENTRY AND STORAGE PROGRAM DATA FLow DIAGRAM
305
MODULATED TRACKING DATA
\
FROM AUDIO OUTPUT OF VCR. 640 f """""""""""""""" ' ';
VIDEO SIGNAL FROM
155 ;
A DEMODULATE AUDIO SIGNAL
I
;
TO FORM SERIAL BIT STREAM.
i
VIDEO OUTPUT OF VCR.
*
l
I
645\ l DECODE. : _ MANCHESTER
CAPTURE VIDEO FRAMES. “‘— 680
5
f
‘
-
i
E
a : I
650\TRANSMIT SERIALLY EVERY OTHER ;
685
: \
SELECT VIDEO//‘/
NIBBLE FOLLOWING INITIAL
:
LEADING 1.
:
:
FRAMES.
I -------------------------------- -J
655
RECEIVE BIT STREAM
\ ENCODING EACH NIBBLE As 690
\
\ \ASSIGN VIDEO FRAME
AN ASCII HEX DIGIT.
I 660
NUMBER.
\\ ASSEMBLE LINES OF
F‘ TRACKING DATA. COMPRESS VIDEO FRAME &
CONCATENATE VIDEO FRAME NUMBER AND
APPEND VIDEO
>
FRAME NUMBER.
1
695
,/
CREATE DATABASE OF NUMBERED, COMPRESSED, VIDEO FRAMES &
STORE TO DISK IN DATABASE FILE: "\/|DEO DATABASE"
700
TRACKING DATA LINES.
I
\
\
665
CREATE DATABASE OF NUMBERED TRACKING DATA LINES & STORE
TO DISK IN \\,\ DATABASE PM 675 “TRACKING DATABASE.“
US. Patent
Apr. 12, 2011
Sheet 9 0130
US RE42,289 E
310
1000
INITIALIZE DISPLACEMENT / AND ORIENTATION VARIABLES
1002 /'/
(IMPLICIT IN DECLARATION).
I
/
1004
/
OPEN INPUT PILE
AND OUTPUT FILES.
READ AND PROCESS INPUT FILE. / WRITE OUTPUT FILES.
/ 1006
(FIG.10) -
1008
US. Patent
Apr. 12, 2011
Fig. 10
Sheet 10 0f 30
US RE42,289 E
m
1006
\
$
READ A DATA FRAME. <~
1130
MORE
\I
NO
YES
DATA
FIND A DATA RECORD.
FRAMES
CLOSE
?
If
FILES.
1112
@
RECORD
"14
FOUND
1120
1122
.
F Zg- 1 1 1120
\l\
4
1210\ READ RECORD. CORRECT - GYRO COUNTERS FOR
OVEHFLOW. 1212
\L
CONVERT TO FLOATING POINT FORM.
1214
\\ CALCULATE CURRENT POSITION (FIG. 12)
1 END
PROCESS DATA RECORD.
PREPEND RESIDUE TO NEXT
(FIG. 11)
FRAME.
J,
\ WRITE
POSITION RECORD. (FIG. 14)
US. Patent
Apr. 12, 2011
Sheet 11 0f 30
US RE42,289 E
Fig. 12
1310 1214
1314
SAVE YAW READING AS ZERO REFERENCE
1318 GET CURRENT
\
YAW, PITCH,
CALCULATE
AND ROLL
R0. PG)
IN REFERENCE FRAME
L '
COORDINATES. 1322 l
dx = o
dy = 0 // dz = 0
\ FORM OUATERNION
YAW, PITCH, AND ROLL REPRESENTATION. FORM INVERSE QUATERNION.
l
1324
TRANSFORM ACCELERATION \
VECTOR TO REFERENCE
COORDINATES. TRANSFORM UNIT VECTOR TO PLATFORM COORDINATES.
1326
l \ INTEGRATE TWICE;
dx = dx + ARX x = x + dx
dy = dy + ARY
y=v+dy
dz = dz + ARZ z = z + d2
1320
US. Patent
Apr. 12, 2011
Sheet 12 0f 30
US RE42,289 E
Fig. 13
3310
3314
\\
\
SAVE YAW READING AS ZERO 331a
REFERENCE
GET CURRENT YAW, PITCH, AND ROLL IN REFERENCE FRAME COORD|NATES_ K
F. CALCULATE ‘______ R0, PIP 3340
FORM QUATEFINION ’ YAW, PITCH, AND ROLL
REPRESENTATION FROM INVERSE
aagox
QUATERNION.
VELOCITY VECTOR FORWARD COMPONENT
VELOCITY VECTOR FORWARD COMPONENT
= "0"
= "WALKING
SPEED"
VELOCITY VECTOR LEFT/RIGHT _._> AND UP/DOWN COMPONENTS 3380 = "o" J
TRANSFORM VELOCITY VECTOR
3390 TO REFERENCE fOORDINATES INTEGRATE: x = x + POS —+P.L Y = Y + POS ->P.F z = z + POS -+ P.u
‘_.____
US. Patent
Apr. 12, 2011
US RE42,289 E
Sheet 13 0f 30
Fig. 14 1 122
\
1402
ANY SKIPPED FRAMES? FOR EACH SKIPPED FRAME: 01406 WRITE SAVED POSITION REOORO.
NO 1408
HAS A POSITION RECORD BEEN WRITTEN FOR THIS FRAME?
1410
“K
WRITE POSITION REOORO FOR THIS FRAME.
1412 \ SAVE THIS POSITION REOORO FOR POSSIBLE FUTURE USE.
I END
US. Patent
Apr. 12, 2011
Sheet 14 0f 30
US RE42,289 E
Fig. 15A ---99.NTBPJ-_ .Q'ELQQLT......... _ Q
\ (455
E
/
TXA1
RANGEFINDER
; MM
5 450, {400
: 455.
/
_,
""2
: 450.\L l
DIRECTIONAL GYRO
DOA. DOB;
VOA!
F’
GYRO
A0 ‘————-—*
QUAD.
JMLé?
\\
vas 5 '
.-- --------- J]?------------ “4'
:
:
GYRO
' :
GYRO
INTZCLR
QDRD1
4-— aonoz
POWER L
: ‘:
90-06
QDRDS ooam
\ 420
GYHON
L ..................................... 435
,/
SIG o
: A2 DCLK
:
i
I 445
SIG1 \
% I
Y ACCELEROMETER
TEMP 1
3 440 \n\ z ACCELEHOMETER
¢
$16 2
TEMP 2
TRACKING DATA AQUISITION UNIT 105
US. Patent
Apr. 12, 2011
Sheet 15 0f 30
US RE42,289 E
"K
Fig 153
""" "566""'"""“"'"""""""""} MODULATED
,(
; TRACKING
CKAOI ——
RTSO
_’
C.P.U.
;
DATA
SHIFT REG.
5
S'GNAL
AND
I
ENCODER
:
’
_,
‘J?lll
115
l
'2
N
_
*
i
O
& 2 c5 k
. 0 Q
/ X 1502 g
475
f 5,
:
I2 2 g
:
L
\
—-
\
—
CONTROL
—
LOG'C
130
PLAY '———
TRACKING
VIDEO
DATA
SIGNAL
:
490;
5
v.C.R.
i MODULATED y
FROM VIDEO CAMERA
RCD
:---~-;,
8 \
; : 1
Do ' DB
1506 Q:
53
120
_______ ?
SIGNAL
145
5 / OEMOOULATOR
I 155 l l
A20ROY
A29
2
—-—l
CONTROL SDATA
E
170
f
.
->
E
’
465 :
MPXR. AND '
——> _,
.._....>
f /
: :
150
TRACKING DATA
SIGNAL
‘
I
A TO O
.
CVTR
: i :
P.C. ,
185/
l
; _____ _-99_"3T59_L_9'_'39P.'T_?-_Z?___1
TRACKING DATA AQUISITION UNIT 105 I
FIG. FIG. 15A
158
US. Patent
Apr. 12, 2011
FZ'
Sheet 16 0f 30
16
g
US RE42,289 E
2406
START CPU PROGRAM
Y
2408 2402
2404
\
\
INT 1
|N\T(2
INITIALIZE SYSTEM
2050
‘L \\
GATHER DATA FROM
2062
" RANGEFINDER
/
455 1
SEND NEXT
2052
SIZE
\
GATHEH
DATA FROM
SHIFT
A CCE L E R O M E T E R S
REGISTER
1
AND
2054
ENCODEH
\ DATA PACKET
475
¢
(STEPS 600,
.,
605,610, 615
NEW
1'
PITCH,
ROLL, 2064
PREPARE
2056
\\
PLACE DATA
AND
MESSAGE BYTES
YAW
IN RAM AWAITING
COUNT VALUES
TRANSMISSION BY INT 1 PGM
FROM GYROS
l 2055
400,410
\ INITIATE lNT1
1'
PROGRAM
EXIT
MORE BYTES TO TRANSMIT ?
300
2060
US. Patent
Apr. 12, 2011
Sheet 17 0f 30
US RE42,289 E
Fig. 17 TRACKING DATA AQUISITION UNIT
185
AND MODULATOR (FIG. 15) [QQ
#— 3102 COMPUTER
TRACKING & VIDEO DATA ENTRY
& STG. PROGRAM (FIG. 7) L05
POSITIDNAL
¢
DAT/EBASE
TRACKING DATABASE
lg
3104
3-25
4'
I
POS- FRAME RETR. PGM.
TRACKING DATABASE TO POSITION DATABASE
QQQ
CONVERSION PROGRAM leQ
Fig. 18
US. Patent
‘Apr.12,2011
US RE42,289 E
Sheet180f30
Mi1_muma.“ 2R:i52c5": -
\
mwa~s2hv“
.2m"E“a.t5
xttxxxx
Ii
23m
DIE meg. was was“
\omhe
