USO0RE38884E
(19) United States (12) Reissued Patent
(10) Patent Number: US RE38,884 E (45) Date of Reissued Patent: Nov. 22, 2005
Kakii (54)
TWO-WAY INTERACTIVE SYSTEM,
5,500,671 A
*
3/1996 Andersson et al. ...... .. 348/14.1
TERMINALE
5,612,733 A
*
3/1997 Flohr .................... .. 348/14.16
UIPMENT AND IMAGE
PICKUP APPA%ATUS HAVING MECHANISM
5,666,153 A *
9/1997 Copeland ................. .. 348/141
5,734,414 A
3/1998
*
Nishimura et al. .... .. 348/1405
FBggvgggTNcfggRLLggggRglgglgoUGH
5,900,907 A *
5/1999 Malloy et................. al. ..
TRANSMISSION MEANS
5,921,523 A
7/1999 South et al. .......... .. 248/551
*
6,172,703 B1 *
(75) Inventor: Toshiaki Kakii, Yokohama (JP)
1/2001
348/141
Lee ....................... .. 348/1416
FOREIGN PATENT DOCUMENTS
(73) Assignee: Sumitomo Electric Industries, Ltd., Osaka (JP)
(21) Appl. No.: 10/278,794 (22) Filed:
Oct. 24, 2002 Related U.S. Patent Documents
JP JP JP JP JP JP
62-269128 4-81088 4-97663 4-213287 4-223688 6-85582
* 11/1987 * 3/1992 * 3/1992 * 8/1992 * 8/1992 * 12/1994
* cited by examiner
Reissue of:
(64) Patent No.:
6,137,526
Issued:
Primary Examiner—Xu Mei (74) Attorney, Agent, or Firm—McDermott Will & Emery
Oct. 24, 2000
Appl. No.:
09/121,915
Filed:
Jul. 24, 1998
LLP
(57)
ABSTRACT
U.S. Applications: (63)
(30)
Continuation of application No. 08/606,044, ?led on Feb. 16, 1996, now Pat. No. 5,815,197.
Foreign Application Priority Data (JP) ........................................... .. 7-028061
Aug. 31, 1995
(JP) ........................................... .. 7-223606
(51) (52)
Int. Cl.7 ................................................ .. H04N 7/14 U.S. Cl. .............................. .. 348/14.16; 348/1408;
(58)
Field of Search ......................... .. 348/1408, 14.16,
348/373; 396/428
348/211.12, 211.13, 373; 396/428 References Cited U.S. PATENT DOCUMENTS 4,655,567
Morley
........
A
*
4/1987
5,012,348 A
*
4/1991 WitZel et al.
5,247,330 A
*
9/1993 Ohyama et al.
..... .. 355/64
5,400,069 A
*
3/1995
.. 348/1416
5,438,357 A
*
8/1995 McNelley ................ .. 348/141
Braun et al. ...... ..
enhance communication through a transmission means When the interlocutors are located in remote areas. Terminal
equipment, located at the site of each interlocutor, includes
Feb. 16, 1995
(56)
A tWo-Way interactive system that includes an apparatus for matching the line of sight of interlocutors in order to
. . . ..
352/243
348/373
a display unit, a position regulator means and an image pickup apparatus. The display unit includes a monitor that displays an image of a partner interlocutor. The position regulating means regulates the position of the interlocutor With respect to the monitor. The image pickup apparatus includes an image pickup unit Which captures an image of the interlocutors and a support means Which sets the image
pickup unit in a predetermined position betWeen the inter locutor and a monitor of the display unit. More particularly, the image pick up unit is positioned in a line-of-sight matching area de?ned by the distance from the interlocutor to the monitor and a parallactic angle formed by the lines of sight of the interlocutor to the image of the partner inter locutor and the image pickup unit.
62 Claims, 13 Drawing Sheets
U.S. Patent
Nov. 22,2005
Sheet 1 0f 13
22A
APOINT 23A 3A
24A
25A
1!
US RE38,884 E
26A
A 5A6A 600,, D “A
A
IA
2IA
32
\
40A
4A
LVZA
/ 3! g
Lr:
:11
LA
‘
2?
I TRANSMISSION
_
_
_
MEANS
208 28
24B
,/ 7
BPOINT
23B
25B 58 6B 39
600B
\ 40B
4 3:8
"28 L
L 26B
"’
U.S. Patent
Nov. 22,2005
Sheet 2 0f 13
US RE38,884 E
K:
R
j
U.S. Patent
Nov. 22, 2005
Sheet 3 0f 13
US RE38,884 E
O
O O
"'
O
A
Q
|
o
O
9.
0
2 9
III“
lOlb
3Fig.
I030
U.S. Patent
Nov. 22,2005
Sheet 4 0f 13
Fig.4
Fig.6
US RE38,884 E
U.S. Patent
Nov. 22,2005
Sheet 5 0f 13
Fig.7
A
US RE38,884 E
U.S. Patent
Nov. 22,2005
Sheet 6 0f 13
US RE38,884 E
F i g . \O 260:: $ \ ‘
5d
\
\
\
~
,1” 4- "'
/
r"
/
g’ 200
S
27
S ‘MAGE PICK-UP DATA TRANSMISS‘ON PROCESSING UNIT 4* MEANS PROCESSING
UNIT
2IO"“’
’
60
U.S. Patent
Nov. 22,2005
Sheet 7 0f 13
US RE38,884 E
U.S. Patent
Nov. 22,2005
Sheet 8 0f 13
US RE38,884 E
Fig.13
@
PICK UP IMAGE OF INTERLOCUTOR WITH CCD CAMERA
"\P STI
I
DISPLAY PICK~UP IMAGE OF INTERLOCUTOR ON MONITOR
~ 8T2
I
MOVE CCD CAMERA UPWARD BY
Ah(MOVEMENT OF CCD CAMERA)
N 8T3
I
MOVE IMAGE PICK—UP DIRECTION OF CCD CAMERA DOWNWARD BY A 0
(ANGLE ADJUSTMENT FOR CCD CAMERA)
@@ Fig.14
@ SPECIFY REFERENCE POINT FROM RECEIVED IMAGE OF PARTNER INTERLOCUTOR
NO SHIFT
“5T5
IS SHIFT BETWEE REFERENCE POINT OF MONITOR IMAGE AND ACTUAL POSITION OF CCD CAMERA ON MONITOR CALCULATED?
ST6
SHIFT IS PRESENT ,_/ ST7
CALCULATE VECTOR STARTING FROM COORDINATES OF REFERENCE POINT ON MONITOR AND TERMINATING AT ACTUAL POSITION OF CCD CAMERA ON MONITOR I
TRANSLATE RECEIVED AND DISPLAYED MONITOR w ST8 IMAGE ON THE BASIS OF OBTAINED VECTOR
@
I
U.S. Patent
Nov. 22,2005
Sheet 9 0f 13
Fig.l5
US RE38,884 E
U.S. Patent
Nov. 22,2005
Sheet 10 0f 13
US RE38,884 E
Fig.l7 IO
5'
600
6
f/
26
(/
6H
6m
PROCESSING UNIT IMAGE PICKUP DATA
PROCESSIN§ UNIT 1
E1) -
;F '
' IMAGE
INPUT OUTPUT
INPUT OUTPUT
l_JT
242 1 1 /
'
H T-I
“
U.S. Patent
Nov. 22,2005
Sheet 11 0f 13
US RE38,884 E
F|g.19 .§0 5
{TI-:11‘: '''''' ‘WROTE'SWGWT—'_]
|.
coo CAMERA
.
nmvme
om PROCESSING
um ‘
l
-
.
!
CONTROL 1300a IMAGE}
MECHAN'SM i! mlsoob
I
AND-SHAPED 6000 .{ PROCESSINGUNIT j RECORDING
\289
MEDIUM
| ''''''' —'
i_-
l READQNIT P i262
274 \_ PERSON \_
B00583 GUN
. ‘DATAPROCESSING I UN” ‘
I DRMNG CONTROL
MECHANIiM 280,? um!
INTEHACHVE
PQPEEATQSJ IT
' VOICE mcousnc
,
288
'
/
l 31
263 275 GKTITHENTICAUON __PT/m$
271
.
I
RECEPTION \"r
SYSTEM
BACKUPSYSTEM BACKUP
0B
264
23
282
283
I MEANS
SWlT/CHING]
SYSTEM 32i
276
277
287
EXTERNAL
PH1NTE
270
#33?“ ROLUN'T
THEUKE
269
I
l
PROCESSING
' 267
i
SYSITEM
!|
-'
'- CHARACTER/GRAPHIC INPUTCONTROLUNIT
KEYBOARD
g POINTING DEVICE PEN TOUCH PANEL
'
I -
lwé?wg? __ INTEGRATED ‘
272
MISSION
RADIO
1b 1
i Tm
(DETECTION!
/
,
K27
MISSION!
NFORMAT'ON
9'5
278
i i
MAIN CONTROL UNIT \ 273 266
i !
..... ___EX_-2_8§_______ _._K.____ __l
'\ 26
U.S. Patent
Nov. 22,2005
Sheet 12 0f 13
Fig.2O 23
US RE38,884 E
U.S. Patent
Nov. 22,2005
Sheet 13 0f 13
US RE38,884 E
A B C ...... ..
E FG...._....
ABC
US RE38,884 E 1
2
TWO-WAY INTERACTIVE SYSTEM, TERMINAL EQUIPMENT AND IMAGE
image pickup apparatus arranged behind the display appa
PICKUP APPARATUS HAVING MECHANISM FOR MATCHING LINES OF SIGHT BETWEEN INTERLOCUTORS THROUGH TRANSMISSION MEANS
display apparatus at Which the interlocutor looks, so that the lines of sight betWeen interlocutors can be matched each other.
ratus at a position offset from the display screen of the
A still another image pickup apparatus is disclosed in
Japanese Patent Laid-Open No. 4-97633 (third prior art) in Matter enclosed in heavy brackets [ ] appears in the original patent but forms no part of this reissue speci? cation; matter printed in italics indicates the additions made by reissue.
Which a small-size mirror or a small-size half mirror is 10
This is a continuation of application Ser. No. 08/606,044, ?led Feb. 16, 1996 now US. Pat. No. 5,815,197. BACKGROUND OF THE INVENTION
arrangement, the image of the interlocutor, Which is re?ected by the mirror, is picked up by the small-size camera arranged at a position offset from the display screen of a display 15
1. Field of the Invention The present invention relates to a tWo-Way interactive
of-sight matching in the vertical direction, so that the lines of sight betWeen interlocutors can be matched each other.
betWeen interlocutors through a transmission means, and a 20
SUMMARY OF THE INVENTION
Upon examination of the above prior arts, the present inventor found the folloWing problems.
betWeen, e.g., a doctor and a patient or a counselor and a client in remote areas through a transmission means and is
mainly oriented to a one-to-one tWo-Way interactive
operation, and an apparatus constituting the system. 2. Related Background Art Conventionally, counseling in the ?eld of education, medical service, psychology, or the like is generally per formed betWeen interlocutors directly facing each other While establishing an intimate reliability relationship
25
30
In the ?rst prior art in Which light is separated from an object by a half mirror, a half mirror is almost the same size as that of the screen of the display apparatus is needed. Inevitably, a large space is required to accommodate such a large mirror. Particularly, it is dif?cult to manufacture an apparatus With a screen size of 40 inches or more. In
addition, since a half mirror is used, an image picked up by
the image pickup apparatus arranged at a position offset
betWeen them. In some cases, hoWever a person Who needs
counseling and a person (counselor) Who has mastered a
counseling technique can hardly share place and time
apparatus at Which the interlocutor looks. Therefore, the angle of elevation of the line of sight of the interlocutor is suppressed to 10° or less, Which is an alloWance for line
system having a mechanism for matching lines of sight
terminal equipment and an image pickup apparatus consti tuting the system and, more particularly, to a counseling system Which realizes a tWo-Way interactive operation
attached to the upper portion of a display apparatus, and the image of an interlocutor is picked up by a small-size camera With an optical axis directed to the mirror. According to this
35
from the screen Where an interlocutor’s image is displayed is darkened, so no ideal counseling environment can be realized.
because a natural disaster has occurred, or they are in remote areas. Along With a progress in communications technology,
The second prior art using a special display apparatus for selecting re?ection/transmission of light cannot be incorpo
a system for realizing counseling betWeen persons in remote areas is strongly demanded. In construction of such a counseling system by using a
rated in a commercially available mass-produced TV set. For this reason, to realize a system using this principle, a dedicated machine must be developed, resulting in an
40
increase in cost of the apparatus. The image display appa ratus for picking up the image of an interlocutor is set to pick
communication technology, to ensure an intimate relation
ship betWeen interlocutors and an ideal counseling environment, as in the conventional face-to-face system, it is important to match the lines of sight betWeen the interlocu tors.
up the image of the interlocutor through the special display apparatus. Therefore, the picked-up image of the interlocu 45
lines of sight betWeen interlocutors through a transmission line is disclosed in Japanese Patent Laid-Open No. 62-269128 (?rst prior art). In the ?rst prior art, a half mirror is provided betWeen an interlocutor and the image pickup
and a small-size camera, the camera is arranged at a position offset from the display screen at Which an interlocutor looks, 50
and the image of the interlocutor is picked up through the small-size re?ecting mirror. For this reason, the picked-up person’s image is darkened. To obtain a large ?eld angle, the re?ecting mirror must have a size larger than that of the small-size camera, and person’s image display on the dis
55
play apparatus is interfered With this large re?ecting mirror.
apparatus, so that light from the interlocutor as an object is
divided toWard the image pickup apparatus and toWard an
image display apparatus. Therefore, according to the arrangement of the ?rst prior art, the image pickup apparatus is arranged at a position offset from the display screen of the
If the position of the person to be picked up changes, the
image display apparatus at Which the interlocutor looks, thereby matching the optical axis of light incidental in the image pickup apparatus With the line of sight of the inter
small-size camera cannot folloW up the movement because
the image pickup direction cannot be changed. Therefore,
locutor.
Another image pickup apparatus is disclosed in Japanese Patent Laid-Open No. 4-213287 (second prior art) in Which a special display apparatus capable of selecting re?ection/ transmission of light is used, and an image pickup apparatus is arranged behind the display apparatus (on the opposite side of an interlocutor through the display apparatus). According to this arrangement, the line of sight of the interlocutor is directed to the display apparatus, i.e., the
tor is darkened, so no ideal counseling environment can be
realized. In the third prior art Which combines a re?ecting mirror
A conventional image pickup apparatus for matching the
60
matching of the lines of sight betWeen interlocutors cannot alWays be ensured. The characteristics of the above-mentioned prior arts are
structurally common in that the image pickup apparatus (including a camera) is arranged at a position offset from a space betWeen the display screen at Which an interlocutor 65
looks (screen on Which the image of a partner interlocutor is displayed) and the interlocutor. In other Words, to match the lines of sight betWeen interlocutors, a re?ecting mirror, a
US RE38,884 E 3
4
half mirror, or a special display apparatus must be set
In the above tWo-Way interactive system, the parallactic angle of matching the line of sight of the interlocutor (?rst
between an interlocutor and the image pickup apparatus. Therefore, the transmitted image of the partner is darkened,
interlocutor) With the line of sight of the image (monitor image) of the person (second interlocutor) displayed on the
so no ideal counseling environment can be realiZed.
monitor is preferably 45° or less in the horiZontal direction of the monitor screen, 12° or less in the upper direction, and 8° or less in the loWer direction When vieWed from the
It is an object of the present invention to provide a
tWo-Way interactive system and an apparatus constituting the system Which has a mechanism for matching lines of sight betWeen interlocutors through a transmission means to minimiZe the sense of incompatibility during an interactive operation betWeen interlocutors in remote areas.
interlocutor. To realiZe an ideal line-of-sight matching state, the parallactic angle is more preferably 3° or less in the 10
Particularly, the tWo-Way interactive system according to the present invention has as its object to create an organic
relationship betWeen interlocutors (realiZe close communication) and provide an ideal counseling environ
ment by integrally processing voice information (including a background music (BGM), character/graphic information,
15
and the ?rst interlocutor falls Within the range of the
The line-of-sight matching area Where the image pickup unit is set is preferably at a position higher than the line of sight of the-interlocutor to be picked up. Particularly, it Was con?rmed as a result of an experiment by the present inventor that the image pickup unit Which shielded the
equipment located at a ?rst point Where a ?rst interlocutor is present, a second terminal equipment located at a second
point Where a second interlocutor is present, and transmis sion means, arranged betWeen the ?rst terminal equipment
monitor from the interlocutor Was preferably set near the
parietal portion of the person’s image displayed on the
and the second terminal equipment, for enabling tWo-Way 25
monitor When vieWed from the interlocutor. To set the image pickup unit in an area (line-of-sight
matching area) de?ned by the distance betWeen the inter locutor and the monitor, Which is speci?ed by the position regulation means, and the predetermined parallactic angle for determining that the line of sight of the interlocutor matches the line of sight of the person’s image displayed on the monitor, a ?rst means for moving the image pickup unit
?rst interlocutor to de?ne at least a distance betWeen the ?rst
interlocutor and the display means, and an image pickup apparatus for transmitting the picked-up image of the ?rst interlocutor to the second terminal equipment through the transmission means.
passing through the photoelectric conversion surface 208
parallactic angle.
and the like, in-addition to the images of the interlocutors. According to the present invention, there is provided a tWo-Way interactive system comprising a ?rst terminal
communication of at least image/voice data of the ?rst interlocutor and the second interlocutor. In this system, at least the ?rst terminal equipment comprises a display unit for displaying an image of the second interlocutor, Which is received through the transmission means, on a monitor, position regulation means for regulating a position of the
horiZontal and vertical directions of the monitor screen With
respect to the line of sight of the interlocutor When vieWed from the interlocutor. Particularly, as shoWn in FIG. 8, an angle formed by a line perpendicular to a photoelectric conversion surface 208 of an image pickup device 205 and
itself to the area Where the lines of sight betWeen the interlocutor and the person’s image on the monitor match, a 35
second means for moving the interlocutor opposing the
The image pickup apparatus has at least an image pickup
monitor to move the area Where the lines of sight betWeen
unit for picking up the image of the ?rst interlocutor and a support mechanism for setting the image pickup unit at a
match, thereby relatively changing the positions of the
predetermined position betWeen the ?rst interlocutor and the monitor While the image pickup direction of the image pickup unit is set directly toWard the ?rst interlocutor. Particularly, the image pickup unit is set in a line-of-sight matching area de?ned by (1) a distance betWeen the monitor and the ?rst interlocutor, Which is regulated by the position regulation means, and (2) a parallactic angle for determining
the interlocutor and the person’s image on the monitor
40
45
sight of the monitor image of the second interlocutor, so that the monitor image of the second interlocutor is partially
The ?rst means can be realiZed by providing a processing
unit for recogniZing the position of the person’s image on the monitor, Which is displayed on the monitor, in the terminal equipment, and providing a driving mechanism for rela tively moving the set position of the image pickup unit With
shielded With respect to the ?rst interlocutor. The second terminal equipment may have the same arrangement. The ?rst terminal equipment can remote-control the second terminal equipment. Each terminal equipment has a sound
respect to the person’s image Whose position is recogniZed by the processing unit in the image pickup apparatus. As an
absorbing member arranged around the interlocutor and having external sound insulating properties. Therefore, men
embodiment, the ?rst means is provided in each terminal 55
concentration is enabled to alloW a close interactive opera
tion.
the second terminal equipment, thereby remote-controlling
As described above, the image pickup unit is set in the line-of-sight matching area betWeen the interlocutor and the monitor, so that the monitor image is partially shielded. This
the second terminal equipment from the ?rst terminal equip ment.
The second means can be realiZed by providing a pro
to because the present inventor con?rmed on the basis of an
cessing unit for recogniZing the position of the person’s
experiment that, even When the image pickup unit Was set to
partially shield the person’s image displayed on the monitor, a tWo-Way interactive operation as far as the siZe of the
image pickup unit Was beloW a predetermined value.
equipment. Alternatively, position recognition of the moni tor image of the second interlocutor is performed by the processing unit of the ?rst terminal equipment, and the processing unit transmits a predetermined control signal to
decreased, and the sound quality is improved, so that mind
the presence of the image pickup unit hardly in?uenced on
area Where the person’s image is displayed on a screen split
by a display control unit (to be described later) can be used.
that a line of sight of the ?rst interlocutor matches a line of
tal obstacles to leakage of interactive operation contents are
image pickup unit and the line-of-sight matching area, or a third means for moving the person’s image on the monitor and similarly moving the area Where the lines of sight betWeen the interlocutor opposing the monitor and the person’s image on the monitor (the third means can be effectively used as an algorithm for moving an image display
65
image on the monitor, Which is displayed on the monitor, in the terminal equipment, and providing a mechanism for adjusting the line of sight of the interlocutor to be picked up With respect to the line of sight of the person’s image Whose
US RE38,884 E 5
6
position is recognized by the processing unit in the position
output from the image pickup unit ?uctuates. When the
regulation means. As an embodiment, the ?rst means is
moving speed is limited, and the frequency of the moving
provided in each terminal equipment. Alternatively, position
operation is minimiZed, ?uctuation for causing a sense of incompatibility can be prevented.
recognition of the monitor image of the second interlocutor is performed by the processing unit of the ?rst terminal equipment, and the processing unit transmits a predeter mined control signal to the second terminal equipment,
For the driving mechanism for moving the image pickup unit, a member positioned betWeen the interlocutor and the
display unit is preferably partially formed of a transparent
thereby remote-controlling the second terminal equipment
material.
from the ?rst terminal equipment. To realiZe the third means, the terminal equipment com
The position regulation means for regulating the position 10
prises a processing unit for adjusting the display position of the person’s image displayed on the monitor. The processing
able (a chair having a sitting height adjustment mechanism) separated from the display unit by a predetermined distance. A table provided betWeen the interlocutor and the display
unit speci?es a reference point at a virtual position on the
monitor image Where the image pickup unit should be originally present, and subsequently, detects a shift of the person’s image displayed on the monitor on the basis of
15
coordinates on the monitor corresponding to the actual set
position of the image pickup unit and coordinates of the obtained reference point on the monitor. If it is determined
that the person’s image shifts from the display position on the monitor Where the person’s image should be originally
unit also function as the position regulation means. The system uses the position regulation means to separate the interlocutor to be picked up from the display unit by a distance of 0.5 m to 5 m. This is because, since the movement of the interlocutor in the left and right directions and in the back-and-fourth directions can be suf?ciently
limited by the position regulation means, moving adjustment in the vertical direction, Which is caused by a physical
displayed, a vector starting from the coordinates of the reference point on the monitor and terminating at the coor dinates on the monitor corresponding to the set position of
the image pickup unit is calculated, and the shifting person’s
of the interlocutor includes a chair Whose height is adjust
difference betWeen interlocutors, is most effective to match 25
image is translated on the monitor. As an embodiment, this
processing is performed for the monitor image of the inter locutor (initial setting before the start of an interactive operation) or for the monitor image of the second interlocu
the lines of sight betWeen interlocutors in this system. The image pickup unit adequate to this system may be arranged independently of the display unit, as shoWn in, e. g., FIG. 2. Alternatively, the image pickup unit may be attached to part of the display unit through the support mechanism, as shoWn in FIG. 7.
tor. Particularly, When the display siZe of the person’s image
The tWo-Way interactive system according to the present
(the monitor image of the ?rst interlocutor or the second interlocutor) is arbitrarily changed in a screen division
invention is oriented to a counseling system betWeen inter locutors in remote areas. Therefore, to acquire useful coun seling data or realiZe an ideal counseling environment, this
operation (including enlargement/reduction of the person’s image to be displayed) performed by a display control unit (to be described later), the area for displaying the image can
system has various arrangements and functions. 35
to best at an arbitrary position on the monitor screen.
Therefore, to match the lines of sight betWeen the interlocu tor and the person’s image in a changed siZe, the third means can be particularly effectively used.
The display positions of both the eyes of the person’s
40
image on the monitor, Which is received and displayed on
the monitor, are detected, and a position separated upWard from the center betWeen the eyes by a predetermined dis tance (e.g., about 10 cm) on the monitor is de?ned as the
reference point. Alternatively, the person’s image received
For eXample, the display control unit for performing display control of the display unit has a functional of enlarging or reducing an image to be displayed on the display unit. The display siZe largely in?uences on the mentality of an interlocutor. By freely setting the siZe of the image display area on the monitor by the display control unit, the degree of freedom of an interactive operation increases, and a comfortable interactive operation environ ment can be realiZed. The display position of a reduced
person’s image (monitor image) on the monitor is adjusted 45
by the above-mentioned third means for matching the lines
and displayed on the monitor may be binariZed to detect the
of sight. Therefore, the line-of-sight matching siZe betWeen
contour of the person’s image, and the references point may
the ?rst interlocutor and the second interlocutor is main tained.
be speci?ed at a position on the monitor corresponding to the
parietal portion of the person’s image.
Transmission information-may be character/graphic
The driving mechanism for moving the image pickup unit may adjust the angle of the image pickup direction of the
information in addition to image/voice information of an interlocutor. With this arrangement, the display unit can also
image pickup unit on the basis of the position of the
display characters or graphic together With a person’s image
interlocutor, Which is temporarily recogniZed by the pro cessing unit, and the position of the image pickup unit. When the angle adjustment for the image pickup unit is
through the transmission means. That is, by displaying not
on the basis of character or graphic information received 55
only image/voice information but also character/graphic
performed by the driving mechanism, the image pickup
information, an interactive operation becomes smoother.
direction of the image pickup unit is automatically set toWard the line of sight of the interlocutor Whose position is
To display character/graphic information, the characters or graphics may overlap a displayed person’s image. The
recogniZed by the processing unit. The driving mechanism
monitor screen may be divided into an area for displaying characters or graphics and an area for displaying a person’s
for moving the image pickup unit may also have an arrange ment such that average matching of the directions of lines of
image. As a preferably form of screen division, (1) the monitor screen is divided along the horiZontal direction, and a person’s image is displayed on the upper side, or (2) the
sight is detected, and if the moving speed in moving and aligning operations is a predetermined value or less, and the ?uctuation in the direction of line of sight is Within a
predetermined range, the aligning operation is not per
65
monitor screen is divided along the vertical direction, and a person’s image is displayed on one side or at the central
formed (determination is performed by the processing unit
portion. The present inventor con?rmed that matching of the
or the like). When the image pickup unit is moved, an image
lines of sight betWeen an interlocutor and a person’s image
US RE38,884 E 8
7 As an
talk. In addition, When the output volume of the interlocutor
application of this form, a plurality of display units including
is displayed on the monitor screen to cause the interlocutor
a ?rst monitor, for displaying a person’s image and a second
to recogniZe the volume, the volume can be fed back to the interlocutor, so that the interlocutor can determine Whether
did not degrade in both the forms (1) and
monitor for displaying character/graphic information may
the output volume is appropriate for transmission. A voice/acoustic output control unit for controlling the
be provided. This system also has a character/graphic input device as a man-machine interface. With this arrangement, the display
unit can display character/graphic information input from the character/graphic input device through a character/ graphic input control unit. In addition, character information
voice output unit has a function of outputting a background
music (BGM) and detecting the interactive operation vol 10
or graphic information received in this manner can be output
to the terminal equipment on the partner side. The character/
graphic input device may be provided to only one of the terminal equipment at the ?rst and second points Where interlocutors are present.
15
ume of the voice of an interlocutor and adjust the volume of
As described above, When this system has a character/
the background music not to impeded listening to the voice of the interlocutor, thereby outputting a background music overlapping the voice of the partner interlocutor from the
graphic input device, an interactive operation using charac ters or graphics is enabled to make the interactive operation
smoother. Such a character/graphic input device, a keyboard device, a pen input device (touch panel) arranged near at hand of an interlocutor can be employed. When both key board device and the pen input device are provided and appropriate selected, a further smoother interactive opera tion can be realiZed. The character/graphic input device may have a pointing device for inputting a pointer position for indicating a displayed character or graphic. An object for an interactive operation at each-point of time can be speci?ed in accordance With position indication by a pointer. Use of
voice output unit to relaX the interlocutor. Particularly, When silence continues, the tension of the interlocutor can be eased or the interlocutor can be calmed by starting a back
ground music or changing the volume of the background music. 25
the character/graphic input device to recogniZe an input
a magnetic-optical disk can be used. Information during an
character information, and display the character information
interactive operation, and particularly, character or graphic
is effective to establish a smoother interactive operation (or
information used during an interactive operation can be stored.
counseling). The tWo-Way interactive system has speakers arranged
In the tWo-Way interactive system, each terminal equip ment has a printer for printing the display screen of the monitor, and a storage means for storing information on the display screen. The storage means is a card-shaped record ing medium having at least an IC (integrated circuit), and an optical memory card, an IC card, an optical memory disk,-or
voice of an interlocutor, convert the voice information into
near ears aside an interlocutor, Whispering into ears
ume to adjust the output volume of the background music. The voice/acoustic output control unit detects that the inter active operation volume is substantially Zero for a predeter mined time, and, on the basis of the detection result, at least starts the background music or performs adjustment to increase or decrease the volume of the background music. The voice/acoustic output control unit may detect the vol
35
improves the intimacy. A headphone is typically used as
To maintain the security of this system, each image pickup apparatus preferably has a person authentication unit for performing person authentication of an interlocutor. When an interactive operation is started after person
such a voice output unit.
This system has a directional high-sensitive microphone
authentication, the security of this system can be suf?ciently
as a voice input unit. A microphone used on a table or 40 maintained. The person authentication unit may use a dedi
clipped at the breast can also be employed. HoWever, a
cated read device to read authentication data from a record
microphone on a table may cause conscious obstacles for an
ing medium in Which the authentication data is Written, and
interactive operation. For a microphone at the breast, if the interlocutor carelessly leaves the seat, the cord may be
perform person authentication on the basis of the read result.
damaged. Therefore, it is preferable that a directional high
The recording medium includes a card-shaped recording 45
medium having at least an IC, and an optical memory card,
sensitive microphone of Which the interlocutor is uncon scious be set above the display unit or incorporated in the
an IC card, or a magnetic card can be used. The person
display unit. Particularly, since the line of sight is already recogniZed, the directionality can be improved by setting the microphone toWard the mouth.
on the basis of the analysis result of the face of an
Avoice/ acoustic data processing unit for receiving a voice signal from the voice input unit has a function of amplifying and adjusting voice input to control an input volume. The
voice/ acoustic data processing unit automatically adjusts the output volume of an input voice signal to a volume at a 55
predetermined level. The voice/acoustic data processing unit may detect the average level of the input volume levels for
a predetermined time, and perform automatic volume adjust ment When the difference betWeen the input volume and the average level eXceeds a predetermined value.
At this time, the display unit displays the output volume in the voice/acoustic data processing unit (the volume of the interlocutor, Which is output to the partner side). An uneX perienced person cannot determine the voice volume When
authentication unit may (1) perform person authentication interlocutor, (2) perform person authentication on the basis of the voice analysis result of the voice of an interlocutor, (3) perform person authentication on the basis of the character
analysis result of the signature of an interlocutor, or (4) perform person authentication on the basis of the text analysis result of the message teXt of an interlocutor. With this person authentication, the interlocutor can be speci?ed. The authentication method is preferably selected in accor dance With con?dential contents. The teXt analysis of the message teXt means collation analysis betWeen the database or cipher database of a designated person and characters for, e.g., X and Y in a teXt “X is Y.” The card-shaped external recording medium such as an IC card may also be used to reserve an interactive operation,
authenticate the designated partner interlocutor, or authen
ticate and control interactive operation conditions, including
the person talks to another person on the screen Without 65 a reserved interactive operation start time and an interactive
having a microphone in sight. Therefore, a smoother inter active operation can be realiZed by adjusting the volume of
operation time set in advance. The recording medium can also be used as an electronic fee settlement means. By using
US RE38,884 E 9
10
the recording medium to record the summary of an interac
terms used by an interlocutor, or a predetermined term in a
tive operation of pending questions, these pieces of infor
voice according to transmitted voice information. The ter
mation can be conveniently used in neXt counseling or communication. When data such as the interactive operation start time or a partner for an interactive operation (an
minal equipment may also have a position change analysis means for analyZing the moving track, the moving change
interlocutor on the partner side) is input in advance to the recording medium such as an IC card, the partner-side
head, the upper half of the body, or a hand of an interlocutor
amount, the moving speed, or the moving acceleration of the
or transmitted person’s image (this analysis is performed by the processing unit or the like). In counseling or the like, When interactive operation situation analysis such as interactive operation time analysis, voice spectrum analysis, predetermined term analysis, or
terminal equipment can be searched on a netWork such as a
BBS on the basis of the data. On the basis of the search
result, terminal equipments at ?rst and second speci?c points can be lined-connected independently of the present positions of the ?rst interlocutor and the second interlocutor,
body action analysis is performed, the interactive operation
so that an interactive operation environment betWeen the ?rst and second interlocutors can be formed. When the
name, face (image data), and interactive operation items
15
situation can be quanti?ed. In the transmission means of the tWo-Way interactive system, a plurality of independent transmission lines are
(theme of an interactive operation) of an interlocutor on the
ensured betWeen the ?rst point and the remote second point.
partner side are input to the recording medium (e.g., on IC card), and these pieces of information are displayed for the
At least one of the plurality of transmission lines is a voice
interlocutor on the monitor for predetermined period (e.g., about 30 seconds) before an interactive operation is started,
transmission means includes a netWork such as a conven
transmission line for transmitting voice information. The
tional telephone netWork, and also includes a BBS (Bulletin
Board System) having various databases. This, system also
the interlocutor can mentally prepare for the interactive operation, so the interlocutor can easily shift to the theme of the interactive operation. At this time, if a BGM or voice is output in an interlocked manner, a more comfortable inter
active operation environment can be provided or the inter locutor. Such monitor display can also be effectively per formed during and/or after the interactive operation as
25
needed. Particularly, When monitor display is performed after the interactive operation is completed, the interlocutor
cause for a line abnormally is an abnormally in a coaXial cable, an optical ?ber cable, or a transmission unit.
can smoothly progress settlement of the interactive opera
Therefore, another transmission line is preferably a radio
tion and the like (the time of monitor display after comple tion of the interactive operation corresponds is a mental preparation time from completion of the interactive opera tion to the neXt operation for the interlocutor). The table serving as the position regulation means is set betWeen the interlocutor and the display unit While one end
system. BRIEF DESCRIPTION OF THE DRAWINGS 35
opposes the interlocutor, and the other end substantially contacts the display unit. The table may be set such as (1) When the interlocutor is sitting on the chair, the upper half of the body is above the table, or (2) When the interlocutor is standing, the upper half of the body is above the table. With this arrangement, since a sense of continuity is gen
FIG. 1 is a logical diagram shoWing the entire tWo-Way interactive system according to the present invention; FIG. 2 is a vieW shoWing an image pickup apparatus in FIG. 1 so as to eXplain a method of setting a CCD camera
(image pickup unit) at a predetermined position; FIG. 3 is a vieW shoWing the ?rst embodiment of a 40
support mechanism of the image pickup apparatus in FIG. 1, in Which a structure for selling the CCD camera at the
predetermined position is shoWing;
erated through the table, a feeling of uneasiness caused When the relative distance to the partner (mental distance) is lost can be reduced. In this case, the display unit and the table
has a backup system for detecting a failure in transmission line in use for an interactive operation and sWitching the transmission line to be used. Therefore, if a transmission line fails during an interactive operation, another transmission line can be used to continue the interactive operation. In this case, another transmission line preferably ensures at least transmission of voice information. Generally, a possible
FIG. 4 is a vieW shoWing the second embodiment of the 45
preferably contact each other. HoWever, a mentally negli gible gap (e.g., several tens cm or less) may be present. In this tWo-Way interactive system, the terminal equip
support mechanism of the image pickup apparatus in FIG. 1, in Which a structure for setting the CCD camera at the
predetermined position is shoWn; FIG. 5 is a vieW shoWing the structure of a camera moving
mechanism of the support mechanism shoWn in FIG. 4
ment has an interactive operation processing unit for per
forming statistical analysis processing of the interactive
(second embodiment), Which mechanism particularly moves
operation time of the interlocutor. The interactive operation processing unit may count the local interactive operation time, the average interactive operation time, and the number of interactive operations, and display the total interactive operation time, the average interactive operation time, the number of interactive operations, and the distribution of the number of interactive operations on the display unit. The voice/acoustic data processing unit performs statistical analysis processing of the voice spectrum of an interlocutor,
the CCD camera in the horiZontal and vertical directions; FIG. 6 is a vieW shoWing the structure of an angle
adjustment mechanism of the support mechanism shoWn in
FIG. 4 (second embodiment), Which mechanism particularly 55
FIG. 7 is a vieW shoWing the second embodiment of a
terminal equipment in the tWo-Way interactive system Whose
support mechanism (third embodiment) for setting the CCD camera at a predetermined position has a structure Wherein the CCD camera is attached to a display unit; FIG. 8 is a sectional vieW shoWing the CCD camera
of a voice spectrum according to transmitted voice infor mation. This statistical analysis processing means that the
attachment structure of the support mechanism shoWn in FIG. 7 (third embodiment), taken along a line X—X in FIG.
voice spectrum of an interlocutor, or a voice spectrum
according to transmitted voice information is analyZed With
7;
time to detect an average spectrum and a voice spectrum
changed beyond a predetermined value. Each terminal equipment of this system has an interactive
operation processing unit for detecting a predetermined
changes the image pickup direction of the CCD camera;
65
FIG. 9 is a perspective vieW shoWing the steps of assem
bling the support mechanism shoWn in FIG. 7 (third
embodiment);
US RE38,884 E 11
12
FIG. 10 is a vieW showing a structure for adjusting the position and image pickup direction of the CCD camera as an application of the support mechanism shoWn in FIG. 7
client as a counselee) sits, (b) a table 2B at Which the
interlocutor 3B sits, and (c) a terminal equipment 20B having a mechanism for matching the lines of sight betWeen
(third embodiment).
the interlocutors through the transmission means 27 are set. The transmission means 27 includes an optical ?ber trans
FIG. 11 is a vieW for explaining a parallactic angle;
mission line 31 (main transmission line) Which enables mass
FIG. 12 is a vieW shoWing an area Where the CCD camera
transmission and a satellite communication transmission
should be set; FIG. 13 is a How chart for explaining the initial setting operation of the CCD camera; FIG. 14 is a How chart for explaining an operation for moving an image displayed on a monitor such that the lines
line 32 (backup transmission line), so that transmission/ reception of image information and voice information betWeen the terminal equipment 20A at the point A and the terminal equipment 20B at the point B is enabled. The
10
transmission means 27 can be either a Wired system or a
radio system. In addition, the transmission means includes a netWork such as a conventional telephone netWork, and also
of sight betWeen interlocutors match each other; FIG. 15 is a vieW for explaining the initial setting opera tion of the CCD camera, Which is explained by the How chart shoWn in FIG. 13; FIG. 16 is a vieW for explaining the operation for moving
15
At this point A, the chain 4A functions to regulate the position of the interlocutor 3A. The chair 4A is preferably
the image displayed on the monitor, Which is explained by
?xed to maintain a predetermined distance to a display unit
the How chart shoWn in FIG. 13; FIG. 17 is a vieW shoWing an application of a single image
21A of the terminal equipment 20A. HoWever, even When the chair 4A is not ?xed, a predetermined distance can be maintained betWeen the chair 4A and the display unit 21A by setting the table 2A betWeen the interlocutor 3A and the
pickup unit (image pickup apparatus) in FIG. 1; FIG. 18 is a vieW shoWing the structure of the rear surface
of the image pickup unit shoWn in FIG. 17; FIG. 19 is a logical block diagram shoWing the entire arrangement of the terminal equipment shoWn in FIG. 1; FIG. 20 is a perspective vieW shoWing the arrangement of
display unit 21A (a fact Was con?rmed as a result of an 25
interlocutor 3A). The terminal equipment 20B at the point B Where the partner interlocutor 3B (e.g., a counselee) is present has the same arrangement as described above.
FIG. 21 is a vieW shoWing a screen con?guration of a
At the point A, the terminal equipment 20A has (1) the
display screen displayed on the monitor in FIG. 1
display unit 21A having a monitor TV 1A and arranged on the table 2A to display the image of the interlocutor 3B on the basis of image information transmitted from the terminal
(particularly in screen division display) (1); FIG. 22 is a vieW shoWing a screen con?guration of a
display screen displayed on the monitor of FIG. 1
(particularly in screen division display) (2); 35
display screen displayed on the monitor in FIG. 1
(particularly in screen division display) (3); and FIG. 24 is a vieW shoWing a screen con?guration of a
display screen displayed on the monitor in FIG. 1
(particularly in screen division display)
equipment 20B at the point B through the transmission means 27, (2) an image pickup unit 6A for receiving the image of the interlocutor 3A, Which is picked up by a CCD camera 5A, as image information and transmitting the image information to the terminal equipment 20B at the point B, (3) a voice output unit 23A (speakers) for outputting the
40
voice of the interlocutor 3B on the basis of voice information
45
transmitted from the terminal equipment 20B at the point B through the transmission means 27, and (4) a voice input unit 24A (microphone) for inputting the voice of the inter locutor 3A as voice information and transmitting the voice information to the terminal equipment 20B at the point B. To
DESCRIPTION OF THE PREFERRED EMBODIMENTS
An embodiment of a tWo-Way interactive system accord ing to the present invention Will be described beloW With
experiment by the present inventor that the presence of the table 2A provided a reference for a mental distance to the
a voice output unit in FIG. 1;
FIG. 23 is a vieW shoWing a screen con?guration of a
includes a BBS (Bulletin Board System) having various databases.
realiZe close communication as a counseling system, this
reference to FIGS. 1 to 24.
system also has (5) a character/graphic input unit 25A (an
FIG. 1 is a logical diagram shoWing the entire tWo-Way interactive system according to the present invention. This
interface such as a keyboard, a pointing device, or a touch
and an interlocutor 3B (second interlocutor) at a point B
panel), used by the interlocutor 3A to input characters or graphics, display the input characters or graphics on the display unit 21A, and transmit the characters or graphics to the terminal equipment 20B at the point B as character/
(second point). Particularly, this system is oriented to a counseling system for establishing an intimate reliability
betWeen the above constituent elements and the transmission
system realiZes a tWo-Way interactive operation betWeen an
interlocutor 3A (?rst interlocutor) at a point A (?rst point)
relationship betWeen interlocutors in remote areas and real iZing close communications, so that at least one of the
graphic information, and (6) a processing unit 26 arranged 55
means 27 to perform signal processing and transmission control. The terminal equipment 20B at the point B has the
interlocutors is assumed to be a counselor such as a teacher,
same arrangement as that of the terminal equipment 20A at
a doctor, or a psychologist. Therefore, this system has
the point A.
various functions and arrangements for providing informa tion necessary for counseling and ideal counseling environ
The arrangement of an image pickup apparatus 600A or 600B of the terminal equipment 20A or 20B at the point A or B Will be described beloW. For the descriptive
ment to the counselor.
At the point A, (a) a chair 4A on Which the interlocutor 3A (e.g., a counselor) sits, (b) a table 2A at Which the inter locutor 3A sits, and (c) a terminal equipment 20A having a mechanism for matching the lines of sight betWeen the
convenience, a constituent element common to the terminal
equipments 20A and 20B at the points A and B Will be referred to as, e.g., an image pickup apparatus 600 by 65
omitting suf?xes A and B Which are added to discriminate
interlocutors through a transmission means 27 are set. At the
betWeen the tWo points. Adescription of the terminal equip
point B, (a) a chair 4B on Which the interlocutor 3B (e.g., a
ment 20A at the point A Will be made beloW in principle
