USO0RE4 l 08 8E
(19) United States (12) Reissued Patent
(10) Patent Number:
Anderson (54)
(75)
(45) Date of Reissued Patent:
APPARATUS AND METHOD FOR ROTATING
4,337,479 A
THE DISPLAY ORIENTATION ()F A CAPTURED IMAGE
4,364,650 A 4,470,067 A
_
.
Inventor.
-
C. Anderson, Gardnerville, NV
.
4,952,920 A
8/1990 Hayashi 2/1991
Altrieth, III
6/1992 6/1993
Saegusa et al. ............ .. 348/375 Parulski et al.
5,227,889 A 5,262,863 A
Appl. No.: 11/206,279
5,270,831 A
_
(22)
5,274,418 A
Filed:
6/ 1982 Tomimoto
4,995,089 A
5,122,827 A 5,218,459 A
.
Aug-16, 2005
4/1 992 Sekine et a1‘ *
7/1993 Yoneyama et 31‘ 11/1993 Okada
12/1993 Parulski et al. * 12/1993
1/1994 Richards et al.
5,448,372 A
9/1995 Axman et a1. _
(comlnued)
6,011,585
Issued:
Jan- 4’ 2000
Appl. NO.Z
08/588,210
JP
FOREIGN PATENT DOCUMENTS
58-222382
12/1983
Filed:
Jan. 19, 1996
JP
4-120889
4/1992
U.S. Applications: (63)
Continuation of application No. 10/040,249, ?led on Jan. 4, 2002, now Pat. No. Re. 38,896.
(51)
KaZamiet al. ............ .. 396/311
5,276,519 A
Related US. Patent Documents
Reissue of: (64) Patent No.:
Int. Cl. H04N 5/228 H04N 3/14 H04N 5/335 H04N 9/04 H04N 9/083 H04N 5/235 G09G 5/00 G06K 9/32
Primary ExamineriLin Ye Assistant Examinerilason Whipkey (74) Attorney, Agent, or FirmiFenwick & West LLP
(57) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01)
ABSTRACT
The apparatus of the present invention preferably comprises an image sensor, an orientation sensor, a memory and a pro
cessing unit. The image sensor is used for generating cap tured image data. The orientation sensor is coupled to the
image sensor, and is used for generating signals relating to the position of the image sensor. The memory, has an auto
(52)
US. Cl. ................ .. 348/272; 348/208.2; 348/208.3;
(58)
Field of Classi?cation Search ................ .. 702/151,
348/222.1; 345/656; 382/296
rotate unit comprising program instructions for transforming the captured image data into rotated image data in response to the orientation sensor signals. The processing unit, executes program instructions stored in the memory, and is coupled to the image sensor, the orientation sensor and the
702/154; 348/208.2, 208.3, 272, 222.1; 345/656;
memory. The method of the present invention preferably
382/296
comprises the steps of: generating image data representative
See application ?le for complete search history. (56)
Jan. 26, 2010
12/ 1982 Terashita et a1. 9/1984 Mino
5,107,293 A
(73) Ass1gnee: Apple Inc., Cupertino, CA (US) (21)
US RE41,088 E
of an object With an image sensor; identifying an orientation
of the image sensor relative to the object during the generat ing step; and selectively transferring the image data to an image processing unit in response to the identifying step.
References Cited U.S. PATENT DOCUMENTS 3,814,227 A
6/1974 Hurd, III et a1.
3,971,065 A
7/1976 Bayer
8 Claims, 17 Drawing Sheets
200 z'axis
Q
E
202
'
:34
I
{
__
TOP
30o
l
—--'r — -—-—>x-axis
304
\.2___1/ \
\
.\
332
336
\
\_ \_
\
K 314
_ _ _ .I ‘
US RE41,088 E Page 2
US. PATENT DOCUMENTS
5,764,291 A 5,796,428 A
5,521,639 5,533,185 5,576,759 5,598,181
A A A A
5/1996 7/1996 11/1996 1/1997
5,634,088 5,649,237
A
7/l997 5/1997
5,734,875 A
Tomuraetal. Lentz et :11. Kawamura et a1. Kermisch Nakano Banton Okazaki et_____________ a1‘ "
3/1998 Cheng
348/2082
5,805,216 5,821,997 5,854,641 5,900,909
A A A A
6,480,288 5,966,116
A B1
6/1998 Fullam *
*
* cited by examiner
8/1998 Matsumoto et a1.
348/207.99
9/1993 10/1998 12/1998 5/1999
Tabeietal, Kawamura et :11. HOW?fdet 31~ Parulski et a1.
10/1999 11/2002
Fujuet a1.. . . . .................. . . . . . . . . . . . . . . . . .. .. 396/98
US. Patent
Jan. 26, 2010
Sheet 1 0f 17
TOP
Fig. 1A (Prior Art)
TOP
Fig. 1B (Prior Art)
US RE41,088 E
US. Patent
Jan. 26, 2010
US RE41,088 E
Sheet 2 0f 17
TOP of an Upright
Landscape Image <<<<< Cohm'ms >>> »
acacsca ,l GRGRGK.
868686.. 5 GRGRGR.
..GBGBGB
, 4 BGBGBG.
TOP of First
Portrait Image
R><
u
y
v
n
1
u
v
a
n
n
u
u
u
I
u
u
-
u
l
n
u
a
-
-
-
-
u
.
u
u
u
u
a
u
I
o
I
c
v
u
‘
I
I
o
rmBGBGBGn .GBGBGB RGRGRG YWGRGRGR....GBGBGB ,WBGBGBGH rmGRGRGRH .RGRGRG
Fig. 1C (Prior Art)
.
TOP of Second
Portrait Image
US. Patent
Jan. 26, 2010
Sheet 3 0f 17
US RE41,088 E
10 /
L\(IM TOP
108
l02\ X 106
A(Fig.1DPrtio)r
US. Patent
Jan. 26, 2010
Sheet 4 0f 17
miwuoi Emzxnsm @19a2m06 Y
wE E 532 .5
mam
US RE41,088 E
US. Patent
20(\
Jan. 26, 2010
Sheet 6 0f 17
US RE41,088 E
z-axis
§ ! !
TOP ‘
I ___.
304
334
i
300
I;
(‘R \ .\
332
/ 336
\
‘\_ \,
\
y-axis
314
US. Patent
Jan. 26, 2010
_
Sheet 7 0f 17
_
u N;
“ 5
l-mgd?w.-tmr1 cow
/
US RE41,088 E
n35: h23;5 nma5.:25m.: . ._ _
E2nv_03?‘
4/2; m w 1Q Eu 2Q J
"omuJmQzo
mmH:15:35mi:...
m@52E0:. m3@:32£;.5 mm;m.@mmmums.
\/\ . _ .. .... n o: .......
_Z
“{lm.-1 i__g
i"235?.Q2hwz
Um.5
US. Patent
Evmcvvow
Jan. 26, 2010
Sheet 8 0f 17
O}2mc&im5uo?
u2E:3s4>02 5:023
Husmp1-i.5n3;gDc9q
l‘-ovI'.
AWY hu. tsn New
11
new
US RE41,088 E
a?wow
US. Patent
Jan. 26, 2010
Sheet 9 0f 17
US RE41,088 E
410
I I I I
I I I
I
Frame
I
I I I
I
Buffer
'
'
I
I I
I I I I
Storage Memory
‘
'
I
I l I
I I l
I I
l I I
I
Fig. 4B
US. Patent
Jan. 26, 2010
Sheet 10 of 17
US RE41,088 E
417
419f __________________ m
Image Data
421\/~
RLOS Signal
421/“
UDOS Signal
1125f
Miscellaneous
Frame Buffer 416
Working Memory 418
US. Patent
Jan. 26, 2010
Sheet 11 0f 17
US RE41,088 E
A
402W
406
4')
Image
H A C Bu 4
R“.a
.mw m1
H Processing
/\J
Unit
8t
e.1 ‘H
¢
.
I
Fig. 4D
. Operatmg
System Unit
m 426
US. Patent
Jan. 26, 2010
Sheet 14 0f 17
US RE41,088 E
Capture a Set of Image Data and a Set of Image /\/60O Sensor Orientation Data Simultaneously l7
Transfer the Captured Image Data and the Image /\_/602 Sensor Orientation Data into the Volatile Memory 1
Generate an Additional Row and Column of Image
603
Data if the Captured Image Data does not already N include an extra Row and Column of Image Data v
Determine an Orientation of the Captured Image Data
606
Is the Captured Image Data in a Portrait_Left
Orientation?
Is the Captured Image Data in a POrtraiLPRight
Orientation?
604
US. Patent
Jan. 26, 2010
Sheet 15 0f 17
US RE41,088 E
(Dl Configure the Image Processing Unit to Accept an Image Data 512 Line Length Corresponding to a Portrait Image N I
Initialize "Column" to a First Pixel Column Required by the Image Processing Unit
V
_
614 N
Initialize "Row" to a Row Containing a First Pixel Color
Required by the Image Processing Unit
616
N
Transfer the Captured Image Data in Frame Buffer Location
("Row", "Column‘) to the Image Processing Unit
A918
I
Decrement "Row" to a Row Containing a Next Pixel Color
/\J620
Required by the Image Processing Unit 622 Has "Last Row" been Transferred’?
Yes
Perform any Required Image Processing on the Image Data
J’ Image Processing Unit
Increment "Column" to a Next Pixel Column Required by the
A923 624
N
626
Has "Last Column" been Transferred? Yes
Fig. 6B
US. Patent
Jan. 26, 2010
Sheet 16 of 17
US RE41,088 E
Configure the Image Processing Unit to Accept an Image Data Line Length Corresponding to a Portrait Image
628 N
v
initialize "Column" to a First Pixel Column Required by the image Processing Unit V
630 N
Initialize "Row" to a Row Containing a First Pixel Color
Required by the Image Processing Unit
632
N
Transfer the Captured Image Data in Frame Buffer Location
("Row", "Column") to the Image Processing Unit
634
r
Increment "Row" to a Row Containing it Next Pixel Color
Required by the image Processing Unit
636
N 638
as "Last Row" been Transferred?
Perform any Required Image Processing on the Image Data ‘I’
.
/\J639 640
Decrement "Column" to 21 Next Pixel Column Required by the (\1
Image Processing Unit
Fig. 6C
US. Patent
Jan. 26, 2010
Sheet 17 of 17
US RE41,088 E
(9l Configure the Image Processing Unit to Accept an Image Data 644 Line Length Corresponding to a Landscape Image N I
Initialize "Row" to a First Pixel Row Required by the Image Processing Unit Initialize "Column" to a Column Containing a First Pixel Color
Required by the Image Processing Unit
646 N 648
N
l Transfer the Captured Image Data in Frame Buffer Location
("Row", "Column") to the Image Processing Unit
A950
Increment "Column" to a Column Containing at Next Pixel Color /\J652
Required by the Image Processing Unit I
Has "Last.Colnmn"\\ been Transferred? 655
Perform any Required Image Processing on the Image Data
N
Increment "Row" to a Next Pixel Row Required by the Image N656
Processing Unit Has "Last Row" bee'n Transferred?
; Yes End
Fig. 6D
US RE41,088 E 1
2
APPARATUS AND METHOD FOR ROTATING THE DISPLAY ORIENTATION OF A CAPTURED IMAGE
Throughout this speci?cation, “G” means “green,” “R” means “red,” and “B” means “blue.”
Once an image is captured by the digital camera, a set of
pixel signals corresponding to the image received by the pixels is processed by an image processing algorithm. Image
Matter enclosed in heavy brackets [ ] appears in the original patent but forms no part of this reissue speci?ca tion; matter printed in italics indicates the additions made by reissue.
processing routines are conventionally designed to process
pixel signals line-by-line, conforming to a speci?c and unchanging pixel pattern format. Thus, image sensors manu factured with the Bayer pixel pattern format will be coupled
Notice: More than one reissue application has been ?led
to image processing routines speci?cally designed to accept pixel signals in alternating sequences of “GRGRGR” and
for the reissue ofU.S. Pat. No. 6,011,585. This reissue appli cation is a continuation of reissue application Ser. No.
“BGBGBG.” Due to possible imperfections in the outer rows and columns of pixels that make up the image sensor, conventional digital cameras sometimes have image sensors
]0/040,249?led Jan. 4, 2002, now RE 38,896.
This application relates to co-pending US. patent applica tion Ser. No. 08/355,031, entitled A System and Method For
large enough so that one or more lines of pixels at the sides of the image sensor can be ignored.
Generating a Contrast Overlay as a Focus Assist for An
Imaging Device, ?led on Dec. 13, 1994, by inventor Eric C. Anderson; and US. patent application Ser. No. 08/384,012, entitled Apparatus and Method for Camera Image and Ori entation Capture, ?led on Feb. 6, 1995, by inventor Scott
20
Fullam. The subject matter of the two applications described
above is hereby incorporated by reference. These related applications are commonly assigned to Apple Computer,
106 and a bottom 108.
As previously described, the image processing routines
Inc. 25
BACKGROUND OF THE INVENTION
1. Field of the Invention The present invention relates generally to an apparatus and method for orienting an image. More particularly, the present invention is an apparatus and method for rotating a captured image to an orientation corresponding to an imag
ing subsystem’s orientation at the time in which the image 35
When a digital camera captures an image of an object, the
camera’s frame of reference with respect to the object pro duces a desired image orientation. Two conventional image orientations exist, namely, a landscape orientation and a por trait orientation. Referring now to FIG. 1A, a prior art graphical representation of an object in a landscape orienta tion is shown, in which the image’s width is greater than its height. Referring also now to FIG. 1B, a prior art graphical
representation of the object in a portrait orientation is shown, in which the image’s height is greater than its width.
40
45
the portrait image corresponds to the right side 104 of the image display 100. This “sideways” orientation for portrait images on the image display 100 is unacceptable and unnatural. This undesirable by-product of conventional digi tal cameras requires that the user rotate portrait images so that they are presented in a more natural and upright viewing
angle on the image display 100. It should be noted that this
problem also exists for digitized images from conventional ?lm cameras. Traditional rotation methods also have the dis advantage of requiring two blocks of memory to rotate a 50
stored image. What is needed is an apparatus and method that e?iciently and automatically rotates a stored photographic image to correspond to the orientation in which the photographic
digital camera so that a ?rst row of pixels (i.e. row r1) corre
image was captured. 55
SUMMARY OF THE INVENTION
The present invention is an apparatus and method for
cally consists of an array of green (G), red (R) and blue (B) pixels. Alternative embodiments include sensors detecting cyan, magenta, yellow and green as is typically used in video cameras. Other image sensor con?gurations are also used.
scape image would correspond to the top 102 of the FIG. 1D
image display 100. Such an orientation for landscape images on the image display 100 is quite natural and is acceptable for ease of viewing the image. However, the presentation of the portrait image of FIG. 1B upon the image display results in the “TOP” of the portrait image corresponding to either 100, depending on how the user had rotated the digital cam era. FIG. 1D explicitly shows the case where the “TOP” of
In a digital camera, an image sensor is comprised of light
sponds to the bottom of an upright and level digital camera. This image sensor orientation is required since as an optical image passes through a conventional camera lens it is inverted. The image sensor in a color digital camera typi
the image display, the “TOP” (i.e. “top portion”) of the land
the right side 104 or the left side 106 of the image display
sensitive devices, such as charge-coupled devices (CCD), that convert an optical image into a set of electrical signals. Referring now to FIG. 1C, a prior art image sensor is shown having a 480 row by 640 column matrix of light collecting pixels. The image sensor is orientated within a body of the
within digital cameras are conventionally designed to pro cess pixel signals on a line-by-line basis according to only one pixel pattern format. Thus, conventional digital cameras process images as if they were always in a landscape format.
In the presentation of the landscape image of FIG. 1A upon 30
was captured.
2. Description of the Background Art
Referring now to FIG. 1D, a prior art graphical represen tation is shown of the object as captured in the portrait orien tation and output upon an image display 100. The image display 100 is typically a conventional stand-alone, personal computer CRT having a top 102, a right side 104, a left side
rotating the display orientation of a captured image. The apparatus of the present invention preferably comprises an 60
image sensor, an orientation sensor, a memory and a pro
The pixels that comprise the image sensor are arranged into
cessing unit. The image sensor is used for capturing image
various patterns or formats. A common image sensor format
data. The orientation sensor is coupled to the image sensor, and is used for generating a portrait signal if the image sen sor is positioned in a portrait orientation relative to the object. The memory, has an auto-rotate unit comprising pro
is called a Bayer pattern. The Bayer pattern format is de?ned as a pixel pattern comprised of 50% green-light responsive
pixels, 25% red-light responsive pixels and 25% blue-light responsive pixels arranged in alternating rows of “GRGRGR” and “BGBGBG,” as shown in FIG. 1C.
65
gram instructions for transforming the captured image data into rotated image data in response to the portrait signal. The
