USO0RE4 l 08 8E

(19) United States (12) Reissued Patent

(10) Patent Number:

Anderson (54)

(75)

(45) Date of Reissued Patent:

APPARATUS AND METHOD FOR ROTATING

4,337,479 A

THE DISPLAY ORIENTATION ()F A CAPTURED IMAGE

4,364,650 A 4,470,067 A

_

.

Inventor.

-

C. Anderson, Gardnerville, NV

.

4,952,920 A

8/1990 Hayashi 2/1991

Altrieth, III

6/1992 6/1993

Saegusa et al. ............ .. 348/375 Parulski et al.

5,227,889 A 5,262,863 A

Appl. No.: 11/206,279

5,270,831 A

_

(22)

5,274,418 A

Filed:

6/ 1982 Tomimoto

4,995,089 A

5,122,827 A 5,218,459 A

.

Aug-16, 2005

4/1 992 Sekine et a1‘ *

7/1993 Yoneyama et 31‘ 11/1993 Okada

12/1993 Parulski et al. * 12/1993

1/1994 Richards et al.

5,448,372 A

9/1995 Axman et a1. _

(comlnued)

6,011,585

Issued:

Jan- 4’ 2000

Appl. NO.Z

08/588,210

JP

FOREIGN PATENT DOCUMENTS

58-222382

12/1983

Filed:

Jan. 19, 1996

JP

4-120889

4/1992

U.S. Applications: (63)

Continuation of application No. 10/040,249, ?led on Jan. 4, 2002, now Pat. No. Re. 38,896.

(51)

KaZamiet al. ............ .. 396/311

5,276,519 A

Related US. Patent Documents

Reissue of: (64) Patent No.:

Int. Cl. H04N 5/228 H04N 3/14 H04N 5/335 H04N 9/04 H04N 9/083 H04N 5/235 G09G 5/00 G06K 9/32

Primary ExamineriLin Ye Assistant Examinerilason Whipkey (74) Attorney, Agent, or FirmiFenwick & West LLP

(57) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01) (2006.01)

ABSTRACT

The apparatus of the present invention preferably comprises an image sensor, an orientation sensor, a memory and a pro

cessing unit. The image sensor is used for generating cap tured image data. The orientation sensor is coupled to the

image sensor, and is used for generating signals relating to the position of the image sensor. The memory, has an auto

(52)

US. Cl. ................ .. 348/272; 348/208.2; 348/208.3;

(58)

Field of Classi?cation Search ................ .. 702/151,

348/222.1; 345/656; 382/296

rotate unit comprising program instructions for transforming the captured image data into rotated image data in response to the orientation sensor signals. The processing unit, executes program instructions stored in the memory, and is coupled to the image sensor, the orientation sensor and the

702/154; 348/208.2, 208.3, 272, 222.1; 345/656;

memory. The method of the present invention preferably

382/296

comprises the steps of: generating image data representative

See application ?le for complete search history. (56)

Jan. 26, 2010

12/ 1982 Terashita et a1. 9/1984 Mino

5,107,293 A

(73) Ass1gnee: Apple Inc., Cupertino, CA (US) (21)

US RE41,088 E

of an object With an image sensor; identifying an orientation

of the image sensor relative to the object during the generat ing step; and selectively transferring the image data to an image processing unit in response to the identifying step.

References Cited U.S. PATENT DOCUMENTS 3,814,227 A

6/1974 Hurd, III et a1.

3,971,065 A

7/1976 Bayer

8 Claims, 17 Drawing Sheets

200 z'axis

Q

E

202

'

:34

I

{

__

TOP

30o

l

—--'r — -—-—>x-axis

304

\.2___1/ \

\

.\

332

336

\

\_ \_

\

K 314

_ _ _ .I ‘

US RE41,088 E Page 2

US. PATENT DOCUMENTS

5,764,291 A 5,796,428 A

5,521,639 5,533,185 5,576,759 5,598,181

A A A A

5/1996 7/1996 11/1996 1/1997

5,634,088 5,649,237

A

7/l997 5/1997

5,734,875 A

Tomuraetal. Lentz et :11. Kawamura et a1. Kermisch Nakano Banton Okazaki et_____________ a1‘ "

3/1998 Cheng

348/2082

5,805,216 5,821,997 5,854,641 5,900,909

A A A A

6,480,288 5,966,116

A B1

6/1998 Fullam *

*

* cited by examiner

8/1998 Matsumoto et a1.

348/207.99

9/1993 10/1998 12/1998 5/1999

Tabeietal, Kawamura et :11. HOW?fdet 31~ Parulski et a1.

10/1999 11/2002

Fujuet a1.. . . . .................. . . . . . . . . . . . . . . . . .. .. 396/98

US. Patent

Jan. 26, 2010

Sheet 1 0f 17

TOP

Fig. 1A (Prior Art)

TOP

Fig. 1B (Prior Art)

US RE41,088 E

US. Patent

Jan. 26, 2010

US RE41,088 E

Sheet 2 0f 17

TOP of an Upright

Landscape Image <<<<< Cohm'ms >>> »

acacsca ,l GRGRGK.

868686.. 5 GRGRGR.

..GBGBGB

, 4 BGBGBG.

TOP of First

Portrait Image

R><

u

y

v

n

1

u

v

a

n

n

u

u

u

I

u

u

-

u

l

n

u

a

-

-

-

-

u

.

u

u

u

u

a

u

I

o

I

c

v

u



I

I

o

rmBGBGBGn .GBGBGB RGRGRG YWGRGRGR....GBGBGB ,WBGBGBGH rmGRGRGRH .RGRGRG

Fig. 1C (Prior Art)

.

TOP of Second

Portrait Image

US. Patent

Jan. 26, 2010

Sheet 3 0f 17

US RE41,088 E

10 /

L\(IM TOP

108

l02\ X 106

A(Fig.1DPrtio)r

US. Patent

Jan. 26, 2010

Sheet 4 0f 17

miwuoi Emzxnsm @19a2m06 Y

wE E 532 .5

mam

US RE41,088 E

US. Patent

20(\

Jan. 26, 2010

Sheet 6 0f 17

US RE41,088 E

z-axis

§ ! !

TOP ‘

I ___.

304

334

i

300

I;

(‘R \ .\

332

/ 336

\

‘\_ \,

\

y-axis

314

US. Patent

Jan. 26, 2010

_

Sheet 7 0f 17

_

u N;

“ 5

l-mgd?w.-tmr1 cow

/

US RE41,088 E

n35: h23;5 nma5.:25m.: . ._ _

E2nv_03?‘

4/2; m w 1Q Eu 2Q J

"omuJmQzo

mmH:15:35mi:...

[email protected]:. [email protected]:32£;.5 mm;[email protected]

\/\ . _ .. .... n o: .......

_Z

“{lm.-1 i__g

i"235?.Q2hwz

Um.5

US. Patent

Evmcvvow

Jan. 26, 2010

Sheet 8 0f 17

O}2mc&im5uo?

u2E:3s4>02 5:023

Husmp1-i.5n3;gDc9q

l‘-ovI'.

AWY hu. tsn New

11

new

US RE41,088 E

a?wow

US. Patent

Jan. 26, 2010

Sheet 9 0f 17

US RE41,088 E

410

I I I I

I I I

I

Frame

I

I I I

I

Buffer

'

'

I

I I

I I I I

Storage Memory



'

I

I l I

I I l

I I

l I I

I

Fig. 4B

US. Patent

Jan. 26, 2010

Sheet 10 of 17

US RE41,088 E

417

419f __________________ m

Image Data

421\/~

RLOS Signal

421/“

UDOS Signal

1125f

Miscellaneous

Frame Buffer 416

Working Memory 418

US. Patent

Jan. 26, 2010

Sheet 11 0f 17

US RE41,088 E

A

402W

406

4')

Image

H A C Bu 4

R“.a

.mw m1

H Processing

/\J

Unit

8t

e.1 ‘H

¢

.

I

Fig. 4D

. Operatmg

System Unit

m 426

US. Patent

Jan. 26, 2010

Sheet 14 0f 17

US RE41,088 E

Capture a Set of Image Data and a Set of Image /\/60O Sensor Orientation Data Simultaneously l7

Transfer the Captured Image Data and the Image /\_/602 Sensor Orientation Data into the Volatile Memory 1

Generate an Additional Row and Column of Image

603

Data if the Captured Image Data does not already N include an extra Row and Column of Image Data v

Determine an Orientation of the Captured Image Data

606

Is the Captured Image Data in a Portrait_Left

Orientation?

Is the Captured Image Data in a POrtraiLPRight

Orientation?

604

US. Patent

Jan. 26, 2010

Sheet 15 0f 17

US RE41,088 E

(Dl Configure the Image Processing Unit to Accept an Image Data 512 Line Length Corresponding to a Portrait Image N I

Initialize "Column" to a First Pixel Column Required by the Image Processing Unit

V

_

614 N

Initialize "Row" to a Row Containing a First Pixel Color

Required by the Image Processing Unit

616

N

Transfer the Captured Image Data in Frame Buffer Location

("Row", "Column‘) to the Image Processing Unit

A918

I

Decrement "Row" to a Row Containing a Next Pixel Color

/\J620

Required by the Image Processing Unit 622 Has "Last Row" been Transferred’?

Yes

Perform any Required Image Processing on the Image Data

J’ Image Processing Unit

Increment "Column" to a Next Pixel Column Required by the

A923 624

N

626

Has "Last Column" been Transferred? Yes

Fig. 6B

US. Patent

Jan. 26, 2010

Sheet 16 of 17

US RE41,088 E

Configure the Image Processing Unit to Accept an Image Data Line Length Corresponding to a Portrait Image

628 N

v

initialize "Column" to a First Pixel Column Required by the image Processing Unit V

630 N

Initialize "Row" to a Row Containing a First Pixel Color

Required by the Image Processing Unit

632

N

Transfer the Captured Image Data in Frame Buffer Location

("Row", "Column") to the Image Processing Unit

634

r

Increment "Row" to a Row Containing it Next Pixel Color

Required by the image Processing Unit

636

N 638

as "Last Row" been Transferred?

Perform any Required Image Processing on the Image Data ‘I’

.

/\J639 640

Decrement "Column" to 21 Next Pixel Column Required by the (\1

Image Processing Unit

Fig. 6C

US. Patent

Jan. 26, 2010

Sheet 17 of 17

US RE41,088 E

(9l Configure the Image Processing Unit to Accept an Image Data 644 Line Length Corresponding to a Landscape Image N I

Initialize "Row" to a First Pixel Row Required by the Image Processing Unit Initialize "Column" to a Column Containing a First Pixel Color

Required by the Image Processing Unit

646 N 648

N

l Transfer the Captured Image Data in Frame Buffer Location

("Row", "Column") to the Image Processing Unit

A950

Increment "Column" to a Column Containing at Next Pixel Color /\J652

Required by the Image Processing Unit I

Has "Last.Colnmn"\\ been Transferred? 655

Perform any Required Image Processing on the Image Data

N

Increment "Row" to a Next Pixel Row Required by the Image N656

Processing Unit Has "Last Row" bee'n Transferred?

; Yes End

Fig. 6D

US RE41,088 E 1

2

APPARATUS AND METHOD FOR ROTATING THE DISPLAY ORIENTATION OF A CAPTURED IMAGE

Throughout this speci?cation, “G” means “green,” “R” means “red,” and “B” means “blue.”

Once an image is captured by the digital camera, a set of

pixel signals corresponding to the image received by the pixels is processed by an image processing algorithm. Image

Matter enclosed in heavy brackets [ ] appears in the original patent but forms no part of this reissue speci?ca tion; matter printed in italics indicates the additions made by reissue.

processing routines are conventionally designed to process

pixel signals line-by-line, conforming to a speci?c and unchanging pixel pattern format. Thus, image sensors manu factured with the Bayer pixel pattern format will be coupled

Notice: More than one reissue application has been ?led

to image processing routines speci?cally designed to accept pixel signals in alternating sequences of “GRGRGR” and

for the reissue ofU.S. Pat. No. 6,011,585. This reissue appli cation is a continuation of reissue application Ser. No.

“BGBGBG.” Due to possible imperfections in the outer rows and columns of pixels that make up the image sensor, conventional digital cameras sometimes have image sensors

]0/040,249?led Jan. 4, 2002, now RE 38,896.

This application relates to co-pending US. patent applica tion Ser. No. 08/355,031, entitled A System and Method For

large enough so that one or more lines of pixels at the sides of the image sensor can be ignored.

Generating a Contrast Overlay as a Focus Assist for An

Imaging Device, ?led on Dec. 13, 1994, by inventor Eric C. Anderson; and US. patent application Ser. No. 08/384,012, entitled Apparatus and Method for Camera Image and Ori entation Capture, ?led on Feb. 6, 1995, by inventor Scott

20

Fullam. The subject matter of the two applications described

above is hereby incorporated by reference. These related applications are commonly assigned to Apple Computer,

106 and a bottom 108.

As previously described, the image processing routines

Inc. 25

BACKGROUND OF THE INVENTION

1. Field of the Invention The present invention relates generally to an apparatus and method for orienting an image. More particularly, the present invention is an apparatus and method for rotating a captured image to an orientation corresponding to an imag

ing subsystem’s orientation at the time in which the image 35

When a digital camera captures an image of an object, the

camera’s frame of reference with respect to the object pro duces a desired image orientation. Two conventional image orientations exist, namely, a landscape orientation and a por trait orientation. Referring now to FIG. 1A, a prior art graphical representation of an object in a landscape orienta tion is shown, in which the image’s width is greater than its height. Referring also now to FIG. 1B, a prior art graphical

representation of the object in a portrait orientation is shown, in which the image’s height is greater than its width.

40

45

the portrait image corresponds to the right side 104 of the image display 100. This “sideways” orientation for portrait images on the image display 100 is unacceptable and unnatural. This undesirable by-product of conventional digi tal cameras requires that the user rotate portrait images so that they are presented in a more natural and upright viewing

angle on the image display 100. It should be noted that this

problem also exists for digitized images from conventional ?lm cameras. Traditional rotation methods also have the dis advantage of requiring two blocks of memory to rotate a 50

stored image. What is needed is an apparatus and method that e?iciently and automatically rotates a stored photographic image to correspond to the orientation in which the photographic

digital camera so that a ?rst row of pixels (i.e. row r1) corre

image was captured. 55

SUMMARY OF THE INVENTION

The present invention is an apparatus and method for

cally consists of an array of green (G), red (R) and blue (B) pixels. Alternative embodiments include sensors detecting cyan, magenta, yellow and green as is typically used in video cameras. Other image sensor con?gurations are also used.

scape image would correspond to the top 102 of the FIG. 1D

image display 100. Such an orientation for landscape images on the image display 100 is quite natural and is acceptable for ease of viewing the image. However, the presentation of the portrait image of FIG. 1B upon the image display results in the “TOP” of the portrait image corresponding to either 100, depending on how the user had rotated the digital cam era. FIG. 1D explicitly shows the case where the “TOP” of

In a digital camera, an image sensor is comprised of light

sponds to the bottom of an upright and level digital camera. This image sensor orientation is required since as an optical image passes through a conventional camera lens it is inverted. The image sensor in a color digital camera typi

the image display, the “TOP” (i.e. “top portion”) of the land

the right side 104 or the left side 106 of the image display

sensitive devices, such as charge-coupled devices (CCD), that convert an optical image into a set of electrical signals. Referring now to FIG. 1C, a prior art image sensor is shown having a 480 row by 640 column matrix of light collecting pixels. The image sensor is orientated within a body of the

within digital cameras are conventionally designed to pro cess pixel signals on a line-by-line basis according to only one pixel pattern format. Thus, conventional digital cameras process images as if they were always in a landscape format.

In the presentation of the landscape image of FIG. 1A upon 30

was captured.

2. Description of the Background Art

Referring now to FIG. 1D, a prior art graphical represen tation is shown of the object as captured in the portrait orien tation and output upon an image display 100. The image display 100 is typically a conventional stand-alone, personal computer CRT having a top 102, a right side 104, a left side

rotating the display orientation of a captured image. The apparatus of the present invention preferably comprises an 60

image sensor, an orientation sensor, a memory and a pro

The pixels that comprise the image sensor are arranged into

cessing unit. The image sensor is used for capturing image

various patterns or formats. A common image sensor format

data. The orientation sensor is coupled to the image sensor, and is used for generating a portrait signal if the image sen sor is positioned in a portrait orientation relative to the object. The memory, has an auto-rotate unit comprising pro

is called a Bayer pattern. The Bayer pattern format is de?ned as a pixel pattern comprised of 50% green-light responsive

pixels, 25% red-light responsive pixels and 25% blue-light responsive pixels arranged in alternating rows of “GRGRGR” and “BGBGBG,” as shown in FIG. 1C.

65

gram instructions for transforming the captured image data into rotated image data in response to the portrait signal. The

Apparatus and method for rotating the display orientation of a captured ...

Aug 16, 2005 - 6/1993 Parulski et al. (73) Ass1gnee: Apple Inc., Cupertino, CA (US) ..... Initialize "Row" to a Row Containing a First Pixel Color. 616. Required by the .... This undesirable by-product of conventional digi tal cameras requires ...

2MB Sizes 3 Downloads 26 Views

Recommend Documents

Apparatus and method for rotating the display orientation of a captured ...
Aug 16, 2005 - (73) Ass1gnee: Apple Inc., Cupertino, CA (US). 5,227,889 A. 7/1993 Yoneyama et 31'. 5,262,863 A. 11/1993 Okada. (21) Appl. No.: 11/206,279.

Method and apparatus for driving the display device, display system ...
Feb 5, 1998 - 345/100. AND DATA PROCESSING DEVICE. 5,604,511 A ... Foreign Application Priority Data. It is an object to .... DATA DRIVER. I l. 'IIII'IIIII'I IJ.

Method and apparatus for driving the display device, display system ...
Feb 5, 1998 - 345/206. THE DISPLAY DEVICE, DISPLAY SYSTEM,. 5,251,051 A * 10/1993 Fujiiyoshi et a1. .. 345/100. AND DATA PROCESSING DEVICE.

Scanning apparatus and method
Dec 24, 2009 - FOREIGN PATENT DOCUMENTS. DE. 3 938 714 A1. 5/1991. EP. 0159187 A1 10/1985. EP. 0159187. 10/1985. EP. 0 328 443. 8/1989. EP. 0 348 247. 12/1989. EP. 0 550 300. 7/1993. EP. 0 589 750. 3/1994. EP. 0 750 175. 12/1996. EP. 0 750 176. 12/19

Apparatus and method for sealing vascular punctures
Oct 22, 1993 - (US); Hans Mische, St. Cloud, MN (US) .... 4,168,708 A * 9/1979 Lepley, Jr. et al. 5,035,695 A * 7/1991 ... 4,404,971 A * 9/1983 LeVeen et al.

Method and apparatus for RFID communication
Nov 26, 2002 - 340/101. 3,713,148 A * 1/1973 Cardullo etal. . 342/42. 3,754,170 A * 8/1973 Tsudaet al. .. 257/659 ..... When a sheet of transponders is aligned, computer 86 directs RF sWitch ..... described in detail in r'Error Control Coding.

Method and apparatus for RFID communication
Nov 26, 2002 - network interface 26 connect to individual peripheral con trollers 20a-20c via ... 16, as well as monitor 22 andperipheral controllers 20a20c are all conventional .... other media will be readily apparent to those skilled in the.

Apparatus and method for enhanced oil recovery
25 Nov 1987 - Appl. No.: Filed: [51} Int. Cl.5 pocket mandrel or other downhole tools. Along with the impingement device, a centralizer to guide tools. Nov. 1, 1985 through the impingement device and to cause a pressure. E21B 43/24. [52] US. Cl. 166/

Method and apparatus for RFID communication
Sep 28, 2007 - wireless communication protocol. 4 Claims ..... The aspects, advantages, and fea ... 15 is connected by cable 18 to subsystem 24 so that signals.

Method and apparatus for RFID communication
Sep 28, 2007 - mized, transponder identity and location are not confused, and test ...... suggestion is practical using the media access control scheme.

Apparatus and method for enhanced oil recovery
Nov 25, 1987 - The vapor phase of the steam ?ows into and is de?ected by the ?ngers of the impinge ment means into the longitudinal ?ow passageway ol.