EP2157547B1 - Object tracking - Google Patents
Object tracking Download PDFInfo
- Publication number
- EP2157547B1 EP2157547B1 EP09168333.4A EP09168333A EP2157547B1 EP 2157547 B1 EP2157547 B1 EP 2157547B1 EP 09168333 A EP09168333 A EP 09168333A EP 2157547 B1 EP2157547 B1 EP 2157547B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- tracking target
- region
- image
- tracking
- target candidate
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Not-in-force
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/61—Control of cameras or camera modules based on recognised objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/61—Control of cameras or camera modules based on recognised objects
- H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
- H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
- H04N23/635—Region indicators; Field of view indicators
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
- H04N23/675—Focus control based on electronic image sensor signals comprising setting of focusing regions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Definitions
- the present invention relates to a technique for tracking an object by acquiring a correlation between images.
- Some digital cameras and video cameras display an image to be captured on a monitor in real time and allow a user to specify a region on the movie displayed on the monitor to control exposure and focus of the object existing in the region.
- Japanese Patent Application Laid-Open No. 2008-011497 discusses a configuration of a camera which continuously tracks an initially-specified main object by calculating correlation between frame images of the movie and continuously detecting a region similar to the initially-specified main object in time series.
- FIGs. 4A, 4B, and 4C rectangular-shaped frames indicated by bold lines are tracking frames indicating the object specified by the user or the region of the main object detected by the tracking processing.
- the tracking frame is set a bit larger than the specified object so that the object (in this example, a person's face) specified by the user can fit into the frame.
- Fig. 4A it is supposed that the person's face which is initially specified as the main object is successfully tracked by correlation calculation. If the person abruptly moves, an image of the person is largely blurred as illustrated in Fig. 4B . Since the tracking frame illustrated in Fig. 4A includes a part of background behind the person, in the frame image illustrated in Fig. 4B , the background is determined as the region that has the highest correlation rather than the person. Thus, the background may be selected as a region of a new main object.
- a part of the background may be selected as a region of a new main object as illustrated in Fig. 4B . If the background is a uniform object having low contrast, many similar regions exist in the periphery. Therefore, as illustrated in Fig. 4C , the tracking frame moves around the background and looks blurred.
- the present invention is directed to an image processing apparatus and an image processing method that can prevent a display of a frame indicating a tracking target from blurring using correlation between images when an object that is the tracking target is missed during tracking processing.
- Fig. 1 is a block diagram illustrating a functional configuration of a digital camera according to an exemplary embodiment of the present invention.
- Fig. 2 is a flowchart illustrating tracking processing according to the exemplary embodiment of the present invention.
- Figs. 3A, 3B, 3C and 3D illustrate states in which the tracking processing is stopped since a tracking target candidate region is set in a uniform background according to the exemplary embodiment of the present invention.
- Figs. 4A, 4B, and 4C illustrate a conventional method for updating a tracking frame.
- an image processing apparatus will be described using a digital camera as an example.
- Fig. 1 is a block diagram illustrating a functional configuration of the digital camera according to the present exemplary embodiment.
- An operation unit 101 includes switches and buttons to be operated by a user to input various instructions to the digital camera.
- the operation unit 101 includes a shutter switch. When the shutter switch is half pressed, a signal SW1 is generated and transmitted from the operation unit 101 to a control unit 102.
- a signal SW2 is generated and transmitted from the operation unit 101 to the control unit 102.
- the control unit 102 controls a movement of each unit illustrated in Fig. 1 according to the instructions from the operation unit 101.
- An image sensor 103 receives light entering through a lens 108a and an exposure mechanism 109a and outputs an electric charge according to a light amount.
- a charge couple device (CCD) and a complementary metal oxide semiconductor (CMOS) sensor which have a photoelectric conversion function may be used for the image sensor 103.
- An analog/digital (A/D) conversion unit 104 performs sampling, gain adjustment and A/D conversion on an analog image signal output from the image sensor 103 and outputs a digital image signal.
- An image processing unit 105 performs various image processing on the digital image signal output from the A/D conversion unit 104 and outputs the processed digital image signal. For example, the image processing unit 105 converts the digital image signal received from the A/D conversion unit 104 into a YUV image signal and outputs the YUV signal.
- a face detection unit 106 performs face detection on the digital image signal received from the A/D conversion unit 104 and notifies the control unit 102 of information about the region.
- the information acquired by the face detection includes a position, an area, and reliability (certainty as a shape of a face) according to brightness in a light shielded region within a screen of the image sensor.
- a monitor unit 107 includes a small liquid crystal screen for displaying the image according to image data processed by the image processing unit 105.
- a tracking processing unit 115 determines where in subsequent image frames a region considered to be the same as the face detected in a frame image exists by using correlation between the frame images. Further, the tracking processing unit 115 calculates a position, an area and a size of the region. A movement of the camera itself is also calculated using an angular speed sensor. According to the present exemplary embodiment, two images placed in a time sequential order are compared to each other to acquire information about a difference thereof. Using the difference information, the regions considered to have the same face are extracted. The tracking processing unit 115 continuously extracts the regions considered to have the same face in order to track a face area detected by the face detection unit 106.
- a determination unit 116 acquires an evaluation value from the correlation calculated by the tracking processing unit 115 and determines whether the current frame is set on the object or on the background uniform surface.
- the determination method is a distinctive feature of the present exemplary embodiment and will be described below in detail.
- an automatic focusing (AF) processing unit 108 Based on an automatic focusing (AF) evaluation value acquired from the image obtained by the image processing unit 105, an automatic focusing (AF) processing unit 108 operates the lens 108a to focus on the object. Further, based on a light metering value acquired from the image obtained by the image processing unit 105, an automatic exposing (AE) processing unit 109 operates the exposure mechanism 109a so that the screen has an optimum brightness.
- AF automatic focusing
- AE automatic exposing
- the control unit 102 When the control unit 102 receives a notification of the signal SW1 from the shutter switch, the AF processing unit 108 and the AE processing unit 109 perform the focus control and the exposure control using the newest image at that point.
- the control unit 102 Upon receiving a notification of the signal SW2 from the shutter switch (more specifically, a notification that the shutter switch is fully pressed), the control unit 102 determines whether to emit the flash. The user may previously set whether to emit the flash using the operation unit 101, so that the control unit 102 can determine by reading the setting data. Further, the control unit 102 can automatically determine by detecting darkness in the circumstance.
- the control unit 102 controls the EF processing unit 110 to cause the flash unit 111 to previously emit the flash to calculate an amount of emitted flash. Then, the control unit 102 causes the flash unit 111 to emit an amount of main flash which is calculated based on the previously emitted flash and performs main image capturing.
- the format conversion unit 112 converts the format of the digital image signal (image data) output from the image processing unit 105 when the main image capturing is performed into the JPEG format, for example, and outputs the digital image signal to the image storage unit 113.
- the image storage unit 113 stores the image data in the converted format received from the format conversion unit 112 in a memory (not illustrated) of the digital camera or an external memory inserted in the digital camera.
- An external connection unit 114 functions as an interface for connecting the digital camera to an external apparatus such as a personal computer (PC) and a printer.
- PC personal computer
- control for determining whether the object is successfully tracked is based on determination on a uniform surface.
- Fig. 2 is a flowchart illustrating tracking processing according to the present exemplary embodiment.
- the processing of this flowchart is started.
- step S201 when the user sets the digital camera to the image capturing mode, the shutter is opened and light enters the image sensor 103 through the lens 108a and the exposure mechanism 109a.
- the frame images continuously created by the image sensor 103 are displayed on the monitor unit 107 as a movie so that the user can observe a state of the object in real time.
- step S202 the control unit 102 performs evaluative light metering (for example, center-weighted light metering) on the movie to control exposure.
- evaluative light metering for example, center-weighted light metering
- step S203 the control unit 102 reads out the newest frame image and transmits the frame image to the face detection unit 106 which detects a face area from the frame image.
- a method for detecting a face for example, a method using characteristics of portions included in the face such as eyes, a nose and a mouth can be used. For example, if many portions that match with standard patterns and characteristics (shapes and positions) of eyes, nose and mouth that are previously stored in the camera are included and arranged in a fine balance, this region can be estimated as the face area.
- all of the plurality of face areas may be set as the tracking target region. Further, points may be given to each of a position and a size of the face, and reliability as a face (a degree of matching with the standard patterns), and only one face which gains the highest points may be set as the tracking target region. Moreover, the user may operate the operation unit 101 to select any one of the plurality of face areas.
- the movie may be displayed with the tracking frame superimposed thereon so that the user can recognize the position and the size of the tracking target region.
- the image data corresponding to the tracking target region is transmitted to the tracking processing unit 115 to be used as reference image data in the tracking processing.
- Fig. 3A illustrates a state described above and the image displayed on the monitor unit 107 where the tracking frame indicated by a bold line is superimposed on the detected face area.
- the tracking frame is set a bit larger than the face area so that the detected face area can be completely included in the tracking frame.
- step S204 the control unit 102 determines whether the face area can be detected in step S203. If the face area is detected (YES in step S204), the processing proceeds to step S205. If the face area is not detected (NO in step S204), the processing proceeds to step S209.
- step S205 the control unit 102 newly reads out the frame image, and performs the tracking processing on the frame image.
- the tracking processing unit 115 cuts out a partial region from the newly read out frame image while the size or the coordinate of the newly read out frame image is being gradually shifted, and an evaluation value of the cut out region is acquired from information about brightness, color difference and contrast of the cut out region.
- step S206 the tracking processing unit 115 compares an evaluation value acquired from the reference image data which has been previously acquired to each evaluation value of all cut out regions. From among the cut out regions, namely the target regions, the region which is determined to have the highest correlation with the reference image data is selected and set as a tracking target candidate region.
- Fig. 3B illustrates a state described above.
- the background indicated by a dotted line is selected as the tracking target candidate region which has the highest correlation with the region surrounded by the tracking frame illustrated in Fig. 3A .
- step S207 the determination unit 116 sets four peripheral regions acquired by shifting the tracking target candidate region to four positions (i.e., upper left, upper right, lower left, and lower right) centering on the set tracking target candidate region and acquires evaluation values of the four peripheral regions.
- a shifting amount for example, 50 to 100 percent of each of a vertical width and a lateral width of the tracking target candidate region may be set.
- the shifting amount may be acquired as an experimentally optimum value considering the size and previous movement of the object set as the tracking target region.
- Fig. 3C illustrates a state described above.
- the peripheral regions shifted in four oblique directions are set centering on the tracking target candidate region illustrated in Fig. 3B .
- the determination unit 116 acquires the differences between each of the evaluation values acquired from four peripheral regions and the evaluation value acquired from the tracking target candidate region. If all of the differences are less than a threshold value (YES in step S207), the processing proceeds to step S208. If not all of the differences are less than the threshold value, more specifically, if even one region has the evaluation value equal to or more than the threshold value, the processing proceeds to S211.
- step S208 the control unit 102 deletes and resets the information about the tracking target region and the reference image data and stops the tracking processing as illustrated in Fig. 3D .
- the control unit 102 deletes the tracking frame from the image displayed on the monitor unit 107.
- the tracking target candidate region may be set in the uniform background which has the low contrast.
- the tracking processing is stopped because, if the tracking processing is continued as it is, many regions which have the high correlation exist in the periphery as described above using Fig. 4A, 4B, and 4C .
- the position of the tracking frame becomes unstable and blurred.
- step S209 since the tracking target is missed, the control unit 102 causes the AF processing unit 108 to perform the focus control so that the object existing on a center part is roughly focused on and the AE processing unit 109 to perform the exposure control by the center-weighted light metering.
- step S210 the control unit 102 monitors whether the signal SW1 is generated by half pressing the shutter switch of the operation unit 101. If the signal SW1 is generated (YES in step S210), the processing proceeds to step S214. If the signal SW1 is not generated (NO in step S210), the processing returns to step S203 and the control unit 102 newly detects the tracking target region and detects the face area from the frame image newly read out by the face detection unit 106.
- step S207 determines that the tracking is successfully performed and updates the tracking target region with the tracking target candidate region as the new tracking target region. Further, the control unit 102 displays the image of the new tracking target region on which the tracking frame is superimposed, on the monitor unit 107.
- step S212 the AF processing unit 108 and the AE processing unit 109 respectively control the focusing and the exposure by performing the greatest weighting on the light metering value and the AF evaluation value that are acquired from the track target region.
- step S213 the control unit 102 monitors whether the signal SW1 is generated by half pressing the shutter switch of the operation unit 101. If the signal SW1 is generated (YES in stepS213), the processing proceeds to step S214. If the signal SW1 is not generated (NO in step S213), the processing returns to step S205 to continue the tracking processing.
- the AF processing unit 108 and the AE processing unit 109 respectively control the focusing and the exposure for the main image capturing. If the tracking target region exists, the control unit 102 controls the AF processing unit 108 and the AE processing unit 109 to adjust the focus on the tracking target region and the brightness to the optimum value. If the tracking target region does not exist, the control unit 102 controls the AF processing unit 108 and the AE processing unit 109 to prioritize the focus and the light metering value at the center part to adjust the whole screen to an optimum state.
- step S215 the control unit 102 waits until the signal SW2 is generated by fully pressing the shutter switch of the operation unit 101. If the signal SW2 is generated (YES in step S215), the processing proceeds to step S216, and the control unit 102 generates still image data and stores the still image data in the image storage unit 113.
- a problem can be prevented that the tracking frame is set to and blurred on the uniform background which has the low contrast.
- peripheral regions are set around the tracking target candidate region to determine whether the tracking can be continued, however, the peripheral regions may be set only to determine the degree of the correlation between the tracking target candidate region and the peripheral region. Therefore, a number of the peripheral regions to be set may be, for example, three or six. Further, the peripheral regions may be set not only in the four oblique directions but also in vertical and lateral directions.
- the face area detected by the face detection unit 106 is set as the tracking target region.
- the user may operate the operation unit 101 to specify the arbitrary tracking target region.
- the tracking processing unit 115 may extract moving objects considering the differences between the frame images, and the user may operate the operation unit 101 to select any one of the plurality of face areas.
- the present invention can be applied to the configuration that includes the tracking processing unit 115 for determining where in a subsequent second image a region considered to be the same as the face detected in a first image exists by using the correlation between the frame images.
- the exemplary embodiment of the present invention is described referring to the digital camera as an example.
- the digital camera can be applied to a video camera.
- it can be applied also to a computer system such as a monitoring system that receives movie data from a camera connected to the system and performs the tracking processing on the received movie data in the image processing unit.
- a computer-readable storage medium (or recording medium) storing a program code of software for realizing functions of the exemplary embodiment described above is supplied to a system or an apparatus.
- a computer or a central processing unit (CPU) or a micro processing unit (MPU) of the system or the apparatus reads out and executes the program code stored in the storage medium.
- the program code itself read out from the storage medium realizes the functions of the present exemplary embodiment described above, and thus the storage medium storing the program code constitutes the present invention.
- the function of the exemplary embodiment described above can be achieved as follows. More specifically, an operating system (OS) running on the computer performs a part of or all of the actual processing based on instructions of the read out program code, and the above-described functions of the exemplary embodiment can be realized by the processing.
- OS operating system
- the computer-readable storage medium for storing the program code includes, for example, a floppy disk, a hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic tape, a non-volatile memory card, a compact disk ROM (CD-ROM), a CD recordable (CD-R), a digital versatile disk (DVD), an optical disk, and a magneto-optical (MO) disk.
- a computer network such as a local area network (LAN) and a wide area network (WAN) can be used for supplying the program code.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
- Image Analysis (AREA)
- Closed-Circuit Television Systems (AREA)
Description
- The present invention relates to a technique for tracking an object by acquiring a correlation between images.
- Some digital cameras and video cameras display an image to be captured on a monitor in real time and allow a user to specify a region on the movie displayed on the monitor to control exposure and focus of the object existing in the region.
- Other digital cameras and video cameras allow the user to specify an arbitrary object on the movie displayed on the monitor in order to specify a main object as a tracking target to be tracked. Japanese Patent Application Laid-Open No.
2008-011497 - However, if the object of the tracking target widely moves, common portions between the frame images decrease even if the frame images include the same object. Thus, the object in the frame image cannot be recognized as the same object, and tracking may fail.
- With reference to
Figs. 4A, 4B, and 4C , an example will be described. InFigs. 4A, 4B, and 4C , rectangular-shaped frames indicated by bold lines are tracking frames indicating the object specified by the user or the region of the main object detected by the tracking processing. The tracking frame is set a bit larger than the specified object so that the object (in this example, a person's face) specified by the user can fit into the frame. - For example, in
Fig. 4A , it is supposed that the person's face which is initially specified as the main object is successfully tracked by correlation calculation. If the person abruptly moves, an image of the person is largely blurred as illustrated inFig. 4B . Since the tracking frame illustrated inFig. 4A includes a part of background behind the person, in the frame image illustrated inFig. 4B , the background is determined as the region that has the highest correlation rather than the person. Thus, the background may be selected as a region of a new main object. - If only a region having low correlation is detected, it may be determined that the main object is missed. However, when a posture or a direction of the main object is changed, high correlation between the regions cannot be always acquired, even though the main object exists in the image. Thus, in order to enable tracking of the object even when the object moves or changes its posture to some degree, it is desired that a region having the highest correlation in an image is selected as the region of the main object, even though only the region having the low correlation exists in the image.
- However, in this case, a part of the background may be selected as a region of a new main object as illustrated in
Fig. 4B . If the background is a uniform object having low contrast, many similar regions exist in the periphery. Therefore, as illustrated inFig. 4C , the tracking frame moves around the background and looks blurred. - 'Model-based real-time head tracking' by Jacob Strom, EURASIP Journal on Applied Signal Processing, January 2002, pages 1039-1052 describes re-initialization of a tracker on failure of object tracking.
- 'Motion estimation using quad-tree phase correlation' by Argyriou et al, Image Processing 2005, ICIP 2005, IEEE, pages 1081-1084 details the use of identification of peaks on a phase correlation surface to identify motion components in a scene.
- 'An adaptive motion decision system for digital image stabilizer based on edge pattern matching' by Joon Ki Paik et al, IEEE Transactions on Consumer Electronics, vol. 38, no. 3, August 1992, pages 607-615 describes the judgment of a motion vector as unreliable if a difference in correlation values becomes too small.
- The present invention is directed to an image processing apparatus and an image processing method that can prevent a display of a frame indicating a tracking target from blurring using correlation between images when an object that is the tracking target is missed during tracking processing.
- According to a first aspect of the present invention, there is provided an image processing apparatus as specified in claims 1 to 4.
- According to a second aspect of the present invention, there is provided a method for processing an image as specified in claim 5.
- Further features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings.
- The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments of the invention and, together with the description, serve to explain the principles of the invention.
-
Fig. 1 is a block diagram illustrating a functional configuration of a digital camera according to an exemplary embodiment of the present invention. -
Fig. 2 is a flowchart illustrating tracking processing according to the exemplary embodiment of the present invention. -
Figs. 3A, 3B, 3C and 3D illustrate states in which the tracking processing is stopped since a tracking target candidate region is set in a uniform background according to the exemplary embodiment of the present invention. -
Figs. 4A, 4B, and 4C illustrate a conventional method for updating a tracking frame. - Various exemplary embodiments, features, and aspects of the invention will be described in detail below with reference to the drawings.
- In an exemplary embodiment of the present invention, an image processing apparatus will be described using a digital camera as an example.
-
Fig. 1 is a block diagram illustrating a functional configuration of the digital camera according to the present exemplary embodiment. - An
operation unit 101 includes switches and buttons to be operated by a user to input various instructions to the digital camera. Theoperation unit 101 includes a shutter switch. When the shutter switch is half pressed, a signal SW1 is generated and transmitted from theoperation unit 101 to acontrol unit 102. - Further, when the shutter switch is fully pressed, a signal SW2 is generated and transmitted from the
operation unit 101 to thecontrol unit 102. - The
control unit 102 controls a movement of each unit illustrated inFig. 1 according to the instructions from theoperation unit 101. - An
image sensor 103 receives light entering through alens 108a and anexposure mechanism 109a and outputs an electric charge according to a light amount. A charge couple device (CCD) and a complementary metal oxide semiconductor (CMOS) sensor which have a photoelectric conversion function may be used for theimage sensor 103. - An analog/digital (A/D)
conversion unit 104 performs sampling, gain adjustment and A/D conversion on an analog image signal output from theimage sensor 103 and outputs a digital image signal. - An
image processing unit 105 performs various image processing on the digital image signal output from the A/D conversion unit 104 and outputs the processed digital image signal. For example, theimage processing unit 105 converts the digital image signal received from the A/D conversion unit 104 into a YUV image signal and outputs the YUV signal. - A
face detection unit 106 performs face detection on the digital image signal received from the A/D conversion unit 104 and notifies thecontrol unit 102 of information about the region. The information acquired by the face detection includes a position, an area, and reliability (certainty as a shape of a face) according to brightness in a light shielded region within a screen of the image sensor. - A
monitor unit 107 includes a small liquid crystal screen for displaying the image according to image data processed by theimage processing unit 105. - A
tracking processing unit 115 determines where in subsequent image frames a region considered to be the same as the face detected in a frame image exists by using correlation between the frame images. Further, thetracking processing unit 115 calculates a position, an area and a size of the region. A movement of the camera itself is also calculated using an angular speed sensor. According to the present exemplary embodiment, two images placed in a time sequential order are compared to each other to acquire information about a difference thereof. Using the difference information, the regions considered to have the same face are extracted. Thetracking processing unit 115 continuously extracts the regions considered to have the same face in order to track a face area detected by theface detection unit 106. - A
determination unit 116 acquires an evaluation value from the correlation calculated by thetracking processing unit 115 and determines whether the current frame is set on the object or on the background uniform surface. The determination method is a distinctive feature of the present exemplary embodiment and will be described below in detail. - Based on an automatic focusing (AF) evaluation value acquired from the image obtained by the
image processing unit 105, an automatic focusing (AF)processing unit 108 operates thelens 108a to focus on the object. Further, based on a light metering value acquired from the image obtained by theimage processing unit 105, an automatic exposing (AE)processing unit 109 operates theexposure mechanism 109a so that the screen has an optimum brightness. - When the
control unit 102 receives a notification of the signal SW1 from the shutter switch, theAF processing unit 108 and theAE processing unit 109 perform the focus control and the exposure control using the newest image at that point. - Upon receiving a notification of the signal SW2 from the shutter switch (more specifically, a notification that the shutter switch is fully pressed), the
control unit 102 determines whether to emit the flash. The user may previously set whether to emit the flash using theoperation unit 101, so that thecontrol unit 102 can determine by reading the setting data. Further, thecontrol unit 102 can automatically determine by detecting darkness in the circumstance. - As a result of the determination, if it is determined that the flash is emitted, the
control unit 102 controls theEF processing unit 110 to cause theflash unit 111 to previously emit the flash to calculate an amount of emitted flash. Then, thecontrol unit 102 causes theflash unit 111 to emit an amount of main flash which is calculated based on the previously emitted flash and performs main image capturing. - The
format conversion unit 112 converts the format of the digital image signal (image data) output from theimage processing unit 105 when the main image capturing is performed into the JPEG format, for example, and outputs the digital image signal to theimage storage unit 113. Theimage storage unit 113 stores the image data in the converted format received from theformat conversion unit 112 in a memory (not illustrated) of the digital camera or an external memory inserted in the digital camera. - An
external connection unit 114 functions as an interface for connecting the digital camera to an external apparatus such as a personal computer (PC) and a printer. - Next, control for determining whether the object is successfully tracked, more specifically, whether tracking of a moving object is to be stopped is based on determination on a uniform surface.
-
Fig. 2 is a flowchart illustrating tracking processing according to the present exemplary embodiment. When the user sets the digital camera to an image capturing mode, the processing of this flowchart is started. - In step S201, when the user sets the digital camera to the image capturing mode, the shutter is opened and light enters the
image sensor 103 through thelens 108a and theexposure mechanism 109a. The frame images continuously created by theimage sensor 103 are displayed on themonitor unit 107 as a movie so that the user can observe a state of the object in real time. - In step S202, the
control unit 102 performs evaluative light metering (for example, center-weighted light metering) on the movie to control exposure. - In step S203, the
control unit 102 reads out the newest frame image and transmits the frame image to theface detection unit 106 which detects a face area from the frame image. As a method for detecting a face, for example, a method using characteristics of portions included in the face such as eyes, a nose and a mouth can be used. For example, if many portions that match with standard patterns and characteristics (shapes and positions) of eyes, nose and mouth that are previously stored in the camera are included and arranged in a fine balance, this region can be estimated as the face area. - When a plurality of face areas is detected, all of the plurality of face areas may be set as the tracking target region. Further, points may be given to each of a position and a size of the face, and reliability as a face (a degree of matching with the standard patterns), and only one face which gains the highest points may be set as the tracking target region. Moreover, the user may operate the
operation unit 101 to select any one of the plurality of face areas. The movie may be displayed with the tracking frame superimposed thereon so that the user can recognize the position and the size of the tracking target region. Furthermore, the image data corresponding to the tracking target region is transmitted to thetracking processing unit 115 to be used as reference image data in the tracking processing. -
Fig. 3A illustrates a state described above and the image displayed on themonitor unit 107 where the tracking frame indicated by a bold line is superimposed on the detected face area. The tracking frame is set a bit larger than the face area so that the detected face area can be completely included in the tracking frame. - In step S204, the
control unit 102 determines whether the face area can be detected in step S203. If the face area is detected (YES in step S204), the processing proceeds to step S205. If the face area is not detected (NO in step S204), the processing proceeds to step S209. - In step S205, the
control unit 102 newly reads out the frame image, and performs the tracking processing on the frame image. Thetracking processing unit 115 cuts out a partial region from the newly read out frame image while the size or the coordinate of the newly read out frame image is being gradually shifted, and an evaluation value of the cut out region is acquired from information about brightness, color difference and contrast of the cut out region. - In step S206, the
tracking processing unit 115 compares an evaluation value acquired from the reference image data which has been previously acquired to each evaluation value of all cut out regions. From among the cut out regions, namely the target regions, the region which is determined to have the highest correlation with the reference image data is selected and set as a tracking target candidate region. -
Fig. 3B illustrates a state described above. InFig. 3B , since the person whose face is detected as the face area moves rapidly and thus the image of the person becomes blurred, the background indicated by a dotted line is selected as the tracking target candidate region which has the highest correlation with the region surrounded by the tracking frame illustrated inFig. 3A . - In step S207, the
determination unit 116 sets four peripheral regions acquired by shifting the tracking target candidate region to four positions (i.e., upper left, upper right, lower left, and lower right) centering on the set tracking target candidate region and acquires evaluation values of the four peripheral regions. As a shifting amount, for example, 50 to 100 percent of each of a vertical width and a lateral width of the tracking target candidate region may be set. The shifting amount may be acquired as an experimentally optimum value considering the size and previous movement of the object set as the tracking target region. -
Fig. 3C illustrates a state described above. InFig. 3C , the peripheral regions shifted in four oblique directions are set centering on the tracking target candidate region illustrated inFig. 3B . - The
determination unit 116 acquires the differences between each of the evaluation values acquired from four peripheral regions and the evaluation value acquired from the tracking target candidate region. If all of the differences are less than a threshold value (YES in step S207), the processing proceeds to step S208. If not all of the differences are less than the threshold value, more specifically, if even one region has the evaluation value equal to or more than the threshold value, the processing proceeds to S211. - In step S208, the
control unit 102 deletes and resets the information about the tracking target region and the reference image data and stops the tracking processing as illustrated inFig. 3D . Thecontrol unit 102 deletes the tracking frame from the image displayed on themonitor unit 107. As a result of the determination in step S207, if the tracking target candidate region is similar to all of the four peripheral regions, the tracking target candidate region may be set in the uniform background which has the low contrast. The tracking processing is stopped because, if the tracking processing is continued as it is, many regions which have the high correlation exist in the periphery as described above usingFig. 4A, 4B, and 4C . Thus, the position of the tracking frame becomes unstable and blurred. - In step S209, since the tracking target is missed, the
control unit 102 causes theAF processing unit 108 to perform the focus control so that the object existing on a center part is roughly focused on and theAE processing unit 109 to perform the exposure control by the center-weighted light metering. - In step S210, the
control unit 102 monitors whether the signal SW1 is generated by half pressing the shutter switch of theoperation unit 101. If the signal SW1 is generated (YES in step S210), the processing proceeds to step S214. If the signal SW1 is not generated (NO in step S210), the processing returns to step S203 and thecontrol unit 102 newly detects the tracking target region and detects the face area from the frame image newly read out by theface detection unit 106. - If the processing proceeds from step S207 to step S211, the
control unit 102 determines that the tracking is successfully performed and updates the tracking target region with the tracking target candidate region as the new tracking target region. Further, thecontrol unit 102 displays the image of the new tracking target region on which the tracking frame is superimposed, on themonitor unit 107. - In step S212, the
AF processing unit 108 and theAE processing unit 109 respectively control the focusing and the exposure by performing the greatest weighting on the light metering value and the AF evaluation value that are acquired from the track target region. - In step S213, the
control unit 102 monitors whether the signal SW1 is generated by half pressing the shutter switch of theoperation unit 101. If the signal SW1 is generated (YES in stepS213), the processing proceeds to step S214. If the signal SW1 is not generated (NO in step S213), the processing returns to step S205 to continue the tracking processing. - If the signal SW1 is generated in steps S210 and S213, then in step S214, the
AF processing unit 108 and theAE processing unit 109 respectively control the focusing and the exposure for the main image capturing. If the tracking target region exists, thecontrol unit 102 controls theAF processing unit 108 and theAE processing unit 109 to adjust the focus on the tracking target region and the brightness to the optimum value. If the tracking target region does not exist, thecontrol unit 102 controls theAF processing unit 108 and theAE processing unit 109 to prioritize the focus and the light metering value at the center part to adjust the whole screen to an optimum state. - In step S215, the
control unit 102 waits until the signal SW2 is generated by fully pressing the shutter switch of theoperation unit 101. If the signal SW2 is generated (YES in step S215), the processing proceeds to step S216, and thecontrol unit 102 generates still image data and stores the still image data in theimage storage unit 113. - As described above, according to the exemplary embodiment, a problem can be prevented that the tracking frame is set to and blurred on the uniform background which has the low contrast.
- Further, according to the exemplary embodiment, four peripheral regions are set around the tracking target candidate region to determine whether the tracking can be continued, however, the peripheral regions may be set only to determine the degree of the correlation between the tracking target candidate region and the peripheral region. Therefore, a number of the peripheral regions to be set may be, for example, three or six. Further, the peripheral regions may be set not only in the four oblique directions but also in vertical and lateral directions.
- In the exemplary embodiment, as an example, the face area detected by the
face detection unit 106 is set as the tracking target region. However, as discussed in Japanese Patent Application Laid-Open No.2008-011497 operation unit 101 to specify the arbitrary tracking target region. By employing the method described above, not only a person's face but also animals and vehicles can be set as the tracking target. - Alternatively, the
tracking processing unit 115 may extract moving objects considering the differences between the frame images, and the user may operate theoperation unit 101 to select any one of the plurality of face areas. - More specifically, the present invention can be applied to the configuration that includes the
tracking processing unit 115 for determining where in a subsequent second image a region considered to be the same as the face detected in a first image exists by using the correlation between the frame images. - As described above, the exemplary embodiment of the present invention is described referring to the digital camera as an example. Instead of the digital camera, the present invention can be applied to a video camera. Moreover, it can be applied also to a computer system such as a monitoring system that receives movie data from a camera connected to the system and performs the tracking processing on the received movie data in the image processing unit.
- Further, the present invention can be achieved as follows. Firstly, a computer-readable storage medium (or recording medium) storing a program code of software for realizing functions of the exemplary embodiment described above is supplied to a system or an apparatus. A computer (or a central processing unit (CPU) or a micro processing unit (MPU)) of the system or the apparatus reads out and executes the program code stored in the storage medium.
- In this case, the program code itself read out from the storage medium realizes the functions of the present exemplary embodiment described above, and thus the storage medium storing the program code constitutes the present invention.
- Furthermore, in addition to execution of the program code read out by the computer, the function of the exemplary embodiment described above can be achieved as follows. More specifically, an operating system (OS) running on the computer performs a part of or all of the actual processing based on instructions of the read out program code, and the above-described functions of the exemplary embodiment can be realized by the processing.
- The computer-readable storage medium for storing the program code includes, for example, a floppy disk, a hard disk, a read only memory (ROM), a random access memory (RAM), a magnetic tape, a non-volatile memory card, a compact disk ROM (CD-ROM), a CD recordable (CD-R), a digital versatile disk (DVD), an optical disk, and a magneto-optical (MO) disk. Further, a computer network such as a local area network (LAN) and a wide area network (WAN) can be used for supplying the program code.
- While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments but by the scope of the appended claims.
Claims (7)
- An image processing apparatus for performing tracking processing in frame images that constitute a movie by detecting in a second image an object existing in a tracking target region set in a first image, the image processing apparatus comprising:setting means configured to set the tracking target region in the first image;tracking means (115) configured to compare an evaluation value which is computed from image information in the tracking target region set by the setting means and an evaluation value which is computed from image information in each region in the second image, and to detect in the second image a tracking target candidate region which has a highest correlation with the tracking target region based on the comparison result;determination means (116) configured to calculate differences between the evaluation value acquired in the tracking target candidate region and an evaluation value which is computed from image information in each of peripheral regions of the tracking target candidate region, wherein each of the peripheral regions is acquired by being set to a region that results from shifting the tracking target candidate region by a shift amount; andcontrol means (102) configured to update the tracking target region with the tracking target candidate region if at least one of the differences is equal to or more than a predetermined threshold value and to cause the setting means to set a new tracking target region in a third image if each of the differences is less than the predetermined threshold value.
- The image processing apparatus according to claim 1,
wherein the setting means is configured to set a region where a person's face is detected, as the tracking target region. - The image processing apparatus according to claim 1 or claim 2,
wherein the setting means is further configured to set the peripheral region about the tracking target candidate region based on a size of the tracking target region. - The image processing apparatus according to any preceding claim,
wherein the setting means is further configured to set the peripheral region about the tracking target candidate region based on a movement of the tracking target region. - A method for processing an image to perform tracking processing in frame images that constitute a movie by detecting in a second image an object existing in a tracking target region set in a first image, the method comprising:setting the tracking target region in the first image;comparing an evaluation value which is computed from image information in the set tracking target region and an evaluation value which is computed from image information in each region in the second image;detecting in the second image a tracking target candidate region which has a highest correlation with the tracking target region based on the comparison result;acquiring peripheral regions, each of which is acquired by being set to a region that results from shifting the tracking target candidate region by a shift amount;calculating differences between the evaluation value acquired in the tracking target candidate region and an evaluation value computed from image information in each of the peripheral regions of the tracking target candidate region; andupdating the tracking target region with the tracking target candidate region if at least one of the differences is equal to or more than a predetermined threshold value and causing the setting of a new tracking target region in a third image if each of the differences is less than the predetermined threshold value.
- A program which, when run on a computer, causes the computer to act as an image processing apparatus according to any one of claims 1 to 4.
- A computer-readable storage medium storing a program according to claim 6.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2008211842 | 2008-08-20 | ||
JP2009155684A JP5279635B2 (en) | 2008-08-20 | 2009-06-30 | Image processing apparatus, image processing method, and program |
Publications (3)
Publication Number | Publication Date |
---|---|
EP2157547A2 EP2157547A2 (en) | 2010-02-24 |
EP2157547A3 EP2157547A3 (en) | 2014-04-09 |
EP2157547B1 true EP2157547B1 (en) | 2016-10-19 |
Family
ID=41403948
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP09168333.4A Not-in-force EP2157547B1 (en) | 2008-08-20 | 2009-08-20 | Object tracking |
Country Status (5)
Country | Link |
---|---|
US (1) | US8284994B2 (en) |
EP (1) | EP2157547B1 (en) |
JP (1) | JP5279635B2 (en) |
KR (1) | KR101206389B1 (en) |
CN (1) | CN101656830B (en) |
Families Citing this family (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2355492B1 (en) * | 2009-10-07 | 2018-04-11 | Panasonic Intellectual Property Corporation of America | Device, method, program, and circuit for selecting subject to be tracked |
JP5631181B2 (en) * | 2010-12-01 | 2014-11-26 | キヤノン株式会社 | Image processing apparatus, control method therefor, and program |
JP5825851B2 (en) * | 2011-05-27 | 2015-12-02 | キヤノン株式会社 | Imaging apparatus and control method thereof |
US8929598B2 (en) * | 2011-06-29 | 2015-01-06 | Olympus Imaging Corp. | Tracking apparatus, tracking method, and storage medium to store tracking program |
US9258462B2 (en) * | 2012-04-18 | 2016-02-09 | Qualcomm Incorporated | Camera guided web browsing based on passive object detection |
JP6525545B2 (en) * | 2014-10-22 | 2019-06-05 | キヤノン株式会社 | INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND COMPUTER PROGRAM |
JP6743337B1 (en) * | 2019-06-04 | 2020-08-19 | エスゼット ディージェイアイ テクノロジー カンパニー リミテッドSz Dji Technology Co.,Ltd | Control device, imaging device, imaging system, control method, and program |
CN110414447B (en) * | 2019-07-31 | 2022-04-15 | 京东方科技集团股份有限公司 | Pedestrian tracking method, device and equipment |
KR20210024859A (en) * | 2019-08-26 | 2021-03-08 | 삼성전자주식회사 | Method and apparatus of image processing |
CN112561963A (en) * | 2020-12-18 | 2021-03-26 | 北京百度网讯科技有限公司 | Target tracking method and device, road side equipment and storage medium |
Family Cites Families (13)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6724915B1 (en) * | 1998-03-13 | 2004-04-20 | Siemens Corporate Research, Inc. | Method for tracking a video object in a time-ordered sequence of image frames |
US6980690B1 (en) * | 2000-01-20 | 2005-12-27 | Canon Kabushiki Kaisha | Image processing apparatus |
JP3846553B2 (en) * | 2001-03-30 | 2006-11-15 | 三菱電機株式会社 | Image processing device |
US7899208B2 (en) * | 2004-01-06 | 2011-03-01 | Sony Corporation | Image processing device and method, recording medium, and program for tracking a desired point in a moving image |
US7710498B2 (en) * | 2004-02-13 | 2010-05-04 | Sony Corporation | Image processing apparatus, image processing method and program |
JP4819380B2 (en) * | 2004-03-23 | 2011-11-24 | キヤノン株式会社 | Surveillance system, imaging setting device, control method, and program |
JP4830650B2 (en) * | 2005-07-05 | 2011-12-07 | オムロン株式会社 | Tracking device |
ATE546800T1 (en) * | 2005-07-05 | 2012-03-15 | Omron Tateisi Electronics Co | TRACKING DEVICE |
CN100417223C (en) * | 2005-12-30 | 2008-09-03 | 浙江工业大学 | Intelligent safety protector based on omnibearing vision sensor |
US7856360B2 (en) * | 2006-01-30 | 2010-12-21 | Hoozware, Inc. | System for providing a service to venues where people aggregate |
JP2008011497A (en) | 2006-06-01 | 2008-01-17 | Canon Inc | Camera apparatus |
JP4930509B2 (en) * | 2006-08-30 | 2012-05-16 | 株式会社ニコン | Image tracking device, image tracking method, and camera |
JP5268433B2 (en) * | 2008-06-02 | 2013-08-21 | キヤノン株式会社 | IMAGING DEVICE AND IMAGING DEVICE CONTROL METHOD |
-
2009
- 2009-06-30 JP JP2009155684A patent/JP5279635B2/en active Active
- 2009-08-03 KR KR1020090071184A patent/KR101206389B1/en active IP Right Grant
- 2009-08-11 US US12/539,036 patent/US8284994B2/en active Active
- 2009-08-20 EP EP09168333.4A patent/EP2157547B1/en not_active Not-in-force
- 2009-08-20 CN CN2009101629772A patent/CN101656830B/en not_active Expired - Fee Related
Also Published As
Publication number | Publication date |
---|---|
US8284994B2 (en) | 2012-10-09 |
JP2010074815A (en) | 2010-04-02 |
CN101656830A (en) | 2010-02-24 |
KR101206389B1 (en) | 2012-11-29 |
KR20100022918A (en) | 2010-03-03 |
JP5279635B2 (en) | 2013-09-04 |
US20100046798A1 (en) | 2010-02-25 |
EP2157547A3 (en) | 2014-04-09 |
EP2157547A2 (en) | 2010-02-24 |
CN101656830B (en) | 2013-01-16 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP2157547B1 (en) | Object tracking | |
EP2015251B1 (en) | Object extracting, object tracking and image synthesizing | |
EP2211306B1 (en) | Imaging apparatus, subject tracking method and storage medium | |
JP2007072606A (en) | Image processor and processing method | |
JP2007279601A (en) | Camera | |
US20180007254A1 (en) | Focus adjusting apparatus, focus adjusting method, and image capturing apparatus | |
JP5289993B2 (en) | TRACKING DEVICE AND TRACKING METHOD | |
JP2007067559A (en) | Image processing method, image processing apparatus, and control method of imaging apparatus | |
JP2009111716A (en) | Imaging apparatus, program and template generating method | |
JP5118590B2 (en) | Subject tracking method and imaging apparatus | |
US20230360229A1 (en) | Image processing apparatus, image capturing apparatus, control method, and storage medium | |
US20220094840A1 (en) | Focus adjustment apparatus and method, and image capturing apparatus | |
US12008773B2 (en) | Object tracking apparatus and control method thereof using weight map based on motion vectors | |
US11750912B2 (en) | Main subject tracking and prioritization using depth and successive temporal location proximity | |
JP4983479B2 (en) | Imaging device | |
JP5111293B2 (en) | Imaging apparatus and control method thereof | |
JP2019041414A (en) | Electronic apparatus, imaging apparatus, and program | |
JP2014023054A (en) | Image processing apparatus, imaging apparatus, and image processing program | |
US20240073522A1 (en) | Image capturing apparatus, control method thereof, and storage medium | |
US20220309706A1 (en) | Image processing apparatus that tracks object and image processing method | |
JP5965653B2 (en) | TRACKING DEVICE AND TRACKING METHOD | |
JP5323243B2 (en) | Image processing apparatus and control method thereof | |
US20230237758A1 (en) | Image processing apparatus, control method therefor, image capturing apparatus, and storage medium | |
JP6443486B2 (en) | Image processing apparatus, imaging apparatus, and image processing program | |
US20240070877A1 (en) | Image processing apparatus, method for controlling the same, imaging apparatus, and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A2 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA RS |
|
PUAL | Search report despatched |
Free format text: ORIGINAL CODE: 0009013 |
|
AK | Designated contracting states |
Kind code of ref document: A3 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA RS |
|
RIC1 | Information provided on ipc code assigned before grant |
Ipc: G06T 7/20 20060101AFI20140304BHEP |
|
17P | Request for examination filed |
Effective date: 20141009 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20160512 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 838910 Country of ref document: AT Kind code of ref document: T Effective date: 20161115 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602009041802 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20161019 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 838910 Country of ref document: AT Kind code of ref document: T Effective date: 20161019 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170119 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170120 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170220 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170219 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602009041802 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20170119 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
26N | No opposition filed |
Effective date: 20170720 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20170820 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170831 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170831 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: ST Effective date: 20180430 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170820 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170820 Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170820 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170831 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20170820 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20181031 Year of fee payment: 10 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20090820 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20161019 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602009041802 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20161019 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200303 |