APPARATUS AND METHOD FOR SEPARATELY PROVIDING
ADDITIONAL INFORMATION ON EACH OBJECT IN DIGITAL
BROADCASTING IMAGE
Technical Field
The present invention relates to an apparatus and method for providing additional information regarding a particular object in a digital broadcast image, and more particularly, to an apparatus and method for recognizing and tracking a particular object corresponding to a user's setting in a digital broadcast image and providing additional information regarding the particular object.
Background Art
With the development of digital broadcast technology, services of providing additional information together with a digital broadcast image (hereinafter, referred to as additional information services) increase.
In such additional information services, when a motion image is displayed on a separate apparatus such as a digital TV, additional information regarding at least one object included in the motion image is also displayed on a screen together with the motion image. Conventionally, additional information regarding all objects in the motion image is provided in units of frames over time.
As described above, when additional information regarding all objects in a frame is provided, additional information regarding a particular object needed by a user cannot be provided efficiently.
Disclosure of the Invention
The present invention provides an apparatus and method for extracting a particular object designated by a user from a digital broadcast image or a normal motion image, recognizing the extracted object, and providing additional information regarding the object while the object is displayed on a screen.
According to an aspect of the present invention, there is provided an apparatus for providing additional information regarding a particular object in a digital broadcast image. The apparatus includes a motion image input unit which receives a motion image signal that is a stream of sequential unit images; a user command input unit which receives a user command; a scene transition detection unit which analyzes a motion image signal received through the motion image input unit and detects scene transition information that is information on a unit image having scene transition; a target object setting unit which receives scene transition information from the scene transition detection unit, a motion image signal from the motion image input unit, and object designation information on an object designated by a user from the user command input unit, detects a unit image corresponding to the object designation information among a unit image corresponding to the scene transition information and unit images succeeding the unit image corresponding to the scene transition information, sets a target area of the object in the detected unit image, and detects an initial position of the object; an object processing unit which receives object target area setting information resulting from the setting of the target area from the target object setting unit, scene transition information from the scene transition detection unit, and a motion image signal from the motion image input unit, sequentially extracts an object from a unit image corresponding to the object target area setting information and unit images succeeding the unit image corresponding to the object target area setting information, and tracks a moving position of the object; an additional information insertion unit which receives object tracking information resulting from tracking the object from the object processing unit, detects a range of unit images, over which the tracking of the object is performed, based on the object tracking information, and inserts additional information regarding the object in the detected range; and an output unit which converts object additional information resulting from the inserting of the additional
information to be suitable for a system, to which the object additional information will be provided, and outputs the converted object additional information to the system.
According to another aspect of the present invention, there is provided a method for providing additional information regarding a particular object in a digital broadcast image. The method includes (a) receiving a motion image signal that is a stream of sequential unit images, analyzing the motion image signal, and detecting scene transition information regarding a unit image having scene transition; (b) receiving object designation information regarding an object designated by a user, detects a unit image corresponding to the object designation information among a unit image corresponding to the scene transition information detected in step (a) and unit images succeeding the unit image corresponding to the scene transition information and preceding a unit image corresponding to next scene transition information, setting an object target area in the detected unit image; (c) sequentially extracting the object from a unit image corresponding to the object target area set in step (b) and unit images succeeding the unit image corresponding to the object target area and preceding the unit image corresponding to the next scene transition information; (d) verifying whether the object extracted from each of the unit images in step (c) exists in each unit image; (e) tracking a moving position of the object over a stream of unit images according to a verification result and detecting object tracking information; and (f) detecting a range of unit images, over which the object is tracked, based on the object tracking information, inserting additional information regarding the object in the detected range, converting the additional information regarding the object to be suitable for a system, to which the additional information will be provided, and outputting the converted additional information to the system.
Brief Description of the Drawings
FIG. 1 is a schematic block diagram of an apparatus for providing additional information regarding an object in a digital broadcast image, according to an embodiment of the present invention.
FIG. 2 is a schematic block diagram of an object processing unit according to an embodiment of the present invention.
FIG. 3 is a flowchart of a method for providing additional information regarding an object in a digital broadcast image, according to an embodiment of the present invention.
FIG. 4 is a flowchart of an operation of detecting scene transition according to an embodiment of the present invention.
FIG. 5 is a flowchart of an operation of tracking an object according to an embodiment of the present invention.
Best mode for carrying out the Invention Hereinafter, embodiments of an apparatus and method for providing additional information regarding an object in a digital broadcast image according to the present invention will be described in detail with reference to the attached drawings.
FIG. 1 is a schematic block diagram of an apparatus for providing additional information regarding an object in a digital broadcast image, according to an embodiment of the present invention. Referring to FIG. 1 , the apparatus according to the embodiment of the present invention includes a motion image input unit 100, a scene transition detection unit 110, a target object setting unit 120, an object processing unit 130, an additional information insertion unit 140, an output unit 150, a first buffer 160, a second buffer 170, and a user command input unit 180.
The motion image input unit 100 receives a motion image signal, i.e., a stream of sequential unit images, for example, frame_1 through frame_n. The user command input unit 180 receives a user's command signals (for example, an object designation signal and an object tracking stop request signal).
The scene transition detection unit 110 sequentially receives the unit images, for example, frame_1 through frame_n, constituting the motion image signal from the motion image input unit 100 and stores them in the first buffer 160. The scene transition detection unit 110 also compares a unit image (e.g., frame_t) currently stored in the first buffer 160 with a unit image (e.g., frame_(t-3)), which corresponds to scene transition information and has already been stored in the second buffer 170, detects a unit image having scene transition according to a comparison result, and stores scene transition information regarding the detected unit image in the second buffer 170.
When there is no scene transition information stored in the second buffer 170, the scene transition detection unit 110 determines the unit image (e.g., frame_t) currently stored in the first buffer 160 as a unit image having scene transition. Thereafter, the scene transition detection unit 110 stores the scene transition information regarding the unit image (e.g., frame_t) in the second buffer 170 and simultaneously stores a next unit image (e.g., frame_(t+1)) in the first buffer 160. Next, the scene transition detection unit 110 compares the unit image (e.g., frame_(t+1)) currently stored in the first buffer 160 with the unit image (e.g., frame_t) corresponding to scene transition information stored in the second buffer 170.
In other words, when a histogram difference calculated by comparing the unit image (e.g., frame_t) stored in the first buffer 160 with the unit frame (e.g., frame_(t-3)) corresponding to scene transition information stored in the second buffer 170 is greater than a predetermined threshold value, the scene transition detection unit 110 detects the unit image (e.g., frame_t) stored in the first buffer 160 as a unit image having scene transition and stores scene transition information regarding the detected unit image (e.g., frame_t) in the second buffer 170.
In addition, the scene transition detection unit 110 determines whether the unit image (e.g., frame_t) stored in the first buffer 160 is the last frame (e.g., frame_n) of the motion image signal. When it is determined that the unit image stored in the first buffer 160 is the last frame of the motion image signal, the scene transition detection unit 110 terminates scene transition detection with respect to the motion image signal. However, when it is determined that the unit image stored in the first buffer 160 is not the last frame of the motion image signal, the scene transition detection unit 110 stores a next unit image (e.g., frame_(t+1)) in the first buffer 160 and compares the unit image (e.g., frame_(t+1)) with a unit image (e.g., frame_t), which is stored in the second buffer 170 and corresponds to scene transition information.
Meanwhile, when the histogram difference calculated by comparing the unit image (e.g., frame_t) stored in the first buffer 160 with the unit frame (e.g., frame_(t-3)) corresponding to scene transition information stored in the second buffer 170 does not exceed the predetermined threshold value, the scene transition detection unit 110 determines that there is no scene transition.
Next, the scene transition detection unit 110 determines whether the unit image (e.g., frame_t) stored in the first buffer 160 is the last frame (e.g., frame i) of the motion image signal. When it is determined that the unit image stored in the first buffer 160 is the last frame of the motion image signal, the scene transition detection unit 110 terminates scene transition detection with respect to the motion image signal. However, when it is determined that the unit image stored in the first buffer 160 is not the last frame of the motion image signal, the scene transition detection unit 110 stores a next unit image (e.g., frame_(t+1)) in the first buffer 160 and compares the unit image (e.g., frame_(t+1)) with a unit image (e.g., frame_(t-3)), which is stored in the second buffer 170 and corresponds to scene transition information.
Various apparatuses and methods already known can be used to detect a unit image having scene transition. A representative of scene transition information detected by the scene transition detection unit 110 is a frame number corresponding to a unit image having scene transition. The target object setting unit 120 receives scene transition information from the scene transition detection unit 110, a motion image signal from the motion image input unit 100, and object designation information regarding object designated by a user from the user command input unit 180. Next, the target object setting unit 120 detects a unit image (e.g., frame_(t+1)) corresponding to the object designation information among a unit image (e.g., frame_t) corresponding to the scene transition information and unit images (e.g., frame_(t+1) through frame_(t+20)) which are input following the unit image (e.g., frame_t) corresponding to the current scene transition information and preceding a unit image (e.g., frame_(t+21)) corresponding to next scene transition information. Next, the target object setting unit 120 sets a target area of an object (hereinafter, referred to as an object target area) in the detected unit image (e.g., frame_(t+1)) and detects an initial position of the object. Thereafter, the target object setting unit 120 transmits object target area setting information, i.e., information on the unit image (e.g., frame_(t+1)) in which the object target area has been set, to the object processing unit 130. The object target area setting information may include the object target area and a frame number of the unit image where the object target area has been set.
The object processing unit 130 receives scene transition information from the scene transition detection unit 110, object target area setting information from the target object setting unit 120, and a motion image signal from the motion image input unit 100. Next, the object processing unit 130 sequentially extracts an object from each of a unit image (e.g., frame_(t+1)) corresponding to the object target area
setting information and unit images (e.g., frame_(t+2) through frame_(t+20)) which are input following the unit image (e.g., frame_(t+1)) corresponding to the object target area setting information and preceding a unit image (e.g., frame_(t+21)) corresponding to next scene transition information. Next, the object processing unit 130 tracks a motion of the extracted object over a stream of the sequential unit images (e.g., frame_(t+1) through frame (t+20)) and transmits tracking information of the object (hereinafter, referred to as object tracking information) to the additional information insertion unit 140. The object tracking information may include frame numbers of unit images, over which the object is extracted and tracked, and basic information regarding the object (such as, a name of the object).
Hereinafter, it is assumed that the object processing unit 130 performs extraction and tracking of an object with respect to unit images (e.g., frame_(t+1) through frame (t+20)) from a unit image (e.g., frame_(t+1)) corresponding to the object target area setting information received from the target object setting unit 120 to a unit image (e.g. frame (t+20)) preceding a unit image (e.g., frame_(t+21)) corresponding to next scene transition information received from the scene transition detection unit 110.
The additional information insertion unit 140 receives object tracking information from the object processing unit 130, detects a range of unit images (e.g., frame_(t+1) through frame (t+20)), over which an object has been tracked, based on the object tracking information, and inserts predetermined additional information regarding the object into the range of the unit images. Various apparatuses and methods already known can be used to insert the additional information regarding the object into the range corresponding to the unit images with respect to which the object has been extracted and tracked. The output unit 150 receives object additional information as an insertion result from the additional information insertion unit 140, converts
the object additional information to be suitable for a system (such as a digital TV, a mobile apparatus, or video on demand (VOD)) to which the object additional information will be provided, and outputs the converted object additional information to the system. FIG. 2 is a schematic block diagram of the object processing unit
130 according to an embodiment of the present invention. Referring to FIG. 2, the object processing unit 130 includes an object extractor 131 , an object recognizer 132, an object tracker 133, and an object management database (DB) 134. The object extractor 131 receives scene transition information from the scene transition detection unit 110 shown in FIG. 1 , object target area setting information from the target object setting unit 120 shown in FIG. 1 , and a motion image signal from the motion image input unit 100 shown in FIG. 1. The object extractor 131 sequentially extracts an object from each of a unit image (e.g., frame_(t+1)) corresponding to the object target area setting information and unit images (e.g., frame_(t+2) through frame_(t+20)) which are input following the unit image (e.g., frame_(t+1)) corresponding to the object target area setting information and preceding a unit image (e.g., frame_(t+21)) corresponding to next scene transition information. Next, the object extractor 131 transmits information on a unit image from which an object is extracted, i.e., object extraction information, to the object recognizer 132. The object extraction information includes basic information regarding the extracted object and a frame number of a unit image from which the object is extracted.
While sequentially extracting an object from each of the unit image (e.g., frame_(t+1)) corresponding to the object target area setting information and the unit images (e.g., frame_(t+2) through frame_(t+20)) which are input following the unit image (e.g., frame_(t+1)) corresponding to the object target area setting information and preceding the unit image (e.g., frame_(t+21)) corresponding to the next scene transition
information, when it is determined that a current unit image (e.g., frame_(t+19)) is the last frame of the motion image signal, the object extractor 131 terminates object extraction immediately after extracting an object from the current unit image (e.g., frame_(t+19)). Various apparatuses and methods already known can be used to extract an object from a unit image.
The object recognizer 132 receives object extraction information from the object extractor 131 and a motion image signal from the motion image input unit 100 and verifies whether an object exists in a unit image (e.g., frame_(t+1)) corresponding to the object extraction information based on the object management DB 134. The object management DB 134 stores basic information (e.g., a name of an object) regarding all objects existing in a motion image.
When an object is verified as existing in the unit image (e.g., frame_(t+1)) corresponding to the object extraction information, the object recognizer 132 transmits object recognition information to the object tracker 133. However, whet an object is not verified as existing in the unit image (e.g., frame_(t+1)) corresponding to the object extraction information, the object recognizer 132 requests the target object setting unit 120 to reset an object. Then, the target object setting unit 120 requests a user to newly designate an object and repeats the-above described operations with respect to the newly designated object.
Similarly, the object recognizer 132 sequentially receives object extraction information from the object extractor 131 and verifies whether an object exists in each of the unit images (e.g., frame_(t+2) through frame_(t+20)) corresponding to the object extraction information.
The object tracker 133 sequentially receives object recognition information regarding each of unit frames (e.g., frame_(t+1) through frame_(t+20)) from the object recognizer 132 and a motion image signal from the motion image input unit 100, tracks a moving position of an object over a stream of the unit images (e.g., frame_(t+1) through
frame_(t+20)) corresponding to the sequentially received object recognition information, and outputs object tracking information according to the motion of the object.
While tracking the object, the object tracker 133 compares a size of the object in a previous unit image (e.g., frame_(t+10)) with a size of the object in a current unit image (e.g., frame_(t+11)). When a difference between the object size in the previous unit image and the object size in the current unit image is greater than a predetermined reference value, the object tracker 133 determines that the size of the object has changed and performs object size compensation with respect to the current unit image (e.g., frame_(t+11)) before performing object tracking over a next unit image (e.g., frame_(t+12)).
When object tracking stop request information generated by a user requesting stop of object tracking is received from the user command input unit 180 during object tracking, the object tracker 133 outputs object tracking information based on a result of tracking a moving position of the object over unit images (e.g., frame_(t+1) through frame_(t+18)) from the unit image (e.g., frame__(t+1)) corresponding to the object recognition information_to a unit image (e.g., frame_(t+18)) corresponding to the object tracking stop request information.
When the object tracker 133 cannot track the motion of the object in a current unit image (e.g., frame_(t+18)), it determines that the object disappears and outputs object tracking information based on a result of tracking a moving position of the object over unit images (e.g., frame_(t+1) through frame_(t+17)) from the unit image (e.g., frame_(t+1)) corresponding to the object recognition information o a unit image (e.g., frame_(t+17)) preceding the current unit image (e.g., frame_(t+18)).
When it is determined that a current frame (e.g., frame_(t+19)) is the last frame of the motion image signal during object tracking, the object tracker 133 outputs object tracking information based on a result of tracking a moving position of the object over unit images (e.g.,
frame_(t+1) through frame_(t+19)) from the unit image (e.g., frame_(t+1)) corresponding to the object recognition information to the current unit image (e.g., frame_(t+19)).
FIG. 3 is a flowchart of a method for providing additional information regarding an object in a digital broadcast image, according to an embodiment of the present invention. Referring to FIG. 3, a motion image signal that is a stream of sequential unit images is received in step S100. Information on a unit image (e.g., frame_t) having scene transition, i.e., scene transition information, is detected by analyzing the motion image signal in step S110.
When object designation information is input by a user in step S120, among a unit image (e.g., frame_t) corresponding to the scene transition information detected in step S110 and unit images (e.g., frame_(t+1) through frame_(t+20)) which are input following the unit image (e.g., frame_t) corresponding to the current scene transition information and preceding a unit image (e.g., frame_(t+21)) corresponding to next scene transition information, a unit image (e.g., frame_(t+1)) corresponding to the object designation information is detected, an object target area is set in the detected unit image (e.g., frame_(t+1)), and an initial position of an object is detected in step S130.
Next, based on the scene transition information detected in step
S110 and information on a unit image (e.g., frame_(t+1)), in which the object target area has been set in step S130, i.e., object target area setting information, the object is sequentially extracted from each of a unit image (e.g., frame_(t+1)) corresponding to the object target area setting information and unit images (e.g., frame_(t+2) through frame_(t+20)) which are input following the unit image (e.g., frame_(t+1)) corresponding to the object target area setting information and preceding the unit image (e.g., frame_(t+21)) corresponding to the next scene transition information in step S140.
It is verified whether the object extracted from the unit images (e.g., frame_(t+1) through frame_(t+20)) exists in each of the unit images (e.g., frame_(t+1) through frame_(t+20)) in step S150. When the object is recognized in all of the unit images (e.g., frame_(t+1) through frame_(t+20)) as a result of verification in step S150, a moving position of the object is tracked over the stream of the unit images in step S160. When the object is not recognized in step S150, the method goes to step S120.
Based on object tracking information obtained as a result of tracking the moving position of the object in step S160, a range of the unit images (e.g., frame_(t+1) through frame_(t+20)), over which the object has been tracked, is detected, and additional information regarding the object is inserted into the detected range in step S170.
A result of inserting the additional information regarding the object, i.e., object additional information, is converted suitable for a system to which the object additional information will be provided, and the converted object additional information is output to the system in step S180.
FIG. 4 is a flowchart of an operation of detecting scene transition in step S110, according to an embodiment of the present invention. Referring to FIG. 4, a unit image is received in step S111 and then stored in the first buffer 160 shown in FIG. 1 in step S112. When there is scene transition information that is information on a unit image (e.g., frame_(t-3)) having scene transition in the second buffer 170 shown in FIG. 1 in step S113, the unit image (e.g., frame_t) stored in the first buffer 160 is compared with a unit image (e.g., frame_(t-3)) corresponding to the scene transition information stored in the second buffer 170 in step S115.
However, when there is no scene transition information in the second buffer 170 in step S113, the unit image (e.g., frame_t) stored in the first buffer 160 is determined as having scene transition, and scene
transition information regarding the unit image (e.g., frame_t) stored in the first buffer 160 is stored in the second buffer 170 in step S114. Next, the operation goes to step S111 in which a unit image (e.g., frame_(t+1)) succeeding the unit image (e.g., frame_t) stored in the first buffer 160 is received. Next, steps S112 and S113 are repeated.
Thereafter, the unit image (e.g., frame_t) stored in the first buffer 160 is compared with a unit image (e.g., frame_(t-3)) corresponding to the scene transition information stored in the second buffer 170 in step S115. As a result of comparing the two unit images in step S115, when it is determined that a histogram difference between the two unit images is greater than a predetermined threshold value in step S116, the unit image (e.g., frame_t) stored in the first buffer 160 is determined as having scene transition, and scene transition information regarding the unit image (e.g., frame_t) stored in the first buffer 160 is stored in the second buffer 170 so that the scene transition information stored in the second buffer 170 is updated in step S117.
Next, it is determined whether the unit image (e.g., frame_t) stored in the first buffer 160 is the last frame of the motion image signal in step S118. When it is determined that the unit image (e.g., frame_t) stored in the first buffer 160 is the last frame of the motion image signal, the operation ends. However, when it is determined that the unit image (e.g., frame_t) stored in the first buffer 160 is not the last frame of the motion image signal, the operation goes to step S111 in which a unit image (e.g., frame_(t+1)) succeeding the unit image (e.g., frame_t) stored in the first buffer 160 is received, and then steps S112 through S118 are performed.
However, when the histogram difference between the unit image (e.g., frame_t) stored in the first buffer 160 and the unit image (e.g., frame_(t-3)) corresponding to the scene transition information stored in the second buffer 170 does not exceed the predetermined threshold
value in step S116, the unit image (e.g., frame_t) is determined as not having scene transition, and it is determined whether the unit image (e.g., frame_t) stored in the first buffer 160 is the last frame of the motion image signal in step S118. Detecting scene transition from a motion image signal and steps shown in FIG. 4 are techniques already known in the field of the present invention, and thus various known techniques can be selectively used.
FIG. 5 is a flowchart of an operation of tracking the object in step S160, according to an embodiment of the present invention. Referring to FIG. 5, after It is verified whether the object extracted from the unit images (e.g., frame_(t+1) through frame_(t+20)) exists in each of the unit images (e.g., frame_(t+1) through frame_(t+20)) in step S150, a moving position of the object is tracked over the stream of the unit images (e.g., frame_(t+1) through frame_(t+20)) in step S161. When object tracking end information is received in step S162, the moving position of the object is tracked up to a specified unit image according to the object tracking end information, and a result of tracking the object is output as object tracking information.
In other words, when the object tracking end information indicates object tracking stop request information generated by a user, the moving position of the object is tracked up to a current unit image (e.g., frame_(t+18)) corresponding to the object tracking stop request information in step S163, and a result of tracking the object is output as object tracking information in step S164. When object tracking end information indicates that the object disappears form a current unit image (e.g., frame_(t+18)), the moving position of the object is tracked up to a unit image (e.g., frame_(t+17)) preceding the current unit image (e.g., frame_(t+18)) from which the object disappears in step S165, and object tracking information is output in step S166.
When object tracking end information is information on the last frame of the motion image signal, it is determined whether a current unit image (e.g., frame_(t+19)), on which object tracking is to be performed next, is the last frame of the motion image signal in step S167. When it is determined that the current unit image (e.g., frame_(t+19)) is the last frame of the motion image signal, the moving position of the object is tracked up to the current unit image (e.g., frame_(t+19)) corresponding to the last frame in step S168, and object tracking information is output in step S169. However, when the current unit image (e.g., frame_(t+19)) is not the last frame of the motion image signal, the operation goes to step S140 shown in FIG. 3.
While this invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes may be made therein without departing from the scope of the invention. Therefore, the above-described embodiments will be considered not in restrictive sense but in descriptive sense only. The scope of the invention will be defined not by the above description but by the appended claims, and it will be construed that all differences made within the scope defined by the claims are included in the present invention.
Industrial Applicability
As described above, in an apparatus and method for providing additional information regarding a particular object in a digital broadcast image according to the present invention, a particular object in a digital broadcast motion image is extracted, recognized, and tracked so that additional information regarding the particular object can be provided.
The present invention can be applied to normal motion images as well as digital broadcast images. In addition, since the present invention enables providing additional information regarding a particular object among objects
appearing in a motion image, the present invention can be widely used in service systems providing detailed information regarding goods online, T-commerce systems, etc.
Moreover, the present invention has an incidental effect of indirectly advertising an object regarding which additional information is provided.