WO2013124923A1 - 興味区間特定装置、興味区間特定方法、興味区間特定プログラム - Google Patents
興味区間特定装置、興味区間特定方法、興味区間特定プログラム Download PDFInfo
- Publication number
- WO2013124923A1 WO2013124923A1 PCT/JP2012/006717 JP2012006717W WO2013124923A1 WO 2013124923 A1 WO2013124923 A1 WO 2013124923A1 JP 2012006717 W JP2012006717 W JP 2012006717W WO 2013124923 A1 WO2013124923 A1 WO 2013124923A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- interest
- section
- specificity
- frame
- motion feature
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims description 47
- 230000033001 locomotion Effects 0.000 claims abstract description 232
- 238000004364 calculation method Methods 0.000 claims abstract description 87
- 238000001514 detection method Methods 0.000 claims abstract description 39
- 230000003287 optical effect Effects 0.000 claims description 25
- 238000000605 extraction Methods 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 8
- 239000000284 extract Substances 0.000 claims description 5
- 238000003860 storage Methods 0.000 claims description 4
- 230000006870 function Effects 0.000 description 25
- 230000008859 change Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 239000000047 product Substances 0.000 description 7
- 239000013598 vector Substances 0.000 description 5
- 241001465754 Metazoa Species 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000003384 imaging method Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 230000000386 athletic effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000004091 panning Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/144—Movement detection
- H04N5/145—Movement estimation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/269—Analysis of motion using gradient-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/255—Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
- G06V20/47—Detecting features for summarising video content
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/102—Programmed access in sequence to addressed parts of tracks of operating record carriers
- G11B27/105—Programmed access in sequence to addressed parts of tracks of operating record carriers of operating discs
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
- G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
- G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/44—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
- H04N21/44008—Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
- H04N21/845—Structuring of content, e.g. decomposing content into time segments
- H04N21/8456—Structuring of content, e.g. decomposing content into time segments by decomposing the content in the time domain, e.g. in time segments
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8549—Creating video summaries, e.g. movie trailer
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/79—Processing of colour television signals in connection with recording
- H04N9/87—Regeneration of colour television signals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B2220/00—Record carriers by type
- G11B2220/20—Disc-shaped record carriers
- G11B2220/25—Disc-shaped record carriers characterised in that the disc is based on a specific recording technology
- G11B2220/2537—Optical discs
- G11B2220/2541—Blu-ray discs; Blue laser DVR discs
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
- G11B2220/00—Record carriers by type
- G11B2220/20—Disc-shaped record carriers
- G11B2220/25—Disc-shaped record carriers characterised in that the disc is based on a specific recording technology
- G11B2220/2537—Optical discs
- G11B2220/2562—DVDs [digital versatile discs]; Digital video discs; MMCDs; HDCDs
Definitions
- the present invention relates to an interest section specifying device for specifying a section that attracts a user's interest in a moving image.
- the digest that the user is interested in is considered to be a moving image
- the motion of the person is detected in the moving image for which the digest is to be created (for example, see Patent Document 1). It is conceivable to extract a section where a person moves.
- the present invention has been made in view of the above problems, and an object thereof is to provide an interest section specifying device that is useful for creating an interesting digest video for a user.
- an interest interval specifying device is an interest interval specifying device that specifies an interest interval that is estimated to be of interest to a user in a moving image, and an object is extracted from a frame extracted from the moving image.
- Object detection means for detecting, motion feature quantity extraction means for calculating a motion feature quantity of the object with respect to the extracted frame, and specificity of motion of the object in the extracted frame using the motion feature quantity of the object
- Specificity calculating means for calculating the interest interval
- interest interval specifying means for specifying the interest interval in the moving image using the specificity.
- An interest section specifying method is an interest section specifying method by an interest section specifying device that specifies an interest section that is presumed to be of interest to a user in a moving image, and an object is extracted from a frame extracted from the moving image.
- An object detection step for detecting; a motion feature amount extracting step for calculating a motion feature amount of the object with respect to the extracted frame; and a specificity of the motion of the object in the extracted frame using the motion feature amount of the object
- a specificity calculating step for calculating the interest interval, and an interest interval specifying step for specifying the interest interval in the moving image using the specificity.
- the interest section specifying program is an interest section specifying program for causing a computer to execute an interest section specifying process for specifying an interest section that is presumed to be interesting by a user in a moving image.
- an object detection step for detecting an object from a frame extracted from the moving image
- a motion feature amount extraction step for calculating a motion feature amount of the object with respect to the extracted frame, and using the motion feature amount of the object
- the method includes a specificity calculation step of calculating the specificity of the movement of the object in the extracted frame, and an interest interval specifying step of specifying the interest interval in the moving image using the specificity.
- the interest interval specifying device specifies the interest interval based on the specificity of the motion of the object in the extracted frame. Therefore, the convenience of creating a digest video interesting for the user can be improved.
- FIG. 10 is a flowchart showing the operation of the interested section identification device according to the second embodiment.
- 12 is a flowchart showing an operation of calculating specificity of the interested area identification device according to the second embodiment.
- the inventor uses the movement of an object as in the technique described in Patent Document 1 to specify the section of interest that the user is likely to be interested in the video for creating the digest video. Considering. However, it has been found that when only the movement of an object is used for specifying an interest section, even a monotonous movement (for example, a movement just walking) may be specified as an interest section. For example, when the technique described in Patent Document 1 is used, the same repeated video (for example, video in which the subject person keeps walking) is extracted, and a digest video that is not interesting to the user is created. There is a problem that will be done.
- FIG. 1 is a block diagram illustrating a functional configuration of the interested section identification device 100. Note that FIG. 1 also shows peripheral devices of the section of interest specifying device 100.
- the interested section specifying device 100 is connected to a photographing device 120 and a display device 130.
- the interest section specifying device 100 extracts the interest section from the moving image photographed by the photographing device 120 and causes the display device 130 to display it.
- the interest section is a moving image having a predetermined fixed time length (for example, 3 minutes).
- the photographing device 120 is a device having a function of photographing and recording a moving image such as a movie camera or a digital camera.
- the imaging device 120 is connected to the section of interest specifying device 100 via a USB (UniversalUSBSerial Bus) cable, for example.
- USB UniversalUSBSerial Bus
- the display device 130 is a monitor having a function of displaying an image such as a digital television, an LCD (Liquid Crystal Display), or a PDP (Plasma Display Panel).
- the display device 130 is connected to the section of interest specifying device 100 via, for example, a USB cable.
- the interested section identification device 100 includes a moving image acquisition unit 101, an object detection unit 102, an object tracking unit 103, a region determination unit 104, a motion feature amount calculation unit 105, and a specificity calculation unit. 106, an interesting section specifying unit 107, and an interesting section extracting unit 108.
- the moving image acquisition unit 101 has a function of acquiring a moving image shot by the shooting device 120 from the shooting device 120.
- the moving image acquisition unit 101 includes software that controls an interface such as a USB port for connecting a USB cable and a USB driver, for example.
- the object detection unit 102 has a function of detecting an object from each frame of the moving image.
- Examples of the object include a person, a person's face, an animal, and a car.
- the object detection unit 102 detects an edge with respect to a frame, or moves a search window within the frame to classify a classifier for identifying an object within the search window.
- a face learning dictionary For example, as shown in FIG. 2, a person's face in a frame can be detected.
- the object 203 is detected from the t-th frame 201 of the moving image, and the object 204 is detected from the t + 1-th frame 202.
- the classifier is described as detecting a person's face, but the classifier may be other than a person such as an animal or a car.
- the object detection unit 102 may hold a plurality of classifiers in order to detect a plurality of different types, and when an object is detected from a frame, the classifier used to detect the object It may also have a function of giving information on what is to be detected as metadata to the detected object.
- the object tracking unit 103 tracks the position of each of the one or more objects detected by the object detection unit 102 in the next frame.
- the object tracking unit 103 has a function of assigning the same identifier (object ID) to objects that are assumed to be the same in each frame as a result of the tracking.
- object ID the same identifier
- An example of the object tracking method will be described with reference to FIG.
- the areas of the detected object 203 and the object 204 in the frames 201 and 202 which are continuous frames have almost the same position in the respective frames.
- 204 is the same object, and is assigned the same object ID. That is, in the present embodiment, in the frame, the objects whose positions in the respective frames of the regions in each frame are close to each other are specified as the same object. In this way, the object tracking unit 103 tracks an object by identifying objects included in a plurality of frames.
- the area determination unit 104 has a function of determining an area for evaluating the movement of the object in each frame of the moving image.
- the area for detecting the movement of the object includes the object to be tracked. Region determination by the region determination unit 104 will be described with reference to FIG.
- FIG. 3 is a diagram illustrating an area determined based on an object in the frame 301.
- the area determination unit 104 determines the area 302a (the left hatched portion in FIG. 3) that includes the object estimated by the object detection unit 102 as the person's face and the size of the person's face.
- a region 302b (right hatched portion in FIG. 3) estimated to have a human body is determined as a region for detecting the movement of the object.
- the size of the region 302b is determined in advance in what direction with respect to the inclination and size of the detected region 302a. In each frame, the position and range where the corresponding region exists may be common between frames or may be different.
- the area determined by the area determination unit 104 is determined as a shape and a region depending on the content of the object detected by the object detection unit 102, and the object detection unit 102 determines the area from the content of the detected object. Determine the shape and extent of the.
- the motion feature amount calculation unit 105 has a function of calculating the motion feature amount of an object in each frame included in the moving image.
- the motion feature amount calculation unit 105 identifies each pixel in the region determined by the region determination unit 104 for the frame included in the moving image (see FIG. 4).
- FIG. 4 shows each pixel in the determined region 302.
- the motion feature amount calculation unit 105 calculates an optical flow of each identified pixel (see FIG. 5).
- FIG. 5 shows an example of a part of the optical flow instead of the optical flow of all the pixels in the region.
- a gradient method is used as an example of the optical flow calculation method.
- the gradient method estimates where a certain pixel of a frame at time t is moving in the frame at time t + 1 under the assumption that “the brightness of a point on the object does not change after movement” A vector is obtained based on the amount of movement.
- the motion feature amount calculation unit 105 determines the motion feature amount of the region determined by the region determination unit 104 using the calculated optical flow. With reference to FIG. 5, a method for determining the motion feature amount of the region will be described.
- the coordinates (x, y) of the pixel at the left end of the first step in the region 302 are (a, b), the right neighbor is (a + 1, b),.
- the coordinates (x, y) of the rightmost pixel are (a + w, b), and the coordinates of the second leftmost pixel are (a, b + 1).
- the optical flow at coordinates (a, b) is (x a , y b ), the optical flow at coordinates (a + 1, b) is (x a + 1 , y b ),..., Coordinates (a + w, b) Is calculated as (x a + w , y b ), the optical flow at coordinates (a, b + 1) is (x a , y b + 1 ), and so on.
- the motion feature amount (feature amount 1, feature amount 2, feature amount 3,..., Feature amount K,...)
- the region 302 is (x a , y b , x a + 1 , y b ,..., X a + w , y b , x a , y b + 1 ,. That is, the motion feature amount of the region 302 determined by the region determination unit 104 is a set of optical flows of each pixel included in the region 302, and the region 302 has a right end from the left end of the first step, a right end from the left end of the second step,. The N-th stage left end to the right end,...
- the motion feature amount calculation unit 105 calculates the motion feature amount of each object detected for each frame, and stores the information in a memory (not shown) or the like.
- FIG. 6 is a data conceptual diagram of information on stored motion feature amounts.
- the motion feature amount information is information in which a frame number for identifying each frame is associated with a numerical value of the feature amount for each feature amount type.
- the type of feature amount is the amount of motion in the x-axis direction or the amount of motion in the y-axis direction of the motion vector of any pixel in the region.
- the numeric value of feature quantity 1 is 6, and the numeric value of feature quantity 2 is 2.
- the specificity calculation unit 106 has a function of calculating the specificity of each frame included in the moving image.
- the specificity of each frame is an index indicating how much the motion of the object included in each frame differs from the motion of the object in other frames. Details of the specificity calculation method will be described later.
- the interest interval specifying unit 107 has a function of specifying an interest interval using the specificity of each frame calculated by the specificity calculating unit 106. With reference to FIG. 7, a method for identifying an interest interval will be described.
- FIG. 7 is a graph showing changes in specificity over time of a moving image, with the time axis on the horizontal axis and the specificity on the vertical axis.
- the interest interval specifying unit 107 adds the specificities of the frames included in the window 701 while shifting the window 701 of the interest interval length from the beginning to the end of the moving image, with respect to the fixed-length interest interval.
- the window 701 at the highest point is specified as the interest section.
- the interest section specifying unit 107 gives information indicating the start point of the specified interest section and information indicating the end point of the interest section to the moving image.
- a tag indicating the start point of the interest section hereinafter referred to as a start point tag
- a tag indicating the end point of the interest section hereinafter referred to as an end point tag
- the interest interval extracting unit 108 Based on the start point tag and end point tag attached to the moving image by the interest interval specifying unit 107, the interest interval extracting unit 108 extracts an interest interval moving image from the moving image, which is a moving image having the start point tag as the start point and the end point tag as the end point. Has a function to extract.
- the output unit 109 has a function of outputting the interest section moving image extracted by the interest section extraction unit 108 to the display device 130.
- the display device 130 reproduces the interest section moving image output from the output unit 109, and the user can recognize the content of the moving image in a short time and watch the interest section moving image with interest by watching this. ⁇ Operation> Next, the operation of the section of interest specifying device 100 according to the present embodiment will be described using the flowchart shown in FIG.
- the moving image acquisition unit 101 of the interested section identification device 100 receives and acquires a moving image from the photographing device 120 via the USB cable (step S801).
- the moving image acquisition unit 101 transmits the acquired moving image to the object detection unit 102.
- the object detection unit 102 detects an object included in the image to be displayed for each frame from the transmitted video (step S802).
- the object detection unit 102 transmits information about the detected one or more objects to the object tracking unit 103.
- the object tracking unit 103 tracks, for each object detected from each frame, at which position in which frame each object exists, and identifies an object common between the frames by assigning the same object ID (Ste S803).
- the region determination unit 104 determines, for each object detected as a result of tracking by the object tracking unit 103, a region in which motion is detected for each object in each frame (step S804).
- the motion feature amount calculation unit 105 determines, for each region of each object determined by the region determination unit 104, how much the region moves from the target frame for which the motion feature amount is calculated to the next frame. Based on this, the motion feature amount of each frame is calculated (step S805). That is, the motion feature amount calculation unit 105 generates, for each frame of the moving image, a vector composed of a set of optical flows of each pixel included in the region as the motion feature amount of the frame.
- the specificity calculation unit 106 calculates the specificity of each frame using the motion feature amount of each frame calculated by the motion feature amount calculation unit 105 (step S806). Details of the calculation of the specificity of the frame will be described later using the flowchart of FIG.
- the interest section specifying unit 107 specifies the section where the sum of the specificities of consecutive frames corresponding to the length of the interest section in the video is the highest as the interest section of the video.
- Information indicating the start point and end point of the section is given (indexed) to the moving image (step S807).
- the interest interval extraction unit 108 extracts the interest interval identified by the interest interval identification unit 107 from the moving image, and transmits the extracted interest interval moving image to the output unit 109. Then, the output unit 109 outputs the transmitted interest section moving image to the display device 130 (step S808).
- the display device 130 displays the interest section moving image transmitted from the interest section specifying device 100, and the user can recognize the content of the moving image only by confirming the contents of the interest section without viewing the entire moving image. .
- FIG. 9 is a flowchart showing the motion feature amount calculation operation by the motion feature amount calculation unit 105.
- the motion feature amount calculation unit 105 calculates the optical flow of each pixel in the region 302 in each frame (step S901).
- the motion feature quantity calculation unit 105 normalizes the optical flow calculated for each frame based on the size of the object in each frame (step S902).
- the motion feature amount calculation unit 105 for example, based on the size information of the face of the object, the optical flow in each pixel is linearly interpolated when enlarging, and the average of the reduction source areas when reducing There is. This is because even when the same object is the subject, the size of the object in the moving image changes depending on the distance between the imaging device 120 and the object of the subject, and the size of the object is determined in the process of determining the motion feature amount of the object. Is determined to be constant.
- the motion feature amount calculation unit 105 calculates, for each frame, a motion feature amount composed of a set of optical flows of each pixel included in the region defined for the object included in each frame (step S903). .
- FIG. 10 is a flowchart showing the calculation operation of specificity by the specificity calculation unit 106.
- FIG. 11 is a diagram for explaining a frame to be used when calculating the specificity.
- FIG. 11 is a three-dimensional graph with time on the x-axis, type of motion feature quantity on the y-axis, and feature quantity on the z-axis, and shows the motion feature quantity of a certain object in each frame of the moving image at each time It is a conceptual diagram.
- the motion feature amount used for calculating the specificity will be described with reference to FIG.
- the specificity of the target frame for which the specificity is calculated indicates how much the motion feature quantity of the target frame is not similar to the motion feature quantity of a frame in a predetermined range before and after the target frame It is.
- the previous M frame M is an integer of 2 or more
- the subsequent M frame are defined as the first section.
- the preceding N N is an integer of 1 or more and less than M
- the following N frames are set as the second section.
- the motion feature amount of the target frame and the average value of the motion feature amounts of the frames included in the third section excluding the second section from the first section are used.
- M is 300 frames and N is 60 frames.
- 300 frames corresponds to 10 seconds of a moving image
- 60 frames corresponds to 2 seconds of a moving image.
- the specificity calculation unit 106 determines a target frame for which the specificity is calculated (step S1001). This can be done in order from the beginning to the end of the movie, in reverse order from the end to the beginning, or at random, as the specificity only needs to be calculated for all frames of the movie. May be. Here, the calculation is performed in order from the beginning of the moving image.
- the specificity calculation unit 106 selects one object included in the target frame (step S1002).
- the specificity calculation unit 106 calculates the average motion feature amount from the motion feature amount of the frame for the third section (2M-2N) with the target frame of the selected object as a reference (step S1003).
- the specificity calculation unit 106 represents the motion feature amount of the object A in the target frame k by the following formula 1.
- the average motion feature amount of the object A in the third section which is a comparison target of the specificity, is expressed by the following formula 2 for the target frame k.
- the average motion feature amount of the object A in the third section can be expressed by the following formula 3.
- the average motion feature amount of the object A in the third section used for calculating the specificity for the target frame k can be expressed by the following equation (4).
- the average motion feature amount of the object A in the target frame k when f max ⁇ N ⁇ k can be expressed by the following formula 5.
- the specificity calculation unit 106 calculates the score of the selected object in the target frame k using the mathematical formula and the average motion feature amount shown in Equation 1 (step S1004).
- the score of the object A in the target frame k can be expressed by the following formula 6.
- the absolute value of the motion feature amount of the object A in the frame k is a value obtained by adding 1 to the inner product of the motion feature amount of the object A in the frame k and the average motion feature amount of the object A in the third section with respect to the frame k.
- the result of division is used as the score of object A in frame k.
- the specificity calculation unit 106 determines whether or not the scores for all objects have been calculated in the selected symmetric frame (step S1005). If scores for all objects have not been calculated (NO in step S1005), the process returns to step S1002.
- step S1005 If the scores for all objects have been calculated (YES in step S1005), the calculated scores of the respective objects are added together to calculate the specificity of the target frame (step S1006).
- step S1007 the specificity calculation unit 106 determines whether or not the specificity for all the frames has been calculated. If the specificity for all frames has not been calculated (NO in step S1007), the process returns to step S1001.
- step S1007 If the specificity for all frames has been calculated (YES in step S1007), the specificity calculation processing is terminated.
- the interest section specifying device 100 specifies a part of the moving image that is particularly rich in the movement of the object as an interest section, and thus provides an interest section that does not bore the user. Will be able to.
- the detection is performed by comparing with the movement of the same object in another frame.
- the method for detecting the specificity of the object is not limited to this.
- a specificity detection method different from that in the first embodiment will be described.
- the description common to the first embodiment will be omitted, and different points will be described.
- the interest section specifying device 1200 includes a moving image acquisition unit 101, an object detection unit 102, a motion feature amount calculation unit 105, a specificity calculation unit 1206, and an interest section specification unit. 1207, an interest section extraction unit 108, and an output unit 109.
- the same name and number are attached
- the object detection unit 102 information on the detected object is transmitted to the motion feature amount calculation unit 105 via the object tracking unit 103 and the region determination unit 104.
- the motion feature amount calculation unit 105 calculates a motion feature amount for each object detected by the object detection unit 102 for each frame. Then, the motion feature amount calculation unit 105 transmits the calculated motion feature amount to the specificity calculation unit 1206.
- the accepting unit 1205 has a function of accepting designation of a specific object selected from the objects detected by the object detecting unit 102 from the user and transmitting information about the specific object to the specificity calculating unit 1206.
- the specificity calculation unit 1206 has a function of calculating how specific a specific object is from other objects photographed in the same frame.
- the specificity calculation unit 1206 selects a specific object received from the reception unit 1205 from among a plurality of objects detected by the object detection unit 102. Then, how specific the motion feature amount of the identified object is calculated from the motion feature amount of the other object is calculated.
- the interest section specifying unit 1207 has a function of specifying an interest section based on the specificity of a specific object in each frame calculated by the specificity calculation unit 1206. Specifically, the interest section specifying unit 1207 is a predetermined period (the moving image length that is desired to be specified as the interest section, for example, 3 minutes, for which the total value of the specificity P of each frame calculated by the specificity calculation unit 1206 is the largest. Alternatively, an equivalent number of frames) is specified as the interest interval.
- ⁇ Operation> The operation of identifying an interest interval of the interest interval identification device 1200 according to Embodiment 2 will be described using the flowcharts of FIGS. 13 and 14. In the flowcharts of FIGS.
- the motion feature amount calculation unit 105 of the interested section identification device 1200 calculates the motion feature amount of each object for each frame (step S805).
- the specificity calculation unit 1206 of the interest section identification device 1200 receives the motion feature amount calculated for each object extracted from each frame of the moving image, and calculates the specificity of the object.
- the accepting unit 1205 accepts designation of a specific object for which the specificity is calculated from the user (step S1401).
- the specificity calculation unit 1206 calculates the specificity of the specific object for the target frame (step S1404). Specifically, the specificity calculation unit 1206 calculates the specificity of the specified specific object from the average value of motion feature amounts of other objects included in the target frame for which the specificity is calculated. Specifically, it is calculated as follows.
- the motion feature amount of the identified object is expressed by the following formula 7 for the target frame for which the specificity is calculated.
- k is an index for identifying other objects.
- n there is an index n in the target frame.
- the average value vector b k, ave of the motion feature values of other objects can be expressed as the following Expression 9.
- the specificity P of the identified object in the target frame can be expressed by the following formula 10.
- the specificity calculation unit 1206 calculates the specificity P as the specificity of the target frame (step S1406).
- the specificity calculation unit 1206 calculates the specificity in each frame.
- the interest section identification unit 1207 receives the specificity for each frame from the specificity calculation unit 1206, and for a predetermined period (the period of the window 701 shown in FIG. 7), the predetermined period in which the combined value of the specificity is the highest. Is identified as the interval of interest.
- the interested section specifying device 1200 moves differently from other children by one person, for example, when the movie is a picture of a group gymnastics scene at a children's athletic meet. This is useful for extracting NG scenes.
- the interest section specifying device 1200 can specify a scene in which there are objects that are operating differently from the plurality of objects as the interest section while a plurality of objects are performing the same operation. .
- ⁇ Modification> Although the interest section specifying device according to the present invention has been described according to the above embodiment, the embodiment of the present invention is not limited to this. Hereinafter, various modifications included as the idea of the present invention will be described.
- the length of the interest section specified by the interest section specifying device 100 (1200) is determined in advance.
- the interest section length need not be fixed.
- the length may be a predetermined ratio (for example, 10%) with respect to the moving image length of the moving image from which the interest interval is extracted.
- the interest section identification device 100 (1200) may change the extracted interest section length according to the type of moving image.
- the type of moving image is a genre of moving image content such as variety, sports, drama, and news.
- the interest interval specifying device holds an interest interval length table in which the interest interval length is determined in association with the genre of the moving image, and the moving image acquisition unit 101 uses the acquired moving image metadata or the like to The type of interest is specified, and the interest interval specifying unit 107 (1200) acquires the interest interval length corresponding to the type of the specified video in the held interest interval length table, and specifies the interest interval of that length. To do.
- the interested section identifying apparatus 100 can identify a more appropriate interested section if it can identify an appropriate length of the interested section according to the type of the moving image.
- the user may input the genre to specify the interest section length, or the predetermined length may be set as the interest section length.
- the interested section specifying device 100 (1200) may include a setting unit (interface) for the user to set the interested section length.
- the interest interval specifying means specifies the interest interval of the interest interval length set by the user.
- the interest section can be set by directly entering the number of seconds of the interest section, or by inputting to an interface having a button for determining the length of the interest section, such as long, normal, or short. It is good to do. Thus, it is good also as a structure which specifies the interest area of a desired length by a user.
- the lengths of the first section and the second section for calculating the specificity of each frame are determined in advance.
- the first section length and the second section length need not be fixed.
- the interested section identification device 100 (1200) may determine the first section length and the second section length according to the type of the moving image.
- the type of moving image is a genre of moving image content such as variety, sports, drama, and news.
- the interested section specifying device 100 (1200) holds a section length table that defines the first section length and the second section length in association with the genre of the moving picture, and the moving picture acquisition unit 101 stores the acquired moving picture.
- the type of the moving image is specified, and the specificity calculation unit 106 (1206) has a first section length and a second section corresponding to the specified type of moving image in the stored section length table. Get the length and calculate the specificity.
- the interested section specifying device 100 (1200) can specify appropriate lengths of the first section and the second section according to the type of the moving image, the specificity of the more accurate value can be obtained. It can be calculated.
- the user may input the genre and specify the first section length and the second section length, or set the predetermined length as the first section length and the second section length. Also good.
- the interested section identification device 100 (1200) may include setting means (interface) for the user to set the first section length and the second section length.
- the specificity calculation unit 106 (1206) calculates the specificity using the first section having the first section length and the second section having the second section length set by the user.
- the method for calculating the specificity by the section of interest specifying device 100 (1200) shown in the above embodiment is an example, and other methods can be used as long as the specificity of the motion of the object in the moving image can be calculated. It may be calculated.
- the motion characteristic amount of the second section is not used in calculating the specificity of the frame, but the second section length is set to 0 and the target frame of the first section is excluded. You may use the motion feature-value of all the frames.
- the numerator may be 1.
- the specificity calculated for the frame may be weighted so that the motion feature amount of the specific frame is more reflected.
- Equation 6 and Equation 10 may be used.
- the interest section specifying device 100 (1200) adds the specificities calculated from the moving feature amount of each object.
- the specificity of the target frame was used. However, another method may be used to calculate the specificity of the target frame.
- the specificity calculated for each object may be weighted and then calculated as the specificity of the target frame.
- the interest section specifying device 100 (1200) includes an object specifying unit for specifying what the object detected by the object detecting unit 102 is. And for classifying an object, a classifier indicating the characteristics of various objects is used.
- the interest section specifying device 100 (1200) holds an object classification table for weighting an object, and the object classification table is associated with a weight value according to the contents of the object (person, animal, building, etc.). It has been.
- an object A, B the weighted values for the C, respectively, W A, W B, and a W C.
- the specificity calculation unit 106 calculates the specificity of the target frame as P A ⁇ W A + P B ⁇ W B + P C ⁇ W C.
- the interest section specifying device 100 (1200) may take such a configuration, calculate the specificity according to the detected object, and extract the interest section.
- the interest section specifying device 100 (1200) includes the receiving unit 1205 as in the second embodiment, and the receiving unit 1205 receives a designation from the user that the object A is the center of the moving image. . Then, interested section identifying unit 100 (1200), so that the weight of the object A is increased, the W A example, set to 1.5, to set the other weights 0.5. In this way, it is possible to execute a specific object priority interest section specification.
- the weight of the object A is 1.5 and the others are 0.5, but this numerical value is an example, and it is sufficient that the weight of the object A is heavier than the weights of the other objects.
- the interested section identification device 100 (1200) may determine the weight as follows.
- the designation of the object A is accepted from the user.
- the section of interest specifying apparatus 100 (1200) may specify the object A as follows.
- the interest interval identification device 100 (1200) includes a recording medium that stores a plurality of moving images other than the moving image from which the interest interval is to be extracted.
- a function of accessing a recording medium (may be a memory on a network) outside the interested section specifying device 100 that stores a plurality of moving images is provided.
- a frame for detecting an object is extracted from an arbitrary frame of each moving image. That is, it may be extracted from all the frames, or may be extracted from one or more randomly selected frames. Then, in order to determine whether or not one or more detected objects are the same object, clustering is performed according to the feature amount of each detected object.
- the interested section identifying apparatus 100 (1200) recognizes each cluster obtained by such clustering as one identical object.
- the object A described above is identified as the object (subject) having the largest number of nodes (object feature quantities) belonging to each cluster is the object (subject) important to the user of the interest section identifying device 100. May be.
- the largest number is specified as the object A. This is because the number of predetermined nodes is determined in advance as a threshold, and the weight corresponding to the cluster exceeding the threshold is weighted with specificity. It is good also as performing.
- the weighting at this time may be as long as the weight of the object corresponding to the cluster exceeding the threshold is larger than the weights of the other objects. In this case, weighting may be performed for a plurality of objects corresponding to clusters exceeding the threshold value. However, this weighting may be a uniform value or in order from the largest number of nodes.
- X1, Y1, Y2 are the coordinate values of the pixels in the frame) in the frame Is determined as the central portion of the frame, and it is determined whether or not each object is within the central portion. Then, the weight of the object in the center portion is set to 1, and the weight of the object outside the center portion is set to 0.
- the weight of the object in the central portion is larger than the weight of the object outside the central portion, and the weight is not limited to 1 or 0.
- the interested section identification device 100 (1200) may have a configuration for determining weights in this way.
- a predetermined period in which the sum of specificities of frames included in the predetermined period is the maximum is specified as the interest interval. It was decided to.
- the method for specifying the interest interval is not limited to this, and other methods may be used as long as a specific segment of motion can be specified.
- a part where the difference in motion feature amount between a certain frame and the next frame exceeds a predetermined threshold may be specified, and a predetermined period including the most part may be specified as an interest section.
- all the difference values of motion feature quantities between consecutive frames in the moving image are obtained, a predetermined period in which the sum of the difference values is maximum is specified, and the difference value used to specify the predetermined period is calculated.
- a period including all frames corresponding to the motion feature amount used in the above may be specified as an interest section.
- the section of interest specifying device 100 (1200) specifies one section of interest from the video.
- the interest interval specified from the moving image may not be one.
- the interest section specifying device 100 may specify a predetermined period in which the combined value of the specificities exceeds a predetermined threshold, and may set all of them as the interest section. At this time, the average value may be compared with a predetermined threshold instead of the sum of the specificities. With such a configuration, a plurality of interest intervals can be extracted. When two predetermined periods satisfying this condition overlap with each other, the start point of the earlier predetermined period is set as the start point of the interest section, and the end point of the later predetermined period is set as the end point of the interest section. It is good also as specifying the interest area exceeding the length which has been.
- the output unit 109 may output a digest video in which these plurality of interest sections are connected.
- the creation of the digest video may be performed by the interest section extraction unit 108.
- the specificity calculating unit 106 (1206) calculates the specificity of all frames
- the interest interval specifying unit 107 (1207) is based on the specificity of all frames. It was decided to identify. However, this is only an example, and it is not necessary to calculate the specificity of all frames as long as the interest interval can be specified to some extent.
- the specificity calculation unit 106 (1206) may calculate only the specificity of even-numbered or odd-numbered frames for each frame of the moving image, or only the specificity of that frame for each specific number of frames. May be calculated. That is, the specificity calculation unit 106 (1206) may calculate the specificity by thinning out the frames of the moving image. Then, the interest interval specifying unit 107 (1207) may specify the interest interval using only the frame for which the specificity is calculated.
- the interest interval specifying device calculates the interest interval specifying device 100 (1200), although the accuracy of the interest interval to be specified is lower than the case where the specificity of all frames is calculated. While reducing the processing burden, it is possible to shorten the time until the interest section is specified.
- Embodiment 1 how specific the motion feature amount of the target frame is as seen from the motion feature amount of each frame in the third section is calculated. It may be peculiarity from the viewpoint of the motion feature amount of each frame of the entire moving image excluding the target frame, not each frame of the section.
- the object tracking by the object tracking unit 103 is performed by tracking the objects close to each other in the position of the object in the frame and the object in the next frame (the coordinate position of the object in the frame) as the same object. It was decided to. However, other methods may be used for tracking the object.
- the similarity between objects detected in one frame and the next frame is detected for each frame detected by the object detection unit 102. It may be performed by calculating. The similarity is obtained by calculating the similarity of image feature amounts obtained by using, for example, a Gabor filter for each detected object.
- the region 302 determined by the region determination unit 104 includes the object (face) detected by the object detection unit 102 and the body that should accompany it.
- this region 302 may be a range other than this.
- the region whose motion should be evaluated is similarly the face region (region 302a in FIG. 3). Only). That is, it may be a region including a part of the object region detected by the object detection unit 102, or may be a region including the detected object region as in the above-described embodiment.
- the interest interval specifying unit 107 (1207) adds up the specificities of the frames included in the window 701, and specifies the place where the sum is the highest as the interest interval. .
- this is not a simple summation and other calculations may be performed.
- the specificities of the rear frames may be weighted and then added together. With such a configuration, it is easy to extract an interest section in which a section rich in changes in the movement of an object exists in the second half rather than the first half.
- the motion feature amount calculation unit 105 calculates the motion feature amount of the object based on the region for evaluating the motion of the object.
- the method of calculating the optical flow of the pixels and using the information obtained by arranging the information as the motion feature amount has been described as an example, but other methods may be used.
- the optical flow of the entire image may be calculated, and the average value thereof may be subtracted from each pixel in the region.
- the difference value in the previous and next frames of the luminance of each pixel in the region where the motion should be evaluated may be calculated, and the information obtained by arranging the information may be used as the motion feature amount.
- the distance between the image capturing device 120 and the object may be estimated based on the size of the region in which the motion in each frame is to be evaluated, and the information may be added as a motion feature amount.
- the luminance change amount may be used as the motion feature amount.
- the motion feature amount of the area is the first stage left end to the right end of the area 302, the second stage left end to the right end,..., The N stage left end to the right end,. It is assumed that this is a set of optical flows of each pixel arranged in the order from the left end to the right end of the lower end, but the order of optical flows is not limited to this. As long as the arrangement order of the pixels of the motion feature amount is common for the objects in each frame, other orders may be used.
- the region 302 may be arranged in order from the first right end of the region 302 to the left end, the second step right end to the left end,..., The N step right end to the left end,. It may be arranged from the lower end to the upper end direction.
- the interest section specifying unit 107 (1207) assigns a start point tag and an end point tag indicating the start point and end point of the specified interest section for the moving image.
- a configuration may be adopted in which only one of them is given. Place one tag away from it by the length of the interest section (if the start tag is attached, then after the interest section length, then if the end tag is attached, the interest The last point of the section length) becomes the end point of the section of interest.
- the interested section specifying device 100 displays the moving picture acquired by the moving picture acquiring unit 101, and the interested section specifying unit 107 (1207) indicates the starting point and the ending point of the interested section.
- the storage unit is realized by, for example, an HDD (Hard Disc Drive) or an SSD (Solid State Drive) flash memory.
- the section of interest specifying device 100 (1200) acquires a moving image from the image capturing device 120, but this may be acquired from other than the image capturing device 120.
- a USB flash memory drive may be mounted on the interested section specifying device 100 to acquire a moving image stored in the USB flash memory, or a network communication function may be provided to the interested section specifying device 100 (1200). It is also possible to download and acquire a video from the Internet.
- the output unit 109 outputs the moving image of the interest section extracted to the display device 130, but the output destination is not limited to the display device 130.
- USB flash memory connected to the section of interest specifying device 100 (1200), or may be uploaded to the Internet.
- the output unit 109 may output the entire moving image indexed with the information indicating the starting point and the ending point of the interested interval, instead of outputting only the moving image of the interested interval part.
- the section of interest specifying device 100 (1200) is supposed to extract a section with a large movement of the object as the section of interest, but in this case, the scene has changed in the video. Therefore, there may be a case where a large movement has been detected.
- the interest section specifying device 100 (1200) determines that a scene change has occurred when the change in the movement feature amount is greater than or equal to a predetermined threshold, and divides the moving image from which the interest section is extracted at that point. And it is good also as extracting an interest area from the animation which became two.
- the detection of the scene change may not be a change in the motion feature amount, but may be based on a case where the change in specificity is larger than a predetermined threshold.
- the interest section specifying device 1200 selects a specific object. That is, instead of the receiving unit 1205 that receives a specific object designation from the user, an object selecting unit that selects an object according to a predetermined algorithm may be provided.
- the object detection unit 102 of the interested section identification device 1200 detects each object included in the frame for each frame of the input moving image. Then, the motion feature amount calculation unit 105 calculates the motion feature amount of each detected object.
- the specificity calculation unit 1206 first calculates the average value of the motion feature amounts of all the objects included in each frame. Then, for each frame, the distance between the motion feature amount of each object included in the frame and the calculated average value is calculated. An object whose distance is greater than or equal to a predetermined threshold value is selected as a specific object, assuming that the object is moving in a specific manner in the frame.
- the interest section may be specified by calculating the specificity for the specific object using the method described in the second embodiment.
- a method other than the method of the second embodiment may be used, and the predetermined section having the largest sum of the distances calculated for selecting a specific object is specified as the interest section. May be.
- the total value of the distance is used, but as a method of specifying the interest interval, for example, a moving image for a predetermined period before and after the frame having the largest distance may be specified as the interest interval.
- the object selection unit may select, as a specific object, an object that is included in a large number of moving images other than a moving image that specifies an interest interval, as in the method for specifying object A described in supplement (4).
- a control program comprising a processor code such as a recording / playback apparatus such as a BD player and program codes to be executed by various circuits connected to the processor is recorded on a recording medium or distributed via various communication paths. It can also be distributed.
- Such recording media include IC cards, hard disks, optical disks, flexible disks, ROMs, and the like.
- the distributed and distributed control program is used by being stored in a memory or the like that can be read by the processor, and the processor executes the control program, thereby realizing various functions as shown in the embodiment. Will come to be.
- the cloud server may execute detection of an object from a moving image included in the object detection unit 102 held by the interested section identification device 100 (1200).
- the cloud server gives information about the detected object to the moving image, and the interested section specifying device 100 (1200) receives the moving image to which the information is added. Then, the interested section specifying device 100 (1200) inputs the moving image to which the detected object information is added to the object tracking unit 103, and causes the subsequent processing to be executed.
- the object detection unit 102 adds information on the detected object to the moving image for the moving image in which the object is detected. Then, the interested section identification device 100 (1200) transmits the moving image to which the object information is added, for example, to the cloud server.
- the cloud server executes the functions of the object tracking unit 103, the region determination unit 104, and the motion feature amount calculation unit 105, using the transmitted moving image to which the object information is added. Then, the cloud server adds feature amount information for each object of each frame to the moving image, and transmits the moving image to the interest section specifying device 100 (1200). Then, the interested section identification device 100 (1200) executes processing after calculation of specificity.
- a part of necessary processing may be requested to an external device outside the section of interest specifying apparatus 100 (1200).
- transmission / reception of data necessary for processing is executed between the interested area identification device 100 (1200) and the external device.
- Each functional component shown in the above embodiment may be realized as a circuit that realizes the function, or may be realized by executing a program by one or a plurality of processors.
- the section of interest specifying device 100 (1200) of the above-described embodiment may be configured as an IC, LSI, or other integrated circuit package. This package is incorporated into various devices and used for use, whereby the various devices realize the functions as described in the embodiments.
- Each functional block is typically realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them.
- the name used here is LSI, but it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.
- the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible.
- An FPGA Field Programmable Gate Array
- a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.
- An interest interval specifying device is an interest interval specifying device that specifies an interest interval that is presumed that a user is interested in a moving image, and detects an object from a frame extracted from the moving image.
- An interest section specifying method is an interest section specifying method by an interest section specifying device that specifies an interest section that is presumed to be of interest to a user in a moving image, and an object is extracted from a frame extracted from the moving image.
- An object detection step for detecting; a motion feature amount extracting step for calculating a motion feature amount of the object with respect to the extracted frame; and a specificity of the motion of the object in the extracted frame using the motion feature amount of the object
- a specificity calculating step for calculating the interest interval, and an interest interval specifying step for specifying the interest interval in the moving image using the specificity.
- the interest section specifying program is an interest section specifying program for causing a computer to execute an interest section specifying process for specifying an interest section that is presumed to be interesting by a user in a moving image.
- an object detection step for detecting an object from a frame extracted from the moving image
- a motion feature amount extraction step for calculating a motion feature amount of the object with respect to the extracted frame, and using the motion feature amount of the object
- the method includes a specificity calculation step of calculating the specificity of the movement of the object in the extracted frame, and an interest interval specifying step of specifying the interest interval in the moving image using the specificity.
- the section of interest specifying device can specify a frame in which the motion of the object is unique in the moving image, it can specify the section of interest that does not get bored for the user.
- the identified interest section is used for creating a digest video or displaying it on a monitor so that the user can recognize the content of the moving image.
- the object detecting means detects the object from a plurality of frames included in the moving image
- the motion feature amount extracting means is configured to detect the object in each of the plurality of frames.
- the motion feature amount of the object is extracted, and the specificity calculation unit calculates the specificity of the motion feature amount of the object in the frame from the motion feature amount of the object in another frame for each of the plurality of frames. It is good to do.
- the interested section specifying device can specify the interested section including the scene when the object moves peculiarly in the time axis direction.
- the specificity calculation means for a target frame for which the specificity of the object is to be calculated, the motion feature quantity of the object in the target frame, and the target frame
- the specificity may be calculated based on the inner product with the average of the motion feature quantities of the object in a plurality of frames other than.
- an indicator of how similar the object motion in the target frame is to the object motion in the other frames As an index suitable for calculating the specificity. As the inner product is closer to 0, the movement of the object in the target frame does not resemble the movement of the object in other frames.
- the specificity calculation means uses the motion feature amount of a frame included in a predetermined first section including the target frame in the moving image to calculate the target frame. It is also possible to calculate the specificity of the movement of the object.
- the specificity of the frame included in the moving image from the other frames can be calculated, and can be used as an index for specifying the section of interest. Further, by limiting to the motion feature amount of the first section instead of the entire moving image, it is possible to reduce the load of calculation processing by the interested section specifying device.
- the specificity calculation means includes a motion feature amount of the object of the target frame that is shorter than the first section including the target frame from the first section.
- the specificity may be calculated from the average value of the motion feature amount of the object in each frame included in the third section excluding the second section.
- the interested section specifying device can allow the interest section to include the monotonous motion for the second section. Even if it is a monotonous movement, if it is about several seconds, the user will not feel painful to see it, so such an interval may be included in the interest interval.
- the specificity calculation means may determine the interval length of the second interval according to the interval length of the first interval.
- the section of interest specifying device is appropriate according to the length of the first section. It becomes possible to set the second section with a long length.
- the interest interval identification device may further include setting means for the user to set the second interval.
- the interested section specifying device can specify the interested section according to the user's preference.
- the specificity calculation means may determine the interval length of the first interval based on the moving image length of the moving image.
- the interested section specifying device can determine the first section for calculating the specificity according to the length of the moving image.
- the interest interval specifying device may further include setting means for the user to set the first interval.
- the interested section specifying device can specify the interested section according to the user's preference.
- the interest interval identification device uses each object included in a predetermined period in the moving image using the specificity of the object calculated for each frame. It is good also as specifying as the said interest area the predetermined period when the total value of specificity of this becomes the maximum.
- the interested section identifying device can identify the interested section where the object makes a lot of unique movements.
- the specificity calculating means calculates the specificity of the motion feature amount of the object from the motion feature amount of another object included in the frame. It is good.
- the interested section specifying device can specify a section including a frame having an object that moves peculiarly from other objects in the frame as the interested section.
- the specificity calculation unit may calculate the motion feature of another object when there are a plurality of other objects included in the frame. It is good also as calculating the specificity from the average value.
- the interest section specifying device can calculate the specificity used to specify the interest section.
- the interest interval specifying device specifies an NG scene as an interest interval, particularly when only one person moves out of the same scene while a plurality of persons are moving the same. be able to.
- the interested section specifying device can specify a section with particularly high movement and high specificity as the interested section.
- the interest interval specifying means may further index information indicating the start point and end point of the specified interest interval into the moving image.
- the interested area specifying device can provide a moving image including information on the interested area.
- indexing the information related to the section of interest it can be used, for example, for specifying a playback position (designating a skip destination) when playing back a moving image.
- the interest interval specifying device further includes an extraction means for extracting the interest interval specified by the interest interval specifying means from the moving image, and the interest extracted by the extraction means.
- an extraction means for extracting the interest interval specified by the interest interval specifying means from the moving image, and the interest extracted by the extraction means.
- the section of interest specifying device further includes object position detecting means for detecting a location in the frame from which the object is extracted, and each of the objects for evaluating the movement of the object.
- object position detecting means for detecting a location in the frame from which the object is extracted, and each of the objects for evaluating the movement of the object.
- a region determining unit that determines a region including the object in the frame, wherein the motion feature amount extracting unit determines the motion of the object in each frame from an average value of the motion feature amount of each feature point in the region of each frame. The feature amount may be calculated.
- the interest section identification device can more easily detect the specificity of the movement of the object than using the movement of the entire frame. it can.
- the specificity calculation unit is configured to detect a motion feature amount calculated for each object when the object detection unit detects a plurality of objects. Weighting may be performed to calculate the specificity.
- the motion feature amount is weighted to calculate the specificity, so that it is possible to specify an interest section that can follow the object focused on by the user. it can.
- the specificity calculating unit may weight the objects existing within the specific range of the frame more heavily than the objects outside the specific range. Good.
- the object that is normally the center is often photographed in the central part of the video, so for example, by setting a specific range to the central part of the frame, the object that is the center of the video can be easily weighted. To identify the section of interest.
- the interest interval specifying device further includes accepting means for accepting designation of a specific object detected from a frame, and the specificity calculating means is the accepting means. It is also possible to give a higher weight to the specific object received in step 1 than for other objects.
- the interest interval specifying device further includes storage means for holding one or more other moving images other than the moving image, and the accepting means includes the one or more interested portions. It is good also as accepting the object contained in many other moving images as the said specific object.
- the interest section specifying device can select a specific object and perform weighting on the object without designation from the user.
- the user holds a plurality of captured videos in the interest section specifying device, there is a high possibility that many objects that are of interest to the user have been captured. Therefore, an object that is frequently photographed in a plurality of other moving images can be identified as an object that is interesting to the user.
- the motion feature amount may be an optical flow.
- the apparatus for specifying a section of interest can be used as a device for creating a digest video from a moving image, for example, a recording / playback device such as a DVD player or a BD player.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Computer Security & Cryptography (AREA)
- Image Analysis (AREA)
- Television Signal Processing For Recording (AREA)
Abstract
Description
発明者は、ダイジェスト映像の作成のための動画においてユーザが興味を示すと思われる興味区間の装置による特定を行うにあたって、上記特許文献1に記載の技術のようにオブジェクトの動きを利用することを考慮した。しかし、単純にオブジェクトの動きだけを興味区間特定に用いた場合、単調な動き(例えば、ただ歩いているだけの動き)でさえも、興味区間として特定される虞があることを発見した。例えば、上記特許文献1に記載の技術を用いた場合、延々と同じことの繰り返し映像(例えば、被写体の人物が歩き続ける映像)などを抽出することになり、ユーザにとって面白味のないダイジェスト映像が作成されることになるという問題がある。
<実施の形態1>
以下、本発明の一実施形態である興味区間特定装置について図面を用いて説明する。
<構成>
図1は、興味区間特定装置100の機能構成を示すブロック図である。なお、図1には、興味区間特定装置100の周辺機器も示している。
<動作>
次に、本実施の形態に係る興味区間特定装置100の動作を図7に示すフローチャートを用いて説明する。
<実施の形態2>
上記実施の形態1においては、オブジェクトの動きの特異性を検出するにあたって、他のフレームにおける同じオブジェクトの動きと比較することで検出することとした。しかし、オブジェクトの特異性を検出する方法はこれに限らない。本実施の形態2においては、上記実施の形態1とは異なる特異性の検出方法を説明する。なお、本実施の形態2においては、上記実施の形態1と共通する内容については、説明を省略し、異なる点について述べることとする。
<構成>
実施の形態2に係る興味区間特定装置1200は、図12に示す通り、動画取得部101と、オブジェクト検出部102と、動き特徴量算出部105と、特異度算出部1206と、興味区間特定部1207と、興味区間抽出部108と、出力部109とを含む。図12においては、図2に示す興味区間特定装置100と同一の機能を果たす機能部については、同一の名称と付番を付し、説明を簡略化もしくは省略する。
<動作>
実施の形態2に係る興味区間特定装置1200の興味区間を特定する動作について、図13、図14のフローチャートを用いて、説明する。なお、図13、図14のフローチャートにおいては、興味区間特定装置1200の動作として、上記実施の形態1に示した興味区間特定装置100と共通する内容については、図8や図10などのフローチャートと同じ符号を付与し、説明を割愛する。
<変形例>
上記実施の形態に従って、本発明に係る興味区間特定装置について説明してきたが、本発明の実施の形態はこれに限られるものではない。以下、本発明の思想として含まれる各種変形例について説明する。
<補足>
ここで、本実施の形態に係る興味区間特定装置の一実施形態の構成と、各構成により奏する効果について説明する。
101 動画取得部
102 オブジェクト検出部
103 オブジェクト追跡部
104 領域決定部
105 動き特徴量算出部
106、1205 特異度算出部
107、1207 興味区間特定部
108 興味区間抽出部
109 出力部
120 撮影装置
130 表示装置
1205 受付部
Claims (22)
- 動画においてユーザが興味を覚えると推測される興味区間を特定する興味区間特定装置であって、
前記動画から抽出したフレームからオブジェクトを検出するオブジェクト検出手段と、
前記抽出したフレームに対する前記オブジェクトの動き特徴量を算出する動き特徴量抽出手段と、
前記オブジェクトの動き特徴量を用いて、前記抽出したフレームにおける前記オブジェクトの動きの特異度を算出する特異度算出手段と、
前記特異度を用いて、前記動画における前記興味区間を特定する興味区間特定手段とを備える
ことを特徴とする興味区間特定装置。 - 前記オブジェクト検出手段は、前記動画に含まれる複数のフレームから前記オブジェクトを検出し、
前記動き特徴量抽出手段は、前記複数のフレーム各々における前記オブジェクトの動き特徴量を抽出し、
前記特異度算出手段は、前記複数のフレーム各々について、当該フレームにおける前記オブジェクトの動き特徴量の、他のフレームにおける前記オブジェクトの動き特徴量からの特異度を算出する
ことを特徴とする請求項1記載の興味区間特定装置。 - 前記特異度算出手段は、前記オブジェクトの特異度を算出する対象となる対象フレームについて、前記対象フレームにおけるオブジェクトの動き特徴量と、前記対象フレーム以外の複数のフレームにおける前記オブジェクトの動き特徴量の平均との内積に基づいて特異度を算出する
ことを特徴とする請求項2記載の興味区間特定装置。 - 前記特異度算出手段は、前記動画中の前記対象フレームを含む所定の第1区間に含まれるフレームの動き特徴量を用いて、前記対象フレームにおける前記オブジェクトの動きの特異度を算出する
ことを特徴とする請求項3記載の興味区間特定装置。 - 前記特異度算出手段は、前記対象フレームの前記オブジェクトの動き特徴量の、前記第1区間から前記対象フレームを含み前記第1区間よりも短い第2区間を除いた第3区間に含まれる各フレームにおける前記オブジェクトの動き特徴量の平均値からの、特異度を算出する
ことを特徴とする請求項4記載の興味区間特定装置。 - 前記特異度算出手段は、前記第2区間の区間長を、前記第1区間の区間長に応じて、定める
ことを特徴とする請求項5記載の興味区間特定装置。 - 前記興味区間特定装置は、更に、
前記第2区間をユーザが設定するための設定手段を備える
ことを特徴とする請求項5記載の興味区間特定装置。 - 前記特異度算出手段は、前記第1区間の区間長を、前記動画の動画長に基づいて定める
ことを特徴とする請求項4記載の興味区間特定装置。 - 前記興味区間特定装置は、更に、
前記第1区間をユーザが設定するための設定手段を備える
ことを特徴とする請求項4記載の興味区間特定装置。 - 前記興味区間特定装置は、各フレームに対して算出された前記オブジェクトの特異度を用いて、前記動画において、所定期間内に含まれる各フレームの特異度の合算値が、最大となる所定期間を前記興味区間として特定する
ことを特徴とする請求項1記載の興味区間特定装置。 - 前記特異度算出手段は、前記オブジェクトの動き特徴量の、前記フレームに含まれる他のオブジェクトの動き特徴量からの、特異度を算出する
ことを特徴とする請求項1記載の興味区間特定装置。 - 前記特異度算出手段は、前記オブジェクトの動き特徴量の、前記フレームに含まれる他のオブジェクトが複数ある場合に、他のオブジェクトの動き特徴量の平均値からの特異度を算出する
ことを特徴とする請求項11記載の興味区間特定装置。 - 前記興味区間特定手段は、更に、特定した興味区間の始点と終点を示す情報を前記動画にインデキシングする
ことを特徴とする請求項1記載の興味区間特定装置。 - 前記興味区間特定装置は、更に、
前記興味区間特定手段が特定した興味区間を前記動画から抽出する抽出手段と、
前記抽出手段が抽出した興味区間が複数ある場合に、複数の興味区間をつなぎ合わせたダイジェスト映像を作成するダイジェスト作成手段とを備える
ことを特徴とする請求項1記載の興味区間特定装置。 - 前記興味区間特定装置は、更に、
前記オブジェクトを抽出したフレームにおける場所を検出するオブジェクト位置検出手段と、
前記オブジェクトの動きを評価するための各フレームにおける前記オブジェクトを含む領域を決定する領域決定手段とを備え、
前記動き特徴量抽出手段は、各フレームの前記領域内における特徴点各々の動き特徴量の平均値から各フレームにおける前記オブジェクトの動き特徴量を算出する
ことを特徴とする請求項1記載の興味区間特定装置。 - 前記特異度算出手段は、前記オブジェクト検出手段が複数のオブジェクトを検出した場合に、それぞれのオブジェクトに対して算出される動き特徴量に対して重み付けを行って、前記特異度を算出する
ことを特徴とする請求項1記載の興味区間特定装置。 - 前記特異度算出手段は、フレームの特定範囲内に存在するオブジェクトに対して、前記特定範囲外にあるオブジェクトよりも重い重み付けを行う
ことを特徴とする請求項16記載の興味区間特定装置。 - 前記興味区間特定装置は、更に、
フレームから検出された特定のオブジェクトの指定を受け付ける受付手段を備え、
前記特異度算出手段は、前記受付手段で受け付けた特定のオブジェクトに対して、他のオブジェクトよりも重い重み付けを行う
ことを特徴とする請求項16記載の興味区間特定装置。 - 前記興味区間特定装置は、更に、
前記動画以外の1以上の他の動画を保持する記憶手段を備え、
前記受付手段は、前記1以上の他の動画に多く含まれるオブジェクトを、前記特定のオブジェクトとして受け付ける
ことを特徴とする請求項18記載の興味区間特定装置。 - 前記動き特徴量は、オプティカルフローである
ことを特徴とする請求項1記載の興味区間特定装置。 - 動画においてユーザが興味を覚えると推測される興味区間を特定する興味区間特定装置による興味区間特定方法であって、
前記動画から抽出したフレームからオブジェクトを検出するオブジェクト検出ステップと、
前記抽出したフレームに対する前記オブジェクトの動き特徴量を算出する動き特徴量抽出ステップと、
前記オブジェクトの動き特徴量を用いて、前記抽出したフレームにおける前記オブジェクトの動きの特異度を算出する特異度算出ステップと、
前記特異度を用いて、前記動画における前記興味区間を特定する興味区間特定ステップとを含む
ことを特徴とする興味区間特定方法。 - 動画においてユーザが興味を覚えると推測される興味区間を特定する興味区間特定処理をコンピュータに実行させる興味区間特定プログラムであって、
前記興味区間特定処理は、
前記動画から抽出したフレームからオブジェクトを検出するオブジェクト検出ステップと、
前記抽出したフレームに対する前記オブジェクトの動き特徴量を算出する動き特徴量抽出ステップと、
前記オブジェクトの動き特徴量を用いて、前記抽出したフレームにおける前記オブジェクトの動きの特異度を算出する特異度算出ステップと、
前記特異度を用いて、前記動画における前記興味区間を特定する興味区間特定ステップとを含む
ことを特徴とする興味区間特定プログラム。
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/824,692 US9288463B2 (en) | 2012-02-23 | 2012-10-19 | Interesting section identification device, interesting section identification method, and interesting section identification program |
JP2013514425A JP5960691B2 (ja) | 2012-02-23 | 2012-10-19 | 興味区間特定装置、興味区間特定方法、興味区間特定プログラム |
CN201280003448.8A CN103404130B (zh) | 2012-02-23 | 2012-10-19 | 兴趣区间确定装置、兴趣区间确定方法 |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2012-037462 | 2012-02-23 | ||
JP2012037462 | 2012-02-23 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2013124923A1 true WO2013124923A1 (ja) | 2013-08-29 |
Family
ID=49005147
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2012/006717 WO2013124923A1 (ja) | 2012-02-23 | 2012-10-19 | 興味区間特定装置、興味区間特定方法、興味区間特定プログラム |
Country Status (4)
Country | Link |
---|---|
US (1) | US9288463B2 (ja) |
JP (1) | JP5960691B2 (ja) |
CN (1) | CN103404130B (ja) |
WO (1) | WO2013124923A1 (ja) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015179438A (ja) * | 2014-03-19 | 2015-10-08 | オムロン株式会社 | 画像処理装置、移動速度判定方法、および移動速度判定プログラム |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017129804A1 (en) * | 2016-01-29 | 2017-08-03 | Kiwisecurity Software Gmbh | Methods and apparatus for using video analytics to detect regions for privacy protection within images from moving cameras |
US10867393B2 (en) | 2018-03-22 | 2020-12-15 | Texas Instruments Incorporated | Video object detection |
CN111461104B (zh) * | 2019-01-22 | 2024-04-09 | 北京京东乾石科技有限公司 | 视觉识别方法、装置、设备及存储介质 |
CN113286194A (zh) * | 2020-02-20 | 2021-08-20 | 北京三星通信技术研究有限公司 | 视频处理方法、装置、电子设备及可读存储介质 |
KR102630705B1 (ko) * | 2023-07-12 | 2024-01-29 | 주식회사 넥스톤 | 에너지 절약형 네트워크 카메라 저장 장치 및 네트워크카메라 저장 장치의 동작 방법 |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2010128947A (ja) * | 2008-11-28 | 2010-06-10 | Nippon Telegr & Teleph Corp <Ntt> | 非定常度推定装置、非定常度推定方法、非定常度推定プログラム |
JP2011175599A (ja) * | 2010-02-25 | 2011-09-08 | Canon Inc | 画像処理装置、その処理方法及びプログラム |
JP2012010133A (ja) * | 2010-06-25 | 2012-01-12 | Nikon Corp | 画像処理装置および画像処理プログラム |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0648458B2 (ja) | 1985-03-01 | 1994-06-22 | 日本電信電話株式会社 | 情報入力装置 |
US7499077B2 (en) * | 2001-06-04 | 2009-03-03 | Sharp Laboratories Of America, Inc. | Summarization of football video content |
JP3947973B2 (ja) * | 2003-02-14 | 2007-07-25 | ソニー株式会社 | 画像処理装置および方法、プログラム、並びに記録媒体 |
JP2005115672A (ja) * | 2003-10-08 | 2005-04-28 | Fuji Photo Film Co Ltd | 画像処理装置 |
JP2006019387A (ja) | 2004-06-30 | 2006-01-19 | Toshiba Corp | 半導体記憶装置およびその製造方法 |
JP4830650B2 (ja) | 2005-07-05 | 2011-12-07 | オムロン株式会社 | 追跡装置 |
EP1811457A1 (en) * | 2006-01-20 | 2007-07-25 | BRITISH TELECOMMUNICATIONS public limited company | Video signal analysis |
JP2008278467A (ja) * | 2007-03-30 | 2008-11-13 | Sanyo Electric Co Ltd | 画像処理装置および画像処理方法 |
JP2010226557A (ja) * | 2009-03-25 | 2010-10-07 | Sony Corp | 画像処理装置、画像処理方法、およびプログラム |
-
2012
- 2012-10-19 JP JP2013514425A patent/JP5960691B2/ja active Active
- 2012-10-19 US US13/824,692 patent/US9288463B2/en active Active
- 2012-10-19 WO PCT/JP2012/006717 patent/WO2013124923A1/ja active Application Filing
- 2012-10-19 CN CN201280003448.8A patent/CN103404130B/zh active Active
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2010128947A (ja) * | 2008-11-28 | 2010-06-10 | Nippon Telegr & Teleph Corp <Ntt> | 非定常度推定装置、非定常度推定方法、非定常度推定プログラム |
JP2011175599A (ja) * | 2010-02-25 | 2011-09-08 | Canon Inc | 画像処理装置、その処理方法及びプログラム |
JP2012010133A (ja) * | 2010-06-25 | 2012-01-12 | Nikon Corp | 画像処理装置および画像処理プログラム |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2015179438A (ja) * | 2014-03-19 | 2015-10-08 | オムロン株式会社 | 画像処理装置、移動速度判定方法、および移動速度判定プログラム |
Also Published As
Publication number | Publication date |
---|---|
US9288463B2 (en) | 2016-03-15 |
CN103404130A (zh) | 2013-11-20 |
US20140072279A1 (en) | 2014-03-13 |
JP5960691B2 (ja) | 2016-08-02 |
JPWO2013124923A1 (ja) | 2015-05-21 |
CN103404130B (zh) | 2017-06-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Lai et al. | Semantic-driven generation of hyperlapse from 360 degree video | |
CN106663196B (zh) | 用于识别主体的方法、***和计算机可读存储介质 | |
JP5960691B2 (ja) | 興味区間特定装置、興味区間特定方法、興味区間特定プログラム | |
US8594488B1 (en) | Methods and systems for video retargeting using motion saliency | |
Higuchi et al. | Egoscanning: Quickly scanning first-person videos with egocentric elastic timelines | |
US20090251421A1 (en) | Method and apparatus for tactile perception of digital images | |
JP6141829B2 (ja) | 動画解析装置、動画解析方法、プログラム、及び集積回路 | |
US20110249904A1 (en) | Face clustering device, face clustering method, and program | |
Dubuisson et al. | A survey of datasets for visual tracking | |
US9691180B2 (en) | Determination of augmented reality information | |
JP2013122763A (ja) | 映像処理装置及び映像処理方法 | |
JP2012105205A (ja) | キーフレーム抽出装置、キーフレーム抽出プログラム、キーフレーム抽出方法、撮像装置、およびサーバ装置 | |
EP3151243B1 (en) | Accessing a video segment | |
CN108960130B (zh) | 视频文件智能处理方法和装置 | |
WO1999005865A1 (en) | Content-based video access | |
JP2007072789A (ja) | 映像構造化方法及び装置及びプログラム | |
WO2023088029A1 (zh) | 一种封面生成方法、装置、设备及介质 | |
US10762395B2 (en) | Image processing apparatus, image processing method, and recording medium | |
TW201222422A (en) | Method and arrangement for identifying virtual visual information in images | |
Choudhary et al. | Real time video summarization on mobile platform | |
JP2014085845A (ja) | 動画処理装置、動画処理方法、プログラム、および集積回路 | |
TWI729322B (zh) | 資訊顯示系統及資訊顯示方法 | |
Lee | Novel video stabilization for real-time optical character recognition applications | |
JP5276609B2 (ja) | 画像処理装置及びプログラム | |
Hamandi | Modeling and Enhancing Deep Learning Accuracy in Computer Vision Applications |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
WWE | Wipo information: entry into national phase |
Ref document number: 13824692 Country of ref document: US |
|
ENP | Entry into the national phase |
Ref document number: 2013514425 Country of ref document: JP Kind code of ref document: A |
|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 12869342 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 12869342 Country of ref document: EP Kind code of ref document: A1 |