CN113190713B - Video searching method and device, electronic equipment and medium - Google Patents
Video searching method and device, electronic equipment and medium Download PDFInfo
- Publication number
- CN113190713B CN113190713B CN202110492157.0A CN202110492157A CN113190713B CN 113190713 B CN113190713 B CN 113190713B CN 202110492157 A CN202110492157 A CN 202110492157A CN 113190713 B CN113190713 B CN 113190713B
- Authority
- CN
- China
- Prior art keywords
- video
- searched
- feature
- candidate
- image frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 239000012634 fragment Substances 0.000 claims abstract description 70
- 238000004590 computer program Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 6
- 230000004044 response Effects 0.000 claims description 6
- 238000000638 solvent extraction Methods 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 15
- 238000012545 processing Methods 0.000 abstract description 13
- 238000013473 artificial intelligence Methods 0.000 abstract description 8
- 238000004891 communication Methods 0.000 description 14
- 230000006870 function Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 8
- 238000013459 approach Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 230000001413 cellular effect Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000010801 machine learning Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 101100421134 Schizosaccharomyces pombe (strain 972 / ATCC 24843) sle1 gene Proteins 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000010267 cellular communication Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000003924 mental process Effects 0.000 description 1
- 230000003278 mimic effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000003062 neural network model Methods 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/783—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/41—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items
- G06V20/42—Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items of sport video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Library & Information Science (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure provides a video searching method and device, electronic equipment and media, relates to the technical field of image and video processing, and in particular relates to artificial intelligence and computer vision technology, and can be applied to video understanding scenes. The implementation scheme is as follows: according to the object corresponding to the image frame of the video to be searched, determining at least one object fragment included in the video to be searched, wherein each image frame in each object fragment corresponds to the same object; splicing object outgoing data pairs of one or more continuous object fragments in the at least one object fragment to obtain a first object outgoing feature of the video to be searched, wherein the object outgoing data pairs comprise objects corresponding to the corresponding object fragments and time differences between a first image frame of the object fragment and a first image frame of a last object fragment; and determining at least one target video matched with the video to be searched from the plurality of candidate videos according to the first object presence feature.
Description
Technical Field
The present disclosure relates to the field of image and video processing technologies, and in particular, to artificial intelligence and computer vision technologies, which may be applied in video understanding scenarios. And more particularly to a method, apparatus, electronic device, computer readable storage medium and computer program product for video searching.
Background
Artificial intelligence is the discipline of studying the process of making a computer mimic certain mental processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.) of a person, both hardware-level and software-level techniques. The artificial intelligence hardware technology generally comprises technologies such as a sensor, a special artificial intelligence chip, cloud computing, distributed storage, big data processing and the like, and the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, a machine learning/deep learning technology, a big data processing technology, a knowledge graph technology and the like.
Video may provide more rich information content than text, images, etc. information transmission media. In some scenarios, a user wishes to obtain their desired video by searching. Existing video search methods are typically text-based searches, i.e., search results are obtained by matching search terms entered by a user with text labels of individual videos in a video library. This search approach is independent of the content of the video itself, and is dependent only on the accuracy of the search terms entered by the user and the labeling of the video text labels, with the search results often being unsatisfactory to the user.
The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, the problems mentioned in this section should not be considered as having been recognized in any prior art unless otherwise indicated.
Disclosure of Invention
The present disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for video searching.
According to an aspect of the present disclosure, there is provided a video searching method, the method including: according to the object corresponding to the image frame of the video to be searched, determining at least one object fragment included in the video to be searched, wherein each image frame in each object fragment corresponds to the same object; splicing object outgoing data pairs of one or more continuous object fragments in the at least one object fragment to obtain a first object outgoing feature of the video to be searched, wherein the object outgoing data pairs comprise objects corresponding to the corresponding object fragments and time differences between a first image frame of the object fragment and a first image frame of a last object fragment; and determining at least one target video matched with the video to be searched from the plurality of candidate videos according to the first object presence feature.
According to another aspect of the present disclosure, there is provided a video search apparatus including: the object segment dividing module is configured to determine at least one object segment included in the video to be searched according to the object corresponding to the image frame of the video to be searched, and each image frame in each object segment corresponds to the same object; the feature extraction module is configured to splice object appearance data pairs of one or more continuous object fragments in the at least one object fragment to obtain a first object appearance feature of the video to be searched, wherein the object appearance data pairs comprise objects corresponding to the corresponding object fragments and time differences between a first image frame of the object fragment and a first image frame of a previous object fragment; and a feature matching module configured to determine at least one target video matching the video to be searched from the plurality of candidate videos according to the first object presence feature.
According to another aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor. The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the aspects.
According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are for causing a computer to perform the method of any of the above aspects.
According to another aspect of the present disclosure, a computer program product is provided, including a computer program. Which, when executed by a processor, carries out the method of any of the above aspects.
In accordance with one or more embodiments of the present disclosure, a video search scheme is provided for searching video in video based on object presence features. Determining at least one object fragment in the video to be searched according to the object corresponding to the image frame of the video to be searched; and generating a first object appearance characteristic of the video to be searched according to the objects corresponding to one or more continuous object fragments and the time difference between the first image frame of the object fragment and the first image frame of the last object fragment. The first object presence feature can express the object presence condition of the video to be searched, and is a refined and accurate summary of the video content to be searched. The target video is determined from the plurality of candidate videos according to the first object departure characteristic, and quick and accurate video searching can be achieved.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.
Drawings
The accompanying drawings illustrate exemplary embodiments and, together with the description, serve to explain exemplary implementations of the embodiments. The illustrated embodiments are for exemplary purposes only and do not limit the scope of the claims. Throughout the drawings, identical reference numerals designate similar, but not necessarily identical, elements.
FIG. 1 illustrates a schematic diagram of an exemplary system in which various methods described herein may be implemented, in accordance with an embodiment of the present disclosure;
FIG. 2 illustrates a flow chart of a video search method according to an embodiment of the present disclosure;
FIG. 3 illustrates a schematic diagram of two exemplary target videos matching a video to be searched according to an embodiment of the present disclosure;
fig. 4 shows a block diagram of a video search apparatus according to an embodiment of the present disclosure; and
Fig. 5 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and should be considered as merely exemplary. Accordingly, one of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
In the present disclosure, the use of the terms "first," "second," and the like to describe various elements is not intended to limit the positional relationship, timing relationship, or importance relationship of the elements, unless otherwise indicated, and such terms are merely used to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, they may also refer to different instances based on the description of the context.
The terminology used in the description of the various illustrated examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, the elements may be one or more if the number of the elements is not specifically limited. Furthermore, the term "and/or" as used in this disclosure encompasses any and all possible combinations of the listed items.
Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.
Fig. 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented, in accordance with an embodiment of the present disclosure. Referring to fig. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. Client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.
In embodiments of the present disclosure, the server 120 may run one or more services or software applications that enable methods of performing video searches.
In some embodiments, server 120 may also provide other services or software applications that may include non-virtual environments and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and/or 106 under a software as a service (SaaS) model.
In the configuration shown in fig. 1, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or a combination thereof that are executable by one or more processors. A user operating client devices 101, 102, 103, 104, 105, and/or 106 may in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be appreciated that a variety of different system configurations are possible, which may differ from system 100. Accordingly, FIG. 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.
The user may conduct a video search using client devices 101, 102, 103, 104, 105, and/or 106. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although fig. 1 depicts only six client devices, those skilled in the art will appreciate that the present disclosure may support any number of any type of client devices.
Client devices 101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and the like. These computer devices may run various types and versions of software applications and operating systems, such as Microsoft Windows, apple iOS, UNIX-like operating systems, linux, or Linux-like operating systems (e.g., *** Chrome OS); or include various mobile operating systems such as Microsoft Windows Mobile OS, iOS, windows Phone, android. Portable handheld devices may include cellular telephones, smart phones, tablet computers, personal Digital Assistants (PDAs), and the like. Wearable devices may include head mounted displays and other devices. The gaming system may include various handheld gaming devices, internet-enabled gaming devices, and the like. The client device is capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), short Message Service (SMS) applications, and may use a variety of communication protocols.
Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a number of available protocols, including but not limited to TCP/IP, SNA, IPX, etc. For example only, the one or more networks 110 may be a Local Area Network (LAN), an ethernet-based network, a token ring, a Wide Area Network (WAN), the internet, a virtual network, a Virtual Private Network (VPN), an intranet, an extranet, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network (e.g., bluetooth, WIFI), and/or any combination of these and/or other networks.
The server 120 may include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-end servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination. The server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture that involves virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices of the server). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.
The computing units in server 120 may run one or more operating systems including any of the operating systems described above as well as any commercially available server operating systems. Server 120 may also run any of a variety of additional server applications and/or middle tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.
In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.
In some implementations, the server 120 may be a server of a distributed system or a server that incorporates a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The cloud server is a host product in a cloud computing service system, so as to solve the defects of large management difficulty and weak service expansibility in the traditional physical host and Virtual special server (VPS PRIVATE SERVER) service.
The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of databases 130 may be used to store information such as audio files and video files. The data store 130 may reside in a variety of locations. For example, the data store used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The data store 130 may be of different types. In some embodiments, the data store used by server 120 may be a database, such as a relational database. One or more of these databases may store, update, and retrieve the databases and data from the databases in response to the commands.
In some embodiments, one or more of databases 130 may also be used by applications to store application data. The databases used by the application may be different types of databases, such as key value stores, object stores, or conventional stores supported by the file system.
The system 100 of fig. 1 may be configured and operated in various ways to enable application of the various methods and apparatus described in accordance with the present disclosure.
For purposes of embodiments of the present disclosure, in the example of fig. 1, client applications for conducting video searches may be included in client devices 101, 102, 103, 104, 105, and 106. The client application may be, for example, an application program that needs to be downloaded and installed before running, a video website that is accessible through a browser, a lightweight applet that runs in a host application, and so on. The client application may provide various video-based functions such as searching, viewing, uploading, downloading, editing, etc. of the video. In response, the server 120 may be a server for use with the client application. Server 120 may provide video services to client applications running in client devices 101, 102, 103, 104, 105, and 106 based on stored video assets, video editing tools, and the like. Specifically, the server 120 may perform the video searching method 200 according to the embodiment of the present disclosure based on the stored video resources, and provide the video searching service to the user, so as to implement fast and accurate video searching.
Fig. 2 is a flowchart illustrating a video search method 200 according to an exemplary embodiment. The method 200 may be performed at a server (e.g., the server 120 shown in fig. 1), i.e., the subject of execution of the steps of the method 200 may be the server 120 shown in fig. 1.
As shown in fig. 2, the method 200 includes:
step 210, determining at least one object segment included in the video to be searched according to the object corresponding to the image frame of the video to be searched, wherein each image frame in each object segment corresponds to the same object;
Step 220, object outgoing data pairs of one or more continuous object segments in the at least one object segment are spliced to obtain a first object outgoing feature of the video to be searched, wherein the object outgoing data pairs comprise objects corresponding to the corresponding object segments and time differences between a first image frame of the object segment and a first image frame of a previous object segment; and
Step 230, determining at least one target video matched with the video to be searched from a plurality of candidate videos according to the first object presence feature.
In accordance with one or more embodiments of the present disclosure, a video search scheme for "video searching" based on object presence features is provided. Determining at least one object fragment in the video to be searched according to the object corresponding to the image frame of the video to be searched; and generating a first object appearance characteristic of the video to be searched according to the objects corresponding to one or more continuous object fragments and the time difference between the first image frame of the object fragment and the first image frame of the last object fragment. The first object presence feature can express the object presence condition of the video to be searched, and is a refined and accurate summary of the video content to be searched. The target video is determined from the plurality of candidate videos according to the first object departure characteristic, and quick and accurate video searching can be achieved.
The video search method 200 of the disclosed embodiments relates to the field of video processing technology, and in particular to artificial intelligence and computer vision technology. The method 200 may be applied in a video understanding scenario, for example, for searching for a target video that matches a user-specified video to be searched, or for making video recommendations to a user based on video content, etc.
In embodiments of the present disclosure, the video to be searched is typically a video of a short duration (e.g., a few seconds or minutes), i.e., a short video. The candidate video, the target video, is typically a video of a long duration (duration at least longer than the video to be searched, e.g. a few minutes or tens of minutes), i.e. a long video.
For example, in some scenes, a user may watch a highlight movie clip through a channel and wish to search for a movie or episode containing the movie clip. In this case, the video to be searched is the movie fragment; candidate videos may be all videos stored in a server or associated database, or all movies/drama videos, movies/drama videos of a certain type (e.g., comedy, suspense, etc.) or starring by certain actors, etc.; the target video is the video which is searched from the candidate video and matches with the highlight video segment.
For another example, in other scenarios, a user may view a highlight in a sporting event, such as a goal segment in a football event, a basketball event, etc., through some channel, and may wish to search for a more complete event video containing the highlight, such as a full-field event video, a half-field event video, etc., containing the highlight. In this case, the video to be searched is a highlight of the sports event; the candidate videos may be all videos stored in a server or associated database, or all sports videos, a certain type of sports video, etc.; the target video is the event video which is searched from the candidate video and matches with the highlight.
The various steps of method 200 are described in detail below.
Referring to fig. 2, in step 210, at least one object segment included in a video to be searched is determined according to an object corresponding to an image frame of the video to be searched, and each image frame in each object segment corresponds to the same object.
The video to be searched in step 210 may be acquired in a variety of ways.
According to some embodiments, a user may upload a video to be searched through a client device and initiate a video search request requesting to search for a video matching the video to be searched. Accordingly, in step 210, the server may directly acquire the video to be searched uploaded by the user.
According to other embodiments, a user may specify an address of a video to be searched through a client device and initiate a video search request. Accordingly, in step 210, the server may obtain the video to be searched from the corresponding address.
According to still other embodiments, the server may take any video that the user has viewed as the video to be searched without specification by the user. In this case, the server may determine a target video matching the video to be searched through the method 200, push the target video to the client device, and provide the video intelligent recommendation service to the user.
After the video to be searched is obtained, an object corresponding to the image frame of the video to be searched can be determined first, and then at least one object fragment included in the video to be searched is determined according to the object corresponding to the image frame.
An object may be any entity including, but not limited to, a person (actor), an object, a virtual cartoon character, and the like.
According to some embodiments, object recognition may be performed on a plurality of image frames of a video to be searched to determine objects corresponding to the plurality of image frames.
Specifically, the "plurality of image frames" in this embodiment may be all image frames in the video to be searched, or may be part of the image frames in the video to be searched (for example, a plurality of image frames obtained by sampling the image frames of the video to be searched, or a plurality of image frames obtained by cutting out a segment of a certain duration in the video to be searched, etc.). Considering that the duration of the video to be searched is generally short, and in order to improve the accuracy of video searching, it is preferable that object recognition be performed on all image frames in the video to be searched.
According to some embodiments, the object recognition may be performed on the image frame through a preset object recognition model. That is, an image frame is input into a preset object recognition model, and the object recognition model outputs the position and class of an object included in the image frame. Specifically, the object recognition model may be, for example, a FASTER RCNN, YOLO, cascades, or other neural network model.
According to some embodiments, an object obtained by performing object recognition on an image frame may be directly taken as an object corresponding to the image frame. The image frame may correspond to one or more objects. For example, by performing object recognition on the image frame1, if only one object a is included in the frame1, the object corresponding to the frame1 is the object a. For example, by performing object recognition on the image frame2, three actors, namely, actor B, actor C, and actor D are included in the frame2, and then the objects corresponding to the frame2 are actor B, actor C, and actor D.
According to other embodiments, objects obtained by object recognition of image frames may be further filtered, leaving only one of the objects. I.e. the image frame corresponds to only one object. Specifically, in this embodiment, for each of a plurality of image frames of a video to be searched: performing object recognition on the image frame to determine candidate objects included in the image frame; and taking any one of the following as an object corresponding to the image frame: the candidate closest to the center of the image frame, the candidate with the largest area, the candidate with the front view.
For example, by performing object recognition on the image frame1, it is obtained that only one object a (candidate object) is included in the frame1, and then the object corresponding to the frame1 is the object a. For another example, by performing object recognition on the image frame2, two objects (candidate objects) are included in the frame2, namely an object B and an object C, where the object B is located at an edge of the frame2 and the object C is located at a center of the frame2, and the object C is taken as an object corresponding to the frame 2. For another example, by performing object recognition on the image frame3, three actors (candidate objects) are included in the frame3, namely, actor D, actor E, and actor F, wherein actor E has a front face (front view) in the frame3, and actors D, F are all side faces, and actor E is taken as the object corresponding to the frame 3.
In the above embodiment, by further screening the candidate objects obtained by object identification, the image frames only correspond to one object, which can reduce the storage space occupied by storing the object information corresponding to each image frame, and is also beneficial to accelerating the calculation speed of subsequently determining the object segment, extracting the first object appearance characteristic and determining the target video.
It should be appreciated that there may also be situations where an object is identified for an image frame but no object is identified. In this case, the object corresponding to the image frame is null (null), that is, there is no corresponding object.
After determining the objects corresponding to the plurality of image frames in the video to be searched, at least one object segment included in the video to be searched can be determined according to the objects corresponding to the image frames, wherein the image frames in each object segment correspond to the same object.
According to some embodiments, one or more consecutive image frames corresponding to the same object may be treated as one object fragment.
For example, the objects corresponding to 10 image frames frame1-frame10 of the video to be searched are B, A, A, A, null, C, C, C, respectively. Frame1 is an object fragment and the corresponding object is B; frame2-frame4 is an object fragment, and the corresponding object is A; frame6-frame10 is an object fragment and the corresponding object is C.
After determining at least one object segment in the video to be searched through step 210, step 220 is performed.
In step 220, object presence data pairs of one or more continuous object segments in the at least one object segment are spliced to obtain a first object presence feature of the video to be searched. The object departure data pair comprises an object corresponding to a corresponding object fragment and a time difference between a first image frame of the object fragment and a first image frame of a last object fragment.
Each object segment corresponds to an object outgoing data pair that includes the object to which the object segment corresponds and a time difference between the first image frame of the object segment and the first image frame of the previous object segment. That is, each object segment corresponds to a data pair in the form of (obj:. DELTA.t), where obj is the object to which the object segment corresponds and DELTA is the time difference between the first image frame of the object segment and the first image frame of the previous object segment. In particular, since the first object segment does not exist in the previous object segment, the time difference corresponding to the first object segment may be set to the time of its first image frame or to 0.
The object corresponding to the object fragment is the object corresponding to each image frame of the object fragment.
The time difference between the first image frame of the object segment and the first image frame of the previous object segment can be calculated according to the frame rate (i.e. the number of image frames per second) of the video to be searched and the difference between the sequence numbers of the first image frame of the object segment and the first image frame of the previous object segment. For example, the frame rate of the video to be searched is 25FPS (FRAMES PER seconds), and the duration of each image frame is 1 s/25=40 ms. The first image frame of a certain object fragment is the 100 th image frame in the video to be searched, and the first image frame of the last object fragment is the 50 th image frame in the video to be searched, so that the time difference between the first image frame of the object fragment and the first image frame of the last object fragment is 40 x (100-50) =2000 ms. The method for calculating the time difference between the first image frame of the object segment and the first image frame of the previous object segment can be expressed by formula 1000 (n 2-n 1)/FR, where n1 and n2 are respectively the sequence numbers of the first image frames of the previous object segment and the current object segment in the video to be searched, FR is the frame rate of the video to be searched, and the calculated time difference is in units of ms.
According to some embodiments, in step 220, object presence data pairs of one or more consecutive object segments of all object segments determined in step 210 may be stitched to obtain a first object presence feature of the video to be searched.
For example, four object fragments seg1, seg2, seg3, seg4 are determined by step 210, with corresponding pairs of object presence data being (obj 1:. DELTA.t1), (obj 2:. DELTA.t2), (obj 3:. DELTA.t3), (obj 4:. DELTA.t4), respectively. An object presence data pair, e.g. (obj 2:. Delta.t2), of one of the object segments may be used as the first object presence feature of the video to be searched. Or, object presence data pairs of a plurality of continuous object segments may be spliced, for example, (obj 1:. DELTA.t1), (obj 2:. DELTA.t2), (obj 3:. DELTA.t3) are spliced, so as to obtain the first object presence feature as obj1: deltat 1; obj2: deltat 2; obj3: Δt3.
According to some embodiments, considering that the duration of the video to be searched is generally shorter, the number of object segments included correspondingly is smaller, and in order to improve the accuracy of video searching, it is preferable to splice the object presence data pairs of all the object segments determined in step 210, as the first object presence feature of the video to be searched.
The first object presence feature can express the object presence condition of the video to be searched, and is a refined and accurate summary of the video content to be searched.
After obtaining the first object presence feature of the video to be searched through step 220, step 230 is performed.
In step 230, at least one target video that matches the video to be searched is determined from the plurality of candidate videos based on the first object presence feature.
The first object presence feature can express the object presence condition of the video to be searched, and is a refined and accurate summary of the video content to be searched, so that the target video is determined from a plurality of candidate videos according to the first object presence feature, and quick and accurate video searching can be realized.
In some embodiments, the plurality of candidate videos in step 230 may be all of the videos in the video library. In other embodiments, the plurality of candidate videos in step 230 may also be a part of the videos in the video library, for example, the videos of the same type as the video to be searched, or videos with a time period greater than a preset threshold (for example, 20 minutes), and so on.
Specifically, in step 230, the first object presence feature of the video to be searched is compared with the second object presence feature of each candidate video, and at least one target video is determined according to the comparison result.
It will be appreciated that the second object presence feature of each candidate video is pre-extracted and stored prior to performing the method 200. And the extraction mode of the second object emergent characteristic of the candidate video is the same as that of the first object emergent characteristic of the video to be searched. That is, the second object presence feature of the candidate video may be extracted as follows: according to the object corresponding to the image frame of the candidate video, determining at least one object fragment included in the candidate video, wherein each image frame in each object fragment corresponds to the same object; and splicing object outgoing data pairs of one or more continuous object fragments in the at least one object fragment to obtain a second object outgoing characteristic of the candidate video, wherein the object outgoing data pairs comprise objects corresponding to the corresponding object fragments and time differences between a first image frame of the object fragment and a first image frame of a last object fragment.
According to some embodiments, the at least one target video may be determined in a manner that locally matches the first object presence feature with the second object presence feature, namely: acquiring a second object presence feature of each candidate video of the plurality of candidate videos, the second object presence feature comprising at least one object presence data pair, each object presence data pair comprising an object and a time difference; and in response to determining that there is at least one object presence data pair having a second object presence feature that is continuously the same as the first object presence feature, regarding the respective candidate video as the target video.
In this embodiment, the second object presence feature of the target video comprises a portion of the first object presence feature.
For example, fig. 3 shows one example of a target video determined in the above-described partial matching manner. As shown in fig. 3, the video 310 to be searched includes three object segments 311, 312, 313, and the corresponding object outgoing data pairs are (a: 0), (B: 7500), and (a: 15000), respectively, wherein A, B is an object, and the unit of time difference is ms. The first object out-of-field feature of the video 310 to be searched is a concatenation of object out-of-field data pairs of object segments 311-313, namely A:0; 7500; a is 15000.
The candidate video 320 includes five object segments 321, 322, 323, 324, 325, and the corresponding pairs of object departure data are (C: 2000), (a: 15000), (B: 7500), (a: 15000), and (B: 4000), respectively, where A, B, C is the object and the time difference is in ms. The second object out-of-field feature of candidate video 320 is a concatenation of object out-of-field data pairs of object segments 321-325, i.e., C:2000; a is 15000; 7500; a is 15000; and B is 4000. 2000 due to the second object presence feature C of candidate video 320; a is 15000; 7500; a is 15000; 4000 and the first object presence feature A0; 7500; 15000 there are two pairs of object presence data that are identical in succession, namely B7500; 15000 (shown as dashed rectangle 326, 314 in fig. 3), the candidate video 320 is thus taken as the target video that matches the video 310 to be searched.
According to some embodiments, in the above-mentioned locally matching manner, further, a matching degree of each of the at least one target video and the video to be searched is determined according to the number of object presence data pairs of which the corresponding second object presence feature is continuously the same as the first object presence feature; and arranging at least one target video according to the sequence from high to low of the matching degree. The more the number of pairs of object presence data of which the second object presence feature is the same as the succession of the first object presence feature, the higher the matching degree of the corresponding target video with the video to be searched. Further, at least one target video ranked from high to low in matching may be returned to the client device to present the video search results to the user.
According to other embodiments, the at least one target video may also be determined by performing a full-scale matching of the first object presence feature with the second object presence feature, i.e.: acquiring a second object departure feature of each candidate video in the plurality of candidate videos; and taking the candidate video including the first object appearance characteristic in the second object appearance characteristic as at least one target video.
In this embodiment, the second object presence feature of the target video includes all of the first object presence feature.
For example, fig. 3 also shows one example of a target video determined in the manner of full-scale matching described above. As shown in fig. 3, the video 310 to be searched includes three object segments 311, 312, 313, and the corresponding object outgoing data pairs are (a: 0), (B: 7500), and (a: 15000), respectively, wherein A, B is an object, and the unit of time difference is ms. The first object out-of-field feature of the video 310 to be searched is a concatenation of object out-of-field data pairs of object segments 311-313, namely A:0; 7500; a is 15000.
The candidate video 330 includes five object segments 331, 332, 333, 334, 335, corresponding pairs of object departure data are (A: 0), (B: 7500), (A: 15000), (B: 4000), (C: 8000), respectively, where A, B, C is the object and the time difference is in units of ms. The second object out-of-field feature of candidate video 330 is a concatenation of object out-of-field data pairs of object segments 331-335, A:0; 7500; a is 15000; 4000; and C is 8000. 0 due to the second object presence feature a of candidate video 330; 7500; a is 15000; 4000; 8000 including a first object presence feature A0; 7500; 15000 (shown in fig. 3 by the dashed rectangle 336), the candidate video 330 is thus the target video that matches the video 310 to be searched.
The first object presence feature can express the object presence condition of the video to be searched, so that the video content to be searched is refined and accurately summarized; the second object presence feature can express the object presence condition of the candidate video, and is a refined and accurate summary of the content of the candidate video, so that the first object presence feature and the second object presence feature are compared to determine the target video matched with the video to be searched, and quick and accurate video searching can be realized.
Further, in the two matching modes, the searching accuracy of the full-quantity matching mode is higher, but the searching accuracy is possibly missed due to matching failure of the first or last object outgoing data pair of the video to be searched, so that the coverage rate of the searching result is lower than that of the partial matching mode; the search accuracy of the local matching mode is slightly inferior to that of the full-quantity matching mode, but the search result is more comprehensive and the coverage rate is higher.
Fig. 4 shows a block diagram of a video search device 400 according to an embodiment of the present disclosure. As shown in fig. 4, the apparatus 400 includes:
The object segment dividing module 410 may be configured to determine at least one object segment included in the video to be searched according to an object corresponding to an image frame of the video to be searched, each image frame in each object segment corresponding to the same object;
The feature extraction module 420 may be configured to splice object appearance data pairs of one or more continuous object segments in the at least one object segment to obtain a first object appearance feature of the video to be searched, where the object appearance data pairs include an object corresponding to the corresponding object segment and a time difference between a first image frame of the object segment and a first image frame of a previous object segment; and
The feature matching module 430 may be configured to determine at least one target video from the plurality of candidate videos that matches the video to be searched based on the first object presence feature.
In accordance with one or more embodiments of the present disclosure, a video search scheme for "video searching" based on object presence features is provided. Determining at least one object fragment in the video to be searched according to the object corresponding to the image frame of the video to be searched; and generating a first object appearance characteristic of the video to be searched according to the objects corresponding to one or more continuous object fragments and the time difference between the first image frame of the object fragment and the first image frame of the last object fragment. The first object presence feature can express the object presence condition of the video to be searched, and is a refined and accurate summary of the video content to be searched. The target video is determined from the plurality of candidate videos according to the first object departure characteristic, and quick and accurate video searching can be achieved.
According to some embodiments, the apparatus 400 further comprises: and the object identification module is configured to identify objects of a plurality of image frames of the video to be searched so as to determine the objects corresponding to the plurality of image frames.
According to some embodiments, the object recognition module further comprises:
a candidate object identifying unit configured to, for each of the plurality of image frames, identify: performing object recognition on the image frame to determine candidate objects included in the image frame; and
A candidate object selection unit configured to take any one of the following as an object corresponding to the image frame: the candidate closest to the center of the image frame, the candidate with the largest area, the candidate with the front view.
According to some embodiments, the object fragment partitioning module 410 is further configured to: one or more consecutive image frames corresponding to the same object are taken as one object segment.
According to some embodiments, feature matching module 430 further comprises:
a feature acquisition unit configured to acquire a second object presence feature of each of the plurality of candidate videos, the second object presence feature including at least one object presence data pair, each object presence data pair including an object and a duration; and
And a local feature matching unit configured to, in response to determining that there is at least one object presence data pair for which a second object presence feature is continuously identical to the first object presence feature, take the corresponding candidate video as the target video.
According to some embodiments, the apparatus 400 further comprises:
A matching degree determining module configured to determine a matching degree of each of the at least one target video and the video to be searched according to the number of object outgoing data pairs of which the corresponding second object outgoing feature is the same as the succession of the first object outgoing feature; and
And the ordering module is configured to order the at least one target video according to the order of the matching degree from high to low.
According to some embodiments, feature matching module 430 further comprises:
a feature acquisition unit configured to acquire a second object departure feature of each of the plurality of candidate videos; and
And a full-scale feature matching unit configured to take, as the at least one target video, a candidate video including the first object presence feature in the second object presence features.
It should be appreciated that the various modules of the apparatus 400 shown in fig. 4 may correspond to the various steps in the method 200 described with reference to fig. 2. Also, the elements included in the various modules of apparatus 400 may correspond to particular implementations of the various steps in method 200. Thus, the operations, features and advantages described above with respect to method 200 apply equally to apparatus 400 and the modules and units comprised thereof. For brevity, certain operations, features and advantages are not described in detail herein.
Although specific functions are discussed above with reference to specific modules, it should be noted that the functions of the various modules discussed herein may be divided into multiple modules and/or at least some of the functions of the multiple modules may be combined into a single module. For example, the object fragment partitioning module 410 and the feature extraction module 420 described above may be combined into a single module in some embodiments.
It should also be appreciated that various techniques may be described herein in the general context of software hardware elements or program modules. The various modules described above with respect to fig. 4 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, the modules may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these modules may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of the object fragment partitioning module 410, the feature extraction module 420, the feature matching module 430 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip including one or more components of a Processor (e.g., a central processing unit (Central Processing Unit, CPU), microcontroller, microprocessor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), etc.), memory, one or more communication interfaces, and/or other circuitry, and may optionally execute received program code and/or include embedded firmware to perform functions.
According to embodiments of the present disclosure, there is also provided an electronic device, a readable storage medium and a computer program product.
Referring to fig. 5, a block diagram of an electronic device 500 that may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 5, the apparatus 500 includes a computing unit 501 that can perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM) 502 or a computer program loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, ROM 502, and RAM 503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
Various components in the device 500 are connected to the I/O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the device 500, the input unit 506 may receive input numeric or character information and generate key signal inputs related to user settings and/or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and/or a remote control. The output unit 507 may be any type of device capable of presenting information and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. Storage unit 508 may include, but is not limited to, magnetic disks, optical disks. The communication unit 509 allows the device 500 to exchange information/data with other devices via a computer network such as the internet and/or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth TM devices, 1302.11 devices, wiFi devices, wiMax devices, cellular communication devices, and/or the like.
The computing unit 501 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 500 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into RAM 503 and executed by computing unit 501, one or more steps of method 200 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the method 200 by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps recited in the present disclosure may be performed in parallel, sequentially or in a different order, provided that the desired results of the disclosed aspects are achieved, and are not limited herein.
Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the foregoing methods, systems, and apparatus are merely illustrative embodiments or examples and that the scope of the present disclosure is not limited by these embodiments or examples but only by the claims following the grant and their equivalents. Various elements of the embodiments or examples may be omitted or replaced with equivalent elements thereof. Furthermore, the steps may be performed in a different order than described in the present disclosure. Further, various elements of the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear after the disclosure.
Claims (15)
1. A video search method, comprising:
According to an object corresponding to an image frame of a video to be searched, determining at least one object fragment included in the video to be searched, wherein each image frame in each object fragment corresponds to the same object;
splicing object outgoing data pairs of one or more continuous object fragments in the at least one object fragment to obtain a first object outgoing characteristic of the video to be searched, wherein the object outgoing data pairs comprise objects corresponding to the corresponding object fragments and time differences between a first image frame of the object fragment and a first image frame of a previous object fragment; and
Determining at least one target video matching the video to be searched from a plurality of candidate videos according to the first object presence feature, wherein the determining at least one target video matching the video to be searched from the plurality of candidate videos according to the first object presence feature comprises:
Acquiring a second object presence feature of each candidate video of the plurality of candidate videos, the second object presence feature comprising at least one object presence data pair, each object presence data pair comprising an object and a time difference; and
In response to determining that there is at least one object presence data pair having a second object presence feature that is continuously the same as the first object presence feature, the respective candidate video is treated as the target video.
2. The method of claim 1, further comprising:
Object recognition is carried out on a plurality of image frames of the video to be searched to determine objects corresponding to the image frames.
3. The method of claim 2, wherein the object recognition of the plurality of image frames of the video to be searched to determine the objects corresponding to the plurality of image frames comprises:
for each of the plurality of image frames:
performing object recognition on the image frame to determine candidate objects included in the image frame; and
Any one of the following is taken as an object corresponding to the image frame: the candidate closest to the center of the image frame, the candidate with the largest area, the candidate with the front view.
4. A method according to any one of claims 1-3, wherein said determining at least one object fragment comprised by the video to be searched from an object corresponding to an image frame of the video to be searched comprises:
one or more consecutive image frames corresponding to the same object are taken as one object segment.
5. The method of claim 1, further comprising:
Determining a matching degree of each target video in the at least one target video and the video to be searched according to the number of the object outgoing data pairs with the corresponding second object outgoing characteristics being the same as the succession of the first object outgoing characteristics; and
And arranging the at least one target video according to the sequence from high to low of the matching degree.
6. The method of claim 1, wherein the determining at least one target video from a plurality of candidate videos that matches the video to be searched based on the first object presence feature comprises:
acquiring a second object departure feature of each candidate video of the plurality of candidate videos; and
And taking the candidate video including the first object appearance characteristic in the second object appearance characteristic as the at least one target video.
7. A video search apparatus comprising:
the object segment dividing module is configured to determine at least one object segment included in the video to be searched according to an object corresponding to an image frame of the video to be searched, wherein each image frame in each object segment corresponds to the same object;
The feature extraction module is configured to splice object appearance data pairs of one or more continuous object fragments in the at least one object fragment to obtain a first object appearance feature of the video to be searched, wherein the object appearance data pairs comprise objects corresponding to corresponding object fragments and time differences between a first image frame of the object fragment and a first image frame of a last object fragment; and
A feature matching module configured to determine at least one target video that matches the video to be searched from a plurality of candidate videos according to the first object presence feature, wherein the feature matching module further comprises:
A feature acquisition unit configured to acquire a second object presence feature of each of the plurality of candidate videos, the second object presence feature including at least one object presence data pair, each object presence data pair including an object and a time difference; and
And a local feature matching unit configured to, in response to determining that there is at least one object presence data pair for which a second object presence feature is continuously identical to the first object presence feature, take the corresponding candidate video as the target video.
8. The apparatus of claim 7, further comprising:
And the object identification module is configured to identify objects of a plurality of image frames of the video to be searched so as to determine the objects corresponding to the plurality of image frames.
9. The apparatus of claim 8, wherein the object recognition module further comprises:
A candidate object identification unit configured to, for each of the plurality of image frames: performing object recognition on the image frame to determine candidate objects included in the image frame; and
A candidate object selection unit configured to take any one of the following as an object corresponding to the image frame: the candidate closest to the center of the image frame, the candidate with the largest area, the candidate with the front view.
10. The apparatus of any of claims 7-9, wherein the object segment partitioning module is further configured to:
one or more consecutive image frames corresponding to the same object are taken as one object segment.
11. The apparatus of claim 7, further comprising:
a matching degree determining module configured to determine a matching degree of each of the at least one target video with the video to be searched according to the number of pairs of object presence data of which the respective second object presence feature is the same as the succession of the first object presence feature; and
And the ordering module is configured to order the at least one target video according to the order of the matching degree from high to low.
12. The apparatus of claim 7, wherein the feature matching module further comprises:
A feature acquisition unit configured to acquire a second object departure feature of each of the plurality of candidate videos; and
And the full-quantity feature matching unit is configured to take candidate videos including the first object out-of-field feature in the second object out-of-field feature as the at least one target video.
13. An electronic device, comprising:
At least one processor; and
A memory communicatively coupled to the at least one processor; wherein the method comprises the steps of
The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
14. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-6.
15. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method of any of claims 1-6.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110492157.0A CN113190713B (en) | 2021-05-06 | 2021-05-06 | Video searching method and device, electronic equipment and medium |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110492157.0A CN113190713B (en) | 2021-05-06 | 2021-05-06 | Video searching method and device, electronic equipment and medium |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113190713A CN113190713A (en) | 2021-07-30 |
CN113190713B true CN113190713B (en) | 2024-06-21 |
Family
ID=76983963
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110492157.0A Active CN113190713B (en) | 2021-05-06 | 2021-05-06 | Video searching method and device, electronic equipment and medium |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113190713B (en) |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106021496A (en) * | 2016-05-19 | 2016-10-12 | 海信集团有限公司 | Video search method and video search device |
CN110826365A (en) * | 2018-08-09 | 2020-02-21 | 阿里巴巴集团控股有限公司 | Video fingerprint generation method and device |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9626567B2 (en) * | 2013-03-13 | 2017-04-18 | Visible Measures Corp. | Automated video campaign building |
KR101485820B1 (en) * | 2013-07-15 | 2015-01-26 | 네무스텍(주) | Intelligent System for Generating Metadata for Video |
US10496650B1 (en) * | 2015-02-25 | 2019-12-03 | Google Llc | Fine-grained video classification |
CN107704525A (en) * | 2017-09-04 | 2018-02-16 | 优酷网络技术(北京)有限公司 | Video searching method and device |
CN110309353A (en) * | 2018-02-06 | 2019-10-08 | 上海全土豆文化传播有限公司 | Video index method and device |
CN111209897B (en) * | 2020-03-09 | 2023-06-20 | 深圳市雅阅科技有限公司 | Video processing method, device and storage medium |
CN111444822B (en) * | 2020-03-24 | 2024-02-06 | 北京奇艺世纪科技有限公司 | Object recognition method and device, storage medium and electronic device |
CN111522996B (en) * | 2020-04-09 | 2023-09-08 | 北京百度网讯科技有限公司 | Video clip retrieval method and device |
-
2021
- 2021-05-06 CN CN202110492157.0A patent/CN113190713B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN106021496A (en) * | 2016-05-19 | 2016-10-12 | 海信集团有限公司 | Video search method and video search device |
CN110826365A (en) * | 2018-08-09 | 2020-02-21 | 阿里巴巴集团控股有限公司 | Video fingerprint generation method and device |
Also Published As
Publication number | Publication date |
---|---|
CN113190713A (en) | 2021-07-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN113190695B (en) | Multimedia data searching method and device, computer equipment and medium | |
CN112749758B (en) | Image processing method, neural network training method, device, equipment and medium | |
CN113810765B (en) | Video processing method, device, equipment and medium | |
US20190227634A1 (en) | Contextual gesture-based image searching | |
CN113723305A (en) | Image and video detection method, device, electronic equipment and medium | |
CN113139093B (en) | Video searching method and device, computer equipment and medium | |
CN115269989B (en) | Object recommendation method, device, electronic equipment and storage medium | |
CN113139092B (en) | Video searching method and device, electronic equipment and medium | |
CN113190713B (en) | Video searching method and device, electronic equipment and medium | |
CN113139095B (en) | Video retrieval method and device, computer equipment and medium | |
CN113868453B (en) | Object recommendation method and device | |
CN113139094B (en) | Video searching method and device, electronic equipment and medium | |
CN116152607A (en) | Target detection method, method and device for training target detection model | |
CN114245171B (en) | Video editing method and device, electronic equipment and medium | |
CN113312511B (en) | Method, apparatus, device and computer readable storage medium for recommending content | |
CN114998963A (en) | Image detection method and method for training image detection model | |
CN113378001B (en) | Video playing progress adjusting method and device, electronic equipment and medium | |
CN116881485B (en) | Method and device for generating image retrieval index, electronic equipment and medium | |
CN113641933B (en) | Abnormal webpage identification method, abnormal site identification method and device | |
CN114390366B (en) | Video processing method and device | |
CN115170536B (en) | Image detection method, training method and device of model | |
CN114140851B (en) | Image detection method and method for training image detection model | |
CN113722534B (en) | Video recommendation method and device | |
CN113126865B (en) | Note generation method and device in video learning process, electronic equipment and medium | |
CN113722523B (en) | Object recommendation method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |