CN115017339A

CN115017339A - Media file multimode retrieval method and system based on AI algorithm

Info

Publication number: CN115017339A
Application number: CN202210604891.6A
Authority: CN
Inventors: 甘江威; 方露露; 杨丛聿
Original assignee: Xinhua Zhiyun Technology Co ltd
Current assignee: Xinhua Zhiyun Technology Co ltd
Priority date: 2022-05-30
Filing date: 2022-05-30
Publication date: 2022-09-06

Abstract

The invention discloses a media file multimode retrieval method and a media file multimode retrieval system based on an AI algorithm, wherein the method comprises the following steps: acquiring text information in the audio and video by adopting the existing corresponding AI algorithm; acquiring basic data in text information, and constructing a primary label according to the basic data; constructing a first-order index according to the first-order label, and storing the first-order index in a relational database; obtaining meta information and content information of each audio/video through the AI algorithm, and constructing a secondary label; and constructing a second-order index according to the second-order label, and inquiring and positioning the audio and video by inquiring the second-order index. The method and the system utilize the ES search engine to carry out search acceleration, and adopt a mode of second-order index and second-order search to carry out accurate positioning and detailed analysis on the record hit by search, thereby greatly improving the accuracy rate of the hit by search.

Description

Media file multimode retrieval method and system based on AI algorithm

Technical Field

The invention relates to a media file retrieval method and a system, in particular to a media file multimode retrieval method and a system based on AI algorithm

Background

The existing retrieval method and system usually adopt an ES-based search engine, and can only support fuzzy search, keyword search, and sorting capability according to time, relevancy and the like for texts (generally understood as textual contents such as titles, descriptions and the like; web pages are also textual contents), wherein the technical problems of the retrieval method and system include: 1. only simple text information can be searched; but for the current social and entertainment platforms mainly comprising pictures (photos) and video media, the search for the content of media resources such as pictures, videos and the like cannot be carried out; 2. the current platform's search capability also fails to provide detail on long-time video, long-time audio segment hits.

Disclosure of Invention

One of the purposes of the invention is to provide a media file multi-mode retrieval method and a media file multi-mode retrieval system based on an AI algorithm, wherein the method and the system are based on the existing AI algorithm, carry out multi-dimensional analysis on media audio and video files, and use a heterogeneous database to persist full-dimensional information, so that the retrieval dimension and the capability of the media files can be improved.

Another object of the present invention is to provide a media file multi-mode retrieval method and system based on AI algorithm, which uses an ES search engine to perform search acceleration and uses a second-order index + second-order search mode to perform precise positioning and detailed parsing on the records of search hits, thereby greatly improving the accuracy of search hits.

The invention also aims to provide a media file multimode retrieval method and a media file multimode retrieval system based on the AI algorithm, wherein the method and the system utilize a second-order index mode to construct tag groups of different segments of a video or construct tag groups of pictures by utilizing the existing AI algorithm, and the specific segments and the picture positions of the video can be accurately positioned through the second-order index.

In order to achieve at least one of the above objects, the present invention further provides a media file multimodal retrieval method based on an AI algorithm, the method comprising:

acquiring tag information in the audio and video by adopting the existing corresponding AI algorithm;

acquiring basic data and mate information in audio and video information, and constructing a primary tag according to the basic data and the mate information;

constructing a first-order index according to the first-order label, and storing the first-order index in a non-relational database;

identifying each audio and video content information through the AI algorithm, and extracting the label information to construct a secondary label;

and constructing a second-order index according to the second-order label, and inquiring and positioning the audio and video by inquiring the second-order index.

According to a preferred embodiment of the invention, the method comprises the steps of obtaining a file ID and a tag ID in audio and video data through an AI algorithm, constructing a joint index by using the file ID and the tag ID, and storing the constructed joint index by using the file ID and the tag ID as unique indexes in a non-relational database.

According to another preferred embodiment of the present invention, the method further comprises: and partitioning the tags acquired by the AI algorithm according to the types of the tags, and acquiring the file ID and the tag ID of each partitioned tag as a joint index of the corresponding partition.

According to another preferred embodiment of the present invention, the first-order index includes basic information of a file and meta information of the file, and the second-order index includes audio and video content information, wherein the audio and video content information includes: and mechanisms, places, people and scenes, and assembling the file ID and the tag ID in the basic information and corresponding file content information into a structured index.

According to another preferred embodiment of the present invention, the method comprises: searching and hitting a file ID and a tag ID in the first-order index through keywords, acquiring content information in the second-order index according to the file ID and the tag ID, and performing full-scale index query on the file according to the content information to acquire the finally queried file.

According to another preferred embodiment of the present invention, the method for constructing the first-order index comprises: and recognizing text information in the corresponding audio and video information by using an ASR algorithm and an OCR algorithm, and performing word segmentation processing on basic data and meta data in the recognized text information to obtain keyword label information for constructing a first-order index.

According to another preferred embodiment of the present invention, the first-order index constructing method comprises: and constructing a knowledge graph by using the NLP and the video tag, and taking the knowledge graph as a first-order index structure to remove time information in the first-order index.

According to another preferred embodiment of the present invention, the method for partitioning according to the tag type comprises: and generating corresponding partition keywords according to different file types, wherein the partition keywords are used for storing the basic information, meta information and content information identified in the corresponding document in a partition mode.

According to another preferred embodiment of the present invention, the method comprises: configuring the weight of the tag type, packaging tags configured with different weights, searching the packaged tag type through an ES search engine, outputting the hit result, further packaging the output content information and executing the full information query.

In order to achieve at least one of the above objects, the present invention further provides a media file multimodal retrieval system based on AI algorithm, which executes the above media file multimodal retrieval method based on AI algorithm.

The present invention further provides a computer-readable storage medium storing a computer program executable by a processor to perform the media file multimodal retrieval method of the AI algorithm.

Drawings

Fig. 1 is a flow chart showing a media file multimodal retrieval method based on AI algorithm according to the present invention.

FIG. 2 is a diagram illustrating a storage structure of a media file data structure according to the present invention.

FIG. 3 is a flow chart of the second-order search scheme based on the ES second-order index according to the present invention.

FIG. 4 is a diagram illustrating persistence of unstructured data on tag partitions in the present invention.

FIG. 5 is a diagram showing the structure of the first-order index and the second-order index according to the present invention.

Detailed Description

The following description is provided to disclose the invention so as to enable any person skilled in the art to practice the invention. The preferred embodiments described below are by way of example only, and other obvious variations will occur to those skilled in the art. The basic principles of the invention, as defined in the following description, may be applied to other embodiments, variations, modifications, equivalents, and other technical solutions without departing from the spirit and scope of the invention.

It is understood that the terms "a" and "an" should be interpreted as meaning that a number of one element or element is one in one embodiment, while a number of other elements is one in another embodiment, and the terms "a" and "an" should not be interpreted as limiting the number.

Referring to fig. 1-5, the present invention discloses a media file multimodal retrieval method and system based on AI algorithm, wherein the method comprises: firstly, audio and video data is obtained, information in the audio and video data is identified by adopting the existing AI algorithm, text information is generated, for example, the audio and video information comprises subtitle information, voice information, face information, other self content information and the like, and corresponding information in the audio and video information can be obtained by adopting a face identification algorithm, a voice identification algorithm, a subtitle identification algorithm, a content tag identification algorithm and a natural language identification algorithm, so as to generate tag information. The basic data and meta information of the file can be obtained through the existing AI algorithm, and the basic data and meta information are input into a relational database for data persistence processing. The AI algorithm can be used for obtaining the tags in the audio/video file, and further establishing a first-order index and a second-order index according to the tags.

Specifically, the invention establishes a first-order index by using basic data and meta information, wherein the first-order index establishing method comprises the following steps: acquiring basic information and meta information of a file by using the existing ASR algorithm or OCR algorithm, wherein the basic information can be directly acquired according to the attributes of an audio/video file, acquiring a pre-configured file ID directly by acquiring the attributes of the audio/video file, constructing a first-order index according to the meta information and the basic information, extracting the file ID from the file basic information, constructing a tag ID from a tag identified by a corresponding AI algorithm, wherein the tag ID is the identification type of the existing AI algorithm under the corresponding file ID, for example, the file ID is a video file of a corresponding character, the file ID can be constructed by combining characters of the corresponding character and a character string including a title, the corresponding tag ID can be a face identification ID or a voice identification ID of the corresponding character, constructing a combined index according to the file ID and the tag ID, and storing the file ID and the tag ID in a non-relational database including but not limited to a NoSQL database and the like The persistent operation is performed, it should be noted that the tag ID is a unique ID in the index. The basic information includes, but is not limited to, a file header and a file brief description, and the meta information includes, but is not limited to, resolution, duration, composition, file type and size, etc. In another preferred embodiment of the present invention, the first order index may construct a knowledge-graph from the NLP and video tags, treat the knowledge-graph as the first order index, and delete information unrelated to the knowledge-graph.

On the basis of the first-order index, the invention further constructs a second-order index, wherein the second-order index constructing method comprises the following steps: based on different tag IDs, tag information under the corresponding tag IDs is identified through the existing AI algorithm, wherein the tag information includes but is not limited to tag content, tag start time, tag end time, tag parent-child information, and the like. For example, the tag ID is face information of a corresponding person, wherein a tag start time, a tag end time, tag entity information, expression information, and the like including, but not limited to, the face information of the corresponding person can be recorded by a corresponding face recognition algorithm. It should be noted that, because the tag types identified by different AI algorithms are different, when setting the tag types, the tag IDs identified by different algorithms are different, and the partitioned storage is performed according to different tag types.

The invention further configures different weight configurations of the tag ID according to the retrieval requirements, for example, in order to preferentially acquire the face image in one video, the weight of the image identification tag ID can be configured to be higher than that of other character identification tag IDs, so that in the searching process, the video information with the face image in the corresponding audio/video is searched and preferentially acquired and displayed. Therefore, the invention can effectively improve the acquisition of the corresponding label content through the weight configuration of the label ID.

It should be noted that, under the corresponding file ID and tag ID, the query content needs to be packaged to generate the structured query content, for example, the file ID is a file ID related to the corresponding news media, and the file ID is packaged with the person tag ID, the scene tag ID, the entity tag ID, and the like, and further packaged with the tag content, the tag time, the tag entity information, and the like identified based on the corresponding AI algorithm under different tag IDs, so as to form the structured query packaging structure.

The first-order index and the second-order index are preferably constructed by an ES (ES) search engine, wherein the first-order index and the second-order index constructed by the ES search engine have wider scene adaptability compared with an index constructed by a pure non-relational database, and can be matched with different scenes.

The invention further provides a query method based on the first-order index and the second-order index of the ES search engine, which comprises the following steps: inputting keywords and corresponding filter conditions to execute first-order index query in the ES search engine, hitting keyword fields existing in the first-order index through the keywords, and returning labels corresponding to the fields hit by the keywords by the ES search engine. For example, if "title" is hit or "asrTag" is hit, a specific ES index field name is returned. The ES search engine acquires a hit specific field in the first-order query process, and because the hit specific field does not contain tag specific information identified by an AI algorithm, the invention further acquires a file ID and a tag ID containing the specific field according to the hit specific field to perform second-order tag query, wherein the second-order tag query is a structured data packet, and the second-order query is performed after the tag content, the tag start time, the tag end time, the tag entity information and other detailed information corresponding to the tag ID are encapsulated, so that the file detailed information meeting the tag ID query condition can be obtained. Detailed information such as names, avatars, traces of people, etc. can be obtained through the second order query.

It is worth mentioning that after the tag information is obtained according to the existing AI algorithm, the invention further performs partition processing according to the type of the tag, where the tag type indicates the tag obtained by processing with different AI algorithms, for example, the tag type obtained by the face recognition algorithm is a face tag, the tag type obtained by the word recognition algorithm is a text tag, the tag type obtained by the voice recognition algorithm is a voice tag, and the tag type obtained by the image recognition algorithm is an image tag. The index construction can be clearer through the partition storage arrangement, and the index can be constructed and searched quickly.

That is, after the first-order query by the ES search engine, the file ID and the tag ID that hit the first-order query result can be obtained, and further, the exact query of the full-scale index is performed according to the service keyword and the index keyword recorded by the tag ID, thereby obtaining the final detailed query result.

In particular, according to the embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In such an embodiment, the computer program may be downloaded and installed from a network via the communication section, and/or installed from a removable medium. The computer program, when executed by a Central Processing Unit (CPU), performs the above-described functions defined in the method of the present application. It should be noted that the computer readable medium mentioned above in the present application may be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wire segments, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present application, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In this application, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless section, wire section, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

It will be understood by those skilled in the art that the embodiments of the present invention described above and illustrated in the drawings are given by way of example only and not by way of limitation, the objects of the invention having been fully and effectively achieved, the functional and structural principles of the present invention having been shown and described in the embodiments, and that various changes or modifications may be made in the embodiments of the present invention without departing from such principles.

Claims

1. A media file multi-mode retrieval method based on AI algorithm is characterized in that the method comprises the following steps:

2. The multimode retrieval method for the media file based on the AI algorithm as claimed in claim 1, wherein the method comprises the steps of obtaining the tag ID included in the audio/video data through the AI algorithm, obtaining the file ID of the file, constructing a joint index by using the file ID and the tag ID, and storing the joint index constructed by using the file ID and the tag ID as a unique index in a non-relational database.

3. The AI algorithm-based multimodal retrieval method of media files according to claim 1, characterized in that the method further comprises: and partitioning the label information acquired by the AI algorithm according to the type of the label information, and acquiring the file ID and the label ID of each partitioned label as a joint index of the corresponding partition.

4. The AI algorithm-based media file multimodal retrieval method of claim 1, wherein the first-order index includes file base information and file meta information, and the second-order index includes audio and video content information, wherein the audio and video content information includes: and mechanisms, places, people and scenes, and assembling the file ID and the tag ID in the basic information and corresponding file content information into a structured index.

5. The AI algorithm-based multimodal retrieval method of media files according to claim 1, characterized in that it comprises: searching and hitting a file ID and a tag ID in the first-order index through keywords, acquiring content information in the second-order index according to the file ID and the tag ID, and performing full-scale index query on the file according to the content information to acquire the finally queried file.

6. The AI algorithm-based multimodal media file retrieval method of claim 1, wherein the first-order index is constructed by the method comprising: and recognizing text information in the corresponding audio and video information by using an ASR algorithm and an OCR algorithm, and performing word segmentation processing on basic data and meta data in the recognized text information to obtain keyword label information for constructing a first-order index.

7. The AI algorithm-based multimodal media file retrieval method of claim 1, wherein the first order index construction method comprises: and constructing a knowledge graph by using the NLP and the video tag, and taking the knowledge graph as a first-order index structure to remove time information in the first-order index.

8. The AI algorithm-based media file multimodal retrieval method of claim 1, wherein the method of partitioning according to the tag type comprises: and generating corresponding partition keywords according to different file types, wherein the partition keywords are used for storing the basic information, meta information and content information identified in the corresponding document in a partition mode.

9. An AI-algorithm-based media file multimodal retrieval system, characterized in that the system executes an AI-algorithm-based media file multimodal retrieval method according to any of claims 1 to 8.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which can be executed by a processor to perform the media file multimodal retrieval method of an AI algorithm according to any one of claims 1 to 8.